Raghavendra K T
2013-Aug-26 08:48 UTC
[PATCH V13 0/4] Paravirtualized ticket spinlocks for KVM host
This series forms the kvm host part of paravirtual spinlock based against kvm tree. Please refer to https://lkml.org/lkml/2013/8/9/265 for kvm guest and Xen, x86 part merged to -tip spinlocks. Please note that: kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi is a common patch for both guest and host. Changes since V12: fold the patch 3 into patch 2 for bisection. (Eric Northup) Raghavendra K T (3): kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi kvm hypervisor: Simplify kvm_for_each_vcpu with kvm_irq_delivery_to_apic Documentation/kvm : Add documentation on Hypercalls and features used for PV spinlock Srivatsa Vaddagiri (1): kvm hypervisor : Add a hypercall to KVM hypervisor to support pv-ticketlocks Documentation/virtual/kvm/cpuid.txt | 4 ++++ Documentation/virtual/kvm/hypercalls.txt | 14 ++++++++++++++ arch/x86/include/asm/kvm_host.h | 5 +++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/cpuid.c | 3 ++- arch/x86/kvm/lapic.c | 5 ++++- arch/x86/kvm/x86.c | 31 ++++++++++++++++++++++++++++++- include/uapi/linux/kvm_para.h | 1 + 8 files changed, 61 insertions(+), 3 deletions(-) -- 1.7.11.7
Raghavendra K T
2013-Aug-26 08:48 UTC
[PATCH V13 1/4] kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi
this is needed by both guest and host. Originally-from: Srivatsa Vaddagiri <vatsa at linux.vnet.ibm.com> Signed-off-by: Raghavendra K T <raghavendra.kt at linux.vnet.ibm.com> Acked-by: Gleb Natapov <gleb at redhat.com> Acked-by: Ingo Molnar <mingo at kernel.org> --- arch/x86/include/uapi/asm/kvm_para.h | 1 + include/uapi/linux/kvm_para.h | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h index 06fdbd9..94dc8ca 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -23,6 +23,7 @@ #define KVM_FEATURE_ASYNC_PF 4 #define KVM_FEATURE_STEAL_TIME 5 #define KVM_FEATURE_PV_EOI 6 +#define KVM_FEATURE_PV_UNHALT 7 /* The last 8 bits are used to indicate how to interpret the flags field * in pvclock structure. If no bits are set, all flags are ignored. diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index cea2c5c..2841f86 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -19,6 +19,7 @@ #define KVM_HC_MMU_OP 2 #define KVM_HC_FEATURES 3 #define KVM_HC_PPC_MAP_MAGIC_PAGE 4 +#define KVM_HC_KICK_CPU 5 /* * hypercalls use architecture specific -- 1.7.11.7
Raghavendra K T
2013-Aug-26 08:48 UTC
[PATCH V13 2/4] kvm hypervisor : Add a hypercall to KVM hypervisor to support pv-ticketlocks
From: Srivatsa Vaddagiri <vatsa at linux.vnet.ibm.com> kvm_hc_kick_cpu allows the calling vcpu to kick another vcpu out of halt state. the presence of these hypercalls is indicated to guest via kvm_feature_pv_unhalt. Fold pv_unhalt flag into GET_MP_STATE ioctl to aid migration During migration, any vcpu that got kicked but did not become runnable (still in halted state) should be runnable after migration. Signed-off-by: Srivatsa Vaddagiri <vatsa at linux.vnet.ibm.com> Signed-off-by: Suzuki Poulose <suzuki at in.ibm.com> [Raghu: Apic related changes, folding pvunhalted into vcpu_runnable Added flags for future use (suggested by Gleb)] [ Raghu: fold pv_unhalt flag as suggested by Eric Northup] Signed-off-by: Raghavendra K T <raghavendra.kt at linux.vnet.ibm.com> Acked-by: Gleb Natapov <gleb at redhat.com> Acked-by: Ingo Molnar <mingo at kernel.org> --- arch/x86/include/asm/kvm_host.h | 5 +++++ arch/x86/kvm/cpuid.c | 3 ++- arch/x86/kvm/x86.c | 44 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 50 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index f87f7fc..1d1f711 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -511,6 +511,11 @@ struct kvm_vcpu_arch { * instruction. */ bool write_fault_to_shadow_pgtable; + + /* pv related host specific info */ + struct { + bool pv_unhalted; + } pv; }; struct kvm_lpage_info { diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index a20ecb5..b110fe6 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -413,7 +413,8 @@ static int do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, (1 << KVM_FEATURE_CLOCKSOURCE2) | (1 << KVM_FEATURE_ASYNC_PF) | (1 << KVM_FEATURE_PV_EOI) | - (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT); + (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT) | + (1 << KVM_FEATURE_PV_UNHALT); if (sched_info_on()) entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index d21bce5..1e73dab 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5495,6 +5495,36 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) return 1; } +/* + * kvm_pv_kick_cpu_op: Kick a vcpu. + * + * @apicid - apicid of vcpu to be kicked. + */ +static void kvm_pv_kick_cpu_op(struct kvm *kvm, unsigned long flags, int apicid) +{ + struct kvm_vcpu *vcpu = NULL; + int i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (!kvm_apic_present(vcpu)) + continue; + + if (kvm_apic_match_dest(vcpu, 0, 0, apicid, 0)) + break; + } + if (vcpu) { + /* + * Setting unhalt flag here can result in spurious runnable + * state when unhalt reset does not happen in vcpu_block. + * But that is harmless since that should soon result in halt. + */ + vcpu->arch.pv.pv_unhalted = true; + /* We need everybody see unhalt before vcpu unblocks */ + smp_wmb(); + kvm_vcpu_kick(vcpu); + } +} + int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) { unsigned long nr, a0, a1, a2, a3, ret; @@ -5528,6 +5558,10 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) case KVM_HC_VAPIC_POLL_IRQ: ret = 0; break; + case KVM_HC_KICK_CPU: + kvm_pv_kick_cpu_op(vcpu->kvm, a0, a1); + ret = 0; + break; default: ret = -KVM_ENOSYS; break; @@ -5950,6 +5984,7 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) kvm_apic_accept_events(vcpu); switch(vcpu->arch.mp_state) { case KVM_MP_STATE_HALTED: + vcpu->arch.pv.pv_unhalted = false; vcpu->arch.mp_state KVM_MP_STATE_RUNNABLE; case KVM_MP_STATE_RUNNABLE: @@ -6249,7 +6284,12 @@ int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu, struct kvm_mp_state *mp_state) { kvm_apic_accept_events(vcpu); - mp_state->mp_state = vcpu->arch.mp_state; + if (vcpu->arch.mp_state == KVM_MP_STATE_HALTED && + vcpu->arch.pv.pv_unhalted) + mp_state->mp_state = KVM_MP_STATE_RUNNABLE; + else + mp_state->mp_state = vcpu->arch.mp_state; + return 0; } @@ -6770,6 +6810,7 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) BUG_ON(vcpu->kvm == NULL); kvm = vcpu->kvm; + vcpu->arch.pv.pv_unhalted = false; vcpu->arch.emulate_ctxt.ops = &emulate_ops; if (!irqchip_in_kernel(kvm) || kvm_vcpu_is_bsp(vcpu)) vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; @@ -7103,6 +7144,7 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) !vcpu->arch.apf.halted) || !list_empty_careful(&vcpu->async_pf.done) || kvm_apic_has_events(vcpu) + || vcpu->arch.pv.pv_unhalted || atomic_read(&vcpu->arch.nmi_queued) || (kvm_arch_interrupt_allowed(vcpu) && kvm_cpu_has_interrupt(vcpu)); -- 1.7.11.7
Raghavendra K T
2013-Aug-26 08:48 UTC
[PATCH V13 3/4] kvm hypervisor: Simplify kvm_for_each_vcpu with kvm_irq_delivery_to_apic
Note that we are using APIC_DM_REMRD which has reserved usage. In future if APIC_DM_REMRD usage is standardized, then we should find some other way or go back to old method. Suggested-by: Gleb Natapov <gleb at redhat.com> Signed-off-by: Raghavendra K T <raghavendra.kt at linux.vnet.ibm.com> Acked-by: Gleb Natapov <gleb at redhat.com> Acked-by: Ingo Molnar <mingo at kernel.org> --- arch/x86/kvm/lapic.c | 5 ++++- arch/x86/kvm/x86.c | 25 ++++++------------------- 2 files changed, 10 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index afc1124..48c13c9 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -706,7 +706,10 @@ out: break; case APIC_DM_REMRD: - apic_debug("Ignoring delivery mode 3\n"); + result = 1; + vcpu->arch.pv.pv_unhalted = 1; + kvm_make_request(KVM_REQ_EVENT, vcpu); + kvm_vcpu_kick(vcpu); break; case APIC_DM_SMI: diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 1e73dab..640d112 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5502,27 +5502,14 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) */ static void kvm_pv_kick_cpu_op(struct kvm *kvm, unsigned long flags, int apicid) { - struct kvm_vcpu *vcpu = NULL; - int i; + struct kvm_lapic_irq lapic_irq; - kvm_for_each_vcpu(i, vcpu, kvm) { - if (!kvm_apic_present(vcpu)) - continue; + lapic_irq.shorthand = 0; + lapic_irq.dest_mode = 0; + lapic_irq.dest_id = apicid; - if (kvm_apic_match_dest(vcpu, 0, 0, apicid, 0)) - break; - } - if (vcpu) { - /* - * Setting unhalt flag here can result in spurious runnable - * state when unhalt reset does not happen in vcpu_block. - * But that is harmless since that should soon result in halt. - */ - vcpu->arch.pv.pv_unhalted = true; - /* We need everybody see unhalt before vcpu unblocks */ - smp_wmb(); - kvm_vcpu_kick(vcpu); - } + lapic_irq.delivery_mode = APIC_DM_REMRD; + kvm_irq_delivery_to_apic(kvm, 0, &lapic_irq, NULL); } int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) -- 1.7.11.7
Raghavendra K T
2013-Aug-26 08:48 UTC
[PATCH V13 4/4] Documentation/kvm : Add documentation on Hypercalls and features used for PV spinlock
KVM_HC_KICK_CPU hypercall added to wakeup halted vcpu in paravirtual spinlock enabled guest. KVM_FEATURE_PV_UNHALT enables guest to check whether pv spinlock can be enabled in guest. Thanks Vatsa for rewriting KVM_HC_KICK_CPU Cc: Rob Landley <rob at landley.net> Signed-off-by: Srivatsa Vaddagiri <vatsa at linux.vnet.ibm.com> Signed-off-by: Raghavendra K T <raghavendra.kt at linux.vnet.ibm.com> Acked-by: Gleb Natapov <gleb at redhat.com> Acked-by: Ingo Molnar <mingo at kernel.org> --- Documentation/virtual/kvm/cpuid.txt | 4 ++++ Documentation/virtual/kvm/hypercalls.txt | 14 ++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/virtual/kvm/cpuid.txt b/Documentation/virtual/kvm/cpuid.txt index 83afe65..22ff659 100644 --- a/Documentation/virtual/kvm/cpuid.txt +++ b/Documentation/virtual/kvm/cpuid.txt @@ -43,6 +43,10 @@ KVM_FEATURE_CLOCKSOURCE2 || 3 || kvmclock available at msrs KVM_FEATURE_ASYNC_PF || 4 || async pf can be enabled by || || writing to msr 0x4b564d02 ------------------------------------------------------------------------------ +KVM_FEATURE_PV_UNHALT || 7 || guest checks this feature bit + || || before enabling paravirtualized + || || spinlock support. +------------------------------------------------------------------------------ KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side || || per-cpu warps are expected in || || kvmclock. diff --git a/Documentation/virtual/kvm/hypercalls.txt b/Documentation/virtual/kvm/hypercalls.txt index ea113b5..022198e 100644 --- a/Documentation/virtual/kvm/hypercalls.txt +++ b/Documentation/virtual/kvm/hypercalls.txt @@ -64,3 +64,17 @@ Purpose: To enable communication between the hypervisor and guest there is a shared page that contains parts of supervisor visible register state. The guest can map this shared page to access its supervisor register through memory using this hypercall. + +5. KVM_HC_KICK_CPU +------------------------ +Architecture: x86 +Status: active +Purpose: Hypercall used to wakeup a vcpu from HLT state +Usage example : A vcpu of a paravirtualized guest that is busywaiting in guest +kernel mode for an event to occur (ex: a spinlock to become available) can +execute HLT instruction once it has busy-waited for more than a threshold +time-interval. Execution of HLT instruction would cause the hypervisor to put +the vcpu to sleep until occurence of an appropriate event. Another vcpu of the +same guest can wakeup the sleeping vcpu by issuing KVM_HC_KICK_CPU hypercall, +specifying APIC ID (a1) of the vcpu to be woken up. An additional argument (a0) +is used in the hypercall for future use. -- 1.7.11.7
Gleb Natapov
2013-Aug-26 10:04 UTC
[PATCH V13 0/4] Paravirtualized ticket spinlocks for KVM host
On Mon, Aug 26, 2013 at 02:18:32PM +0530, Raghavendra K T wrote:> > This series forms the kvm host part of paravirtual spinlock > based against kvm tree. > > Please refer to https://lkml.org/lkml/2013/8/9/265 for > kvm guest and Xen, x86 part merged to -tip spinlocks. > > Please note that: > kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi is a common patch > for both guest and host. >Thanks, applied. The patchset is not against kvm.git queue though, so I had to fix one minor conflict manually.> Changes since V12: > fold the patch 3 into patch 2 for bisection. (Eric Northup) > > Raghavendra K T (3): > kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi > kvm hypervisor: Simplify kvm_for_each_vcpu with > kvm_irq_delivery_to_apic > Documentation/kvm : Add documentation on Hypercalls and features used > for PV spinlock > > Srivatsa Vaddagiri (1): > kvm hypervisor : Add a hypercall to KVM hypervisor to support > pv-ticketlocks > > Documentation/virtual/kvm/cpuid.txt | 4 ++++ > Documentation/virtual/kvm/hypercalls.txt | 14 ++++++++++++++ > arch/x86/include/asm/kvm_host.h | 5 +++++ > arch/x86/include/uapi/asm/kvm_para.h | 1 + > arch/x86/kvm/cpuid.c | 3 ++- > arch/x86/kvm/lapic.c | 5 ++++- > arch/x86/kvm/x86.c | 31 ++++++++++++++++++++++++++++++- > include/uapi/linux/kvm_para.h | 1 + > 8 files changed, 61 insertions(+), 3 deletions(-) > > -- > 1.7.11.7-- Gleb.
Raghavendra K T
2013-Aug-26 10:28 UTC
[PATCH V13 0/4] Paravirtualized ticket spinlocks for KVM host
On 08/26/2013 03:34 PM, Gleb Natapov wrote:> On Mon, Aug 26, 2013 at 02:18:32PM +0530, Raghavendra K T wrote: >> >> This series forms the kvm host part of paravirtual spinlock >> based against kvm tree. >> >> Please refer to https://lkml.org/lkml/2013/8/9/265 for >> kvm guest and Xen, x86 part merged to -tip spinlocks. >> >> Please note that: >> kvm uapi: Add KICK_CPU and PV_UNHALT definition to uapi is a common patch >> for both guest and host. >> > Thanks, applied. The patchset is not against kvm.git queue though, so I > had to fix one minor conflict manually.Thank you Gleb.
Rob Landley
2013-Aug-28 18:01 UTC
[PATCH V13 4/4] Documentation/kvm : Add documentation on Hypercalls and features used for PV spinlock
On 08/26/2013 03:48:36 AM, Raghavendra K T wrote:> KVM_HC_KICK_CPU hypercall added to wakeup halted vcpu in paravirtual > spinlock > enabled guest. > > KVM_FEATURE_PV_UNHALT enables guest to check whether pv spinlock can > be enabled > in guest. > > Thanks Vatsa for rewriting KVM_HC_KICK_CPU > Cc: Rob Landley <rob at landley.net>Looks like documentation to me: Acked-by: Rob Landley <rob at landley.net> Rob
Reasonably Related Threads
- [PATCH V13 0/4] Paravirtualized ticket spinlocks for KVM host
- [PATCH V12 0/5] Paravirtualized ticket spinlocks for KVM host
- [PATCH V12 0/5] Paravirtualized ticket spinlocks for KVM host
- [PATCH RFC V6 0/5] kvm : Paravirt-spinlock support for KVM guests
- [PATCH RFC V6 0/5] kvm : Paravirt-spinlock support for KVM guests