Xen devel - Jun 2009 - [PATCH 0/2] support kexec/kdump on hvm

Hi,

kexec/kdump on hvm sometimes hangs up. After some investigation,
I found the cause is that kdump halts the vcpu0(bsp). 
Current xen hypervisor seems to assume that the bsp never be halted
and delivery of some IRQs relies on running bsp.

This series of patches fix it at least kexec/kdump works.

Thanks,
Kouya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This patch is needed for kexec/kdump on hvm since kdump halts bsp.

- allow another vcpu than bsp to receive PIT IRQ
- deliver PIT IRQ to the vcpu whose LINT0 is ExtINT mode

Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This patch is needed for kexec/kdump on hvm since kdump halts bsp.

Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30/06/2009 07:57, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
> This patch is needed for kexec/kdump on hvm since kdump halts bsp.
Really? Why would it do that?

These patches are pretty nasty. We can perhaps come up with a neater smaller
version of them if there''s a real need.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser writes:
> On 30/06/2009 07:57, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
> 
> > This patch is needed for kexec/kdump on hvm since kdump halts bsp.
> 
> Really? Why would it do that?
Not always. kdump leaves one cpu which has received a crash signal
and halts all other cpus. I don''t know why.
> These patches are pretty nasty. We can perhaps come up with a neater
smaller
> version of them if there''s a real need.
Hmm, though I tried to make them small...
Could you refine them?

Thanks,
Kouya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Kouya Shimura wrote:
> This patch is needed for kexec/kdump on hvm since kdump halts bsp.
> 
> Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>
It kicks every online vcpu. The result will be more unnecessary IPIs, especially
the vcpu# is big. Therefore increase cost of interrupt handling of assigned
devices.

Regards,
Weidong
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Han, Weidong wrote:
> Kouya Shimura wrote:
>> This patch is needed for kexec/kdump on hvm since kdump halts bsp.
>> 
>> Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>
> 
> It kicks every online vcpu. The result will be more unnecessary IPIs,
> especially the vcpu# is big. Therefore increase cost of interrupt
> handling of assigned devices.  
> 
Sorry, I misunderstood your patch. It only kicks one online vcpu. It is no
problem.

Regards,
Weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30/06/2009 08:57, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
>> Really? Why would it do that?
> 
> Not always. kdump leaves one cpu which has received a crash signal
> and halts all other cpus. I don''t know why.
That seems very brave of it, if it also expects interrupts to still work!
Does it really require *all* interrupts still to be functional?
>> These patches are pretty nasty. We can perhaps come up with a neater
smaller
>> version of them if there''s a real need.
> 
> Hmm, though I tried to make them small...
> Could you refine them?
Well, looking at patch 1/2 to start with, some of the issues you address are
not particular to the PIT, but to any interrupt delivered via vpt.c. I
suspect what you really need is a hook into vpt.c when a vcpu goes down, to
move all its vpt''s (except its own LAPIC one) onto another
vcpu''s list. And
on create_periodic_time() ignore the suggested vcpu given by the caller if
that vcpu is down. And disable the IRQ0_SPECIAL_ROUTING in vioapic.c if
vcpu0 is down.

Patch 2/2 I suppose something like that might be acceptable. Still it''d
be
nice to not have every VCPU piling into hvm_dirq_assist(). Perhaps that and
vpt.c could agree on who their current ''fallback vcpu'' is, and
have that
inherit orphaned vpts and do all dirq assistance.

It''s still a bit fragile and a bit of a pain in the butt though. Is
kdump in
HVM guests super important?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30/06/2009 09:53, "Han, Weidong" <weidong.han@intel.com>
wrote:
> Kouya Shimura wrote:
>> This patch is needed for kexec/kdump on hvm since kdump halts bsp.
>> 
>> Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>
> 
> It kicks every online vcpu. The result will be more unnecessary IPIs,
> especially the vcpu# is big. Therefore increase cost of interrupt handling
of
> assigned devices.
Actually it breaks out of the loop as soon as it kicks one. But still all
vcpus will pile into hvm_dirq_assist() on every vmexit, which is quite
undesirable too.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> @@ -437,7 +444,7 @@ void hvm_dirq_assist(struct vcpu *v)
>      struct hvm_irq_dpci *hvm_irq_dpci = d->arch.hvm_domain.irq.dpci;
>      struct dev_intx_gsi_link *digl;
>  
> -    if ( !iommu_enabled || (v->vcpu_id != 0) || (hvm_irq_dpci == NULL)
)
> +    if ( !iommu_enabled || (hvm_irq_dpci == NULL) )
>          return;
>  
>      for ( irq = find_first_bit(hvm_irq_dpci->dirq_mask,
d->nr_pirqs);
With the patch applied, every vcpu of a domain can access the
hvm_irq_dpci->dirq_mask concurrently!
I think some usage of spinlock must be introduced if you''d like to do
so.

Thanks,
-- Dexuan



-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Kouya Shimura
Sent: 2009?6?30? 14:57
To: xen-devel@lists.xensource.com
Cc: Keir Fraser
Subject: [Xen-devel] [PATCH 2/2] passthrough: deliver IRQs even if bsp is halted
(support kexec/kdump on hvm)

This patch is needed for kexec/kdump on hvm since kdump halts bsp.

Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 30/06/2009 10:09, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:
>> @@ -437,7 +444,7 @@ void hvm_dirq_assist(struct vcpu *v)
>>      struct hvm_irq_dpci *hvm_irq_dpci =
d->arch.hvm_domain.irq.dpci;
>>      struct dev_intx_gsi_link *digl;
>>  
>> -    if ( !iommu_enabled || (v->vcpu_id != 0) || (hvm_irq_dpci ==
NULL) )
>> +    if ( !iommu_enabled || (hvm_irq_dpci == NULL) )
>>          return;
>>  
>>      for ( irq = find_first_bit(hvm_irq_dpci->dirq_mask,
d->nr_pirqs);
> 
> With the patch applied, every vcpu of a domain can access the
> hvm_irq_dpci->dirq_mask concurrently!
> I think some usage of spinlock must be introduced if you''d like to
do so.
The test_and_clear_bit() at the start of the loop body probably provides the
required mutual exclusion.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

Thanks for your review and comments.

Keir Fraser writes:
> Well, looking at patch 1/2 to start with, some of the issues you address
are
> not particular to the PIT, but to any interrupt delivered via vpt.c. I
> suspect what you really need is a hook into vpt.c when a vcpu goes down, to
> move all its vpt''s (except its own LAPIC one) onto another
vcpu''s list. And
> on create_periodic_time() ignore the suggested vcpu given by the caller if
> that vcpu is down. And disable the IRQ0_SPECIAL_ROUTING in vioapic.c if
> vcpu0 is down.
This doesn''t work since the ioapic is completely disabled.

I''ve found a interesting thread in LKML archives.
Subject: [PATCH] kexec: force x86_64 arches to boot kdump kernels on boot cpu
http://lkml.org/lkml/2007/11/26/346

Probably we should fix the guest bios(MP table, ioapic initialization).

I''m looking into it.

Thanks,
Kouya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 01/07/2009 05:49, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
> This doesn''t work since the ioapic is completely disabled.
>
> I''ve found a interesting thread in LKML archives.
> Subject: [PATCH] kexec: force x86_64 arches to boot kdump kernels on boot
cpu
> http://lkml.org/lkml/2007/11/26/346
>
> Probably we should fix the guest bios(MP table, ioapic initialization).
So... You want virtual wire through IO-APIC?

Still some logic changes to vpt.c will be required, since platform timers
only tick and generate interrupts when vpt logic is executed on vcpu0''s
return-to-guest path. Same for irq passthrough (handled by your patch 2/2).

Hmmm... Okay I think this might be workable. Device passthru and vpt
subsystems could be fired off of whichever vcpu owns the LAPIC identifier
currently programmed into IOAPIC0:PIN0, rather than being hardwired to
VCPU0.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser writes:
> So... You want virtual wire through IO-APIC?
Yes, I''m struggling. 
> Still some logic changes to vpt.c will be required, since platform timers
> only tick and generate interrupts when vpt logic is executed on
vcpu0''s
> return-to-guest path. Same for irq passthrough (handled by your patch 2/2).
Indeed, also vioapic.c doesn''t support virtual wire mode.
> Hmmm... Okay I think this might be workable. Device passthru and vpt
> subsystems could be fired off of whichever vcpu owns the LAPIC identifier
> currently programmed into IOAPIC0:PIN0, rather than being hardwired to
> VCPU0.
Okay, but 2/2 patch can wait. First of all, I''ll remake 1/2 patch.

Thanks,
Kouya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 01/07/2009 09:48, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
>> Hmmm... Okay I think this might be workable. Device passthru and vpt
>> subsystems could be fired off of whichever vcpu owns the LAPIC
identifier
>> currently programmed into IOAPIC0:PIN0, rather than being hardwired to
>> VCPU0.
> 
> Okay, but 2/2 patch can wait. First of all, I''ll remake 1/2 patch.
I''ll deal with virtual wire via ioapic myself.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 01/07/2009 14:09, "Keir Fraser" <keir.fraser@eu.citrix.com>
wrote:
> On 01/07/2009 09:48, "Kouya Shimura" <kouya@jp.fujitsu.com>
wrote:
> 
>>> Hmmm... Okay I think this might be workable. Device passthru and
vpt
>>> subsystems could be fired off of whichever vcpu owns the LAPIC
identifier
>>> currently programmed into IOAPIC0:PIN0, rather than being hardwired
to
>>> VCPU0.
>> 
>> Okay, but 2/2 patch can wait. First of all, I''ll remake 1/2
patch.
> 
> I''ll deal with virtual wire via ioapic myself.
Take a look at changeset 19877. It does the basics of what you need. Vpt.c
logic probably still will need tweaking however.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser writes:
> Take a look at changeset 19877. It does the basics of what you need. Vpt.c
> logic probably still will need tweaking however.
Great! This is really what I want. Thanks a lot.
I confirmed it works with my i8254.c patch.

Just one thing. An attached patch will be necessary with debug=y build.

Thanks,
Kouya

Signed-off-by: Kouya Shimura <kouya@jp.fujitsu.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel