Hey folks, I am sending this on behalf of Andrew since our internal email system is dropping all xen-devel mailing lists :-( Anyhow: This is with xen-4.1-testing cs 23201:1c89f7d29fbb and using the default "credit" scheduler. I''ve run into an interesting issue with HVM guests which make use of Pause Loop Exiting (ie. on westmere systems; and also on romley systems): after yielding the cpu, guests don''t seem to receive timer interrupts correctly.. Some background: for historical reasons (ie old templates) we boot OL/RHEL guests with the following settings: kernel parameters: clock=pit nohpet nopmtimer vm.cfg: timer_mode = 2 With PLE enabled, 2.6.32 guests will crash early on with: ..MP-BIOS bug: 8254 timer not connected to IO-APIC # a few lines omitted.. Kernel panic - not syncing: IO-APIC + timer doesn''t work! Boot with apic=debug While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but continue and lock up in the serial line initialization. ..MP-BIOS bug: 8254 timer not connected to IO-APIC # continues until lock up here: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that jiffies isn''t advancing (or only 1 or 2 ticks are being received, which is insufficient for "working"). This is on a "quiet" system with no other activity. So, even though the guest has voluntarily yielded the cpu (through PLE), I would still expect it to receive every clock tick (even with timer_mode=2) as there is no other work to do on the system. Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an aside, so does setting ple_gap to 41 (ie prior to 21355:727ccaaa6cce) -- the perf counters show no exits happening, so this is equivalent to disabling PLE.] I''m hoping someone who knows the scheduler well will be able to quickly decide whether this is a bug or a feature... Andrew
On Tue, Dec 20, 2011 at 04:41:07PM -0400, Konrad Rzeszutek Wilk wrote:> Hey folks, > > I am sending this on behalf of Andrew since our internal email system > is dropping all xen-devel mailing lists :-(<hits his head> And I forgot to CC andrew on it. Added here.> > Anyhow: > > This is with xen-4.1-testing cs 23201:1c89f7d29fbb > and using the default "credit" scheduler. > > I''ve run into an interesting issue with HVM guests which > make use of Pause Loop Exiting (ie. on westmere systems; > and also on romley systems): after yielding the cpu, guests > don''t seem to receive timer interrupts correctly.. > > Some background: for historical reasons (ie old templates) we boot > OL/RHEL guests with the following settings: > > kernel parameters: clock=pit nohpet nopmtimer > vm.cfg: timer_mode = 2 > > With PLE enabled, 2.6.32 guests will crash early on with: > ..MP-BIOS bug: 8254 timer not connected to IO-APIC > # a few lines omitted.. > Kernel panic - not syncing: IO-APIC + timer doesn''t work! Boot with > apic=debug > > While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but > continue and lock up in the serial line initialization. > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC > # continues until lock up here: > Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled > > Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that jiffies > isn''t advancing (or only 1 or 2 ticks are being received, which is insufficient > for "working"). This is on a "quiet" system with no other activity. > So, even though the guest has voluntarily yielded the cpu (through PLE), > I would still expect it to receive every clock tick (even with timer_mode=2) > as there is no other work to do on the system. > > Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an > aside, so does setting ple_gap to 41 (ie prior to 21355:727ccaaa6cce) -- > the perf counters show no exits happening, so this is equivalent to disabling PLE.] > > I''m hoping someone who knows the scheduler well will be able to quickly > decide whether this is a bug or a feature... > > Andrew
We have reproduced your problem locally and are looking into this issue. It seems "PLE with timer mode 2" will trigger the issue. We can post our findings as soon as possible. Shan Haitao> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Konrad Rzeszutek Wilk > Sent: Wednesday, December 21, 2011 4:42 AM > To: xen-devel@lists.xensource.com; konrad.wilk@oracle.com; > George.Dunlap@eu.citrix.com; keir@xen.org; andrew.thomas@oracle.com > Subject: Re: [Xen-devel] issues with PLE and/or scheduler. > > On Tue, Dec 20, 2011 at 04:41:07PM -0400, Konrad Rzeszutek Wilk wrote: > > Hey folks, > > > > I am sending this on behalf of Andrew since our internal email system > > is dropping all xen-devel mailing lists :-( > > <hits his head> And I forgot to CC andrew on it. Added here. > > > > Anyhow: > > > > This is with xen-4.1-testing cs 23201:1c89f7d29fbb and using the > > default "credit" scheduler. > > > > I''ve run into an interesting issue with HVM guests which make use of > > Pause Loop Exiting (ie. on westmere systems; and also on romley > > systems): after yielding the cpu, guests don''t seem to receive timer > > interrupts correctly.. > > > > Some background: for historical reasons (ie old templates) we boot > > OL/RHEL guests with the following settings: > > > > kernel parameters: clock=pit nohpet nopmtimer > > vm.cfg: timer_mode = 2 > > > > With PLE enabled, 2.6.32 guests will crash early on with: > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC # a few lines > > omitted.. > > Kernel panic - not syncing: IO-APIC + timer doesn''t work! Boot with > > apic=debug > > > > While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but > > continue and lock up in the serial line initialization. > > > > ..MP-BIOS bug: 8254 timer not connected to IO-APIC # continues until > > lock up here: > > Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing > > enabled > > > > Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that > > jiffies isn''t advancing (or only 1 or 2 ticks are being received, > > which is insufficient for "working"). This is on a "quiet" system withno> other activity. > > So, even though the guest has voluntarily yielded the cpu (through > > PLE), I would still expect it to receive every clock tick (even with > > timer_mode=2) as there is no other work to do on the system. > > > > Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an > > aside, so does setting ple_gap to 41 (ie prior to 21355:727ccaaa6cce) > > -- the perf counters show no exits happening, so this is equivalent to > > disabling PLE.] > > > > I''m hoping someone who knows the scheduler well will be able to > > quickly decide whether this is a bug or a feature... > > > > Andrew > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
diff -r 381ab77db71a xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c Mon Apr 18 10:10:02 2011 +0100
+++ b/xen/arch/x86/hvm/vpt.c Thu Dec 22 05:54:54 2011 +0800
@@ -129,7 +129,7 @@
if ( missed_ticks <= 0 )
return;
- missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
+ missed_ticks = missed_ticks / (s_time_t) pt->period;
if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
pt->do_not_freeze = !pt->pending_intr_nr;
else
Anyone can explain the above "plus one" logic ? why assume at least
one tick is missed in pt_process_missed_ticks ?
In the guest kernel, ioapic''s check_timer logic is used to determine
how to set IRQ0, and it uses mdelay to delay 10 ticks totally. If kernel can
receive 4+ ticks during the delay, kernel deems IRQ0 is routed correctly through
ioapic.
Unfortunately, mdelay is implemented as a tight pause loop, when PLE is
enabled, the tight pause loop will trigger PLE vmexit. In the PLE vmexit
handler, scheduler yields the CPU, but the yield operation triggers
guest''s time save/restore logic,
eventually pt_process_missed_ticks gets called. Once pt_process_missed_ticks
is called, pt->scheduled is plused by one pt->period due to the above
"plus one" logic.
By default, ple_window is 4096, so each 4096 cycles in guest''s mdelay
triggers one PLE vmexit, and each vmexit delays the vpt timer by one
pt->period, so the vpt timer maybe never be fired during the guest''s
delay. This is why jiffies is not increased during the 10-tick mdelay.
Thanks!
Xiantao
-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Shan, Haitao
Sent: Wednesday, December 21, 2011 9:28 AM
To: Konrad Rzeszutek Wilk; xen-devel@lists.xensource.com;
konrad.wilk@oracle.com; George.Dunlap@eu.citrix.com; keir@xen.org;
andrew.thomas@oracle.com
Subject: Re: [Xen-devel] issues with PLE and/or scheduler.
We have reproduced your problem locally and are looking into this issue. It
seems "PLE with timer mode 2" will trigger the issue. We can post our
findings as soon as possible.
Shan Haitao
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Konrad Rzeszutek Wilk
> Sent: Wednesday, December 21, 2011 4:42 AM
> To: xen-devel@lists.xensource.com; konrad.wilk@oracle.com;
> George.Dunlap@eu.citrix.com; keir@xen.org; andrew.thomas@oracle.com
> Subject: Re: [Xen-devel] issues with PLE and/or scheduler.
>
> On Tue, Dec 20, 2011 at 04:41:07PM -0400, Konrad Rzeszutek Wilk wrote:
> > Hey folks,
> >
> > I am sending this on behalf of Andrew since our internal email
> > system is dropping all xen-devel mailing lists :-(
>
> <hits his head> And I forgot to CC andrew on it. Added here.
> >
> > Anyhow:
> >
> > This is with xen-4.1-testing cs 23201:1c89f7d29fbb and using the
> > default "credit" scheduler.
> >
> > I''ve run into an interesting issue with HVM guests which make
use of
> > Pause Loop Exiting (ie. on westmere systems; and also on romley
> > systems): after yielding the cpu, guests don''t seem to
receive
> > timer interrupts correctly..
> >
> > Some background: for historical reasons (ie old templates) we boot
> > OL/RHEL guests with the following settings:
> >
> > kernel parameters: clock=pit nohpet nopmtimer
> > vm.cfg: timer_mode = 2
> >
> > With PLE enabled, 2.6.32 guests will crash early on with:
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC # a few lines
> > omitted..
> > Kernel panic - not syncing: IO-APIC + timer doesn''t work!
Boot
> > with apic=debug
> >
> > While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but
> > continue and lock up in the serial line initialization.
> >
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC # continues
> > until lock up here:
> > Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing
> > enabled
> >
> > Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that
> > jiffies isn''t advancing (or only 1 or 2 ticks are being
received,
> > which is insufficient for "working"). This is on a
"quiet" system
> > with
no> other activity.
> > So, even though the guest has voluntarily yielded the cpu (through
> > PLE), I would still expect it to receive every clock tick (even with
> > timer_mode=2) as there is no other work to do on the system.
> >
> > Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an
> > aside, so does setting ple_gap to 41 (ie prior to
> > 21355:727ccaaa6cce)
> > -- the perf counters show no exits happening, so this is equivalent
> > to disabling PLE.]
> >
> > I''m hoping someone who knows the scheduler well will be able
to
> > quickly decide whether this is a bug or a feature...
> >
> > Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
On 21/12/2011 10:34, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:> diff -r 381ab77db71a xen/arch/x86/hvm/vpt.c > --- a/xen/arch/x86/hvm/vpt.c Mon Apr 18 10:10:02 2011 +0100 > +++ b/xen/arch/x86/hvm/vpt.c Thu Dec 22 05:54:54 2011 +0800 > @@ -129,7 +129,7 @@ > if ( missed_ticks <= 0 ) > return; > > - missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > + missed_ticks = missed_ticks / (s_time_t) pt->period; > if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > pt->do_not_freeze = !pt->pending_intr_nr; > else > > Anyone can explain the above "plus one" logic ? why assume at least one tick > is missed in pt_process_missed_ticks ?missed_ticks = now - pt->scheduled pt->scheduled was deadline for next tick. Hence the number of missed ticks is the total number of period that have passed since pt->scheduled, plus one. If we had not missed at least one tick, we would return early, from the if-statement at the top of your patch fragment, above. What''s the guest timer_mode? If there was at least one missed tick I would have expected a timer interrupt to get injected straight away. Except perhaps for timer_mode=2=no_missed_ticks_pending -- I don''t understand that timer mode, and perhaps there could be a bad interaction with its specific interrupt-holdoff logic. -- Keir> In the guest kernel, ioapic''s check_timer logic is used to determine how to > set IRQ0, and it uses mdelay to delay 10 ticks totally. If kernel can receive > 4+ ticks during the delay, kernel deems IRQ0 is routed correctly through > ioapic. > Unfortunately, mdelay is implemented as a tight pause loop, when PLE is > enabled, the tight pause loop will trigger PLE vmexit. In the PLE vmexit > handler, scheduler yields the CPU, but the yield operation triggers guest''s > time save/restore logic, > eventually pt_process_missed_ticks gets called. Once pt_process_missed_ticks > is called, pt->scheduled is plused by one pt->period due to the above "plus > one" logic. > By default, ple_window is 4096, so each 4096 cycles in guest''s mdelay > triggers one PLE vmexit, and each vmexit delays the vpt timer by one > pt->period, so the vpt timer maybe never be fired during the guest''s delay. > This is why jiffies is not increased during the 10-tick mdelay. > > Thanks! > Xiantao > > > > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Shan, Haitao > Sent: Wednesday, December 21, 2011 9:28 AM > To: Konrad Rzeszutek Wilk; xen-devel@lists.xensource.com; > konrad.wilk@oracle.com; George.Dunlap@eu.citrix.com; keir@xen.org; > andrew.thomas@oracle.com > Subject: Re: [Xen-devel] issues with PLE and/or scheduler. > > We have reproduced your problem locally and are looking into this issue. It > seems "PLE with timer mode 2" will trigger the issue. We can post our findings > as soon as possible. > > Shan Haitao > >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of Konrad Rzeszutek Wilk >> Sent: Wednesday, December 21, 2011 4:42 AM >> To: xen-devel@lists.xensource.com; konrad.wilk@oracle.com; >> George.Dunlap@eu.citrix.com; keir@xen.org; andrew.thomas@oracle.com >> Subject: Re: [Xen-devel] issues with PLE and/or scheduler. >> >> On Tue, Dec 20, 2011 at 04:41:07PM -0400, Konrad Rzeszutek Wilk wrote: >>> Hey folks, >>> >>> I am sending this on behalf of Andrew since our internal email >>> system is dropping all xen-devel mailing lists :-( >> >> <hits his head> And I forgot to CC andrew on it. Added here. >>> >>> Anyhow: >>> >>> This is with xen-4.1-testing cs 23201:1c89f7d29fbb and using the >>> default "credit" scheduler. >>> >>> I''ve run into an interesting issue with HVM guests which make use of >>> Pause Loop Exiting (ie. on westmere systems; and also on romley >>> systems): after yielding the cpu, guests don''t seem to receive >>> timer interrupts correctly.. >>> >>> Some background: for historical reasons (ie old templates) we boot >>> OL/RHEL guests with the following settings: >>> >>> kernel parameters: clock=pit nohpet nopmtimer >>> vm.cfg: timer_mode = 2 >>> >>> With PLE enabled, 2.6.32 guests will crash early on with: >>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC # a few lines >>> omitted.. >>> Kernel panic - not syncing: IO-APIC + timer doesn''t work! Boot >>> with apic=debug >>> >>> While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but >>> continue and lock up in the serial line initialization. >>> >>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC # continues >>> until lock up here: >>> Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing >>> enabled >>> >>> Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that >>> jiffies isn''t advancing (or only 1 or 2 ticks are being received, >>> which is insufficient for "working"). This is on a "quiet" system >>> with > no >> other activity. >>> So, even though the guest has voluntarily yielded the cpu (through >>> PLE), I would still expect it to receive every clock tick (even with >>> timer_mode=2) as there is no other work to do on the system. >>> >>> Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an >>> aside, so does setting ple_gap to 41 (ie prior to >>> 21355:727ccaaa6cce) >>> -- the perf counters show no exits happening, so this is equivalent >>> to disabling PLE.] >>> >>> I''m hoping someone who knows the scheduler well will be able to >>> quickly decide whether this is a bug or a feature... >>> >>> Andrew >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel
05:54:54 2011 +0800> > @@ -129,7 +129,7 @@ > > if ( missed_ticks <= 0 ) > > return; > > > > - missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > > + missed_ticks = missed_ticks / (s_time_t) pt->period; > > if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > > pt->do_not_freeze = !pt->pending_intr_nr; > > else > > > > Anyone can explain the above "plus one" logic ? why assume at least one > tick > > is missed in pt_process_missed_ticks ? > > missed_ticks = now - pt->scheduled > > pt->scheduled was deadline for next tick. Hence the number of missed > pt->ticks > is the total number of period that have passed since pt->scheduled, plus one. > If we had not missed at least one tick, we would return early, from the if- > statement at the top of your patch fragment, above. > What''s the guest timer_mode? If there was at least one missed tick I would > have expected a timer interrupt to get injected straight away. Except > perhaps for timer_mode=2=no_missed_ticks_pending -- I don''t understand > that timer mode, and perhaps there could be a bad interaction with its > specific interrupt-holdoff logic.Agree, in the if-statement, pt->do_not_freeze is set 1 if pt->pending_intr_nr ==0, so the timer is not stopped in next round''s pt_save_timer. In this case, we expect the timer can be fired after expiration. However, even if expiration occurs, the timer maybe not fired (because timer handler is executed in late softirq). In this case, pt_process_missd_tick may see one missed tick and then delay this timer by one period, but pt->pending_intr_nr is not increased due to not-fired timer. For example, in yield operation, due to short runq, scheduler may always select the previous vcpu as the next vcpu to run after yielding, so between pt_save_timer and pt_restore_timer, even if expiration occurs, it has no chance to fire the timer. However, pt_restore_timer may call pt_process_missed_tick to re-calculate pt->scheduled which will be plused one pt->period once expiration occurs, then the timer is delayed without increasing pt->pending_intr_nr. I think the below patch can fix this issue. if the timer is not stopped( in save logic )and not fired yet( in restore logic), we should keep the timer running until it is fired in softirq, and we shouldn''t delay it through re-calculating its expiration time, because this may lead to guest''s lost timer interrupts in some cases. diff -r 381ab77db71a xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Mon Apr 18 10:10:02 2011 +0100 +++ b/xen/arch/x86/hvm/vpt.c Thu Dec 22 11:35:36 2011 +0800 @@ -185,7 +185,7 @@ list_for_each_entry ( pt, head, list ) { - if ( pt->pending_intr_nr == 0 ) + if ( pt->pending_intr_nr == 0 && !pt->do_not_freeze) { pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled);> > We have reproduced your problem locally and are looking into this > > issue. It seems "PLE with timer mode 2" will trigger the issue. We can > > post our findings as soon as possible. > > > Shan Haitao > > > >> -----Original Message----- > >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > >> bounces@lists.xensource.com] On Behalf Of Konrad Rzeszutek Wilk > >> Sent: Wednesday, December 21, 2011 4:42 AM > >> To: xen-devel@lists.xensource.com; konrad.wilk@oracle.com; > >> George.Dunlap@eu.citrix.com; keir@xen.org; > andrew.thomas@oracle.com > >> Subject: Re: [Xen-devel] issues with PLE and/or scheduler. > >> > >> On Tue, Dec 20, 2011 at 04:41:07PM -0400, Konrad Rzeszutek Wilk wrote: > >>> Hey folks, > >>> > >>> I am sending this on behalf of Andrew since our internal email > >>> system is dropping all xen-devel mailing lists :-( > >> > >> <hits his head> And I forgot to CC andrew on it. Added here. > >>> > >>> Anyhow: > >>> > >>> This is with xen-4.1-testing cs 23201:1c89f7d29fbb and using the > >>> default "credit" scheduler. > >>> > >>> I''ve run into an interesting issue with HVM guests which make use of > >>> Pause Loop Exiting (ie. on westmere systems; and also on romley > >>> systems): after yielding the cpu, guests don''t seem to receive > >>> timer interrupts correctly.. > >>> > >>> Some background: for historical reasons (ie old templates) we boot > >>> OL/RHEL guests with the following settings: > >>> > >>> kernel parameters: clock=pit nohpet nopmtimer > >>> vm.cfg: timer_mode = 2 > >>> > >>> With PLE enabled, 2.6.32 guests will crash early on with: > >>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC # a few lines > >>> omitted.. > >>> Kernel panic - not syncing: IO-APIC + timer doesn''t work! Boot > >>> with apic=debug > >>> > >>> While 2.6.18-238 (ie OL/RHEL5u6) will fail to find the timer, but > >>> continue and lock up in the serial line initialization. > >>> > >>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC # continues > >>> until lock up here: > >>> Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing > >>> enabled > >>> > >>> Instrumenting the 2.6.32 code (ie timer_irq_works()) shows that > >>> jiffies isn''t advancing (or only 1 or 2 ticks are being received, > >>> which is insufficient for "working"). This is on a "quiet" system > >>> with > > no > >> other activity. > >>> So, even though the guest has voluntarily yielded the cpu (through > >>> PLE), I would still expect it to receive every clock tick (even with > >>> timer_mode=2) as there is no other work to do on the system. > >>> > >>> Disabling PLE allows both 2.6.18 and 2.6.32 guests to boot.. [As an > >>> aside, so does setting ple_gap to 41 (ie prior to > >>> 21355:727ccaaa6cce) > >>> -- the perf counters show no exits happening, so this is equivalent > >>> to disabling PLE.] > >>> > >>> I''m hoping someone who knows the scheduler well will be able to > >>> quickly decide whether this is a bug or a feature... > >>> > >>> Andrew > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel >