Xen devel - Feb 2009 - Align periodic vpts to reduce timer interrupts and save power

Hi,

After c/s 18694 changed vHPET to vpt, for single HVM RHEL 5u1 guest idle case,
our box will consume ~0.8W more power than before. The reason is two periodical
vpts'' expires are hard to be aligned in the 50us soft timer SLOP. So we
are considering a vpt specific enhancement which could try to just align
periodical timers within vpt.

A generic enhancement is to add a new interface like align_timer:
(create_periodic_time)
if ( period )
    pt->scheduled = align_timer(NOW() + delta)
...
set_timer(&pt->timer, pt->scheduled);

Then align_timer tries to find another timer closest to expected expiration, and
return to caller. This new interface only serves periodical timer, and just
called once at creation. Internal implementation could be more complex in the
future, e.g. align to timer with same period.

The key point for above changes, is to create more chances for timer alignment
even when timer slop is small. Large slop can align timers passively, but with
heavier impact to single shot timer.

Below is a simple align_timer() which aligns to the end of every ms (we can add
some boot option to control it):
s_time_t align_timer(s_time_t time)
{
    return time | (MILLISECS(1) - 1);
}

Is there any objections, comments or concerns for such a enhancement?

BTW, current vpt code try to handle vLAPIC specially to give vLAPIC ticks a
offset from other timer ticks like below:
    pt->scheduled = NOW() + delta;
    /*
     * Offset LAPIC ticks from other timer ticks. Otherwise guests which use
     * LAPIC ticks for process accounting can see long sequences of process
     * ticks incorrectly accounted to interrupt processing.
     */
    if ( pt->source == PTSRC_lapic )
        pt->scheduled += delta >> 1;

But I don''t think this way can really reach the offset purpose, because
the additional (delta >> 1) can''t prevent other timers created in
other time to expires right before lapic ticks.

Will it really bring issues if vlapic ticks and vhpet ticks were aligned?

Jimmy
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2009-02-09 at 14:26 +0800, Wei, Gang wrote:
...
> Is there any objections, comments or concerns for such a enhancement?
> 
> BTW, current vpt code try to handle vLAPIC specially to give vLAPIC ticks a
offset from other timer ticks like below:
>     pt->scheduled = NOW() + delta;
>     /*
>      * Offset LAPIC ticks from other timer ticks. Otherwise guests which
use
>      * LAPIC ticks for process accounting can see long sequences of process
>      * ticks incorrectly accounted to interrupt processing.
>      */
>     if ( pt->source == PTSRC_lapic )
>         pt->scheduled += delta >> 1;
> 
> But I don''t think this way can really reach the offset purpose,
because the additional (delta >> 1) can''t prevent other timers
created in other time to expires right before lapic ticks.
> 
> Will it really bring issues if vlapic ticks and vhpet ticks were aligned?
In that case, it was the vlapic timer ticking too close to the pit.  We
saw a userspace application getting the wrong answer because long CPU
bound sequences appeared to run with zero CPU time.  Skewing the vlapic
was sufficient to fix the problem.  This only showed up with old Linux
kernels (IIRC, it was with Red Hat 3 U8). 

If the effect of your proposed change is to always deliver all timer
interrupts to the guest in immediate succession, then I think you will
cause a regression on this problem.

-- 
Gary Grebus
Virtual Iron Software, Inc.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tuesday, February 10, 2009 2:44 AM, Gary Grebus
wrote:
> On Mon, 2009-02-09 at 14:26 +0800, Wei, Gang wrote:
> ...
>> Is there any objections, comments or concerns for such a enhancement?
>> 
>> BTW, current vpt code try to handle vLAPIC specially to give vLAPIC
ticks a
>>     offset from other timer ticks like below: pt->scheduled = NOW()
+ delta;
>>     /*
>>      * Offset LAPIC ticks from other timer ticks. Otherwise guests
which use
>>      * LAPIC ticks for process accounting can see long sequences of
process
>>      * ticks incorrectly accounted to interrupt processing.      */
>>     if ( pt->source == PTSRC_lapic )
>>         pt->scheduled += delta >> 1;
>> 
>> But I don''t think this way can really reach the offset
purpose, because the
>> additional (delta >> 1) can''t prevent other timers
created in other time to
>> expires right before lapic ticks.  
>> 
>> Will it really bring issues if vlapic ticks and vhpet ticks were
aligned?
> 
> In that case, it was the vlapic timer ticking too close to the pit.  We
> saw a userspace application getting the wrong answer because long CPU
> bound sequences appeared to run with zero CPU time.  Skewing the vlapic
> was sufficient to fix the problem.  This only showed up with old Linux
> kernels (IIRC, it was with Red Hat 3 U8).
> 
> If the effect of your proposed change is to always deliver all timer
> interrupts to the guest in immediate succession, then I think you will
> cause a regression on this problem.
Thanks for the reply. First, we have the same concern. But it''s only
for specific purpose (accounting) in specific old guest. For most other guests I
am afraid this offset can''t really resist vpit or other vpts being
delivered with vlapic in immediate succession if guest initialize PIT or other
platform timer a bit far away from LAPIC init. Anyway, we could mention it as
the alerts for this feature when people''d like to save power. BTW, the
range timer has similar effect which could nullify the offset effect too.

Second, our solution does not intend to remove this offset. We just tries to
provide a power-saving feature via a configurable way. For such feature some
side effect may exist but the overall gain is more power efficient. I will make
my patch ready soon, and we can do further discussion on the implementation
level then.

Jimmy
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2009-02-10 at 22:42 +0800, Wei, Gang wrote:
> On Tuesday, February 10, 2009 2:44 AM, Gary Grebus wrote:
> > On Mon, 2009-02-09 at 14:26 +0800, Wei, Gang wrote:
> > ...
> >> Is there any objections, comments or concerns for such a
enhancement?
> >> 
> >> BTW, current vpt code try to handle vLAPIC specially to give
vLAPIC ticks a
> >>     offset from other timer ticks like below: pt->scheduled =
NOW() + delta;
> >>     /*
> >>      * Offset LAPIC ticks from other timer ticks. Otherwise guests
which use
> >>      * LAPIC ticks for process accounting can see long sequences
of process
> >>      * ticks incorrectly accounted to interrupt processing.     
*/
> >>     if ( pt->source == PTSRC_lapic )
> >>         pt->scheduled += delta >> 1;
> >> 
> >> But I don''t think this way can really reach the offset
purpose, because the
> >> additional (delta >> 1) can''t prevent other timers
created in other time to
> >> expires right before lapic ticks.  
> >> 
> >> Will it really bring issues if vlapic ticks and vhpet ticks were
aligned?
> > 
> > In that case, it was the vlapic timer ticking too close to the pit. 
We
> > saw a userspace application getting the wrong answer because long CPU
> > bound sequences appeared to run with zero CPU time.  Skewing the
vlapic
> > was sufficient to fix the problem.  This only showed up with old Linux
> > kernels (IIRC, it was with Red Hat 3 U8).
> > 
> > If the effect of your proposed change is to always deliver all timer
> > interrupts to the guest in immediate succession, then I think you will
> > cause a regression on this problem.
> 
> Thanks for the reply. First, we have the same concern. But it''s
only for specific purpose (accounting) in specific old guest. For most other
guests I am afraid this offset can''t really resist vpit or other vpts
being delivered with vlapic in immediate succession if guest initialize PIT or
other platform timer a bit far away from LAPIC init. Anyway, we could mention it
as the alerts for this feature when people''d like to save power. BTW,
the range timer has similar effect which could nullify the offset effect too.
> 
But the customer using that accounting data won''t be happy, even if
they
are saving a couple watts.  Especially since the data is likely to be
only intermittently wrong.  It sounds like, for correctness, there
should be more done to avoid having the virtual timers tick in immediate
succession, not less as you propose.  

I suppose if there is a configuration parameter, we can deal with it by
adding one more guest-specific configuration rule.

	/gary
> Second, our solution does not intend to remove this offset. We just tries
to provide a power-saving feature via a configurable way. For such feature some
side effect may exist but the overall gain is more power efficient. I will make
my patch ready soon, and we can do further discussion on the implementation
level then.
> 
> Jimmy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tuesday, February 10, 2009 11:56 PM, Gary Grebus wrote:
...
> But the customer using that accounting data won''t be happy, even
if they
> are saving a couple watts.  Especially since the data is likely to be
> only intermittently wrong.  It sounds like, for correctness, there
> should be more done to avoid having the virtual timers tick in immediate
> succession, not less as you propose.
The customer would be happy if he keeps getting the correct accounting data
while he want it, meanwhile he have another choice to save more power while he
don''t care the accounting data. That is what we try to do.
> 
> I suppose if there is a configuration parameter, we can deal with it by
> adding one more guest-specific configuration rule.
Good suggestion. The feature would rather fit in guest-specific category, so we
can consider how to do it well.

Jimmy
> 
> 	/gary
> 
>> Second, our solution does not intend to remove this offset. We just
tries to
>> provide a power-saving feature via a configurable way. For such feature
some
>> side effect may exist but the overall gain is more power efficient. I
will
>> make my patch ready soon, and we can do further discussion on the
>> implementation level then.    
>> 
>> Jimmy
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel