Xen devel - Oct 2007 - [PATCH] Add a timer mode that disables pending missed ticks

Hi, Keir,

This patch adds a new timer mode, in which no missed ticks is
calculated. This can be used with latest x86_64 linux guest, since it
can pick up missed ticks themselves.
 <<no_missed_ticks.patch>> 

Best Regards
Haitao Shan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Applied as c/s 16274. Please take a look and make sure the mode works as you
expect.

 -- Keir

On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:
> Hi, Keir, 
> 
> This patch adds a new timer mode, in which no missed ticks is calculated.
This
> can be used with latest x86_64 linux guest, since it can pick up missed
ticks
> themselves.
> 
>  <<no_missed_ticks.patch>>
> 
> Best Regards 
> Haitao Shan 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

Here are my comments on your change.
I''ve attached an updated vpt.c with the changes.

1. For the no_missed_tick_accounting method, we still need the update
    to pt->scheduled taking into account the time that has elapsed when
    missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
    is called from pt_restore_timer() and pt_timer_fn() and, thus, its
    easiest to put the check for no_missed_tick_accounting method in
    missed_ticks() itself.

2. In pt_timer_fn you don''t want to increment pending_intr_nr beyond 1
    for no_missed_tick_accounting option.

thanks,
Dave


Keir Fraser wrote:
>
> Applied as c/s 16274. Please take a look and make sure the mode works 
> as you expect.
>
>  -- Keir
>
> On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com>
wrote:
>
>     Hi, Keir,
>
>     This patch adds a new timer mode, in which no missed ticks is
>     calculated. This can be used with latest x86_64 linux guest, since
>     it can pick up missed ticks themselves.
>
>      <<no_missed_ticks.patch>>
>
>     Best Regards
>     Haitao Shan
>
>    
------------------------------------------------------------------------
>     _______________________________________________
>     Xen-devel mailing list
>     Xen-devel@lists.xensource.com
>     http://lists.xensource.com/xen-devel
>
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Keir, Dave
 
Sorry, I made a silly mistake for my last patch. The attached will fix it.
 
This piece of code should actually be added in pt_restore_timer rather than
pt_timer_fn.
+        else if ( (NOW() - pt->scheduled) >= 0 )
+        {
+            pt->pending_intr_nr++;
+            pt->scheduled = NOW() + pt->period;
+        }

Best Regards 
Haitao Shan 

 

________________________________

From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: 2007年10月31日 0:13
To: Shan, Haitao
Cc: xen-devel@lists.xensource.com; Dong, Eddie; Jiang, Yunhong; Dave Winchell
Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending missed
ticks



Applied as c/s 16274. Please take a look and make sure the mode works as you
expect.

 -- Keir

On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> wrote:



	Hi, Keir, 
	
	This patch adds a new timer mode, in which no missed ticks is calculated. This
can be used with latest x86_64 linux guest, since it can pick up missed ticks
themselves.
	
	 <<no_missed_ticks.patch>> 
	
	Best Regards 
	Haitao Shan 
	
	
________________________________

	_______________________________________________
	Xen-devel mailing list
	Xen-devel@lists.xensource.com
	http://lists.xensource.com/xen-devel
	






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''m going to apply Haitao''s no_missed_ticks_fix.patch. If you
think fixes
are needed apart from that, please provide a unified diff.

 -- Keir

On 30/10/07 21:16, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Keir,
> 
> Here are my comments on your change.
> I''ve attached an updated vpt.c with the changes.
> 
> 1. For the no_missed_tick_accounting method, we still need the update
>     to pt->scheduled taking into account the time that has elapsed when
>     missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
>     is called from pt_restore_timer() and pt_timer_fn() and, thus, its
>     easiest to put the check for no_missed_tick_accounting method in
>     missed_ticks() itself.
> 
> 2. In pt_timer_fn you don''t want to increment pending_intr_nr
beyond 1
>     for no_missed_tick_accounting option.
> 
> thanks,
> Dave
> 
> 
> Keir Fraser wrote:
> 
>> 
>> Applied as c/s 16274. Please take a look and make sure the mode works
>> as you expect.
>> 
>>  -- Keir
>> 
>> On 30/10/07 14:28, "Shan, Haitao"
<haitao.shan@intel.com> wrote:
>> 
>>     Hi, Keir,
>> 
>>     This patch adds a new timer mode, in which no missed ticks is
>>     calculated. This can be used with latest x86_64 linux guest, since
>>     it can pick up missed ticks themselves.
>> 
>>      <<no_missed_ticks.patch>>
>> 
>>     Best Regards
>>     Haitao Shan
>> 
>>    
------------------------------------------------------------------------
>>     _______________________________________________
>>     Xen-devel mailing list
>>     Xen-devel@lists.xensource.com
>>     http://lists.xensource.com/xen-devel
>> 
>> 
> 
> *** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400
> --- vpt.c 2007-10-30 15:30:57.000000000 -0400
> ***************
> *** 57,73 ****
>           return;
>   
>       missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> ! 
> !     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
> !  if ( missed_ticks > 1000 )
> !      {
> !   /* TODO: Adjust guest time together */
> !   pt->pending_intr_nr++;
> !      }
> !  else
> !      {
> !   pt->pending_intr_nr += missed_ticks;
> !      }
>       }
>   
>       pt->scheduled += missed_ticks * pt->period;
> --- 57,70 ----
>           return;
>   
>       missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> !     if ( missed_ticks > 1000 )
> !     {
> !         /* TODO: Adjust guest time together */
> !         pt->pending_intr_nr++;
> !     }
> !     else
> !     {
> !         pt->pending_intr_nr += missed_ticks;
>       }
>   
>       pt->scheduled += missed_ticks * pt->period;
> ***************
> *** 120,126 ****
>   
>       list_for_each_entry ( pt, head, list )
>       {
> !  pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> --- 117,124 ----
>   
>       list_for_each_entry ( pt, head, list )
>       {
> !         if ( !mode_is(v->domain, no_missed_tick_accounting) )
> !             pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> ***************
> *** 135,151 ****
>   
>       pt_lock(pt);
>   
> !     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
> !  if(!pt->pending_intr_nr)
> !      pt->pending_intr_nr++;
> !     }
> !     else
> !  pt->pending_intr_nr++;
>   
>       if ( !pt->one_shot )
>       {
>           pt->scheduled += pt->period;
> !  pt_process_missed_ticks(pt);
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> --- 133,152 ----
>   
>       pt_lock(pt);
>   
> !     pt->pending_intr_nr++;
>   
>       if ( !pt->one_shot )
>       {
>           pt->scheduled += pt->period;
> !         if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting)
)
> !         {
> !             pt_process_missed_ticks(pt);
> !         }
> !         else if ( (NOW() - pt->scheduled) >= 0 )
> !         {
> !             pt->pending_intr_nr++;
> !             pt->scheduled = NOW() + pt->period;
> !         }
>           set_timer(&pt->timer, pt->scheduled);
>       }
>   
> /*
>  * vpt.c: Virtual Platform Timer
>  *
>  * Copyright (c) 2006, Xiaowei Yang, Intel Corporation.
>  *
>  * This program is free software; you can redistribute it and/or modify it
>  * under the terms and conditions of the GNU General Public License,
>  * version 2, as published by the Free Software Foundation.
>  *
>  * This program is distributed in the hope it will be useful, but WITHOUT
>  * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>  * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
for
>  * more details.
>  *
>  * You should have received a copy of the GNU General Public License along
> with
>  * this program; if not, write to the Free Software Foundation, Inc., 59
> Temple
>  * Place - Suite 330, Boston, MA 02111-1307 USA.
>  *
>  */
> 
> #include <xen/time.h>
> #include <asm/hvm/support.h>
> #include <asm/hvm/vpt.h>
> #include <asm/event.h>
> 
> #define mode_is(d, name) \
>     ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name)
> 
> static void pt_lock(struct periodic_time *pt)
> {
>     struct vcpu *v;
> 
>     for ( ; ; )
>     {
>         v = pt->vcpu;
>         spin_lock(&v->arch.hvm_vcpu.tm_lock);
>         if ( likely(pt->vcpu == v) )
>             break;
>         spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>     }
> }
> 
> static void pt_unlock(struct periodic_time *pt)
> {
>     spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
> }
> 
> static void pt_process_missed_ticks(struct periodic_time *pt)
> {
>     s_time_t missed_ticks;
> 
>     if ( pt->one_shot )
>         return;
> 
>     missed_ticks = NOW() - pt->scheduled;
>     if ( missed_ticks <= 0 )
>         return;
> 
>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> 
>     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
> if ( missed_ticks > 1000 )
>    {
> /* TODO: Adjust guest time together */
> pt->pending_intr_nr++;
>    }
> else
>    {
> pt->pending_intr_nr += missed_ticks;
>    }
>     }
> 
>     pt->scheduled += missed_ticks * pt->period;
> }
> 
> static void pt_freeze_time(struct vcpu *v)
> {
>     if ( !mode_is(v->domain, delay_for_missed_ticks) )
>         return;
> 
>     v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v);
> }
> 
> static void pt_thaw_time(struct vcpu *v)
> {
>     if ( !mode_is(v->domain, delay_for_missed_ticks) )
>         return;
> 
>     if ( v->arch.hvm_vcpu.guest_time == 0 )
>         return;
> 
>     hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time);
>     v->arch.hvm_vcpu.guest_time = 0;
> }
> 
> void pt_save_timer(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     if ( test_bit(_VPF_blocked, &v->pause_flags) )
>         return;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>         stop_timer(&pt->timer);
> 
>     pt_freeze_time(v);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void pt_restore_timer(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
> pt_process_missed_ticks(pt);
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     pt_thaw_time(v);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> static void pt_timer_fn(void *data)
> {
>     struct periodic_time *pt = data;
> 
>     pt_lock(pt);
> 
>     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
> if(!pt->pending_intr_nr)
>    pt->pending_intr_nr++;
>     }
>     else
> pt->pending_intr_nr++;
> 
>     if ( !pt->one_shot )
>     {
>         pt->scheduled += pt->period;
> pt_process_missed_ticks(pt);
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     vcpu_kick(pt->vcpu);
> 
>     pt_unlock(pt);
> }
> 
> void pt_update_irq(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
>     uint64_t max_lag = -1ULL;
>     int irq = -1;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         if ( !is_isa_irq_masked(v, pt->irq) &&
pt->pending_intr_nr &&
>              ((pt->last_plt_gtime + pt->period_cycles) < max_lag)
)
>         {
>             max_lag = pt->last_plt_gtime + pt->period_cycles;
>             irq = pt->irq;
>         }
>     }
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> 
>     if ( is_lvtt(v, irq) )
>     {
>         vlapic_set_irq(vcpu_vlapic(v), irq, 0);
>     }
>     else if ( irq >= 0 )
>     {
>         hvm_isa_irq_deassert(v->domain, irq);
>         hvm_isa_irq_assert(v->domain, irq);
>     }
> }
> 
> static struct periodic_time *is_pt_irq(
>     struct vcpu *v, struct hvm_intack intack)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
>     struct RTCState *rtc =
&v->domain->arch.hvm_domain.pl_time.vrtc;
>     int vector;
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         if ( !pt->pending_intr_nr )
>             continue;
> 
>         if ( is_lvtt(v, pt->irq) )
>         {
>             if ( pt->irq != intack.vector )
>                 continue;
>             return pt;
>         }
> 
>         vector = get_isa_irq_vector(v, pt->irq, intack.source);
> 
>         /* RTC irq need special care */
>         if ( (intack.vector != vector) ||
>              ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) )
>             continue;
> 
>         return pt;
>     }
> 
>     return NULL;
> }
> 
> void pt_intr_post(struct vcpu *v, struct hvm_intack intack)
> {
>     struct periodic_time *pt;
>     time_cb *cb;
>     void *cb_priv;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     pt = is_pt_irq(v, intack);
>     if ( pt == NULL )
>     {
>         spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>         return;
>     }
> 
>     if ( pt->one_shot )
>     {
>         pt->enabled = 0;
>         list_del(&pt->list);
>     }
>     else
>     {
>         pt->pending_intr_nr--;
>         if ( mode_is(v->domain, no_missed_tick_accounting) )
>             pt->last_plt_gtime = hvm_get_guest_time(v);
>         else
>             pt->last_plt_gtime += pt->period_cycles;
>     }
> 
>     if ( mode_is(v->domain, delay_for_missed_ticks) &&
>          (hvm_get_guest_time(v) < pt->last_plt_gtime) )
>         hvm_set_guest_time(v, pt->last_plt_gtime);
> 
>     cb = pt->cb;
>     cb_priv = pt->priv;
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> 
>     if ( cb != NULL )
>         cb(v, cb_priv);
> }
> 
> void pt_reset(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>     {
>         pt->pending_intr_nr = 0;
>         pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>         pt->scheduled = NOW() + pt->period;
>         set_timer(&pt->timer, pt->scheduled);
>     }
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void pt_migrate(struct vcpu *v)
> {
>     struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>     struct periodic_time *pt;
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     list_for_each_entry ( pt, head, list )
>         migrate_timer(&pt->timer, v->processor);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void create_periodic_time(
>     struct vcpu *v, struct periodic_time *pt, uint64_t period,
>     uint8_t irq, char one_shot, time_cb *cb, void *data)
> {
>     destroy_periodic_time(pt);
> 
>     spin_lock(&v->arch.hvm_vcpu.tm_lock);
> 
>     pt->enabled = 1;
>     pt->pending_intr_nr = 0;
> 
>     /* Periodic timer must be at least 0.9ms. */
>     if ( (period < 900000) && !one_shot )
>     {
>         gdprintk(XENLOG_WARNING,
>                  "HVM_PlatformTime: program too small period
%"PRIu64"\n",
>                  period);
>         period = 900000;
>     }
> 
>     pt->period = period;
>     pt->vcpu = v;
>     pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>     pt->irq = irq;
>     pt->period_cycles = (u64)period * cpu_khz / 1000000L;
>     pt->one_shot = one_shot;
>     pt->scheduled = NOW() + period;
>     /*
>      * Offset LAPIC ticks from other timer ticks. Otherwise guests which
use
>      * LAPIC ticks for process accounting can see long sequences of process
>      * ticks incorrectly accounted to interrupt processing.
>      */
>     if ( is_lvtt(v, irq) )
>         pt->scheduled += period >> 1;
>     pt->cb = cb;
>     pt->priv = data;
> 
>     list_add(&pt->list, &v->arch.hvm_vcpu.tm_list);
> 
>     init_timer(&pt->timer, pt_timer_fn, pt, v->processor);
>     set_timer(&pt->timer, pt->scheduled);
> 
>     spin_unlock(&v->arch.hvm_vcpu.tm_lock);
> }
> 
> void destroy_periodic_time(struct periodic_time *pt)
> {
>     if ( !pt->enabled )
>         return;
> 
>     pt_lock(pt);
>     pt->enabled = 0;
>     list_del(&pt->list);
>     pt_unlock(pt);
> 
>     /*
>      * pt_timer_fn() can run until this kill_timer() returns. We must do
this
>      * outside pt_lock() otherwise we can deadlock with pt_timer_fn().
>      */
>     kill_timer(&pt->timer);
> }


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir, Haitao, Eddie:

Here is the patch for timekeeping.
There are three components.

1. Increase missed ticks threshold to 100 seconds. Per the comment in 
the patch,
    I have seen scheduling delays of 5 seconds in a test lasting 30 minutes.
    The result is an immediate error of 5 seconds in guest time that
    persists.

2.  In pt_timer_fn for no_missed_tick_accounting option, only
    increment pt->pending_intr_nr if its found to be zero.
    This deals with the case where the guest has interrupts disabled
    for longer than a tick period.

3.  Call pt_process_missed_ticks() unconditionally and place the
    test for no_missed_tick_accounting inside pt_process_missed_ticks().
    This returns the calculation for the next timer expiration
    to the original method. The original method is important
    as it sets up a theoretical time space at t=0 where expirations
    occur only at n*period, where n is any integer. This, in turn, removes
    rounding errors.

Accuracy tests:

A.  32bit Linux guest (Item ''1'' above relevant)
    Test: 2 64bit Linux guests and 1 32bit Linux guest stacked.
          8 vcpus for each guest.
            2 physical processors on the node.
          All vcpus heavily loaded with work.
          Test duration: 3 hours.

        Result: clock error < .02%.
        (Before this patch guest time was 16 seconds slow in a 3.5 
minute test.)

B.  64bit Linux guest (Items ''2'' and ''3''
above relevant)
    Test: 2 64bit Linux guests.
          8 vcpus for each guest.
          2 physical processors on the node.
          All vcpus heavily loaded with work.
          Test duration: 3 hours.

        Result: clock error  .02%.
        (Before this patch clock error was 1.3%. The contributions of
    items ''2'' and ''3'' above to the
improvement in accuracy are .4%
    and .9% respectively.

Of course, these tests are without ntpd running in the guest.
Ntpd requires a local clock error of < .05%.
With the .02% accuracy, one can used ntpd to synchronize the clock.


thanks,
Dave


Keir Fraser wrote:
>I''m going to apply Haitao''s no_missed_ticks_fix.patch. If
you think fixes
>are needed apart from that, please provide a unified diff.
>
> -- Keir
>
>On 30/10/07 21:16, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Keir,
>>
>>Here are my comments on your change.
>>I''ve attached an updated vpt.c with the changes.
>>
>>1. For the no_missed_tick_accounting method, we still need the update
>>    to pt->scheduled taking into account the time that has elapsed
when
>>    missed ticks are calculated. missed_ticks (pt_process_missed_ticks)
>>    is called from pt_restore_timer() and pt_timer_fn() and, thus, its
>>    easiest to put the check for no_missed_tick_accounting method in
>>    missed_ticks() itself.
>>
>>2. In pt_timer_fn you don''t want to increment pending_intr_nr
beyond 1
>>    for no_missed_tick_accounting option.
>>
>>thanks,
>>Dave
>>
>>
>>Keir Fraser wrote:
>>
>>    
>>
>>>Applied as c/s 16274. Please take a look and make sure the mode
works
>>>as you expect.
>>>
>>> -- Keir
>>>
>>>On 30/10/07 14:28, "Shan, Haitao"
<haitao.shan@intel.com> wrote:
>>>
>>>    Hi, Keir,
>>>
>>>    This patch adds a new timer mode, in which no missed ticks is
>>>    calculated. This can be used with latest x86_64 linux guest,
since
>>>    it can pick up missed ticks themselves.
>>>
>>>     <<no_missed_ticks.patch>>
>>>
>>>    Best Regards
>>>    Haitao Shan
>>>
>>>   
------------------------------------------------------------------------
>>>    _______________________________________________
>>>    Xen-devel mailing list
>>>    Xen-devel@lists.xensource.com
>>>    http://lists.xensource.com/xen-devel
>>>
>>>
>>>      
>>>
>>*** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400
>>--- vpt.c 2007-10-30 15:30:57.000000000 -0400
>>***************
>>*** 57,73 ****
>>          return;
>>  
>>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>! 
>>!     if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
{
>>!  if ( missed_ticks > 1000 )
>>!      {
>>!   /* TODO: Adjust guest time together */
>>!   pt->pending_intr_nr++;
>>!      }
>>!  else
>>!      {
>>!   pt->pending_intr_nr += missed_ticks;
>>!      }
>>      }
>>  
>>      pt->scheduled += missed_ticks * pt->period;
>>--- 57,70 ----
>>          return;
>>  
>>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>!     if ( missed_ticks > 1000 )
>>!     {
>>!         /* TODO: Adjust guest time together */
>>!         pt->pending_intr_nr++;
>>!     }
>>!     else
>>!     {
>>!         pt->pending_intr_nr += missed_ticks;
>>      }
>>  
>>      pt->scheduled += missed_ticks * pt->period;
>>***************
>>*** 120,126 ****
>>  
>>      list_for_each_entry ( pt, head, list )
>>      {
>>!  pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>--- 117,124 ----
>>  
>>      list_for_each_entry ( pt, head, list )
>>      {
>>!         if ( !mode_is(v->domain, no_missed_tick_accounting) )
>>!             pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>***************
>>*** 135,151 ****
>>  
>>      pt_lock(pt);
>>  
>>!     if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
>>!  if(!pt->pending_intr_nr)
>>!      pt->pending_intr_nr++;
>>!     }
>>!     else
>>!  pt->pending_intr_nr++;
>>  
>>      if ( !pt->one_shot )
>>      {
>>          pt->scheduled += pt->period;
>>!  pt_process_missed_ticks(pt);
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>--- 133,152 ----
>>  
>>      pt_lock(pt);
>>  
>>!     pt->pending_intr_nr++;
>>  
>>      if ( !pt->one_shot )
>>      {
>>          pt->scheduled += pt->period;
>>!         if ( !mode_is(pt->vcpu->domain,
no_missed_tick_accounting) )
>>!         {
>>!             pt_process_missed_ticks(pt);
>>!         }
>>!         else if ( (NOW() - pt->scheduled) >= 0 )
>>!         {
>>!             pt->pending_intr_nr++;
>>!             pt->scheduled = NOW() + pt->period;
>>!         }
>>          set_timer(&pt->timer, pt->scheduled);
>>      }
>>  
>>/*
>> * vpt.c: Virtual Platform Timer
>> *
>> * Copyright (c) 2006, Xiaowei Yang, Intel Corporation.
>> *
>> * This program is free software; you can redistribute it and/or modify
it
>> * under the terms and conditions of the GNU General Public License,
>> * version 2, as published by the Free Software Foundation.
>> *
>> * This program is distributed in the hope it will be useful, but
WITHOUT
>> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
for
>> * more details.
>> *
>> * You should have received a copy of the GNU General Public License
along
>>with
>> * this program; if not, write to the Free Software Foundation, Inc., 59
>>Temple
>> * Place - Suite 330, Boston, MA 02111-1307 USA.
>> *
>> */
>>
>>#include <xen/time.h>
>>#include <asm/hvm/support.h>
>>#include <asm/hvm/vpt.h>
>>#include <asm/event.h>
>>
>>#define mode_is(d, name) \
>>    ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] ==
HVMPTM_##name)
>>
>>static void pt_lock(struct periodic_time *pt)
>>{
>>    struct vcpu *v;
>>
>>    for ( ; ; )
>>    {
>>        v = pt->vcpu;
>>        spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>        if ( likely(pt->vcpu == v) )
>>            break;
>>        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>    }
>>}
>>
>>static void pt_unlock(struct periodic_time *pt)
>>{
>>    spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock);
>>}
>>
>>static void pt_process_missed_ticks(struct periodic_time *pt)
>>{
>>    s_time_t missed_ticks;
>>
>>    if ( pt->one_shot )
>>        return;
>>
>>    missed_ticks = NOW() - pt->scheduled;
>>    if ( missed_ticks <= 0 )
>>        return;
>>
>>    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>
>>    if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) {
>>if ( missed_ticks > 1000 )
>>   {
>>/* TODO: Adjust guest time together */
>>pt->pending_intr_nr++;
>>   }
>>else
>>   {
>>pt->pending_intr_nr += missed_ticks;
>>   }
>>    }
>>
>>    pt->scheduled += missed_ticks * pt->period;
>>}
>>
>>static void pt_freeze_time(struct vcpu *v)
>>{
>>    if ( !mode_is(v->domain, delay_for_missed_ticks) )
>>        return;
>>
>>    v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v);
>>}
>>
>>static void pt_thaw_time(struct vcpu *v)
>>{
>>    if ( !mode_is(v->domain, delay_for_missed_ticks) )
>>        return;
>>
>>    if ( v->arch.hvm_vcpu.guest_time == 0 )
>>        return;
>>
>>    hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time);
>>    v->arch.hvm_vcpu.guest_time = 0;
>>}
>>
>>void pt_save_timer(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    if ( test_bit(_VPF_blocked, &v->pause_flags) )
>>        return;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>        stop_timer(&pt->timer);
>>
>>    pt_freeze_time(v);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void pt_restore_timer(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>pt_process_missed_ticks(pt);
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    pt_thaw_time(v);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>static void pt_timer_fn(void *data)
>>{
>>    struct periodic_time *pt = data;
>>
>>    pt_lock(pt);
>>
>>    if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) {
>>if(!pt->pending_intr_nr)
>>   pt->pending_intr_nr++;
>>    }
>>    else
>>pt->pending_intr_nr++;
>>
>>    if ( !pt->one_shot )
>>    {
>>        pt->scheduled += pt->period;
>>pt_process_missed_ticks(pt);
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    vcpu_kick(pt->vcpu);
>>
>>    pt_unlock(pt);
>>}
>>
>>void pt_update_irq(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>    uint64_t max_lag = -1ULL;
>>    int irq = -1;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        if ( !is_isa_irq_masked(v, pt->irq) &&
pt->pending_intr_nr &&
>>             ((pt->last_plt_gtime + pt->period_cycles) <
max_lag) )
>>        {
>>            max_lag = pt->last_plt_gtime + pt->period_cycles;
>>            irq = pt->irq;
>>        }
>>    }
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    if ( is_lvtt(v, irq) )
>>    {
>>        vlapic_set_irq(vcpu_vlapic(v), irq, 0);
>>    }
>>    else if ( irq >= 0 )
>>    {
>>        hvm_isa_irq_deassert(v->domain, irq);
>>        hvm_isa_irq_assert(v->domain, irq);
>>    }
>>}
>>
>>static struct periodic_time *is_pt_irq(
>>    struct vcpu *v, struct hvm_intack intack)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>    struct RTCState *rtc =
&v->domain->arch.hvm_domain.pl_time.vrtc;
>>    int vector;
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        if ( !pt->pending_intr_nr )
>>            continue;
>>
>>        if ( is_lvtt(v, pt->irq) )
>>        {
>>            if ( pt->irq != intack.vector )
>>                continue;
>>            return pt;
>>        }
>>
>>        vector = get_isa_irq_vector(v, pt->irq, intack.source);
>>
>>        /* RTC irq need special care */
>>        if ( (intack.vector != vector) ||
>>             ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) )
>>            continue;
>>
>>        return pt;
>>    }
>>
>>    return NULL;
>>}
>>
>>void pt_intr_post(struct vcpu *v, struct hvm_intack intack)
>>{
>>    struct periodic_time *pt;
>>    time_cb *cb;
>>    void *cb_priv;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    pt = is_pt_irq(v, intack);
>>    if ( pt == NULL )
>>    {
>>        spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>        return;
>>    }
>>
>>    if ( pt->one_shot )
>>    {
>>        pt->enabled = 0;
>>        list_del(&pt->list);
>>    }
>>    else
>>    {
>>        pt->pending_intr_nr--;
>>        if ( mode_is(v->domain, no_missed_tick_accounting) )
>>            pt->last_plt_gtime = hvm_get_guest_time(v);
>>        else
>>            pt->last_plt_gtime += pt->period_cycles;
>>    }
>>
>>    if ( mode_is(v->domain, delay_for_missed_ticks) &&
>>         (hvm_get_guest_time(v) < pt->last_plt_gtime) )
>>        hvm_set_guest_time(v, pt->last_plt_gtime);
>>
>>    cb = pt->cb;
>>    cb_priv = pt->priv;
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    if ( cb != NULL )
>>        cb(v, cb_priv);
>>}
>>
>>void pt_reset(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>    {
>>        pt->pending_intr_nr = 0;
>>        pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>>        pt->scheduled = NOW() + pt->period;
>>        set_timer(&pt->timer, pt->scheduled);
>>    }
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void pt_migrate(struct vcpu *v)
>>{
>>    struct list_head *head = &v->arch.hvm_vcpu.tm_list;
>>    struct periodic_time *pt;
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    list_for_each_entry ( pt, head, list )
>>        migrate_timer(&pt->timer, v->processor);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void create_periodic_time(
>>    struct vcpu *v, struct periodic_time *pt, uint64_t period,
>>    uint8_t irq, char one_shot, time_cb *cb, void *data)
>>{
>>    destroy_periodic_time(pt);
>>
>>    spin_lock(&v->arch.hvm_vcpu.tm_lock);
>>
>>    pt->enabled = 1;
>>    pt->pending_intr_nr = 0;
>>
>>    /* Periodic timer must be at least 0.9ms. */
>>    if ( (period < 900000) && !one_shot )
>>    {
>>        gdprintk(XENLOG_WARNING,
>>                 "HVM_PlatformTime: program too small period
%"PRIu64"\n",
>>                 period);
>>        period = 900000;
>>    }
>>
>>    pt->period = period;
>>    pt->vcpu = v;
>>    pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu);
>>    pt->irq = irq;
>>    pt->period_cycles = (u64)period * cpu_khz / 1000000L;
>>    pt->one_shot = one_shot;
>>    pt->scheduled = NOW() + period;
>>    /*
>>     * Offset LAPIC ticks from other timer ticks. Otherwise guests which
use
>>     * LAPIC ticks for process accounting can see long sequences of
process
>>     * ticks incorrectly accounted to interrupt processing.
>>     */
>>    if ( is_lvtt(v, irq) )
>>        pt->scheduled += period >> 1;
>>    pt->cb = cb;
>>    pt->priv = data;
>>
>>    list_add(&pt->list, &v->arch.hvm_vcpu.tm_list);
>>
>>    init_timer(&pt->timer, pt_timer_fn, pt, v->processor);
>>    set_timer(&pt->timer, pt->scheduled);
>>
>>    spin_unlock(&v->arch.hvm_vcpu.tm_lock);
>>}
>>
>>void destroy_periodic_time(struct periodic_time *pt)
>>{
>>    if ( !pt->enabled )
>>        return;
>>
>>    pt_lock(pt);
>>    pt->enabled = 0;
>>    list_del(&pt->list);
>>    pt_unlock(pt);
>>
>>    /*
>>     * pt_timer_fn() can run until this kill_timer() returns. We must do
this
>>     * outside pt_lock() otherwise we can deadlock with pt_timer_fn().
>>     */
>>    kill_timer(&pt->timer);
>>}
>>    
>>
>
>
>  
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

In the previous mail this

       (Before this patch clock error was 1.3%. The contributions of
   items ''2'' and ''3'' above to the
improvement in accuracy are .4%
   and .9% respectively.

should be

       (Before this patch clock error was 1.3%. The contributions of
   items ''2'' and ''3'' above to the
improvement in accuracy are .9%
   and .4% respectively.

thanks,
Dave



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 1/11/07 21:14, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> 1. Increase missed ticks threshold to 100 seconds. Per the comment in
> the patch,
I assume that''s the magic number 100000 in the patch? I think removing
the
check entirely would be better than an arbitrarily high threshold that
we''re
baiscally hoping will never trigger.
> 3.  Call pt_process_missed_ticks() unconditionally and place the
>     test for no_missed_tick_accounting inside pt_process_missed_ticks().
>     This returns the calculation for the next timer expiration
>     to the original method. The original method is important
>     as it sets up a theoretical time space at t=0 where expirations
>     occur only at n*period, where n is any integer. This, in turn, removes
>     rounding errors.
Why do ''rounding errors'' matter? I thought that
no_missed_tick_accounting
was for guests which sampled the TSC on tick interrupt and used that to
determine elapsed wall time, in which case why would it matter when exactly
the tick interrupt is delivered?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

I agree with removing the check.

Sorry, I was a bit vague with the rounding term.

My experiments show that delivering clock interrupts
in a constant N*period time space results in a .4% improvement
in accuracy over delivering interrupts at context switch (in) time
and resetting the N*period time space.
Lets call the former technique method S (synch) and the later AS
(async).

Now I will offer an explanation without proof.
If you would like proof, I can instrument the hypervisor to see
if this explanation is correct.

Let D be the time that the clock vcpu is descheduled and P be
the clock period.  When D < P, I think there is an issue.

The reason is that Linux''s offset calculation, which affects the
last clock interrupt tsc that is recorded, doesn''t kick in if the
time since the last interrupt is less than P. In this case it sets
offset to zero. Linux records the tsc of the current (last) clock interrupt
as (current tsc - offset).

Interrupt delivery method AS delivers a clock interrupt
at context switch (in) time. When D < P, Linux records the
interrupt delivery time as current tsc and bumps jiffies.
This results in a gain of time equal to P-D over wall time.

For method S this never happens because the interrupt isn''t
delivered until the next boundary.

regards,
Dave




Keir Fraser wrote:
>On 1/11/07 21:14, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>1. Increase missed ticks threshold to 100 seconds. Per the comment in
>>the patch,
>>    
>>
>
>I assume that''s the magic number 100000 in the patch? I think
removing the
>check entirely would be better than an arbitrarily high threshold that
we''re
>baiscally hoping will never trigger.
>  
>
>  
>
>>3.  Call pt_process_missed_ticks() unconditionally and place the
>>    test for no_missed_tick_accounting inside pt_process_missed_ticks().
>>    This returns the calculation for the next timer expiration
>>    to the original method. The original method is important
>>    as it sets up a theoretical time space at t=0 where expirations
>>    occur only at n*period, where n is any integer. This, in turn,
removes
>>    rounding errors.
>>    
>>
>
>Why do ''rounding errors'' matter? I thought that
no_missed_tick_accounting
>was for guests which sampled the TSC on tick interrupt and used that to
>determine elapsed wall time, in which case why would it matter when exactly
>the tick interrupt is delivered?
>  
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for the explanation. I think you are mis-describing the current
behaviour of vpt.c though (If I understand it correctly myself, which I may
not!). As I understand it, we do *not* unconditionally deliver a tick and
reset the time space when a vcpu is scheduled onto a cpu. We only do that if
(NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
passed. Otherwise we wait to deliver the next tick exactly one period after
the previous tick was delivered. The remainder of your explanation seems to
be predicated on the (impossible?) case that we deliver a tick less than one
period after the previous one.

Am I confused? :-) Also, what version of Linux are we talking about here,
and what periodic timer, and x86/64 or i386? There are lots of different
Linux timer-handling logics out there in the wild!

 -- Keir

On 2/11/07 15:51, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Let D be the time that the clock vcpu is descheduled and P be
> the clock period.  When D < P, I think there is an issue.
> 
> The reason is that Linux''s offset calculation, which affects the
> last clock interrupt tsc that is recorded, doesn''t kick in if the
> time since the last interrupt is less than P. In this case it sets
> offset to zero. Linux records the tsc of the current (last) clock interrupt
> as (current tsc - offset).
> 
> Interrupt delivery method AS delivers a clock interrupt
> at context switch (in) time. When D < P, Linux records the
> interrupt delivery time as current tsc and bumps jiffies.
> This results in a gain of time equal to P-D over wall time.
> 
> For method S this never happens because the interrupt isn''t
> delivered until the next boundary.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

By the way I checked in the other bits we agree on as changeset 16312. It
now looks quite sane to me.

 -- Keir

On 2/11/07 16:14, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> Thanks for the explanation. I think you are mis-describing the current
> behaviour of vpt.c though (If I understand it correctly myself, which I may
> not!). As I understand it, we do *not* unconditionally deliver a tick and
> reset the time space when a vcpu is scheduled onto a cpu. We only do that
if
> (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period
has
> passed. Otherwise we wait to deliver the next tick exactly one period after
> the previous tick was delivered. The remainder of your explanation seems to
> be predicated on the (impossible?) case that we deliver a tick less than
one
> period after the previous one.
> 
> Am I confused? :-) Also, what version of Linux are we talking about here,
> and what periodic timer, and x86/64 or i386? There are lots of different
> Linux timer-handling logics out there in the wild!
> 
>  -- Keir
> 
> On 2/11/07 15:51, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
> 
>> Let D be the time that the clock vcpu is descheduled and P be
>> the clock period.  When D < P, I think there is an issue.
>> 
>> The reason is that Linux''s offset calculation, which affects
the
>> last clock interrupt tsc that is recorded, doesn''t kick in if
the
>> time since the last interrupt is less than P. In this case it sets
>> offset to zero. Linux records the tsc of the current (last) clock
interrupt
>> as (current tsc - offset).
>> 
>> Interrupt delivery method AS delivers a clock interrupt
>> at context switch (in) time. When D < P, Linux records the
>> interrupt delivery time as current tsc and bumps jiffies.
>> This results in a gain of time equal to P-D over wall time.
>> 
>> For method S this never happens because the interrupt isn''t
>> delivered until the next boundary.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

oops, you''re right. I missed the  ( (NOW() - pt->scheduled) >= 0
).

This means I will have to come up with another explanation.

-Dave


Keir Fraser wrote:
>Thanks for the explanation. I think you are mis-describing the current
>behaviour of vpt.c though (If I understand it correctly myself, which I may
>not!). As I understand it, we do *not* unconditionally deliver a tick and
>reset the time space when a vcpu is scheduled onto a cpu. We only do that if
>(NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has
>passed. Otherwise we wait to deliver the next tick exactly one period after
>the previous tick was delivered. The remainder of your explanation seems to
>be predicated on the (impossible?) case that we deliver a tick less than one
>period after the previous one.
>
>Am I confused? :-) Also, what version of Linux are we talking about here,
>and what periodic timer, and x86/64 or i386? There are lots of different
>Linux timer-handling logics out there in the wild!
>
> -- Keir
>
>On 2/11/07 15:51, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Let D be the time that the clock vcpu is descheduled and P be
>>the clock period.  When D < P, I think there is an issue.
>>
>>The reason is that Linux''s offset calculation, which affects
the
>>last clock interrupt tsc that is recorded, doesn''t kick in if
the
>>time since the last interrupt is less than P. In this case it sets
>>offset to zero. Linux records the tsc of the current (last) clock
interrupt
>>as (current tsc - offset).
>>
>>Interrupt delivery method AS delivers a clock interrupt
>>at context switch (in) time. When D < P, Linux records the
>>interrupt delivery time as current tsc and bumps jiffies.
>>This results in a gain of time equal to P-D over wall time.
>>
>>For method S this never happens because the interrupt isn''t
>>delivered until the next boundary.
>>    
>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

Thanks for applying the fixes in the last submit.
In moving the test for no_missed_tick_accounting into 
pt_process_missed_ticks()
you forgot to add the scheduling part.

This patch adds the scheduling.
It also defines two options for scheduling, PT_SCHEDULE_SYNC and 
PT_SCHEDULE_ASYNC.
The default is PT_SCHEDULE_SYNC.

I am not providing any means for selecting between these two options.
My purpose is to have something to discuss and you can do as you like with
this patch or tell me what you want.

We have been discussing the difference between SYNC and ASYNC scheduling.
I found the reason the code checked in as of 10.31 had an error of .23%.
In the following fragment from pt_restore_timer()

        if ( !mode_is(v->domain, no_missed_tick_accounting) )
        {
            pt_process_missed_ticks(pt);
        }
        else if ( (NOW() - pt->scheduled) >= 0 )
        {
            pt->pending_intr_nr++;
            pt->scheduled = NOW() + pt->period;
        }
        set_timer(&pt->timer, pt->scheduled);

the pt->pending_intr_nr++; line is the problem because if the value of
pt->pending_intr_nr is non-zero before the increment, then you get extra
timer injections. We can either test for zero, set unconditionally to 1,
or leave the line out altogether and let the timeout take care of it.
I have chosen the later in the patch.

The result with this fixed, and with the patch submitted here
is as follows with the usual test I have been describing.

For both SYNC and ASYNC methods, with a sles9sp3-64 and
rh4u4-64 guest running at the same time, the errors were less than
.028%.  I have run some tests with single guests and big loads
and found sles to be in error by .1% with the ASYNC method.
All of the SYNC tests have been under .03%.

But my tests are short, usually 10 to 20 minutes. And there is quite a bit
of variability. Looking at the timer code for the two guests,
they look identical to me.

In my review of the Linux code, I see no reason why the interrupt
deliveries need to be "SYNC". This has been your intuition all along.

So, I leave the choice up to you.

thanks,
Dave


Dave Winchell wrote:
> oops, you''re right. I missed the  ( (NOW() - pt->scheduled)
>= 0 ).
>
> This means I will have to come up with another explanation.
>
> -Dave
>
>
> Keir Fraser wrote:
>
>> Thanks for the explanation. I think you are mis-describing the current
>> behaviour of vpt.c though (If I understand it correctly myself, which 
>> I may
>> not!). As I understand it, we do *not* unconditionally deliver a tick 
>> and
>> reset the time space when a vcpu is scheduled onto a cpu. We only do 
>> that if
>> (NOW() - pt->scheduled) >= 0 -- that is if more than a tick
period has
>> passed. Otherwise we wait to deliver the next tick exactly one period 
>> after
>> the previous tick was delivered. The remainder of your explanation 
>> seems to
>> be predicated on the (impossible?) case that we deliver a tick less 
>> than one
>> period after the previous one.
>>
>> Am I confused? :-) Also, what version of Linux are we talking about 
>> here,
>> and what periodic timer, and x86/64 or i386? There are lots of
different
>> Linux timer-handling logics out there in the wild!
>>
>> -- Keir
>>
>> On 2/11/07 15:51, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> Let D be the time that the clock vcpu is descheduled and P be
>>> the clock period.  When D < P, I think there is an issue.
>>>
>>> The reason is that Linux''s offset calculation, which
affects the
>>> last clock interrupt tsc that is recorded, doesn''t kick in
if the
>>> time since the last interrupt is less than P. In this case it sets
>>> offset to zero. Linux records the tsc of the current (last) clock 
>>> interrupt
>>> as (current tsc - offset).
>>>
>>> Interrupt delivery method AS delivers a clock interrupt
>>> at context switch (in) time. When D < P, Linux records the
>>> interrupt delivery time as current tsc and bumps jiffies.
>>> This results in a gain of time equal to P-D over wall time.
>>>
>>> For method S this never happens because the interrupt
isn''t
>>> delivered until the next boundary.
>>>   
>>
>>
>>
>>  
>>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 3/11/07 21:17, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Thanks for applying the fixes in the last submit.
> In moving the test for no_missed_tick_accounting into
> pt_process_missed_ticks()
> you forgot to add the scheduling part.
Actually it was deliberate, but clearly it was one code simplification too
far: thanks for spotting it! I''ll go the async route, but we do need to
set
pending_intr_nr to 1. We can''t leave that out -- the point of the async
route is to send a tick to the vcpu immediately, since it hasn''t had
one for
more than a tick period. If we wait for the timeout to do that then we have
to wait a whole extra period after the vcpu is re-scheduled.

Attached is my proposed patch. I think it''s quite neat. :-)

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
>On 3/11/07 21:17, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Thanks for applying the fixes in the last submit.
>>In moving the test for no_missed_tick_accounting into
>>pt_process_missed_ticks()
>>you forgot to add the scheduling part.
>>    
>>
>
>Actually it was deliberate, but clearly it was one code simplification too
>far: thanks for spotting it! I''ll go the async route, but we do
need to set
>pending_intr_nr to 1. We can''t leave that out -- the point of the
async
>route is to send a tick to the vcpu immediately, since it hasn''t
had one for
>more than a tick period. If we wait for the timeout to do that then we have
>to wait a whole extra period after the vcpu is re-scheduled.
>
>Attached is my proposed patch. I think it''s quite neat. :-)
>  
>
It looks good to me.

thanks,
Dave
> -- Keir
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

Attached is a fix for pt_process_missed_ticks().
Without this fix, systems run very erratically and some
guests panic on boot complaining that timer interrupts
are not working. As you can imagine.

Also, I have some longer term measurements of the
accuracy of the sync and async methods.
The hardware is an eight cpu AMD box. Two eight
vcpu guests, rh4u4-64 and sles9sp3-64.
All vcpus running loads.

Method   Test duration          Clock errors

SYNC       56000 sec         6.4, 6.7 sec   (.012%)
ASYNC      52000 sec         13, 19 sec     (.036%)

More testing should be done to validate the significance
of this difference.

regards,
Dave


Dave Winchell wrote:
>
> Keir Fraser wrote:
>
>> On 3/11/07 21:17, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> Thanks for applying the fixes in the last submit.
>>> In moving the test for no_missed_tick_accounting into
>>> pt_process_missed_ticks()
>>> you forgot to add the scheduling part.
>>>   
>>
>>
>> Actually it was deliberate, but clearly it was one code 
>> simplification too
>> far: thanks for spotting it! I''ll go the async route, but we
do need
>> to set
>> pending_intr_nr to 1. We can''t leave that out -- the point of
the async
>> route is to send a tick to the vcpu immediately, since it
hasn''t had
>> one for
>> more than a tick period. If we wait for the timeout to do that then 
>> we have
>> to wait a whole extra period after the vcpu is re-scheduled.
>>
>> Attached is my proposed patch. I think it''s quite neat. :-)
>>  
>>
> It looks good to me.
>
> thanks,
> Dave
>
>> -- Keir
>>
>>  
>>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Oh dear, that was a silly typo on my part. Thanks for tracking it down!

Well, I''m surprised that SYNC vs ASYNC differs, if the guest is really
tracking time via the TSC value. But it sounds like the actual time-tracking
algorithm in the guest must be a bit more complicated than that? I''d be
very
happy to help with any further analysis.

 -- Keir

On 7/11/07 14:39, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Keir,
> 
> Attached is a fix for pt_process_missed_ticks().
> Without this fix, systems run very erratically and some
> guests panic on boot complaining that timer interrupts
> are not working. As you can imagine.
> 
> Also, I have some longer term measurements of the
> accuracy of the sync and async methods.
> The hardware is an eight cpu AMD box. Two eight
> vcpu guests, rh4u4-64 and sles9sp3-64.
> All vcpus running loads.
> 
> Method   Test duration          Clock errors
> 
> SYNC       56000 sec         6.4, 6.7 sec   (.012%)
> ASYNC      52000 sec         13, 19 sec     (.036%)
> 
> More testing should be done to validate the significance
> of this difference.
> 
> regards,
> Dave
> 
> 
> Dave Winchell wrote:
> 
>> 
>> Keir Fraser wrote:
>> 
>>> On 3/11/07 21:17, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>> 
>>>  
>>> 
>>>> Thanks for applying the fixes in the last submit.
>>>> In moving the test for no_missed_tick_accounting into
>>>> pt_process_missed_ticks()
>>>> you forgot to add the scheduling part.
>>>>   
>>> 
>>> 
>>> Actually it was deliberate, but clearly it was one code
>>> simplification too
>>> far: thanks for spotting it! I''ll go the async route, but
we do need
>>> to set
>>> pending_intr_nr to 1. We can''t leave that out -- the point
of the async
>>> route is to send a tick to the vcpu immediately, since it
hasn''t had
>>> one for
>>> more than a tick period. If we wait for the timeout to do that then
>>> we have
>>> to wait a whole extra period after the vcpu is re-scheduled.
>>> 
>>> Attached is my proposed patch. I think it''s quite neat.
:-)
>>>  
>>> 
>> It looks good to me.
>> 
>> thanks,
>> Dave
>> 
>>> -- Keir
>>> 
>>>  
>>> 
>> 
> 
> diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c
> --- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000
> +++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500
> @@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru
>      if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
>      {
>          pt->pending_intr_nr = 1;
> -        pt->scheduled = now + pt->scheduled;
> +        pt->scheduled = now + pt->period;
>      }
>      else
>      {


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

You can help me analyze the Linux code.
Here is the code from sles (linux-2.6.5-7.244) x86_64.


static irqreturn_t timer_interrupt(int irq, void *dev_id, struct pt_regs 
*regs)

        spin_lock(&i8253_lock);
        outb_p(0x00, 0x43);
        delay = inb_p(0x40);
        delay |= inb(0x40) << 8;
        spin_unlock(&i8253_lock);
        delay = LATCH - 1 - delay;

     rdtscll_sync(&tsc);
       
        offset = (((tsc - vxtime.last_tsc) *
               vxtime.tsc_quot) >> 32) - (USEC_PER_SEC / HZ);

        if (offset < 0)
            offset = 0;

        if (offset > (USEC_PER_SEC / HZ)) {
            lost = offset / (USEC_PER_SEC / HZ);
            offset %= (USEC_PER_SEC / HZ);
        }

        monotonic_base += (tsc - vxtime.last_tsc)*1000000/cpu_khz ;

        vxtime.last_tsc = tsc - vxtime.quot * delay / vxtime.tsc_quot;

        if ((((tsc - vxtime.last_tsc) *
              vxtime.tsc_quot) >> 32) < offset)
            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;
   

    if (lost > 0) {
        if (report_lost_ticks) {
            printk(KERN_WARNING "time.c: Lost %ld timer "
                   "tick(s)! ", lost);
            print_symbol("rip %s)\n", regs->rip);
        }
        jiffies += lost;
    }
    do_timer(regs);


Now as ''offset'' gets modified it represents the remainder in
the
lost calculation. Our data indicates that when offset is close to zero,
timekeeping is more accurate. So this leads us to this line of code:

        if ((((tsc - vxtime.last_tsc) *
              vxtime.tsc_quot) >> 32) < offset)
            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;

Thus the problem seems to be the way the code switches from adjusting 
last_tsc
by the delay which is the normal mode when interrupts are timely or SYNCed
to handling offsets larger than the delay.

I''m concerned about the way delay is blown off in the final calculation

            vxtime.last_tsc = tsc -
                (((long) offset << 32) / vxtime.tsc_quot) - 1;


If an interrupt is right on time, and the delay is the same as last time,
then the offset should be zero in my mind. In the code above, offset will
equal delay for this example.

What do you think?

thanks,
Dave



Keir Fraser wrote:
>Oh dear, that was a silly typo on my part. Thanks for tracking it down!
>
>Well, I''m surprised that SYNC vs ASYNC differs, if the guest is
really
>tracking time via the TSC value. But it sounds like the actual time-tracking
>algorithm in the guest must be a bit more complicated than that?
I''d be very
>happy to help with any further analysis.
>
> -- Keir
>
>On 7/11/07 14:39, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Keir,
>>
>>Attached is a fix for pt_process_missed_ticks().
>>Without this fix, systems run very erratically and some
>>guests panic on boot complaining that timer interrupts
>>are not working. As you can imagine.
>>
>>Also, I have some longer term measurements of the
>>accuracy of the sync and async methods.
>>The hardware is an eight cpu AMD box. Two eight
>>vcpu guests, rh4u4-64 and sles9sp3-64.
>>All vcpus running loads.
>>
>>Method   Test duration          Clock errors
>>
>>SYNC       56000 sec         6.4, 6.7 sec   (.012%)
>>ASYNC      52000 sec         13, 19 sec     (.036%)
>>
>>More testing should be done to validate the significance
>>of this difference.
>>
>>regards,
>>Dave
>>
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 3/11/07 21:17, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>>>>Thanks for applying the fixes in the last submit.
>>>>>In moving the test for no_missed_tick_accounting into
>>>>>pt_process_missed_ticks()
>>>>>you forgot to add the scheduling part.
>>>>>  
>>>>>          
>>>>>
>>>>Actually it was deliberate, but clearly it was one code
>>>>simplification too
>>>>far: thanks for spotting it! I''ll go the async route,
but we do need
>>>>to set
>>>>pending_intr_nr to 1. We can''t leave that out -- the
point of the async
>>>>route is to send a tick to the vcpu immediately, since it
hasn''t had
>>>>one for
>>>>more than a tick period. If we wait for the timeout to do that
then
>>>>we have
>>>>to wait a whole extra period after the vcpu is re-scheduled.
>>>>
>>>>Attached is my proposed patch. I think it''s quite neat.
:-)
>>>> 
>>>>
>>>>        
>>>>
>>>It looks good to me.
>>>
>>>thanks,
>>>Dave
>>>
>>>      
>>>
>>>>-- Keir
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c
>>--- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000
>>+++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500
>>@@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru
>>     if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) )
>>     {
>>         pt->pending_intr_nr = 1;
>>-        pt->scheduled = now + pt->scheduled;
>>+        pt->scheduled = now + pt->period;
>>     }
>>     else
>>     {
>>    
>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/11/07 16:23, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> I''m concerned about the way delay is blown off in the final
calculation
> 
>             vxtime.last_tsc = tsc -
>                 (((long) offset << 32) / vxtime.tsc_quot) - 1;
> 
> If an interrupt is right on time, and the delay is the same as last time,
> then the offset should be zero in my mind. In the code above, offset will
> equal delay for this example.
> 
> What do you think?
I don''t think this code likes the ASYNC method *at all*. The reason is
that
it estimates how late the tick interrupt is by reading the PIT counter. It
knows the interrupt should have been triggered when the counter wrapped at
zero. But of course, if we are delivering ticks in ASYNC mode then quickly
the time we deliver an interrupt has *nothing* to do with the current PIT
counter value! Hence the lines that effectively fold max(offset, delay) into
last_tsc are just not going to work properly, because delay is a uniform
random variable, and hence we probably lose time.

Let me see if I can come up with a patch that gets the best of ASYNC and
SYNC in one handy mode...

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/11/07 17:10, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> I don''t think this code likes the ASYNC method *at all*. The
reason is that
> it estimates how late the tick interrupt is by reading the PIT counter. It
> knows the interrupt should have been triggered when the counter wrapped at
> zero. But of course, if we are delivering ticks in ASYNC mode then quickly
> the time we deliver an interrupt has *nothing* to do with the current PIT
> counter value! Hence the lines that effectively fold max(offset, delay)
into
> last_tsc are just not going to work properly, because delay is a uniform
> random variable, and hence we probably lose time.
Actually, no, I don''t understand why this code is less accurate with
ASYNC
mode. My comments on ''delay'' above are not true I think. Both
''delay'' and
''offset'' are accurate estimates of the time elapsed since the
last actual
tick threshold, when the PIT counter rolled over. And hence the logic that
is being applied *should* work...

I wonder if something silly like the following is happening: Because we send
unsynchronised tick interrupts, they can actually be delivered when the PIT
is just about to roll over (but hasn''t yet). If that happens we could
get a
worst-case disagreement between offset (derived from TSC) and delay (derived
from PIT). Let''s say that when we read the TSC it looks like we have
just
passed a tick rollover. In that case we will count some whole number of lost
ticks and offset will end up being a very small number (very little
''slop''
after a whole number of ticks is accounted). However, when we read the PIT
it looks like we are just about to roll over (but haven''t yet) and so
delay
will be a very large number: about as large as it can be. In this case we
will subtract a lot from last_tsc, because it looks like we''ve had
nearly a
full tick of delay. But actually the amount of delay that is getting
subtracted from last_tsc has already been accounted for elsewhere as a full
tick!

If this analysis is the truth of the matter then we would actually expect
the ASYNC mode to cause the guest timeofday to run *fast* rather than slow.

Anyhow, it''s bound to be something silly like this. :-)

The SYNC/ASYNC method I was going to propose by the way was going to be, in
pt_process_missed_ticks():
    pt->scheduled += missed_ticks * pt->period;
    if ( mode_is(no_missed_tick_accounting) )
        pt->pending_intr_nr = 1;
    else
        pt->pending_intr_nr += missed_ticks;

So, you can see we send an interrupt immediately (and ASYNC) if any ticks
have been missed, but then successive ticks are delivered ''on the
beat''. A
possible middleground? Or perhaps we should just go with SYNC after all...

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> So, you can see we send an interrupt immediately (and ASYNC) if any ticks
> have been missed, but then successive ticks are delivered ''on the
beat''. A
> possible middleground? Or perhaps we should just go with SYNC after all...
How do these Linux x64 guests fare with the original and default timer mode,
by the way? I would expect that time should be accounted pretty accurately
in that mode, albeit with more interrupts than you''d like. Or is the
lack of
synchronisation of TSCs across VCPUs causing issues that you''re trying
to
avoid?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

My feeling is that we should go full SYNC. Yes, in theory the
guests should be able to handle ASYNC, but in reality it appears that
some do not. Since it is easy for us to give them SYNC,
lets just do it and not stress them out.

In terms of sending the one ASYNC interrupt, I would say no.
Lets simply give it to him on the next boundary. If we give
it to him ASYNC he will have a big offset to deal with, exactly what
gives him trouble.

 > How do these Linux x64 guests fare with the original and default 
timer mode,
 > by the way? I would expect that time should be accounted pretty 
accurately
 > in that mode, albeit with more interrupts than you''d like.

For default mode as checked into unstable is now,
64 bit guests should run quite fast as missed is calculated and then a bunch
of additional interrupts are delivered. On the other hand
32bit guests very well in default mode.

For the original code, before we put in the constant tsc offset business,
64bit guests run poorly and 32bit quests very well time-wise.

 > Or is the lack of
 > synchronization of TSCs across VCPUs causing issues that you''re
trying to
 > avoid?

This does cause issues, but its not the only contributor to poor timing.
Having TSCs synchronized across vcpus will help some of the time going
backwards problems we have seen, I think.

Regards,
Dave

Keir Fraser wrote:
>On 7/11/07 17:29, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
>
>  
>
>>So, you can see we send an interrupt immediately (and ASYNC) if any
ticks
>>have been missed, but then successive ticks are delivered ''on
the beat''. A
>>possible middleground? Or perhaps we should just go with SYNC after
all...
>>    
>>
>
>How do these Linux x64 guests fare with the original and default timer mode,
>by the way? I would expect that time should be accounted pretty accurately
>in that mode, albeit with more interrupts than you''d like. Or is
the lack of
>synchronisation of TSCs across VCPUs causing issues that you''re
trying to
>avoid?
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/11/07 19:38, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> My feeling is that we should go full SYNC. Yes, in theory the
> guests should be able to handle ASYNC, but in reality it appears that
> some do not. Since it is easy for us to give them SYNC,
> lets just do it and not stress them out.
One problem with pure SYNC is there''s a fair chance you won''t
deliver any
ticks at all for a long time, if the guest only runs in short bursts (e.g.,
I/O bound) and happens not to be running on any tick boundary. I''m not
sure
how much that matters. It could cause time goes backwards if the time
extrapolation via the TSC is not perfectly accurate, or cause problems if
there are any assumptions that TSC delta since last tick fits in 32 bits
(less likely in x64 code I suppose). Anyway, my point is that only testing
VCPUs under full load may cause us to optimise in ways that have nasty
unexpected effects for other workloads.
> For default mode as checked into unstable is now,
> 64 bit guests should run quite fast as missed is calculated and then a
bunch
> of additional interrupts are delivered. On the other hand
> 32bit guests very well in default mode.
> 
> For the original code, before we put in the constant tsc offset business,
> 64bit guests run poorly and 32bit quests very well time-wise.
The default mode hasn''t changed. Are you under the impression that
missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests
run badly with that because they treat every one of the missed ticks they
receive as a full tick.

 -- Keir
>> Or is the lack of
>> synchronization of TSCs across VCPUs causing issues that
you''re trying to
>> avoid?
> 
> This does cause issues, but its not the only contributor to poor timing.
> Having TSCs synchronized across vcpus will help some of the time going
> backwards problems we have seen, I think.
> 
> Regards,
> Dave
> 
> Keir Fraser wrote:
> 
>> On 7/11/07 17:29, "Keir Fraser"
<Keir.Fraser@cl.cam.ac.uk> wrote:
>> 
>>  
>> 
>>> So, you can see we send an interrupt immediately (and ASYNC) if any
ticks
>>> have been missed, but then successive ticks are delivered
''on the beat''. A
>>> possible middleground? Or perhaps we should just go with SYNC after
all...
>>>    
>>> 
>> 
>> How do these Linux x64 guests fare with the original and default timer
mode,
>> by the way? I would expect that time should be accounted pretty
accurately
>> in that mode, albeit with more interrupts than you''d like. Or
is the lack of
>> synchronisation of TSCs across VCPUs causing issues that
you''re trying to
>> avoid?
>> 
>> -- Keir
>> 
>> 
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

I''ve added comments below.
See my next mail on some interesting performance numbers.

thanks,
Dave

Keir Fraser wrote:
>On 7/11/07 19:38, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>My feeling is that we should go full SYNC. Yes, in theory the
>>guests should be able to handle ASYNC, but in reality it appears that
>>some do not. Since it is easy for us to give them SYNC,
>>lets just do it and not stress them out.
>>    
>>
>
>One problem with pure SYNC is there''s a fair chance you
won''t deliver any
>ticks at all for a long time, if the guest only runs in short bursts (e.g.,
>I/O bound) and happens not to be running on any tick boundary. I''m
not sure
>how much that matters. It could cause time goes backwards if the time
>extrapolation via the TSC is not perfectly accurate, or cause problems if
>there are any assumptions that TSC delta since last tick fits in 32 bits
>(less likely in x64 code I suppose). Anyway, my point is that only testing
>VCPUs under full load may cause us to optimise in ways that have nasty
>unexpected effects for other workloads.
>  
>
I agree that this could be a problem. I have an idea that could give us full
SYNC and eliminate the long periods without clock interrupts.
In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1.
In pt_save_timer():

    list_for_each_entry ( pt, head, list )
        if(!pt->run_timer)
             stop_timer(&pt->timer);

And in pt_timer_fn():

    pt->run_timer = 0;

So, for a guest that misses a tick, we will interrupt him once from the
descheduled state and then leave him alone in the descheduled state.
>  
>
>>For default mode as checked into unstable is now,
>>64 bit guests should run quite fast as missed is calculated and then a
bunch
>>of additional interrupts are delivered. On the other hand
>>32bit guests very well in default mode.
>>
>>For the original code, before we put in the constant tsc offset
business,
>>64bit guests run poorly and 32bit quests very well time-wise.
>>    
>>
>
>The default mode hasn''t changed. Are you under the impression that
>missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests
>run badly with that because they treat every one of the missed ticks they
>receive as a full tick.
>  
>
Sorry, I was confused.
However, the default mode will still run poorly for 64 bit guests because
of the pending_nr''s accumulated while the guest has interrupts
disabled.
As I recall, the effect is quite large, on the order of 10% error.
I''ll get you a number later today.
> -- Keir
>
>  
>
>>>Or is the lack of
>>>synchronization of TSCs across VCPUs causing issues that
you''re trying to
>>>avoid?
>>>      
>>>
>>This does cause issues, but its not the only contributor to poor timing.
>>Having TSCs synchronized across vcpus will help some of the time going
>>backwards problems we have seen, I think.
>>
>>Regards,
>>Dave
>>
>>Keir Fraser wrote:
>>
>>    
>>
>>>On 7/11/07 17:29, "Keir Fraser"
<Keir.Fraser@cl.cam.ac.uk> wrote:
>>>
>>> 
>>>
>>>      
>>>
>>>>So, you can see we send an interrupt immediately (and ASYNC) if
any ticks
>>>>have been missed, but then successive ticks are delivered
''on the beat''. A
>>>>possible middleground? Or perhaps we should just go with SYNC
after all...
>>>>   
>>>>
>>>>        
>>>>
>>>How do these Linux x64 guests fare with the original and default
timer mode,
>>>by the way? I would expect that time should be accounted pretty
accurately
>>>in that mode, albeit with more interrupts than you''d like.
Or is the lack of
>>>synchronisation of TSCs across VCPUs causing issues that
you''re trying to
>>>avoid?
>>>
>>>-- Keir
>>>
>>>
>>> 
>>>
>>>      
>>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8/11/07 14:43, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> I agree that this could be a problem. I have an idea that could give us
full
> SYNC and eliminate the long periods without clock interrupts.
> In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer
= 1.
> In pt_save_timer():
> 
>     list_for_each_entry ( pt, head, list )
>         if(!pt->run_timer)
>              stop_timer(&pt->timer);
> 
> And in pt_timer_fn():
> 
>     pt->run_timer = 0;
> 
> So, for a guest that misses a tick, we will interrupt him once from the
> descheduled state and then leave him alone in the descheduled state.
Well, I''d rather not complicate the code if it''s avoidable. I
checked in a
SYNC/ASYNC combo and code simplification as changeset 16341, and it''d
be
interesting to know how that fares against your suggested scheme. I suppose
as long as we''re better than ntpd''s tolerance it
doesn''t actually matter all
that much.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

I ran a 24 hour (23hr:40min) test. The usual setup. Protocol was ASYNC.

Errors:
  sles9sp3-64:  -4.96 sec  -.0058%
  rh4u4-64:     +4.42 sec  +.0052%

So, lets leave it ASYNC unless someone produces some test cases
where the error gets up to close to .05%.
I''ll do some testing here with overnight runs or, perhaps,
different loads.

thanks,
Dave

Dave Winchell wrote:
> Hi Keir,
>
> I''ve added comments below.
> See my next mail on some interesting performance numbers.
>
> thanks,
> Dave
>
> Keir Fraser wrote:
>
>> On 7/11/07 19:38, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>
>>  
>>
>>> My feeling is that we should go full SYNC. Yes, in theory the
>>> guests should be able to handle ASYNC, but in reality it appears
that
>>> some do not. Since it is easy for us to give them SYNC,
>>> lets just do it and not stress them out.
>>>   
>>
>>
>> One problem with pure SYNC is there''s a fair chance you
won''t deliver
>> any
>> ticks at all for a long time, if the guest only runs in short bursts 
>> (e.g.,
>> I/O bound) and happens not to be running on any tick boundary.
I''m
>> not sure
>> how much that matters. It could cause time goes backwards if the time
>> extrapolation via the TSC is not perfectly accurate, or cause 
>> problems if
>> there are any assumptions that TSC delta since last tick fits in 32
bits
>> (less likely in x64 code I suppose). Anyway, my point is that only 
>> testing
>> VCPUs under full load may cause us to optimise in ways that have nasty
>> unexpected effects for other workloads.
>>  
>>
> I agree that this could be a problem. I have an idea that could give 
> us full
> SYNC and eliminate the long periods without clock interrupts.
> In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer
= 1.
> In pt_save_timer():
>
>    list_for_each_entry ( pt, head, list )
>        if(!pt->run_timer)
>             stop_timer(&pt->timer);
>
> And in pt_timer_fn():
>
>    pt->run_timer = 0;
>
> So, for a guest that misses a tick, we will interrupt him once from the
> descheduled state and then leave him alone in the descheduled state.
>
>>  
>>
>>> For default mode as checked into unstable is now,
>>> 64 bit guests should run quite fast as missed is calculated and
then
>>> a bunch
>>> of additional interrupts are delivered. On the other hand
>>> 32bit guests very well in default mode.
>>>
>>> For the original code, before we put in the constant tsc offset 
>>> business,
>>> 64bit guests run poorly and 32bit quests very well time-wise.
>>>   
>>
>>
>> The default mode hasn''t changed. Are you under the impression
that
>> missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 
>> guests
>> run badly with that because they treat every one of the missed ticks 
>> they
>> receive as a full tick.
>>  
>>
> Sorry, I was confused.
> However, the default mode will still run poorly for 64 bit guests because
> of the pending_nr''s accumulated while the guest has interrupts
disabled.
> As I recall, the effect is quite large, on the order of 10% error.
> I''ll get you a number later today.
>
>> -- Keir
>>
>>  
>>
>>>> Or is the lack of
>>>> synchronization of TSCs across VCPUs causing issues that
you''re
>>>> trying to
>>>> avoid?
>>>>     
>>>
>>> This does cause issues, but its not the only contributor to poor 
>>> timing.
>>> Having TSCs synchronized across vcpus will help some of the time
going
>>> backwards problems we have seen, I think.
>>>
>>> Regards,
>>> Dave
>>>
>>> Keir Fraser wrote:
>>>
>>>   
>>>
>>>> On 7/11/07 17:29, "Keir Fraser"
<Keir.Fraser@cl.cam.ac.uk> wrote:
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>> So, you can see we send an interrupt immediately (and
ASYNC) if
>>>>> any ticks
>>>>> have been missed, but then successive ticks are delivered
''on the
>>>>> beat''. A
>>>>> possible middleground? Or perhaps we should just go with
SYNC
>>>>> after all...
>>>>>  
>>>>>       
>>>>
>>>> How do these Linux x64 guests fare with the original and
default
>>>> timer mode,
>>>> by the way? I would expect that time should be accounted pretty
>>>> accurately
>>>> in that mode, albeit with more interrupts than you''d
like. Or is
>>>> the lack of
>>>> synchronisation of TSCs across VCPUs causing issues that
you''re
>>>> trying to
>>>> avoid?
>>>>
>>>> -- Keir
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>
>>
>>
>>  
>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

ok, I''ll try your new code and let you know what
I find.

-Dave

Keir Fraser wrote:
>On 8/11/07 14:43, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>I agree that this could be a problem. I have an idea that could give us
full
>>SYNC and eliminate the long periods without clock interrupts.
>>In pt_process_missed_ticks() when missed_ticks > 0 set
pt->run_timer = 1.
>>In pt_save_timer():
>>
>>    list_for_each_entry ( pt, head, list )
>>        if(!pt->run_timer)
>>             stop_timer(&pt->timer);
>>
>>And in pt_timer_fn():
>>
>>    pt->run_timer = 0;
>>
>>So, for a guest that misses a tick, we will interrupt him once from the
>>descheduled state and then leave him alone in the descheduled state.
>>    
>>
>
>Well, I''d rather not complicate the code if it''s
avoidable. I checked in a
>SYNC/ASYNC combo and code simplification as changeset 16341, and
it''d be
>interesting to know how that fares against your suggested scheme. I suppose
>as long as we''re better than ntpd''s tolerance it
doesn''t actually matter all
>that much.
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

Here are the results of the testing I''ve done recently.
There are three protocols tested. I''ll call them ASYNC for the code
as it stood just before your last check-in, MIXED (for ASYNC+SYNC) for
the code you recently checked in, and SYNC for the method I proposed
in the last mail.



Date     Duration       Protocol                 sles, rhat error

11/07  23 hrs 40 min     ASYNC         -4.96 sec, +4.42 sec    -.006%,  
+.005%
11/08         28 min     MIXED          -.75 sec,  -.67 sec    -.045%,  
-.040%
11/08   2 hrs 21 min      SYNC          -.80 sec,  -.34 sec,   -.009%,  
-.004%
11/08   1 hr  25 min      SYNC          -.24 sec,  -.26 sec,   -.005%,  
-.005%
11/08  15 hrs 39 min     MIXED        -19.   sec,-17.4  sec,   -.034%,  
-.031%
11/09   3 hrs 19 min     ASYNC          -.13 sec, +1.44 sec,   -.001%,  
+.012%


Each protocol was run a couple of times to gage the repeatability.

Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
I ran another ASYNC test. I think there may have been something
wrong with the code I used a couple of days ago for ASYNC. It may have been
missing the immediate delivery of interrupt after context switch in.

My results indicate that either SYNC or ASYNC give acceptable accuracy,
each running consistently around or under .01%. MIXED has a fairly high 
error of
greater than .03%. Probably too close to .05% ntp threshold for comfort.
I don''t have an overnight run with SYNC. I plan to leave SYNC running
over the weekend. If you''d rather I can leave MIXED running instead.

It may be too early to pick the protocol and I can run more overnight tests
next week.

I have attached a patch based on what''s checked-in now that shows what
I
tested for SYNC and ASYNC. For MIXED I used the code that is checked-in now.
The patch looks more complicated than the change really is as
it backs out the MIXED code you put in.

Regards,
Dave


Keir Fraser wrote:
>On 8/11/07 14:43, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>I agree that this could be a problem. I have an idea that could give us
full
>>SYNC and eliminate the long periods without clock interrupts.
>>In pt_process_missed_ticks() when missed_ticks > 0 set
pt->run_timer = 1.
>>In pt_save_timer():
>>
>>    list_for_each_entry ( pt, head, list )
>>        if(!pt->run_timer)
>>             stop_timer(&pt->timer);
>>
>>And in pt_timer_fn():
>>
>>    pt->run_timer = 0;
>>
>>So, for a guest that misses a tick, we will interrupt him once from the
>>descheduled state and then leave him alone in the descheduled state.
>>    
>>
>
>Well, I''d rather not complicate the code if it''s
avoidable. I checked in a
>SYNC/ASYNC combo and code simplification as changeset 16341, and
it''d be
>interesting to know how that fares against your suggested scheme. I suppose
>as long as we''re better than ntpd''s tolerance it
doesn''t actually matter all
>that much.
>
> -- Keir
>
>
>  
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 9/11/07 19:22, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Since I had a high error (~.03%) for the ASYNC method a couple of days ago,
> I ran another ASYNC test. I think there may have been something
> wrong with the code I used a couple of days ago for ASYNC. It may have been
> missing the immediate delivery of interrupt after context switch in.
> 
> My results indicate that either SYNC or ASYNC give acceptable accuracy,
> each running consistently around or under .01%. MIXED has a fairly high
> error of
> greater than .03%. Probably too close to .05% ntp threshold for comfort.
> I don''t have an overnight run with SYNC. I plan to leave SYNC
running
> over the weekend. If you''d rather I can leave MIXED running
instead.
> 
> It may be too early to pick the protocol and I can run more overnight tests
> next week.
I''m a bit worried about any unwanted side effects of the SYNC+run_timer
approach -- e.g., whether timer wakeups will cause higher system-wide CPU
contention. I find it easier to think through the implications of ASYNC.
I''m
surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
delivers more timer interrupts than the other approaches, and each interrupt
event causes a small accumulated error?

Overall I would consider MIXED and ASYNC as favourites and if the latter is
actually more accurate then I can simply revert the changeset that
implemented MIXED.

Perhaps rather than running more of the same workloads you could try idle
VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
don''t have any data on workloads that aren''t CPU bound, so
that''s really an
obvious place to put any further effort imo.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

I''ll run the I/O and idle tests you suggested.
Since the tests will be run in the evenings, they won''t be
completed until the end of the week.

Below is the table of tests updated with the weekend SYNC with cpu load 
test,
where the error was less than .01% for both linux guests.

 > I''m a bit worried about any unwanted side effects of the
SYNC+run_timer
 > approach -- e.g., whether timer wakeups will cause higher system-wide CPU
 > contention.


I agree. If the SYNC model turns out to be desirable from an accuracy
standpoint, then the significance of the additional wakeups should
characterized.

Regards,
Dave


Date     Duration       Protocol                 sles, rhat 
error                      load

11/07  23 hrs 40 min     ASYNC         -4.96 sec, +4.42 sec    -.006%,  
+.005%         cpu
11/09   3 hrs 19 min     ASYNC          -.13 sec, +1.44 sec,   -.001%,  
+.012%         cpu

11/08   2 hrs 21 min      SYNC          -.80 sec,  -.34 sec,   -.009%,  
-.004%         cpu
11/08   1 hr  25 min      SYNC          -.24 sec,  -.26 sec,   -.005%,  
-.005%         cpu
11/12  65 hrs 40 min      SYNC        -18    sec, -8    sec,   -.008%,  
-.003%         cpu

11/08         28 min     MIXED          -.75 sec,  -.67 sec    -.045%,  
-.040%         cpu
11/08  15 hrs 39 min     MIXED        -19.   sec,-17.4  sec,   -.034%,  
-.031%         cpu


Keir Fraser wrote:
>On 9/11/07 19:22, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Since I had a high error (~.03%) for the ASYNC method a couple of days
ago,
>>I ran another ASYNC test. I think there may have been something
>>wrong with the code I used a couple of days ago for ASYNC. It may have
been
>>missing the immediate delivery of interrupt after context switch in.
>>
>>My results indicate that either SYNC or ASYNC give acceptable accuracy,
>>each running consistently around or under .01%. MIXED has a fairly high
>>error of
>>greater than .03%. Probably too close to .05% ntp threshold for comfort.
>>I don''t have an overnight run with SYNC. I plan to leave SYNC
running
>>over the weekend. If you''d rather I can leave MIXED running
instead.
>>
>>It may be too early to pick the protocol and I can run more overnight
tests
>>next week.
>>    
>>
>
>I''m a bit worried about any unwanted side effects of the
SYNC+run_timer
>approach -- e.g., whether timer wakeups will cause higher system-wide CPU
>contention. I find it easier to think through the implications of ASYNC.
I''m
>surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
>delivers more timer interrupts than the other approaches, and each interrupt
>event causes a small accumulated error?
>
>Overall I would consider MIXED and ASYNC as favourites and if the latter is
>actually more accurate then I can simply revert the changeset that
>implemented MIXED.
>
>Perhaps rather than running more of the same workloads you could try idle
>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
>don''t have any data on workloads that aren''t CPU bound, so
that''s really an
>obvious place to put any further effort imo.
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

The accuracy data I''ve collected for i/o loads for the
various time protocols follows. In addition, the data
for cpu loads is shown.

The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
Two guests, red hat and sles 64 bit, 8 vcpu each.
The cpu load is usex -e36 on each guest.
(usex is available at http://people.redhat.com/anderson/usex.)
i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.

The loads labeled i/o-32 are 32 instances of dd.
Also, these are run on 4 cpu AMD box.
In addition, there is an idle rh-32bit guest.
All three guests are 8vcpu.

The loads labeled i/o-4/32 are the same as i/o-32
except that the redhat-64 guest has 4 instances of dd.

Date Duration Protocol sles, rhat error load

11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu

11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu

11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu


11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8

11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8

11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8



11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32

11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32


Overhead measurements:

Progress in terms of number of passes through a fixed system workload
on an 8 vcpu red hat with an 8 vcpu sles idle.
The workload was usex -b48.


ASYNC 167 min 145 passes .868 passes/min
SYNC 167 min 144 passes .862 passes/min
SYNC 1065 min 919 passes .863 passes/min
MIXED 221 min 196 passes .887 passes/min


Conclusions:

The only protocol which meets the .05% accuracy requirement for ntp 
tracking under the loads
above is the SYNC protocol. The worst case accuracies for SYNC, MIXED, 
and ASYNC
are .022%, .12%, and .14%, respectively.

We could reduce the cost of the SYNC method by only scheduling the extra 
wakeups if a certain number
of ticks are missed.

Regards,
Dave

Keir Fraser wrote:
>On 9/11/07 19:22, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Since I had a high error (~.03%) for the ASYNC method a couple of days
ago,
>>I ran another ASYNC test. I think there may have been something
>>wrong with the code I used a couple of days ago for ASYNC. It may have
been
>>missing the immediate delivery of interrupt after context switch in.
>>
>>My results indicate that either SYNC or ASYNC give acceptable accuracy,
>>each running consistently around or under .01%. MIXED has a fairly high
>>error of
>>greater than .03%. Probably too close to .05% ntp threshold for comfort.
>>I don''t have an overnight run with SYNC. I plan to leave SYNC
running
>>over the weekend. If you''d rather I can leave MIXED running
instead.
>>
>>It may be too early to pick the protocol and I can run more overnight
tests
>>next week.
>>    
>>
>
>I''m a bit worried about any unwanted side effects of the
SYNC+run_timer
>approach -- e.g., whether timer wakeups will cause higher system-wide CPU
>contention. I find it easier to think through the implications of ASYNC.
I''m
>surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it
>delivers more timer interrupts than the other approaches, and each interrupt
>event causes a small accumulated error?
>
>Overall I would consider MIXED and ASYNC as favourites and if the latter is
>actually more accurate then I can simply revert the changeset that
>implemented MIXED.
>
>Perhaps rather than running more of the same workloads you could try idle
>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We
>don''t have any data on workloads that aren''t CPU bound, so
that''s really an
>obvious place to put any further effort imo.
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Please take a look at xen-unstable changeset 16545.

 -- Keir

On 26/11/07 20:57, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Keir,
> 
> The accuracy data I''ve collected for i/o loads for the
> various time protocols follows. In addition, the data
> for cpu loads is shown.
> 
> The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> Two guests, red hat and sles 64 bit, 8 vcpu each.
> The cpu load is usex -e36 on each guest.
> (usex is available at http://people.redhat.com/anderson/usex.)
> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> 
> The loads labeled i/o-32 are 32 instances of dd.
> Also, these are run on 4 cpu AMD box.
> In addition, there is an idle rh-32bit guest.
> All three guests are 8vcpu.
> 
> The loads labeled i/o-4/32 are the same as i/o-32
> except that the redhat-64 guest has 4 instances of dd.
> 
> Date Duration Protocol sles, rhat error load
> 
> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
> 
> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> 
> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
> 
> 
> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
> 
> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> 
> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
> 
> 
> 
> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> 
> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
> 
> 
> Overhead measurements:
> 
> Progress in terms of number of passes through a fixed system workload
> on an 8 vcpu red hat with an 8 vcpu sles idle.
> The workload was usex -b48.
> 
> 
> ASYNC 167 min 145 passes .868 passes/min
> SYNC 167 min 144 passes .862 passes/min
> SYNC 1065 min 919 passes .863 passes/min
> MIXED 221 min 196 passes .887 passes/min
> 
> 
> Conclusions:
> 
> The only protocol which meets the .05% accuracy requirement for ntp
> tracking under the loads
> above is the SYNC protocol. The worst case accuracies for SYNC, MIXED,
> and ASYNC
> are .022%, .12%, and .14%, respectively.
> 
> We could reduce the cost of the SYNC method by only scheduling the extra
> wakeups if a certain number
> of ticks are missed.
> 
> Regards,
> Dave
> 
> Keir Fraser wrote:
> 
>> On 9/11/07 19:22, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>> 
>>  
>> 
>>> Since I had a high error (~.03%) for the ASYNC method a couple of
days ago,
>>> I ran another ASYNC test. I think there may have been something
>>> wrong with the code I used a couple of days ago for ASYNC. It may
have been
>>> missing the immediate delivery of interrupt after context switch
in.
>>> 
>>> My results indicate that either SYNC or ASYNC give acceptable
accuracy,
>>> each running consistently around or under .01%. MIXED has a fairly
high
>>> error of
>>> greater than .03%. Probably too close to .05% ntp threshold for
comfort.
>>> I don''t have an overnight run with SYNC. I plan to leave
SYNC running
>>> over the weekend. If you''d rather I can leave MIXED
running instead.
>>> 
>>> It may be too early to pick the protocol and I can run more
overnight tests
>>> next week.
>>>    
>>> 
>> 
>> I''m a bit worried about any unwanted side effects of the
SYNC+run_timer
>> approach -- e.g., whether timer wakeups will cause higher system-wide
CPU
>> contention. I find it easier to think through the implications of
ASYNC. I''m
>> surprised that MIXED loses time, and is less accurate than ASYNC.
Perhaps it
>> delivers more timer interrupts than the other approaches, and each
interrupt
>> event causes a small accumulated error?
>> 
>> Overall I would consider MIXED and ASYNC as favourites and if the
latter is
>> actually more accurate then I can simply revert the changeset that
>> implemented MIXED.
>> 
>> Perhaps rather than running more of the same workloads you could try
idle
>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to
/dev/null)? We
>> don''t have any data on workloads that aren''t CPU
bound, so that''s really an
>> obvious place to put any further effort imo.
>> 
>> -- Keir
>> 
>> 
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Anyone make measurements on the final patch?

I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no
load.  This was xen-unstable tip today with no options specified.  32-bit was
about 0.01%.

I think I missed something... how do I run the various accounting choices and
which ones are known to be appropriate for which kernels?

Thanks,
Dan
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
> Sent: Thursday, December 06, 2007 4:57 AM
> To: Dave Winchell
> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
> Yunhong
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Please take a look at xen-unstable changeset 16545.
> 
>  -- Keir
> 
> On 26/11/07 20:57, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
> 
> > Keir,
> >
> > The accuracy data I''ve collected for i/o loads for the
> > various time protocols follows. In addition, the data
> > for cpu loads is shown.
> >
> > The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> > Two guests, red hat and sles 64 bit, 8 vcpu each.
> > The cpu load is usex -e36 on each guest.
> > (usex is available at http://people.redhat.com/anderson/usex.)
> > i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >
> > The loads labeled i/o-32 are 32 instances of dd.
> > Also, these are run on 4 cpu AMD box.
> > In addition, there is an idle rh-32bit guest.
> > All three guests are 8vcpu.
> >
> > The loads labeled i/o-4/32 are the same as i/o-32
> > except that the redhat-64 guest has 4 instances of dd.
> >
> > Date Duration Protocol sles, rhat error load
> >
> > 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
> > 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
> >
> > 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> > 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> > 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >
> > 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> > 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
> >
> >
> > 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
> > 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
> >
> > 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
> > 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >
> > 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
> > 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
> >
> >
> >
> > 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> > 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
> > 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >
> > 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
> > 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
> >
> >
> > Overhead measurements:
> >
> > Progress in terms of number of passes through a fixed 
> system workload
> > on an 8 vcpu red hat with an 8 vcpu sles idle.
> > The workload was usex -b48.
> >
> >
> > ASYNC 167 min 145 passes .868 passes/min
> > SYNC 167 min 144 passes .862 passes/min
> > SYNC 1065 min 919 passes .863 passes/min
> > MIXED 221 min 196 passes .887 passes/min
> >
> >
> > Conclusions:
> >
> > The only protocol which meets the .05% accuracy requirement for ntp
> > tracking under the loads
> > above is the SYNC protocol. The worst case accuracies for 
> SYNC, MIXED,
> > and ASYNC
> > are .022%, .12%, and .14%, respectively.
> >
> > We could reduce the cost of the SYNC method by only 
> scheduling the extra
> > wakeups if a certain number
> > of ticks are missed.
> >
> > Regards,
> > Dave
> >
> > Keir Fraser wrote:
> >
> >> On 9/11/07 19:22, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>
> >>> Since I had a high error (~.03%) for the ASYNC method a 
> couple of days ago,
> >>> I ran another ASYNC test. I think there may have been
something
> >>> wrong with the code I used a couple of days ago for 
> ASYNC. It may have been
> >>> missing the immediate delivery of interrupt after context 
> switch in.
> >>>
> >>> My results indicate that either SYNC or ASYNC give 
> acceptable accuracy,
> >>> each running consistently around or under .01%. MIXED has 
> a fairly high
> >>> error of
> >>> greater than .03%. Probably too close to .05% ntp 
> threshold for comfort.
> >>> I don''t have an overnight run with SYNC. I plan to
leave
> SYNC running
> >>> over the weekend. If you''d rather I can leave MIXED 
> running instead.
> >>>
> >>> It may be too early to pick the protocol and I can run 
> more overnight tests
> >>> next week.
> >>>
> >>>
> >>
> >> I''m a bit worried about any unwanted side effects of the 
> SYNC+run_timer
> >> approach -- e.g., whether timer wakeups will cause higher 
> system-wide CPU
> >> contention. I find it easier to think through the 
> implications of ASYNC. I''m
> >> surprised that MIXED loses time, and is less accurate than 
> ASYNC. Perhaps it
> >> delivers more timer interrupts than the other approaches, 
> and each interrupt
> >> event causes a small accumulated error?
> >>
> >> Overall I would consider MIXED and ASYNC as favourites and 
> if the latter is
> >> actually more accurate then I can simply revert the changeset that
> >> implemented MIXED.
> >>
> >> Perhaps rather than running more of the same workloads you 
> could try idle
> >> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
> to /dev/null)? We
> >> don''t have any data on workloads that aren''t CPU
bound, so
> that''s really an
> >> obvious place to put any further effort imo.
> >>
> >> -- Keir
> >>
> >>
> >>
> >>
> >
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan,

I did some testing with the constant tsc offset SYNC method (now called 
no_missed_ticks_pending)
and found the error to be very high, much larger than 1 %, as I recall.
I have not had a chance to submit a correction. I will try to do it later
this week or the first week in January. My version of constant tsc 
offset SYNC method
produces .02 % error, so I just need to port that into the current code.

The error you got for both of those kernels is what I would expect
for the default mode, delay_for_missed_ticks.

I''ll let Keir answer on how to set the time mode.

Regards,
Dave

Dan Magenheimer wrote:
>Anyone make measurements on the final patch?
>
>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no
load.  This was xen-unstable tip today with no options specified.  32-bit was
about 0.01%.
>
>I think I missed something... how do I run the various accounting choices
and which ones are known to be appropriate for which kernels?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
>>Sent: Thursday, December 06, 2007 4:57 AM
>>To: Dave Winchell
>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>Yunhong
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Please take a look at xen-unstable changeset 16545.
>>
>> -- Keir
>>
>>On 26/11/07 20:57, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>
>>    
>>
>>>Keir,
>>>
>>>The accuracy data I''ve collected for i/o loads for the
>>>various time protocols follows. In addition, the data
>>>for cpu loads is shown.
>>>
>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>The cpu load is usex -e36 on each guest.
>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>
>>>The loads labeled i/o-32 are 32 instances of dd.
>>>Also, these are run on 4 cpu AMD box.
>>>In addition, there is an idle rh-32bit guest.
>>>All three guests are 8vcpu.
>>>
>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>except that the redhat-64 guest has 4 instances of dd.
>>>
>>>Date Duration Protocol sles, rhat error load
>>>
>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>
>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>
>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>
>>>
>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>
>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>
>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>
>>>
>>>
>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>
>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>
>>>
>>>Overhead measurements:
>>>
>>>Progress in terms of number of passes through a fixed 
>>>      
>>>
>>system workload
>>    
>>
>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>The workload was usex -b48.
>>>
>>>
>>>ASYNC 167 min 145 passes .868 passes/min
>>>SYNC 167 min 144 passes .862 passes/min
>>>SYNC 1065 min 919 passes .863 passes/min
>>>MIXED 221 min 196 passes .887 passes/min
>>>
>>>
>>>Conclusions:
>>>
>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>tracking under the loads
>>>above is the SYNC protocol. The worst case accuracies for 
>>>      
>>>
>>SYNC, MIXED,
>>    
>>
>>>and ASYNC
>>>are .022%, .12%, and .14%, respectively.
>>>
>>>We could reduce the cost of the SYNC method by only 
>>>      
>>>
>>scheduling the extra
>>    
>>
>>>wakeups if a certain number
>>>of ticks are missed.
>>>
>>>Regards,
>>>Dave
>>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 9/11/07 19:22, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Since I had a high error (~.03%) for the ASYNC method a 
>>>>>          
>>>>>
>>couple of days ago,
>>    
>>
>>>>>I ran another ASYNC test. I think there may have been
something
>>>>>wrong with the code I used a couple of days ago for 
>>>>>          
>>>>>
>>ASYNC. It may have been
>>    
>>
>>>>>missing the immediate delivery of interrupt after context 
>>>>>          
>>>>>
>>switch in.
>>    
>>
>>>>>My results indicate that either SYNC or ASYNC give 
>>>>>          
>>>>>
>>acceptable accuracy,
>>    
>>
>>>>>each running consistently around or under .01%. MIXED has 
>>>>>          
>>>>>
>>a fairly high
>>    
>>
>>>>>error of
>>>>>greater than .03%. Probably too close to .05% ntp 
>>>>>          
>>>>>
>>threshold for comfort.
>>    
>>
>>>>>I don''t have an overnight run with SYNC. I plan to
leave
>>>>>          
>>>>>
>>SYNC running
>>    
>>
>>>>>over the weekend. If you''d rather I can leave MIXED
>>>>>          
>>>>>
>>running instead.
>>    
>>
>>>>>It may be too early to pick the protocol and I can run 
>>>>>          
>>>>>
>>more overnight tests
>>    
>>
>>>>>next week.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>I''m a bit worried about any unwanted side effects of
the
>>>>        
>>>>
>>SYNC+run_timer
>>    
>>
>>>>approach -- e.g., whether timer wakeups will cause higher 
>>>>        
>>>>
>>system-wide CPU
>>    
>>
>>>>contention. I find it easier to think through the 
>>>>        
>>>>
>>implications of ASYNC. I''m
>>    
>>
>>>>surprised that MIXED loses time, and is less accurate than 
>>>>        
>>>>
>>ASYNC. Perhaps it
>>    
>>
>>>>delivers more timer interrupts than the other approaches, 
>>>>        
>>>>
>>and each interrupt
>>    
>>
>>>>event causes a small accumulated error?
>>>>
>>>>Overall I would consider MIXED and ASYNC as favourites and 
>>>>        
>>>>
>>if the latter is
>>    
>>
>>>>actually more accurate then I can simply revert the changeset
that
>>>>implemented MIXED.
>>>>
>>>>Perhaps rather than running more of the same workloads you 
>>>>        
>>>>
>>could try idle
>>    
>>
>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
>>>>        
>>>>
>>to /dev/null)? We
>>    
>>
>>>>don''t have any data on workloads that aren''t
CPU bound, so
>>>>        
>>>>
>>that''s really an
>>    
>>
>>>>obvious place to put any further effort imo.
>>>>
>>>>-- Keir
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> and which ones are known to be appropriate for which kernels?
The default, delay_for_missed_ticks is good for 32 bit Linux.
no_missed_ticks_pending is the most accurate for 64 bit Linux, but the version
of no_missed_ticks_pending checked in has some errors in it, as I mentioned.

Last I checked, before the most recent changeset, the version now called
one_missed_tick_pending had an error > .05 % for some loads.
This is called the MIXED method in the data I''ve published.

Regards,
Dave



Dan Magenheimer wrote:
>Anyone make measurements on the final patch?
>
>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no
load.  This was xen-unstable tip today with no options specified.  32-bit was
about 0.01%.
>
>I think I missed something... how do I run the various accounting choices
and which ones are known to be appropriate for which kernels?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser
>>Sent: Thursday, December 06, 2007 4:57 AM
>>To: Dave Winchell
>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang,
>>Yunhong
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Please take a look at xen-unstable changeset 16545.
>>
>> -- Keir
>>
>>On 26/11/07 20:57, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>
>>    
>>
>>>Keir,
>>>
>>>The accuracy data I''ve collected for i/o loads for the
>>>various time protocols follows. In addition, the data
>>>for cpu loads is shown.
>>>
>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>The cpu load is usex -e36 on each guest.
>>>(usex is available at http://people.redhat.com/anderson/usex.)
>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>
>>>The loads labeled i/o-32 are 32 instances of dd.
>>>Also, these are run on 4 cpu AMD box.
>>>In addition, there is an idle rh-32bit guest.
>>>All three guests are 8vcpu.
>>>
>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>except that the redhat-64 guest has 4 instances of dd.
>>>
>>>Date Duration Protocol sles, rhat error load
>>>
>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu
>>>
>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>
>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu
>>>
>>>
>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8
>>>
>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>
>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8
>>>
>>>
>>>
>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>
>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32
>>>
>>>
>>>Overhead measurements:
>>>
>>>Progress in terms of number of passes through a fixed 
>>>      
>>>
>>system workload
>>    
>>
>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>The workload was usex -b48.
>>>
>>>
>>>ASYNC 167 min 145 passes .868 passes/min
>>>SYNC 167 min 144 passes .862 passes/min
>>>SYNC 1065 min 919 passes .863 passes/min
>>>MIXED 221 min 196 passes .887 passes/min
>>>
>>>
>>>Conclusions:
>>>
>>>The only protocol which meets the .05% accuracy requirement for ntp
>>>tracking under the loads
>>>above is the SYNC protocol. The worst case accuracies for 
>>>      
>>>
>>SYNC, MIXED,
>>    
>>
>>>and ASYNC
>>>are .022%, .12%, and .14%, respectively.
>>>
>>>We could reduce the cost of the SYNC method by only 
>>>      
>>>
>>scheduling the extra
>>    
>>
>>>wakeups if a certain number
>>>of ticks are missed.
>>>
>>>Regards,
>>>Dave
>>>
>>>Keir Fraser wrote:
>>>
>>>      
>>>
>>>>On 9/11/07 19:22, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Since I had a high error (~.03%) for the ASYNC method a 
>>>>>          
>>>>>
>>couple of days ago,
>>    
>>
>>>>>I ran another ASYNC test. I think there may have been
something
>>>>>wrong with the code I used a couple of days ago for 
>>>>>          
>>>>>
>>ASYNC. It may have been
>>    
>>
>>>>>missing the immediate delivery of interrupt after context 
>>>>>          
>>>>>
>>switch in.
>>    
>>
>>>>>My results indicate that either SYNC or ASYNC give 
>>>>>          
>>>>>
>>acceptable accuracy,
>>    
>>
>>>>>each running consistently around or under .01%. MIXED has 
>>>>>          
>>>>>
>>a fairly high
>>    
>>
>>>>>error of
>>>>>greater than .03%. Probably too close to .05% ntp 
>>>>>          
>>>>>
>>threshold for comfort.
>>    
>>
>>>>>I don''t have an overnight run with SYNC. I plan to
leave
>>>>>          
>>>>>
>>SYNC running
>>    
>>
>>>>>over the weekend. If you''d rather I can leave MIXED
>>>>>          
>>>>>
>>running instead.
>>    
>>
>>>>>It may be too early to pick the protocol and I can run 
>>>>>          
>>>>>
>>more overnight tests
>>    
>>
>>>>>next week.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>I''m a bit worried about any unwanted side effects of
the
>>>>        
>>>>
>>SYNC+run_timer
>>    
>>
>>>>approach -- e.g., whether timer wakeups will cause higher 
>>>>        
>>>>
>>system-wide CPU
>>    
>>
>>>>contention. I find it easier to think through the 
>>>>        
>>>>
>>implications of ASYNC. I''m
>>    
>>
>>>>surprised that MIXED loses time, and is less accurate than 
>>>>        
>>>>
>>ASYNC. Perhaps it
>>    
>>
>>>>delivers more timer interrupts than the other approaches, 
>>>>        
>>>>
>>and each interrupt
>>    
>>
>>>>event causes a small accumulated error?
>>>>
>>>>Overall I would consider MIXED and ASYNC as favourites and 
>>>>        
>>>>
>>if the latter is
>>    
>>
>>>>actually more accurate then I can simply revert the changeset
that
>>>>implemented MIXED.
>>>>
>>>>Perhaps rather than running more of the same workloads you 
>>>>        
>>>>
>>could try idle
>>    
>>
>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads 
>>>>        
>>>>
>>to /dev/null)? We
>>    
>>
>>>>don''t have any data on workloads that aren''t
CPU bound, so
>>>>        
>>>>
>>that''s really an
>>    
>>
>>>>obvious place to put any further effort imo.
>>>>
>>>>-- Keir
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave --

Did you get your correction ported?  If so, it would be nice to see this get
into 3.1.3.

Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed
ticks pending)
on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error
I''ve seen so far
is 0.012%.  But I haven''t tried any exotic loads, just LTP.

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, December 19, 2007 12:33 PM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
> Eddie; Jiang, Yunhong; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> I did some testing with the constant tsc offset SYNC method 
> (now called
> no_missed_ticks_pending)
> and found the error to be very high, much larger than 1 %, as 
> I recall.
> I have not had a chance to submit a correction. I will try to 
> do it later
> this week or the first week in January. My version of constant tsc
> offset SYNC method
> produces .02 % error, so I just need to port that into the 
> current code.
> 
> The error you got for both of those kernels is what I would expect
> for the default mode, delay_for_missed_ticks.
> 
> I''ll let Keir answer on how to set the time mode.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
> >Anyone make measurements on the final patch?
> >
> >I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
> about 0.2% with no load.  This was xen-unstable tip today 
> with no options specified.  32-bit was about 0.01%.
> >
> >I think I missed something... how do I run the various 
> accounting choices and which ones are known to be appropriate 
> for which kernels?
> >
> >Thanks,
> >Dan
> >
> >
> >
> >>-----Original Message-----
> >>From: xen-devel-bounces@lists.xensource.com
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
> Keir Fraser
> >>Sent: Thursday, December 06, 2007 4:57 AM
> >>To: Dave Winchell
> >>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie;
Jiang,
> >>Yunhong
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>Please take a look at xen-unstable changeset 16545.
> >>
> >> -- Keir
> >>
> >>On 26/11/07 20:57, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>
> >>>Keir,
> >>>
> >>>The accuracy data I''ve collected for i/o loads for the
> >>>various time protocols follows. In addition, the data
> >>>for cpu loads is shown.
> >>>
> >>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
> >>>Two guests, red hat and sles 64 bit, 8 vcpu each.
> >>>The cpu load is usex -e36 on each guest.
> >>>(usex is available at http://people.redhat.com/anderson/usex.)
> >>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
> >>>
> >>>The loads labeled i/o-32 are 32 instances of dd.
> >>>Also, these are run on 4 cpu AMD box.
> >>>In addition, there is an idle rh-32bit guest.
> >>>All three guests are 8vcpu.
> >>>
> >>>The loads labeled i/o-4/32 are the same as i/o-32
> >>>except that the redhat-64 guest has 4 instances of dd.
> >>>
> >>>Date Duration Protocol sles, rhat error load
> >>>
> >>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005%
cpu
> >>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012%
cpu
> >>>
> >>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> >>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
> >>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> >>>
> >>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> >>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031%
cpu
> >>>
> >>>
> >>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09%
i/o-8
> >>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14%
i/o-8
> >>>
> >>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022%
i/o-8
> >>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
> >>>
> >>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029%
i/o-8
> >>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031%
i/o-8
> >>>
> >>>
> >>>
> >>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> >>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005%
i/o-32
> >>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> >>>
> >>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003%
i/o-4/32
> >>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01%
i/o-4/32
> >>>
> >>>
> >>>Overhead measurements:
> >>>
> >>>Progress in terms of number of passes through a fixed
> >>>
> >>>
> >>system workload
> >>
> >>
> >>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>The workload was usex -b48.
> >>>
> >>>
> >>>ASYNC 167 min 145 passes .868 passes/min
> >>>SYNC 167 min 144 passes .862 passes/min
> >>>SYNC 1065 min 919 passes .863 passes/min
> >>>MIXED 221 min 196 passes .887 passes/min
> >>>
> >>>
> >>>Conclusions:
> >>>
> >>>The only protocol which meets the .05% accuracy requirement for
ntp
> >>>tracking under the loads
> >>>above is the SYNC protocol. The worst case accuracies for
> >>>
> >>>
> >>SYNC, MIXED,
> >>
> >>
> >>>and ASYNC
> >>>are .022%, .12%, and .14%, respectively.
> >>>
> >>>We could reduce the cost of the SYNC method by only
> >>>
> >>>
> >>scheduling the extra
> >>
> >>
> >>>wakeups if a certain number
> >>>of ticks are missed.
> >>>
> >>>Regards,
> >>>Dave
> >>>
> >>>Keir Fraser wrote:
> >>>
> >>>
> >>>
> >>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>
> >>>>
> >><dwinchell@virtualiron.com> wrote:
> >>
> >>
> >>>>
> >>>>
> >>>>
> >>>>>Since I had a high error (~.03%) for the ASYNC method a
> >>>>>
> >>>>>
> >>couple of days ago,
> >>
> >>
> >>>>>I ran another ASYNC test. I think there may have been
something
> >>>>>wrong with the code I used a couple of days ago for
> >>>>>
> >>>>>
> >>ASYNC. It may have been
> >>
> >>
> >>>>>missing the immediate delivery of interrupt after
context
> >>>>>
> >>>>>
> >>switch in.
> >>
> >>
> >>>>>My results indicate that either SYNC or ASYNC give
> >>>>>
> >>>>>
> >>acceptable accuracy,
> >>
> >>
> >>>>>each running consistently around or under .01%. MIXED
has
> >>>>>
> >>>>>
> >>a fairly high
> >>
> >>
> >>>>>error of
> >>>>>greater than .03%. Probably too close to .05% ntp
> >>>>>
> >>>>>
> >>threshold for comfort.
> >>
> >>
> >>>>>I don''t have an overnight run with SYNC. I
plan to leave
> >>>>>
> >>>>>
> >>SYNC running
> >>
> >>
> >>>>>over the weekend. If you''d rather I can leave
MIXED
> >>>>>
> >>>>>
> >>running instead.
> >>
> >>
> >>>>>It may be too early to pick the protocol and I can run
> >>>>>
> >>>>>
> >>more overnight tests
> >>
> >>
> >>>>>next week.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>I''m a bit worried about any unwanted side effects
of the
> >>>>
> >>>>
> >>SYNC+run_timer
> >>
> >>
> >>>>approach -- e.g., whether timer wakeups will cause higher
> >>>>
> >>>>
> >>system-wide CPU
> >>
> >>
> >>>>contention. I find it easier to think through the
> >>>>
> >>>>
> >>implications of ASYNC. I''m
> >>
> >>
> >>>>surprised that MIXED loses time, and is less accurate than
> >>>>
> >>>>
> >>ASYNC. Perhaps it
> >>
> >>
> >>>>delivers more timer interrupts than the other approaches,
> >>>>
> >>>>
> >>and each interrupt
> >>
> >>
> >>>>event causes a small accumulated error?
> >>>>
> >>>>Overall I would consider MIXED and ASYNC as favourites and
> >>>>
> >>>>
> >>if the latter is
> >>
> >>
> >>>>actually more accurate then I can simply revert the
changeset that
> >>>>implemented MIXED.
> >>>>
> >>>>Perhaps rather than running more of the same workloads you
> >>>>
> >>>>
> >>could try idle
> >>
> >>
> >>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads
> >>>>
> >>>>
> >>to /dev/null)? We
> >>
> >>
> >>>>don''t have any data on workloads that
aren''t CPU bound, so
> >>>>
> >>>>
> >>that''s really an
> >>
> >>
> >>>>obvious place to put any further effort imo.
> >>>>
> >>>>-- Keir
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xensource.com
> >>http://lists.xensource.com/xen-devel
> >>
> >>
> >>
> >>
> >
> >
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

I had to put that port aside with other pressing issues here.
I will try to get the port done in the next couple of days.
Is that soon enough for 3.1.3?

To see the error with timer_mode=2(=SYNC=no missed ticks pending)
you probably need to over commit the physical processors with
virtual processors, say 2:1, and then load up all the virtual processors.

Regards,
Dave

Dan Magenheimer wrote:
>Hi Dave --
>
>Did you get your correction ported?  If so, it would be nice to see this get
into 3.1.3.
>
>Note that I just did some very limited testing with timer_mode=2(=SYNC=no
missed ticks pending)
>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error
I''ve seen so far
>is 0.012%.  But I haven''t tried any exotic loads, just LTP.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, December 19, 2007 12:33 PM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>Eddie; Jiang, Yunhong; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I did some testing with the constant tsc offset SYNC method 
>>(now called
>>no_missed_ticks_pending)
>>and found the error to be very high, much larger than 1 %, as 
>>I recall.
>>I have not had a chance to submit a correction. I will try to 
>>do it later
>>this week or the first week in January. My version of constant tsc
>>offset SYNC method
>>produces .02 % error, so I just need to port that into the 
>>current code.
>>
>>The error you got for both of those kernels is what I would expect
>>for the default mode, delay_for_missed_ticks.
>>
>>I''ll let Keir answer on how to set the time mode.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Anyone make measurements on the final patch?
>>>
>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
>>>      
>>>
>>about 0.2% with no load.  This was xen-unstable tip today 
>>with no options specified.  32-bit was about 0.01%.
>>    
>>
>>>I think I missed something... how do I run the various 
>>>      
>>>
>>accounting choices and which ones are known to be appropriate 
>>for which kernels?
>>    
>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
>>>>        
>>>>
>>Keir Fraser
>>    
>>
>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>To: Dave Winchell
>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie;
Jiang,
>>>>Yunhong
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Please take a look at xen-unstable changeset 16545.
>>>>
>>>>-- Keir
>>>>
>>>>On 26/11/07 20:57, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Keir,
>>>>>
>>>>>The accuracy data I''ve collected for i/o loads for
the
>>>>>various time protocols follows. In addition, the data
>>>>>for cpu loads is shown.
>>>>>
>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD
box.
>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>The cpu load is usex -e36 on each guest.
>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>
>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>Also, these are run on 4 cpu AMD box.
>>>>>In addition, there is an idle rh-32bit guest.
>>>>>All three guests are 8vcpu.
>>>>>
>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>
>>>>>Date Duration Protocol sles, rhat error load
>>>>>
>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
+.005% cpu
>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012%
cpu
>>>>>
>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004%
cpu
>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005%
cpu
>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>
>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031%
cpu
>>>>>
>>>>>
>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09%
i/o-8
>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14%
i/o-8
>>>>>
>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022%
i/o-8
>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>
>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029%
i/o-8
>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031%
i/o-8
>>>>>
>>>>>
>>>>>
>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005%
i/o-32
>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>
>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003%
i/o-4/32
>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01%
i/o-4/32
>>>>>
>>>>>
>>>>>Overhead measurements:
>>>>>
>>>>>Progress in terms of number of passes through a fixed
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>system workload
>>>>
>>>>
>>>>        
>>>>
>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>The workload was usex -b48.
>>>>>
>>>>>
>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>
>>>>>
>>>>>Conclusions:
>>>>>
>>>>>The only protocol which meets the .05% accuracy requirement
for ntp
>>>>>tracking under the loads
>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>SYNC, MIXED,
>>>>
>>>>
>>>>        
>>>>
>>>>>and ASYNC
>>>>>are .022%, .12%, and .14%, respectively.
>>>>>
>>>>>We could reduce the cost of the SYNC method by only
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>scheduling the extra
>>>>
>>>>
>>>>        
>>>>
>>>>>wakeups if a certain number
>>>>>of ticks are missed.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>Keir Fraser wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Since I had a high error (~.03%) for the ASYNC
method a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>couple of days ago,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I ran another ASYNC test. I think there may have
been something
>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ASYNC. It may have been
>>>>
>>>>
>>>>        
>>>>
>>>>>>>missing the immediate delivery of interrupt after
context
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>switch in.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>acceptable accuracy,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>each running consistently around or under .01%.
MIXED has
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>a fairly high
>>>>
>>>>
>>>>        
>>>>
>>>>>>>error of
>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>threshold for comfort.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I don''t have an overnight run with SYNC. I
plan to leave
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>SYNC running
>>>>
>>>>
>>>>        
>>>>
>>>>>>>over the weekend. If you''d rather I can
leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>running instead.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>It may be too early to pick the protocol and I can
run
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>more overnight tests
>>>>
>>>>
>>>>        
>>>>
>>>>>>>next week.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I''m a bit worried about any unwanted side
effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>SYNC+run_timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>approach -- e.g., whether timer wakeups will cause
higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>system-wide CPU
>>>>
>>>>
>>>>        
>>>>
>>>>>>contention. I find it easier to think through the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>implications of ASYNC. I''m
>>>>
>>>>
>>>>        
>>>>
>>>>>>surprised that MIXED loses time, and is less accurate
than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>ASYNC. Perhaps it
>>>>
>>>>
>>>>        
>>>>
>>>>>>delivers more timer interrupts than the other
approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>and each interrupt
>>>>
>>>>
>>>>        
>>>>
>>>>>>event causes a small accumulated error?
>>>>>>
>>>>>>Overall I would consider MIXED and ASYNC as favourites
and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>if the latter is
>>>>
>>>>
>>>>        
>>>>
>>>>>>actually more accurate then I can simply revert the
changeset that
>>>>>>implemented MIXED.
>>>>>>
>>>>>>Perhaps rather than running more of the same workloads
you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>could try idle
>>>>
>>>>
>>>>        
>>>>
>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc
reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>to /dev/null)? We
>>>>
>>>>
>>>>        
>>>>
>>>>>>don''t have any data on workloads that
aren''t CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>that''s really an
>>>>
>>>>
>>>>        
>>>>
>>>>>>obvious place to put any further effort imo.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan and Keir,

Attached is a patch that fixes some issues with the SYNC policy 
(no_missed_ticks_pending).
I have not tried to make the change the minimal one, but, rather, just 
ported into
the new code what I know to work well. The error for 
no_missed_ticks_pending goes from
over 3% to .03% with this change according to my testing.

Regards,
Dave

Dan Magenheimer wrote:
>Hi Dave --
>
>Did you get your correction ported?  If so, it would be nice to see this get
into 3.1.3.
>
>Note that I just did some very limited testing with timer_mode=2(=SYNC=no
missed ticks pending)
>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error
I''ve seen so far
>is 0.012%.  But I haven''t tried any exotic loads, just LTP.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, December 19, 2007 12:33 PM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>Eddie; Jiang, Yunhong; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I did some testing with the constant tsc offset SYNC method 
>>(now called
>>no_missed_ticks_pending)
>>and found the error to be very high, much larger than 1 %, as 
>>I recall.
>>I have not had a chance to submit a correction. I will try to 
>>do it later
>>this week or the first week in January. My version of constant tsc
>>offset SYNC method
>>produces .02 % error, so I just need to port that into the 
>>current code.
>>
>>The error you got for both of those kernels is what I would expect
>>for the default mode, delay_for_missed_ticks.
>>
>>I''ll let Keir answer on how to set the time mode.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Anyone make measurements on the final patch?
>>>
>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of 
>>>      
>>>
>>about 0.2% with no load.  This was xen-unstable tip today 
>>with no options specified.  32-bit was about 0.01%.
>>    
>>
>>>I think I missed something... how do I run the various 
>>>      
>>>
>>accounting choices and which ones are known to be appropriate 
>>for which kernels?
>>    
>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: xen-devel-bounces@lists.xensource.com
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
>>>>        
>>>>
>>Keir Fraser
>>    
>>
>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>To: Dave Winchell
>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie;
Jiang,
>>>>Yunhong
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Please take a look at xen-unstable changeset 16545.
>>>>
>>>>-- Keir
>>>>
>>>>On 26/11/07 20:57, "Dave Winchell" 
>>>>        
>>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>>
>>>>        
>>>>
>>>>>Keir,
>>>>>
>>>>>The accuracy data I''ve collected for i/o loads for
the
>>>>>various time protocols follows. In addition, the data
>>>>>for cpu loads is shown.
>>>>>
>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD
box.
>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>The cpu load is usex -e36 on each guest.
>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.
>>>>>
>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>Also, these are run on 4 cpu AMD box.
>>>>>In addition, there is an idle rh-32bit guest.
>>>>>All three guests are 8vcpu.
>>>>>
>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>except that the redhat-64 guest has 4 instances of dd.
>>>>>
>>>>>Date Duration Protocol sles, rhat error load
>>>>>
>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
+.005% cpu
>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012%
cpu
>>>>>
>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004%
cpu
>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005%
cpu
>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
>>>>>
>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031%
cpu
>>>>>
>>>>>
>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09%
i/o-8
>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14%
i/o-8
>>>>>
>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022%
i/o-8
>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8
>>>>>
>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029%
i/o-8
>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031%
i/o-8
>>>>>
>>>>>
>>>>>
>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005%
i/o-32
>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
>>>>>
>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003%
i/o-4/32
>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01%
i/o-4/32
>>>>>
>>>>>
>>>>>Overhead measurements:
>>>>>
>>>>>Progress in terms of number of passes through a fixed
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>system workload
>>>>
>>>>
>>>>        
>>>>
>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>The workload was usex -b48.
>>>>>
>>>>>
>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>
>>>>>
>>>>>Conclusions:
>>>>>
>>>>>The only protocol which meets the .05% accuracy requirement
for ntp
>>>>>tracking under the loads
>>>>>above is the SYNC protocol. The worst case accuracies for
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>SYNC, MIXED,
>>>>
>>>>
>>>>        
>>>>
>>>>>and ASYNC
>>>>>are .022%, .12%, and .14%, respectively.
>>>>>
>>>>>We could reduce the cost of the SYNC method by only
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>scheduling the extra
>>>>
>>>>
>>>>        
>>>>
>>>>>wakeups if a certain number
>>>>>of ticks are missed.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>Keir Fraser wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Since I had a high error (~.03%) for the ASYNC
method a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>couple of days ago,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I ran another ASYNC test. I think there may have
been something
>>>>>>>wrong with the code I used a couple of days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ASYNC. It may have been
>>>>
>>>>
>>>>        
>>>>
>>>>>>>missing the immediate delivery of interrupt after
context
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>switch in.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>My results indicate that either SYNC or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>acceptable accuracy,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>each running consistently around or under .01%.
MIXED has
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>a fairly high
>>>>
>>>>
>>>>        
>>>>
>>>>>>>error of
>>>>>>>greater than .03%. Probably too close to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>threshold for comfort.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>I don''t have an overnight run with SYNC. I
plan to leave
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>SYNC running
>>>>
>>>>
>>>>        
>>>>
>>>>>>>over the weekend. If you''d rather I can
leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>running instead.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>It may be too early to pick the protocol and I can
run
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>more overnight tests
>>>>
>>>>
>>>>        
>>>>
>>>>>>>next week.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>I''m a bit worried about any unwanted side
effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>SYNC+run_timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>approach -- e.g., whether timer wakeups will cause
higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>system-wide CPU
>>>>
>>>>
>>>>        
>>>>
>>>>>>contention. I find it easier to think through the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>implications of ASYNC. I''m
>>>>
>>>>
>>>>        
>>>>
>>>>>>surprised that MIXED loses time, and is less accurate
than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>ASYNC. Perhaps it
>>>>
>>>>
>>>>        
>>>>
>>>>>>delivers more timer interrupts than the other
approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>and each interrupt
>>>>
>>>>
>>>>        
>>>>
>>>>>>event causes a small accumulated error?
>>>>>>
>>>>>>Overall I would consider MIXED and ASYNC as favourites
and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>if the latter is
>>>>
>>>>
>>>>        
>>>>
>>>>>>actually more accurate then I can simply revert the
changeset that
>>>>>>implemented MIXED.
>>>>>>
>>>>>>Perhaps rather than running more of the same workloads
you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>could try idle
>>>>
>>>>
>>>>        
>>>>
>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc
reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>to /dev/null)? We
>>>>
>>>>
>>>>        
>>>>
>>>>>>don''t have any data on workloads that
aren''t CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>that''s really an
>>>>
>>>>
>>>>        
>>>>
>>>>>>obvious place to put any further effort imo.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>_______________________________________________
>>>>Xen-devel mailing list
>>>>Xen-devel@lists.xensource.com
>>>>http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Applied as c/s 16690, although the checked-in patch is smaller. I think the
only important fix is to pt_intr_post() and the only bit of the patch I
totally omitted was the change to pt_process_missed_ticks(). I don''t
think
that change can be important, but let''s see what happens to the error
percentage...

 -- Keir

On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Hi Dan and Keir,
> 
> Attached is a patch that fixes some issues with the SYNC policy
> (no_missed_ticks_pending).
> I have not tried to make the change the minimal one, but, rather, just
> ported into
> the new code what I know to work well. The error for
> no_missed_ticks_pending goes from
> over 3% to .03% with this change according to my testing.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
>> Hi Dave --
>> 
>> Did you get your correction ported?  If so, it would be nice to see
this get
>> into 3.1.3.
>> 
>> Note that I just did some very limited testing with
timer_mode=2(=SYNC=no
>> missed ticks pending)
>> on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error
I''ve
>> seen so far
>> is 0.012%.  But I haven''t tried any exotic loads, just LTP.
>> 
>> Thanks,
>> Dan
>> 
>>  
>> 
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Wednesday, December 19, 2007 12:33 PM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong,
>>> Eddie; Jiang, Yunhong; Dave Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>> disables pending
>>> missed ticks
>>> 
>>> 
>>> Dan,
>>> 
>>> I did some testing with the constant tsc offset SYNC method
>>> (now called
>>> no_missed_ticks_pending)
>>> and found the error to be very high, much larger than 1 %, as
>>> I recall.
>>> I have not had a chance to submit a correction. I will try to
>>> do it later
>>> this week or the first week in January. My version of constant tsc
>>> offset SYNC method
>>> produces .02 % error, so I just need to port that into the
>>> current code.
>>> 
>>> The error you got for both of those kernels is what I would expect
>>> for the default mode, delay_for_missed_ticks.
>>> 
>>> I''ll let Keir answer on how to set the time mode.
>>> 
>>> Regards,
>>> Dave
>>> 
>>> Dan Magenheimer wrote:
>>> 
>>>    
>>> 
>>>> Anyone make measurements on the final patch?
>>>> 
>>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>      
>>>> 
>>> about 0.2% with no load.  This was xen-unstable tip today
>>> with no options specified.  32-bit was about 0.01%.
>>>    
>>> 
>>>> I think I missed something... how do I run the various
>>>>      
>>>> 
>>> accounting choices and which ones are known to be appropriate
>>> for which kernels?
>>>    
>>> 
>>>> Thanks,
>>>> Dan
>>>> 
>>>> 
>>>> 
>>>>      
>>>> 
>>>>> -----Original Message-----
>>>>> From: xen-devel-bounces@lists.xensource.com
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>        
>>>>> 
>>> Keir Fraser
>>>    
>>> 
>>>>> Sent: Thursday, December 06, 2007 4:57 AM
>>>>> To: Dave Winchell
>>>>> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
Eddie; Jiang,
>>>>> Yunhong
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>> disables pending
>>>>> missed ticks
>>>>> 
>>>>> 
>>>>> Please take a look at xen-unstable changeset 16545.
>>>>> 
>>>>> -- Keir
>>>>> 
>>>>> On 26/11/07 20:57, "Dave Winchell"
>>>>>        
>>>>> 
>>> <dwinchell@virtualiron.com> wrote:
>>>    
>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> Keir,
>>>>>> 
>>>>>> The accuracy data I''ve collected for i/o loads
for the
>>>>>> various time protocols follows. In addition, the data
>>>>>> for cpu loads is shown.
>>>>>> 
>>>>>> The loads labeled cpu and i/o-8 are on an 8 processor
AMD box.
>>>>>> Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>> The cpu load is usex -e36 on each guest.
>>>>>> (usex is available at
http://people.redhat.com/anderson/usex.)
>>>>>> i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
>>>>>> 
>>>>>> The loads labeled i/o-32 are 32 instances of dd.
>>>>>> Also, these are run on 4 cpu AMD box.
>>>>>> In addition, there is an idle rh-32bit guest.
>>>>>> All three guests are 8vcpu.
>>>>>> 
>>>>>> The loads labeled i/o-4/32 are the same as i/o-32
>>>>>> except that the redhat-64 guest has 4 instances of dd.
>>>>>> 
>>>>>> Date Duration Protocol sles, rhat error load
>>>>>> 
>>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
+.005% cpu
>>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%,
+.012% cpu
>>>>>> 
>>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
-.004% cpu
>>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%,
-.005% cpu
>>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%,
-.003% cpu
>>>>>> 
>>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040%
cpu
>>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%,
-.031% cpu
>>>>>> 
>>>>>> 
>>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
-.09% i/o-8
>>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
-.14% i/o-8
>>>>>> 
>>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
-.022% i/o-8
>>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018%
i/o-8
>>>>>> 
>>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
-.029% i/o-8
>>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
-.031% i/o-8
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04%
i/o-32
>>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
-.005% i/o-32
>>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11%
i/o-32
>>>>>> 
>>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
.003% i/o-4/32
>>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
.01% i/o-4/32
>>>>>> 
>>>>>> 
>>>>>> Overhead measurements:
>>>>>> 
>>>>>> Progress in terms of number of passes through a fixed
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> system workload
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>> The workload was usex -b48.
>>>>>> 
>>>>>> 
>>>>>> ASYNC 167 min 145 passes .868 passes/min
>>>>>> SYNC 167 min 144 passes .862 passes/min
>>>>>> SYNC 1065 min 919 passes .863 passes/min
>>>>>> MIXED 221 min 196 passes .887 passes/min
>>>>>> 
>>>>>> 
>>>>>> Conclusions:
>>>>>> 
>>>>>> The only protocol which meets the .05% accuracy
requirement for ntp
>>>>>> tracking under the loads
>>>>>> above is the SYNC protocol. The worst case accuracies
for
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> SYNC, MIXED,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> and ASYNC
>>>>>> are .022%, .12%, and .14%, respectively.
>>>>>> 
>>>>>> We could reduce the cost of the SYNC method by only
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>> scheduling the extra
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>> wakeups if a certain number
>>>>>> of ticks are missed.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>> Keir Fraser wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>          
>>>>>> 
>>>>>>> On 9/11/07 19:22, "Dave Winchell"
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> <dwinchell@virtualiron.com> wrote:
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>>>>> Since I had a high error (~.03%) for the ASYNC
method a
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> couple of days ago,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> I ran another ASYNC test. I think there may
have been something
>>>>>>>> wrong with the code I used a couple of days ago
for
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> ASYNC. It may have been
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> missing the immediate delivery of interrupt
after context
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> switch in.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> My results indicate that either SYNC or ASYNC
give
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> acceptable accuracy,
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> each running consistently around or under .01%.
MIXED has
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> a fairly high
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> error of
>>>>>>>> greater than .03%. Probably too close to .05%
ntp
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> threshold for comfort.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> I don''t have an overnight run with
SYNC. I plan to leave
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> SYNC running
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> over the weekend. If you''d rather I
can leave MIXED
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> running instead.
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> It may be too early to pick the protocol and I
can run
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>> more overnight tests
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>>> next week.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>           
>>>>>>>> 
>>>>>>> I''m a bit worried about any unwanted side
effects of the
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> SYNC+run_timer
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> approach -- e.g., whether timer wakeups will cause
higher
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> system-wide CPU
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> contention. I find it easier to think through the
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> implications of ASYNC. I''m
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> surprised that MIXED loses time, and is less
accurate than
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> ASYNC. Perhaps it
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> delivers more timer interrupts than the other
approaches,
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> and each interrupt
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> event causes a small accumulated error?
>>>>>>> 
>>>>>>> Overall I would consider MIXED and ASYNC as
favourites and
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> if the latter is
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> actually more accurate then I can simply revert the
changeset that
>>>>>>> implemented MIXED.
>>>>>>> 
>>>>>>> Perhaps rather than running more of the same
workloads you
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> could try idle
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> VCPUs and I/O bound VCPUs (e.g., repeated large
disc reads
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> to /dev/null)? We
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> don''t have any data on workloads that
aren''t CPU bound, so
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> that''s really an
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>>>>> obvious place to put any further effort imo.
>>>>>>> 
>>>>>>> -- Keir
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            
>>>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>        
>>>>> 
>>>> 
>>>>      
>>>> 
>>>    
>>> 
>> 
>>  
>> 
> 
> diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>  
>      missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>      if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> -        pt->do_not_freeze = !pt->pending_intr_nr;
> +        pt->do_not_freeze = 1;
>      else
>          pt->pending_intr_nr += missed_ticks;
>      pt->scheduled += missed_ticks * pt->period;
> @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>  
>      pt_lock(pt);
>  
> -    pt->pending_intr_nr++;
> +    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
> +        pt->pending_intr_nr = 1;
> + pt->do_not_freeze = 0;
> +    }
> +    else
> + pt->pending_intr_nr++;
>  
>      if ( !pt->one_shot )
>      {
> @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>          return;
>      }
>  
> -    pt->do_not_freeze = 0;
> -
>      if ( pt->one_shot )
>      {
>          pt->enabled = 0;
> @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>              pt->last_plt_gtime = hvm_get_guest_time(v);
>              pt->pending_intr_nr = 0; /* ''collapse''
all missed ticks */
>          }
> + else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> +     pt->pending_intr_nr--;
> +     pt->last_plt_gtime = hvm_get_guest_time(v);
> + }
>          else
>          {
>              pt->last_plt_gtime += pt->period_cycles;


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

The latest change, c/s 16690, looks fine.
I agree that the code in c/s 16690 is equivalent to
the code I submitted. Also, your version is more
concise.

The error tests confirm the equivalence. With overnight cpu loads,
the checked in version was accurate to +.048% for sles
and +.038% for red hat. My version was +.046% and +.032% in a 2 hour test.
I don''t think the difference is significant.

i/o loads produced errors of +.01%.

Thanks for all your efforts on this issue.

Regards,
Dave



Keir Fraser wrote:
>Applied as c/s 16690, although the checked-in patch is smaller. I think the
>only important fix is to pt_intr_post() and the only bit of the patch I
>totally omitted was the change to pt_process_missed_ticks(). I
don''t think
>that change can be important, but let''s see what happens to the
error
>percentage...
>
> -- Keir
>
>On 4/1/08 23:24, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
>
>  
>
>>Hi Dan and Keir,
>>
>>Attached is a patch that fixes some issues with the SYNC policy
>>(no_missed_ticks_pending).
>>I have not tried to make the change the minimal one, but, rather, just
>>ported into
>>the new code what I know to work well. The error for
>>no_missed_ticks_pending goes from
>>over 3% to .03% with this change according to my testing.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>Did you get your correction ported?  If so, it would be nice to see
this get
>>>into 3.1.3.
>>>
>>>Note that I just did some very limited testing with
timer_mode=2(=SYNC=no
>>>missed ticks pending)
>>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst
error I''ve
>>>seen so far
>>>is 0.012%.  But I haven''t tried any exotic loads, just LTP.
>>>
>>>Thanks,
>>>Dan
>>>
>>> 
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>To: dan.magenheimer@oracle.com
>>>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com;
Dong,
>>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Dan,
>>>>
>>>>I did some testing with the constant tsc offset SYNC method
>>>>(now called
>>>>no_missed_ticks_pending)
>>>>and found the error to be very high, much larger than 1 %, as
>>>>I recall.
>>>>I have not had a chance to submit a correction. I will try to
>>>>do it later
>>>>this week or the first week in January. My version of constant
tsc
>>>>offset SYNC method
>>>>produces .02 % error, so I just need to port that into the
>>>>current code.
>>>>
>>>>The error you got for both of those kernels is what I would
expect
>>>>for the default mode, delay_for_missed_ticks.
>>>>
>>>>I''ll let Keir answer on how to set the time mode.
>>>>
>>>>Regards,
>>>>Dave
>>>>
>>>>Dan Magenheimer wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>Anyone make measurements on the final patch?
>>>>>
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>about 0.2% with no load.  This was xen-unstable tip today
>>>>with no options specified.  32-bit was about 0.01%.
>>>>   
>>>>
>>>>        
>>>>
>>>>>I think I missed something... how do I run the various
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>accounting choices and which ones are known to be appropriate
>>>>for which kernels?
>>>>   
>>>>
>>>>        
>>>>
>>>>>Thanks,
>>>>>Dan
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf
Of
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>Keir Fraser
>>>>   
>>>>
>>>>        
>>>>
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>To: Dave Winchell
>>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong,
Eddie; Jiang,
>>>>>>Yunhong
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>>disables pending
>>>>>>missed ticks
>>>>>>
>>>>>>
>>>>>>Please take a look at xen-unstable changeset 16545.
>>>>>>
>>>>>>-- Keir
>>>>>>
>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>><dwinchell@virtualiron.com> wrote:
>>>>   
>>>>
>>>>        
>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Keir,
>>>>>>>
>>>>>>>The accuracy data I''ve collected for i/o
loads for the
>>>>>>>various time protocols follows. In addition, the
data
>>>>>>>for cpu loads is shown.
>>>>>>>
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
>>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
>>>>>>>
>>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>All three guests are 8vcpu.
>>>>>>>
>>>>>>>The loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>except that the redhat-64 guest has 4 instances of
dd.
>>>>>>>
>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec
-.006%, +.005% cpu
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec,
-.001%, +.012% cpu
>>>>>>>
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
-.004% cpu
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%,
-.005% cpu
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%,
-.003% cpu
>>>>>>>
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040%
cpu
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec,
-.034%, -.031% cpu
>>>>>>>
>>>>>>>
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
-.09% i/o-8
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
-.14% i/o-8
>>>>>>>
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
-.022% i/o-8
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%,
-.018% i/o-8
>>>>>>>
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
-.029% i/o-8
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec,
-.023%, -.031% i/o-8
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04%
i/o-32
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
-.005% i/o-32
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11%
i/o-32
>>>>>>>
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
.003% i/o-4/32
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
.01% i/o-4/32
>>>>>>>
>>>>>>>
>>>>>>>Overhead measurements:
>>>>>>>
>>>>>>>Progress in terms of number of passes through a
fixed
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>system workload
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>The workload was usex -b48.
>>>>>>>
>>>>>>>
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>
>>>>>>>
>>>>>>>Conclusions:
>>>>>>>
>>>>>>>The only protocol which meets the .05% accuracy
requirement for ntp
>>>>>>>tracking under the loads
>>>>>>>above is the SYNC protocol. The worst case
accuracies for
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>SYNC, MIXED,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>and ASYNC
>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>
>>>>>>>We could reduce the cost of the SYNC method by only
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>scheduling the extra
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>wakeups if a certain number
>>>>>>>of ticks are missed.
>>>>>>>
>>>>>>>Regards,
>>>>>>>Dave
>>>>>>>
>>>>>>>Keir Fraser wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Since I had a high error (~.03%) for the
ASYNC method a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>couple of days ago,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>I ran another ASYNC test. I think there may
have been something
>>>>>>>>>wrong with the code I used a couple of days
ago for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>ASYNC. It may have been
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>missing the immediate delivery of interrupt
after context
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>switch in.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>My results indicate that either SYNC or
ASYNC give
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>acceptable accuracy,
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>each running consistently around or under
.01%. MIXED has
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>a fairly high
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>error of
>>>>>>>>>greater than .03%. Probably too close to
.05% ntp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>threshold for comfort.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>I don''t have an overnight run with
SYNC. I plan to leave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>SYNC running
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>over the weekend. If you''d rather I
can leave MIXED
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>running instead.
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>It may be too early to pick the protocol and
I can run
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>more overnight tests
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>>next week.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>          
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>I''m a bit worried about any unwanted
side effects of the
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>SYNC+run_timer
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>approach -- e.g., whether timer wakeups will
cause higher
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>system-wide CPU
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>contention. I find it easier to think through
the
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>implications of ASYNC. I''m
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>surprised that MIXED loses time, and is less
accurate than
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>ASYNC. Perhaps it
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>delivers more timer interrupts than the other
approaches,
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>and each interrupt
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>event causes a small accumulated error?
>>>>>>>>
>>>>>>>>Overall I would consider MIXED and ASYNC as
favourites and
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>if the latter is
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>actually more accurate then I can simply revert
the changeset that
>>>>>>>>implemented MIXED.
>>>>>>>>
>>>>>>>>Perhaps rather than running more of the same
workloads you
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>could try idle
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large
disc reads
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>to /dev/null)? We
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>don''t have any data on workloads that
aren''t CPU bound, so
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>that''s really an
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>
>>>>>>>>-- Keir
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>_______________________________________________
>>>>>>Xen-devel mailing list
>>>>>>Xen-devel@lists.xensource.com
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>   
>>>>
>>>>        
>>>>
>>> 
>>>
>>>      
>>>
>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>> 
>>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
>>-        pt->do_not_freeze = !pt->pending_intr_nr;
>>+        pt->do_not_freeze = 1;
>>     else
>>         pt->pending_intr_nr += missed_ticks;
>>     pt->scheduled += missed_ticks * pt->period;
>>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>> 
>>     pt_lock(pt);
>> 
>>-    pt->pending_intr_nr++;
>>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
>>+        pt->pending_intr_nr = 1;
>>+ pt->do_not_freeze = 0;
>>+    }
>>+    else
>>+ pt->pending_intr_nr++;
>> 
>>     if ( !pt->one_shot )
>>     {
>>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>         return;
>>     }
>> 
>>-    pt->do_not_freeze = 0;
>>-
>>     if ( pt->one_shot )
>>     {
>>         pt->enabled = 0;
>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>             pt->pending_intr_nr = 0; /* ''collapse''
all missed ticks */
>>         }
>>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>+     pt->pending_intr_nr--;
>>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>>+ }
>>         else
>>         {
>>             pt->last_plt_gtime += pt->period_cycles;
>>    
>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks very much Dave and Keir!

Keir, if its not too late already, can this patch be added for 3.1.3?

Which reminds me, when will 3.1.3 go final?

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Keir,
> 
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
> 
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a 
> 2 hour test.
> I don''t think the difference is significant.
> 
> i/o loads produced errors of +.01%.
> 
> Thanks for all your efforts on this issue.
> 
> Regards,
> Dave
> 
> 
> 
> Keir Fraser wrote:
> 
> >Applied as c/s 16690, although the checked-in patch is 
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of 
> the patch I
> >totally omitted was the change to pt_process_missed_ticks(). 
> I don''t think
> >that change can be important, but let''s see what happens to
the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but, 
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be 
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with 
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the 
> worst error I''ve
> >>>seen so far
> >>>is 0.012%.  But I haven''t tried any exotic loads, just
LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %,
as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try
to
> >>>>do it later
> >>>>this week or the first week in January. My version of
constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would
expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I''ll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss
of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be
appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com;
Dong,
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I''ve collected for
i/o loads for the
> >>>>>>>various time protocols follows. In addition,
the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu
each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of
dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as
i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances
of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec
-.006%,
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec,
-.001%,
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec,
-.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
-.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
-.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
-.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec,
-.034%,
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec,
-.01%,
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec,
-.015%
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec,
-.017%,
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%,
-.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec,
-.020%,
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec,
-.023%,
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
-.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec,
-.011%,
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
-.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec
-.07%,
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec,
-.017%,
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a
fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case
accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by
only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for
the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there
may have
> been something
> >>>>>>>>>wrong with the code I used a couple of
days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of
interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or
ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or
under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close
to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don''t have an overnight run
with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you''d
rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the
protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I''m a bit worried about any
unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups
will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think
through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I''m
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is
less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the
other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as
favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply
revert the
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the
same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don''t have any data on workloads
that aren''t CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that''s really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort
imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
)
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /*
''collapse'' all
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yes, it''ll be added. 3.1.3 won''t be for a couple of weeks. I
want to get
3.2.0 squared away first.

 -- Keir

On 9/1/08 17:19, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> Thanks very much Dave and Keir!
> 
> Keir, if its not too late already, can this patch be added for 3.1.3?
> 
> Which reminds me, when will 3.1.3 go final?
> 
> Thanks,
> Dan
> 
>> -----Original Message-----
>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>> Sent: Wednesday, January 09, 2008 9:53 AM
>> To: Keir Fraser
>> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
>> Winchell
>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>> disables pending
>> missed ticks
>> 
>> 
>> Hi Keir,
>> 
>> The latest change, c/s 16690, looks fine.
>> I agree that the code in c/s 16690 is equivalent to
>> the code I submitted. Also, your version is more
>> concise.
>> 
>> The error tests confirm the equivalence. With overnight cpu loads,
>> the checked in version was accurate to +.048% for sles
>> and +.038% for red hat. My version was +.046% and +.032% in a
>> 2 hour test.
>> I don''t think the difference is significant.
>> 
>> i/o loads produced errors of +.01%.
>> 
>> Thanks for all your efforts on this issue.
>> 
>> Regards,
>> Dave
>> 
>> 
>> 
>> Keir Fraser wrote:
>> 
>>> Applied as c/s 16690, although the checked-in patch is
>> smaller. I think the
>>> only important fix is to pt_intr_post() and the only bit of
>> the patch I
>>> totally omitted was the change to pt_process_missed_ticks().
>> I don''t think
>>> that change can be important, but let''s see what happens
to the error
>>> percentage...
>>> 
>>> -- Keir
>>> 
>>> On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>>> 
>>> 
>>> 
>>>> Hi Dan and Keir,
>>>> 
>>>> Attached is a patch that fixes some issues with the SYNC policy
>>>> (no_missed_ticks_pending).
>>>> I have not tried to make the change the minimal one, but,
>> rather, just
>>>> ported into
>>>> the new code what I know to work well. The error for
>>>> no_missed_ticks_pending goes from
>>>> over 3% to .03% with this change according to my testing.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>> Dan Magenheimer wrote:
>>>> 
>>>> 
>>>> 
>>>>> Hi Dave --
>>>>> 
>>>>> Did you get your correction ported?  If so, it would be
>> nice to see this get
>>>>> into 3.1.3.
>>>>> 
>>>>> Note that I just did some very limited testing with
>> timer_mode=2(=SYNC=no
>>>>> missed ticks pending)
>>>>> on tip of xen-3.1-testing (64-bit Linux hv guest) and the
>> worst error I''ve
>>>>> seen so far
>>>>> is 0.012%.  But I haven''t tried any exotic loads,
just LTP.
>>>>> 
>>>>> Thanks,
>>>>> Dan
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>> Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>> To: dan.magenheimer@oracle.com
>>>>>> Cc: Keir Fraser; Shan, Haitao;
>> xen-devel@lists.xensource.com; Dong,
>>>>>> Eddie; Jiang, Yunhong; Dave Winchell
>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>>> disables pending
>>>>>> missed ticks
>>>>>> 
>>>>>> 
>>>>>> Dan,
>>>>>> 
>>>>>> I did some testing with the constant tsc offset SYNC
method
>>>>>> (now called
>>>>>> no_missed_ticks_pending)
>>>>>> and found the error to be very high, much larger than 1
%, as
>>>>>> I recall.
>>>>>> I have not had a chance to submit a correction. I will
try to
>>>>>> do it later
>>>>>> this week or the first week in January. My version of
constant tsc
>>>>>> offset SYNC method
>>>>>> produces .02 % error, so I just need to port that into
the
>>>>>> current code.
>>>>>> 
>>>>>> The error you got for both of those kernels is what I
would expect
>>>>>> for the default mode, delay_for_missed_ticks.
>>>>>> 
>>>>>> I''ll let Keir answer on how to set the time
mode.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>> Dan Magenheimer wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Anyone make measurements on the final patch?
>>>>>>> 
>>>>>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a
loss of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> about 0.2% with no load.  This was xen-unstable tip
today
>>>>>> with no options specified.  32-bit was about 0.01%.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> I think I missed something... how do I run the
various
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> accounting choices and which ones are known to be
appropriate
>>>>>> for which kernels?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: xen-devel-bounces@lists.xensource.com
>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> Keir Fraser
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>>> Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>> To: Dave Winchell
>>>>>>>> Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
>> Eddie; Jiang,
>>>>>>>> Yunhong
>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that
>>>>>>>> disables pending
>>>>>>>> missed ticks
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Please take a look at xen-unstable changeset
16545.
>>>>>>>> 
>>>>>>>> -- Keir
>>>>>>>> 
>>>>>>>> On 26/11/07 20:57, "Dave Winchell"
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Keir,
>>>>>>>>> 
>>>>>>>>> The accuracy data I''ve collected
for i/o loads for the
>>>>>>>>> various time protocols follows. In
addition, the data
>>>>>>>>> for cpu loads is shown.
>>>>>>>>> 
>>>>>>>>> The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
>>>>>>>>> Two guests, red hat and sles 64 bit, 8 vcpu
each.
>>>>>>>>> The cpu load is usex -e36 on each guest.
>>>>>>>>> (usex is available at
http://people.redhat.com/anderson/usex.)
>>>>>>>>> i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
>>>>>>>>> 
>>>>>>>>> The loads labeled i/o-32 are 32 instances
of dd.
>>>>>>>>> Also, these are run on 4 cpu AMD box.
>>>>>>>>> In addition, there is an idle rh-32bit
guest.
>>>>>>>>> All three guests are 8vcpu.
>>>>>>>>> 
>>>>>>>>> The loads labeled i/o-4/32 are the same as
i/o-32
>>>>>>>>> except that the redhat-64 guest has 4
instances of dd.
>>>>>>>>> 
>>>>>>>>> Date Duration Protocol sles, rhat error
load
>>>>>>>>> 
>>>>>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42
sec -.006%,
>> +.005% cpu
>>>>>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
sec, -.001%,
>> +.012% cpu
>>>>>>>>> 
>>>>>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec,
-.009%, -.004% cpu
>>>>>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
-.005%, -.005% cpu
>>>>>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
-.008%, -.003% cpu
>>>>>>>>> 
>>>>>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec
-.045%, -.040% cpu
>>>>>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4
sec, -.034%,
>> -.031% cpu
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7
sec, -.01%,
>> -.09% i/o-8
>>>>>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0
sec, -.015%
>> -.14% i/o-8
>>>>>>>>> 
>>>>>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
sec, -.017%,
>> -.022% i/o-8
>>>>>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec,
-.017%, -.018% i/o-8
>>>>>>>>> 
>>>>>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
sec, -.020%,
>> -.029% i/o-8
>>>>>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
sec, -.023%,
>> -.031% i/o-8
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec,
-.12%, -.04% i/o-32
>>>>>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec,
-.011%,
>> -.005% i/o-32
>>>>>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec
-.10%, -.11% i/o-32
>>>>>>>>> 
>>>>>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13.
sec -.07%,
>> .003% i/o-4/32
>>>>>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
sec, -.017%,
>> .01% i/o-4/32
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Overhead measurements:
>>>>>>>>> 
>>>>>>>>> Progress in terms of number of passes
through a fixed
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> system workload
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> on an 8 vcpu red hat with an 8 vcpu sles
idle.
>>>>>>>>> The workload was usex -b48.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>> SYNC 167 min 144 passes .862 passes/min
>>>>>>>>> SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>> MIXED 221 min 196 passes .887 passes/min
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Conclusions:
>>>>>>>>> 
>>>>>>>>> The only protocol which meets the .05%
accuracy
>> requirement for ntp
>>>>>>>>> tracking under the loads
>>>>>>>>> above is the SYNC protocol. The worst case
accuracies for
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> SYNC, MIXED,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> and ASYNC
>>>>>>>>> are .022%, .12%, and .14%, respectively.
>>>>>>>>> 
>>>>>>>>> We could reduce the cost of the SYNC method
by only
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> scheduling the extra
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> wakeups if a certain number
>>>>>>>>> of ticks are missed.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>> 
>>>>>>>>> Keir Fraser wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 9/11/07 19:22, "Dave
Winchell"
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Since I had a high error (~.03%)
for the ASYNC method a
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> couple of days ago,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> I ran another ASYNC test. I think
there may have
>> been something
>>>>>>>>>>> wrong with the code I used a couple
of days ago for
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> ASYNC. It may have been
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> missing the immediate delivery of
interrupt after context
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> switch in.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> My results indicate that either
SYNC or ASYNC give
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> acceptable accuracy,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> each running consistently around or
under .01%. MIXED has
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> a fairly high
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> error of
>>>>>>>>>>> greater than .03%. Probably too
close to .05% ntp
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> threshold for comfort.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> I don''t have an overnight
run with SYNC. I plan to leave
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> SYNC running
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> over the weekend. If you''d
rather I can leave MIXED
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> running instead.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> It may be too early to pick the
protocol and I can run
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> more overnight tests
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> next week.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> I''m a bit worried about any
unwanted side effects of the
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> SYNC+run_timer
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> approach -- e.g., whether timer wakeups
will cause higher
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> system-wide CPU
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> contention. I find it easier to think
through the
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> implications of ASYNC. I''m
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> surprised that MIXED loses time, and is
less accurate than
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> ASYNC. Perhaps it
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> delivers more timer interrupts than the
other approaches,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> and each interrupt
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> event causes a small accumulated error?
>>>>>>>>>> 
>>>>>>>>>> Overall I would consider MIXED and
ASYNC as favourites and
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> if the latter is
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> actually more accurate then I can
simply revert the
>> changeset that
>>>>>>>>>> implemented MIXED.
>>>>>>>>>> 
>>>>>>>>>> Perhaps rather than running more of the
same workloads you
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> could try idle
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> VCPUs and I/O bound VCPUs (e.g.,
repeated large disc reads
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> to /dev/null)? We
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> don''t have any data on
workloads that aren''t CPU bound, so
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> that''s really an
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> obvious place to put any further effort
imo.
>>>>>>>>>> 
>>>>>>>>>> -- Keir
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> Xen-devel mailing list
>>>>>>>> Xen-devel@lists.xensource.com
>>>>>>>> http://lists.xensource.com/xen-devel
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>> --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
>>>> +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
>>>> @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>>>> 
>>>>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
>>>>     if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) )
>>>> -        pt->do_not_freeze = !pt->pending_intr_nr;
>>>> +        pt->do_not_freeze = 1;
>>>>     else
>>>>         pt->pending_intr_nr += missed_ticks;
>>>>     pt->scheduled += missed_ticks * pt->period;
>>>> @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>>>> 
>>>>     pt_lock(pt);
>>>> 
>>>> -    pt->pending_intr_nr++;
>>>> +    if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) ) {
>>>> +        pt->pending_intr_nr = 1;
>>>> + pt->do_not_freeze = 0;
>>>> +    }
>>>> +    else
>>>> + pt->pending_intr_nr++;
>>>> 
>>>>     if ( !pt->one_shot )
>>>>     {
>>>> @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
>>>>         return;
>>>>     }
>>>> 
>>>> -    pt->do_not_freeze = 0;
>>>> -
>>>>     if ( pt->one_shot )
>>>>     {
>>>>         pt->enabled = 0;
>>>> @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
>>>>             pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>             pt->pending_intr_nr = 0; /*
''collapse'' all
>> missed ticks */
>>>>         }
>>>> + else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
>>>> +     pt->pending_intr_nr--;
>>>> +     pt->last_plt_gtime = hvm_get_guest_time(v);
>>>> + }
>>>>         else
>>>>         {
>>>>             pt->last_plt_gtime += pt->period_cycles;
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sorry for the very late followup on this but we finally were able
to get our testing set up again on stable 3.1 bits and have
seen some very bad results on 3.1.3-rc1, on the order of 1%.

Test enviroment was a 4-socket dual core machine with 24GB of
memory running six two-vcpu 2GB domains, four hvm plus two pv.
All six guests were running LTP simultaneously.  The four hvm
guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
All four hvm guests experienced skew around -1%, even the 32-bit
guest.  Less intensive testing didn''t exhibit much skew at all.

A representative graph is attached.

Dave, I wonder if some portion of your patches didn''t end up in
the xen trees?

(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
14.318MHz HPET.)

Thanks,
Dan

P.S. Many thanks to Deepak and Akira for running tests.
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of 
> Dave Winchell
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Keir,
> 
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
> 
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a 
> 2 hour test.
> I don''t think the difference is significant.
> 
> i/o loads produced errors of +.01%.
> 
> Thanks for all your efforts on this issue.
> 
> Regards,
> Dave
> 
> 
> 
> Keir Fraser wrote:
> 
> >Applied as c/s 16690, although the checked-in patch is 
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of 
> the patch I
> >totally omitted was the change to pt_process_missed_ticks(). 
> I don''t think
> >that change can be important, but let''s see what happens to
the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but, 
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be 
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with 
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the 
> worst error I''ve
> >>>seen so far
> >>>is 0.012%.  But I haven''t tried any exotic loads, just
LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %,
as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try
to
> >>>>do it later
> >>>>this week or the first week in January. My version of
constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would
expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I''ll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss
of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be
appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com;
Dong,
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I''ve collected for
i/o loads for the
> >>>>>>>various time protocols follows. In addition,
the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu
each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of
dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as
i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances
of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec
-.006%,
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec,
-.001%,
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec,
-.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
-.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
-.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
-.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec,
-.034%,
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec,
-.01%,
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec,
-.015%
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec,
-.017%,
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%,
-.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec,
-.020%,
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec,
-.023%,
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
-.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec,
-.011%,
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
-.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec
-.07%,
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec,
-.017%,
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a
fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case
accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by
only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for
the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there
may have
> been something
> >>>>>>>>>wrong with the code I used a couple of
days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of
interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or
ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or
under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close
to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don''t have an overnight run
with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you''d
rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the
protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I''m a bit worried about any
unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups
will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think
through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I''m
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is
less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the
other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as
favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply
revert the
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the
same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don''t have any data on workloads
that aren''t CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that''s really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort
imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
)
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /*
''collapse'' all
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,
 
Hpet timer does have a fairly large error, as I was trying this one recently.
I don''t remember what I got for error, but 1% sounds about right.
The problem is that hpet is not built on top of vpt.c, the module Keir and I did
all the recent work in, for its periodic timer needs. Try specifying clock=pit
on the linux boot line. If it still picks the hpet, which it might, let me know
and I''ll tell you how to get around this.
 
Regards,
Dave
 
 
 

________________________________

From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
Sent: Fri 1/25/2008 6:50 PM
To: Dave Winchell; Keir Fraser
Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com;
akira.ijuin@oracle.com
Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed
ticks



Sorry for the very late followup on this but we finally were able
to get our testing set up again on stable 3.1 bits and have
seen some very bad results on 3.1.3-rc1, on the order of 1%.

Test enviroment was a 4-socket dual core machine with 24GB of
memory running six two-vcpu 2GB domains, four hvm plus two pv.
All six guests were running LTP simultaneously.  The four hvm
guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
All four hvm guests experienced skew around -1%, even the 32-bit
guest.  Less intensive testing didn''t exhibit much skew at all.

A representative graph is attached.

Dave, I wonder if some portion of your patches didn''t end up in
the xen trees?

(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
14.318MHz HPET.)

Thanks,
Dan

P.S. Many thanks to Deepak and Akira for running tests.
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> Dave Winchell
> Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> disables pending
> missed ticks
>
>
> Hi Keir,
>
> The latest change, c/s 16690, looks fine.
> I agree that the code in c/s 16690 is equivalent to
> the code I submitted. Also, your version is more
> concise.
>
> The error tests confirm the equivalence. With overnight cpu loads,
> the checked in version was accurate to +.048% for sles
> and +.038% for red hat. My version was +.046% and +.032% in a
> 2 hour test.
> I don''t think the difference is significant.
>
> i/o loads produced errors of +.01%.
>
> Thanks for all your efforts on this issue.
>
> Regards,
> Dave
>
>
>
> Keir Fraser wrote:
>
> >Applied as c/s 16690, although the checked-in patch is
> smaller. I think the
> >only important fix is to pt_intr_post() and the only bit of
> the patch I
> >totally omitted was the change to pt_process_missed_ticks().
> I don''t think
> >that change can be important, but let''s see what happens to
the error
> >percentage...
> >
> > -- Keir
> >
> >On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
> >
> >
> >
> >>Hi Dan and Keir,
> >>
> >>Attached is a patch that fixes some issues with the SYNC policy
> >>(no_missed_ticks_pending).
> >>I have not tried to make the change the minimal one, but,
> rather, just
> >>ported into
> >>the new code what I know to work well. The error for
> >>no_missed_ticks_pending goes from
> >>over 3% to .03% with this change according to my testing.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dan Magenheimer wrote:
> >>
> >>
> >>
> >>>Hi Dave --
> >>>
> >>>Did you get your correction ported?  If so, it would be
> nice to see this get
> >>>into 3.1.3.
> >>>
> >>>Note that I just did some very limited testing with
> timer_mode=2(=SYNC=no
> >>>missed ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
> worst error I''ve
> >>>seen so far
> >>>is 0.012%.  But I haven''t tried any exotic loads, just
LTP.
> >>>
> >>>Thanks,
> >>>Dan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>To: dan.magenheimer@oracle.com
> >>>>Cc: Keir Fraser; Shan, Haitao;
> xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>disables pending
> >>>>missed ticks
> >>>>
> >>>>
> >>>>Dan,
> >>>>
> >>>>I did some testing with the constant tsc offset SYNC method
> >>>>(now called
> >>>>no_missed_ticks_pending)
> >>>>and found the error to be very high, much larger than 1 %,
as
> >>>>I recall.
> >>>>I have not had a chance to submit a correction. I will try
to
> >>>>do it later
> >>>>this week or the first week in January. My version of
constant tsc
> >>>>offset SYNC method
> >>>>produces .02 % error, so I just need to port that into the
> >>>>current code.
> >>>>
> >>>>The error you got for both of those kernels is what I would
expect
> >>>>for the default mode, delay_for_missed_ticks.
> >>>>
> >>>>I''ll let Keir answer on how to set the time mode.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Anyone make measurements on the final patch?
> >>>>>
> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss
of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>about 0.2% with no load.  This was xen-unstable tip today
> >>>>with no options specified.  32-bit was about 0.01%.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>I think I missed something... how do I run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>accounting choices and which ones are known to be
appropriate
> >>>>for which kernels?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Thanks,
> >>>>>Dan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>Keir Fraser
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>To: Dave Winchell
> >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com;
Dong,
> Eddie; Jiang,
> >>>>>>Yunhong
> >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
> >>>>>>disables pending
> >>>>>>missed ticks
> >>>>>>
> >>>>>>
> >>>>>>Please take a look at xen-unstable changeset 16545.
> >>>>>>
> >>>>>>-- Keir
> >>>>>>
> >>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>><dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Keir,
> >>>>>>>
> >>>>>>>The accuracy data I''ve collected for
i/o loads for the
> >>>>>>>various time protocols follows. In addition,
the data
> >>>>>>>for cpu loads is shown.
> >>>>>>>
> >>>>>>>The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
> >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu
each.
> >>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
> >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
> >>>>>>>
> >>>>>>>The loads labeled i/o-32 are 32 instances of
dd.
> >>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>All three guests are 8vcpu.
> >>>>>>>
> >>>>>>>The loads labeled i/o-4/32 are the same as
i/o-32
> >>>>>>>except that the redhat-64 guest has 4 instances
of dd.
> >>>>>>>
> >>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>
> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec
-.006%,
> +.005% cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec,
-.001%,
> +.012% cpu
> >>>>>>>
> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec,
-.009%, -.004% cpu
> >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
-.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
-.008%, -.003% cpu
> >>>>>>>
> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
-.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec,
-.034%,
> -.031% cpu
> >>>>>>>
> >>>>>>>
> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec,
-.01%,
> -.09% i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec,
-.015%
> -.14% i/o-8
> >>>>>>>
> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec,
-.017%,
> -.022% i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%,
-.018% i/o-8
> >>>>>>>
> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec,
-.020%,
> -.029% i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec,
-.023%,
> -.031% i/o-8
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
-.04% i/o-32
> >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec,
-.011%,
> -.005% i/o-32
> >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
-.11% i/o-32
> >>>>>>>
> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec
-.07%,
> .003% i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec,
-.017%,
> .01% i/o-4/32
> >>>>>>>
> >>>>>>>
> >>>>>>>Overhead measurements:
> >>>>>>>
> >>>>>>>Progress in terms of number of passes through a
fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>system workload
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>The workload was usex -b48.
> >>>>>>>
> >>>>>>>
> >>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>
> >>>>>>>
> >>>>>>>Conclusions:
> >>>>>>>
> >>>>>>>The only protocol which meets the .05% accuracy
> requirement for ntp
> >>>>>>>tracking under the loads
> >>>>>>>above is the SYNC protocol. The worst case
accuracies for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>SYNC, MIXED,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>and ASYNC
> >>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>
> >>>>>>>We could reduce the cost of the SYNC method by
only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>scheduling the extra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>wakeups if a certain number
> >>>>>>>of ticks are missed.
> >>>>>>>
> >>>>>>>Regards,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>Keir Fraser wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Since I had a high error (~.03%) for
the ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>couple of days ago,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I ran another ASYNC test. I think there
may have
> been something
> >>>>>>>>>wrong with the code I used a couple of
days ago for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>ASYNC. It may have been
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>missing the immediate delivery of
interrupt after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>switch in.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>My results indicate that either SYNC or
ASYNC give
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>acceptable accuracy,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>each running consistently around or
under .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>a fairly high
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>error of
> >>>>>>>>>greater than .03%. Probably too close
to .05% ntp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>threshold for comfort.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>I don''t have an overnight run
with SYNC. I plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>SYNC running
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>over the weekend. If you''d
rather I can leave MIXED
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>running instead.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>It may be too early to pick the
protocol and I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>more overnight tests
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>>next week.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I''m a bit worried about any
unwanted side effects of the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>SYNC+run_timer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>approach -- e.g., whether timer wakeups
will cause higher
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>system-wide CPU
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>contention. I find it easier to think
through the
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>implications of ASYNC. I''m
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>surprised that MIXED loses time, and is
less accurate than
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>ASYNC. Perhaps it
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>delivers more timer interrupts than the
other approaches,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>and each interrupt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>event causes a small accumulated error?
> >>>>>>>>
> >>>>>>>>Overall I would consider MIXED and ASYNC as
favourites and
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>if the latter is
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>actually more accurate then I can simply
revert the
> changeset that
> >>>>>>>>implemented MIXED.
> >>>>>>>>
> >>>>>>>>Perhaps rather than running more of the
same workloads you
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>could try idle
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
large disc reads
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>to /dev/null)? We
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>don''t have any data on workloads
that aren''t CPU bound, so
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>that''s really an
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>obvious place to put any further effort
imo.
> >>>>>>>>
> >>>>>>>>-- Keir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>_______________________________________________
> >>>>>>Xen-devel mailing list
> >>>>>>Xen-devel@lists.xensource.com
> >>>>>>http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
> >>
> >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
> >>     if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
)
> >>-        pt->do_not_freeze = !pt->pending_intr_nr;
> >>+        pt->do_not_freeze = 1;
> >>     else
> >>         pt->pending_intr_nr += missed_ticks;
> >>     pt->scheduled += missed_ticks * pt->period;
> >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
> >>
> >>     pt_lock(pt);
> >>
> >>-    pt->pending_intr_nr++;
> >>+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending)
) {
> >>+        pt->pending_intr_nr = 1;
> >>+ pt->do_not_freeze = 0;
> >>+    }
> >>+    else
> >>+ pt->pending_intr_nr++;
> >>
> >>     if ( !pt->one_shot )
> >>     {
> >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
> >>         return;
> >>     }
> >>
> >>-    pt->do_not_freeze = 0;
> >>-
> >>     if ( pt->one_shot )
> >>     {
> >>         pt->enabled = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
> >>             pt->last_plt_gtime = hvm_get_guest_time(v);
> >>             pt->pending_intr_nr = 0; /*
''collapse'' all
> missed ticks */
> >>         }
> >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> >>+     pt->pending_intr_nr--;
> >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >>+ }
> >>         else
> >>         {
> >>             pt->last_plt_gtime += pt->period_cycles;
> >>
> >>
> >
> >
> >
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed
ticksThanks, I hadn''t realized that!  No wonder we didn''t see
the same improvement you saw!
> Try specifying clock=pit on the linux boot line...
I''m confused... do you mean "clocksource=pit" on the Xen
command line or
"nohpet" / "clock=pit" / "clocksource=pit" on the
guest (or dom0?) command
line?  Or both places?  Since the tests take awhile, it would be nice
to get this right the first time.  Do the Xen and guest clocksources need
to be the same?

Thanks,
Dan
 
 -----Original Message-----
From: Dave Winchell [mailto:dwinchell@virtualiron.com]
Sent: Sunday, January 27, 2008 2:22 PM
To: dan.magenheimer@oracle.com; Keir Fraser
Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com;
akira.ijuin@oracle.com; Dave Winchell
Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed
ticks


  Hi Dan,

  Hpet timer does have a fairly large error, as I was trying this one recently.
  I don''t remember what I got for error, but 1% sounds about right.
  The problem is that hpet is not built on top of vpt.c, the module Keir and I
did
  all the recent work in, for its periodic timer needs. Try specifying clock=pit
  on the linux boot line. If it still picks the hpet, which it might, let me
know
  and I''ll tell you how to get around this.

  Regards,
  Dave





------------------------------------------------------------------------------
  From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
  Sent: Fri 1/25/2008 6:50 PM
  To: Dave Winchell; Keir Fraser
  Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com;
akira.ijuin@oracle.com
  Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed
ticks


  Sorry for the very late followup on this but we finally were able
  to get our testing set up again on stable 3.1 bits and have
  seen some very bad results on 3.1.3-rc1, on the order of 1%.

  Test enviroment was a 4-socket dual core machine with 24GB of
  memory running six two-vcpu 2GB domains, four hvm plus two pv.
  All six guests were running LTP simultaneously.  The four hvm
  guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
  Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
  All four hvm guests experienced skew around -1%, even the 32-bit
  guest.  Less intensive testing didn''t exhibit much skew at all.

  A representative graph is attached.

  Dave, I wonder if some portion of your patches didn''t end up in
  the xen trees?

  (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
  14.318MHz HPET.)

  Thanks,
  Dan

  P.S. Many thanks to Deepak and Akira for running tests.

  > -----Original Message-----
  > From: xen-devel-bounces@lists.xensource.com
  > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
  > Dave Winchell
  > Sent: Wednesday, January 09, 2008 9:53 AM
  > To: Keir Fraser
  > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave
  > Winchell
  > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
  > disables pending
  > missed ticks
  >
  >
  > Hi Keir,
  >
  > The latest change, c/s 16690, looks fine.
  > I agree that the code in c/s 16690 is equivalent to
  > the code I submitted. Also, your version is more
  > concise.
  >
  > The error tests confirm the equivalence. With overnight cpu loads,
  > the checked in version was accurate to +.048% for sles
  > and +.038% for red hat. My version was +.046% and +.032% in a
  > 2 hour test.
  > I don''t think the difference is significant.
  >
  > i/o loads produced errors of +.01%.
  >
  > Thanks for all your efforts on this issue.
  >
  > Regards,
  > Dave
  >
  >
  >
  > Keir Fraser wrote:
  >
  > >Applied as c/s 16690, although the checked-in patch is
  > smaller. I think the
  > >only important fix is to pt_intr_post() and the only bit of
  > the patch I
  > >totally omitted was the change to pt_process_missed_ticks().
  > I don''t think
  > >that change can be important, but let''s see what happens to
the error
  > >percentage...
  > >
  > > -- Keir
  > >
  > >On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
  > >
  > >
  > >
  > >>Hi Dan and Keir,
  > >>
  > >>Attached is a patch that fixes some issues with the SYNC policy
  > >>(no_missed_ticks_pending).
  > >>I have not tried to make the change the minimal one, but,
  > rather, just
  > >>ported into
  > >>the new code what I know to work well. The error for
  > >>no_missed_ticks_pending goes from
  > >>over 3% to .03% with this change according to my testing.
  > >>
  > >>Regards,
  > >>Dave
  > >>
  > >>Dan Magenheimer wrote:
  > >>
  > >>
  > >>
  > >>>Hi Dave --
  > >>>
  > >>>Did you get your correction ported?  If so, it would be
  > nice to see this get
  > >>>into 3.1.3.
  > >>>
  > >>>Note that I just did some very limited testing with
  > timer_mode=2(=SYNC=no
  > >>>missed ticks pending)
  > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the
  > worst error I''ve
  > >>>seen so far
  > >>>is 0.012%.  But I haven''t tried any exotic loads,
just LTP.
  > >>>
  > >>>Thanks,
  > >>>Dan
  > >>>
  > >>>
  > >>>
  > >>>
  > >>>
  > >>>>-----Original Message-----
  > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
  > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
  > >>>>To: dan.magenheimer@oracle.com
  > >>>>Cc: Keir Fraser; Shan, Haitao;
  > xen-devel@lists.xensource.com; Dong,
  > >>>>Eddie; Jiang, Yunhong; Dave Winchell
  > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
  > >>>>disables pending
  > >>>>missed ticks
  > >>>>
  > >>>>
  > >>>>Dan,
  > >>>>
  > >>>>I did some testing with the constant tsc offset SYNC
method
  > >>>>(now called
  > >>>>no_missed_ticks_pending)
  > >>>>and found the error to be very high, much larger than 1
%, as
  > >>>>I recall.
  > >>>>I have not had a chance to submit a correction. I will
try to
  > >>>>do it later
  > >>>>this week or the first week in January. My version of
constant tsc
  > >>>>offset SYNC method
  > >>>>produces .02 % error, so I just need to port that into
the
  > >>>>current code.
  > >>>>
  > >>>>The error you got for both of those kernels is what I
would expect
  > >>>>for the default mode, delay_for_missed_ticks.
  > >>>>
  > >>>>I''ll let Keir answer on how to set the time
mode.
  > >>>>
  > >>>>Regards,
  > >>>>Dave
  > >>>>
  > >>>>Dan Magenheimer wrote:
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>Anyone make measurements on the final patch?
  > >>>>>
  > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss
of
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>about 0.2% with no load.  This was xen-unstable tip today
  > >>>>with no options specified.  32-bit was about 0.01%.
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>I think I missed something... how do I run the
various
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>accounting choices and which ones are known to be
appropriate
  > >>>>for which kernels?
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>Thanks,
  > >>>>>Dan
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>>-----Original Message-----
  > >>>>>>From: xen-devel-bounces@lists.xensource.com
  > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>Keir Fraser
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
  > >>>>>>To: Dave Winchell
  > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com;
Dong,
  > Eddie; Jiang,
  > >>>>>>Yunhong
  > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
  > >>>>>>disables pending
  > >>>>>>missed ticks
  > >>>>>>
  > >>>>>>
  > >>>>>>Please take a look at xen-unstable changeset
16545.
  > >>>>>>
  > >>>>>>-- Keir
  > >>>>>>
  > >>>>>>On 26/11/07 20:57, "Dave Winchell"
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>><dwinchell@virtualiron.com> wrote:
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>Keir,
  > >>>>>>>
  > >>>>>>>The accuracy data I''ve collected for
i/o loads for the
  > >>>>>>>various time protocols follows. In addition,
the data
  > >>>>>>>for cpu loads is shown.
  > >>>>>>>
  > >>>>>>>The loads labeled cpu and i/o-8 are on an 8
processor AMD box.
  > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu
each.
  > >>>>>>>The cpu load is usex -e36 on each guest.
  > >>>>>>>(usex is available at
http://people.redhat.com/anderson/usex.)
  > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6
of=/dev/null.
  > >>>>>>>
  > >>>>>>>The loads labeled i/o-32 are 32 instances of
dd.
  > >>>>>>>Also, these are run on 4 cpu AMD box.
  > >>>>>>>In addition, there is an idle rh-32bit guest.
  > >>>>>>>All three guests are 8vcpu.
  > >>>>>>>
  > >>>>>>>The loads labeled i/o-4/32 are the same as
i/o-32
  > >>>>>>>except that the redhat-64 guest has 4
instances of dd.
  > >>>>>>>
  > >>>>>>>Date Duration Protocol sles, rhat error load
  > >>>>>>>
  > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42
sec -.006%,
  > +.005% cpu
  > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec,
-.001%,
  > +.012% cpu
  > >>>>>>>
  > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec,
-.009%, -.004% cpu
  > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
-.005%, -.005% cpu
  > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
-.008%, -.003% cpu
  > >>>>>>>
  > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
-.040% cpu
  > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec,
-.034%,
  > -.031% cpu
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec,
-.01%,
  > -.09% i/o-8
  > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec,
-.015%
  > -.14% i/o-8
  > >>>>>>>
  > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec,
-.017%,
  > -.022% i/o-8
  > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
-.017%, -.018% i/o-8
  > >>>>>>>
  > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec,
-.020%,
  > -.029% i/o-8
  > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec,
-.023%,
  > -.031% i/o-8
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec,
-.12%, -.04% i/o-32
  > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec,
-.011%,
  > -.005% i/o-32
  > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec
-.10%, -.11% i/o-32
  > >>>>>>>
  > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec
-.07%,
  > .003% i/o-4/32
  > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec,
-.017%,
  > .01% i/o-4/32
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>Overhead measurements:
  > >>>>>>>
  > >>>>>>>Progress in terms of number of passes through
a fixed
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>system workload
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles
idle.
  > >>>>>>>The workload was usex -b48.
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>ASYNC 167 min 145 passes .868 passes/min
  > >>>>>>>SYNC 167 min 144 passes .862 passes/min
  > >>>>>>>SYNC 1065 min 919 passes .863 passes/min
  > >>>>>>>MIXED 221 min 196 passes .887 passes/min
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>Conclusions:
  > >>>>>>>
  > >>>>>>>The only protocol which meets the .05%
accuracy
  > requirement for ntp
  > >>>>>>>tracking under the loads
  > >>>>>>>above is the SYNC protocol. The worst case
accuracies for
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>SYNC, MIXED,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>and ASYNC
  > >>>>>>>are .022%, .12%, and .14%, respectively.
  > >>>>>>>
  > >>>>>>>We could reduce the cost of the SYNC method
by only
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>scheduling the extra
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>wakeups if a certain number
  > >>>>>>>of ticks are missed.
  > >>>>>>>
  > >>>>>>>Regards,
  > >>>>>>>Dave
  > >>>>>>>
  > >>>>>>>Keir Fraser wrote:
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>
  > >>>>>>>>On 9/11/07 19:22, "Dave
Winchell"
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>><dwinchell@virtualiron.com> wrote:
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>>Since I had a high error (~.03%) for
the ASYNC method a
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>couple of days ago,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>I ran another ASYNC test. I think
there may have
  > been something
  > >>>>>>>>>wrong with the code I used a couple
of days ago for
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>ASYNC. It may have been
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>missing the immediate delivery of
interrupt after context
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>switch in.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>My results indicate that either SYNC
or ASYNC give
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>acceptable accuracy,
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>each running consistently around or
under .01%. MIXED has
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>a fairly high
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>error of
  > >>>>>>>>>greater than .03%. Probably too close
to .05% ntp
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>threshold for comfort.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>I don''t have an overnight
run with SYNC. I plan to leave
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>SYNC running
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>over the weekend. If you''d
rather I can leave MIXED
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>running instead.
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>It may be too early to pick the
protocol and I can run
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>more overnight tests
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>>next week.
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>>
  > >>>>>>>>I''m a bit worried about any
unwanted side effects of the
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>SYNC+run_timer
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>approach -- e.g., whether timer wakeups
will cause higher
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>system-wide CPU
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>contention. I find it easier to think
through the
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>implications of ASYNC. I''m
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>surprised that MIXED loses time, and is
less accurate than
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>ASYNC. Perhaps it
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>delivers more timer interrupts than the
other approaches,
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>and each interrupt
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>event causes a small accumulated error?
  > >>>>>>>>
  > >>>>>>>>Overall I would consider MIXED and ASYNC
as favourites and
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>if the latter is
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>actually more accurate then I can simply
revert the
  > changeset that
  > >>>>>>>>implemented MIXED.
  > >>>>>>>>
  > >>>>>>>>Perhaps rather than running more of the
same workloads you
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>could try idle
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
large disc reads
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>to /dev/null)? We
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>don''t have any data on workloads
that aren''t CPU bound, so
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>that''s really an
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>>>obvious place to put any further effort
imo.
  > >>>>>>>>
  > >>>>>>>>-- Keir
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>>>
  > >>>>>>_______________________________________________
  > >>>>>>Xen-devel mailing list
  > >>>>>>Xen-devel@lists.xensource.com
  > >>>>>>http://lists.xensource.com/xen-devel
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>>
  > >>>
  > >>>
  > >>>
  > >>>
  > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
  > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
  > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
  > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
  > >>
  > >>     missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
  > >>     if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) )
  > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
  > >>+        pt->do_not_freeze = 1;
  > >>     else
  > >>         pt->pending_intr_nr += missed_ticks;
  > >>     pt->scheduled += missed_ticks * pt->period;
  > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
  > >>
  > >>     pt_lock(pt);
  > >>
  > >>-    pt->pending_intr_nr++;
  > >>+    if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) ) {
  > >>+        pt->pending_intr_nr = 1;
  > >>+ pt->do_not_freeze = 0;
  > >>+    }
  > >>+    else
  > >>+ pt->pending_intr_nr++;
  > >>
  > >>     if ( !pt->one_shot )
  > >>     {
  > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
  > >>         return;
  > >>     }
  > >>
  > >>-    pt->do_not_freeze = 0;
  > >>-
  > >>     if ( pt->one_shot )
  > >>     {
  > >>         pt->enabled = 0;
  > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
  > >>             pt->last_plt_gtime = hvm_get_guest_time(v);
  > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
  > missed ticks */
  > >>         }
  > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
  > >>+     pt->pending_intr_nr--;
  > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
  > >>+ }
  > >>         else
  > >>         {
  > >>             pt->last_plt_gtime += pt->period_cycles;
  > >>
  > >>
  > >
  > >
  > >
  > >
  >
  >
  > _______________________________________________
  > Xen-devel mailing list
  > Xen-devel@lists.xensource.com
  > http://lists.xensource.com/xen-devel
  >



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan,

I guess I''m a bit out of date calling for clock= usage.
Looking at linux 2.6.20.4 sources, I think you should specify
"clocksource=pit nohpet" on the linux guest bootline.

You can leave the xen and dom0 bootlines as they are.
The xen and guest clocksources do not need to be the same.
In my tests, xen is using the hpet for its timekeeping and
that appears to be the default.

When you boot the guests you should see
    time.c: Using PIT/TSC based timekeeping.
on the rh4u5-64 guest, and something similar on the others.

 > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
 > 14.318MHz HPET.)

This appears to be the xen state, which is fine.
I was wrongly assuming that this was the guest state.
You might want to look in your guest logs and see what they were picking
for a clock source.

Regards,
Dave




Dan Magenheimer wrote:
> Thanks, I hadn''t realized that!  No wonder we didn''t see
the same
> improvement you saw!
>  
>> Try specifying clock=pit on the linux boot line...
>  
> I''m confused... do you mean "clocksource=pit" on the Xen
command line or
> "nohpet" / "clock=pit" / "clocksource=pit" on
the guest (or dom0?) command
> line?  Or both places?  Since the tests take awhile, it would be nice
> to get this right the first time.  Do the Xen and guest clocksources need
> to be the same?
>  
> Thanks,
> Dan
>  
>  -----Original Message-----
> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> *Sent:* Sunday, January 27, 2008 2:22 PM
> *To:* dan.magenheimer@oracle.com; Keir Fraser
> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; 
> akira.ijuin@oracle.com; Dave Winchell
> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables 
> pending missed ticks
>
>     Hi Dan,
>      
>     Hpet timer does have a fairly large error, as I was trying this
>     one recently.
>     I don''t remember what I got for error, but 1% sounds about
right.
>     The problem is that hpet is not built on top of vpt.c, the module
>     Keir and I did
>     all the recent work in, for its periodic timer needs. Try
>     specifying clock=pit
>     on the linux boot line. If it still picks the hpet, which it
>     might, let me know
>     and I''ll tell you how to get around this.
>      
>     Regards,
>     Dave
>      
>      
>      
>
>    
------------------------------------------------------------------------
>     *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>     *Sent:* Fri 1/25/2008 6:50 PM
>     *To:* Dave Winchell; Keir Fraser
>     *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>     akira.ijuin@oracle.com
>     *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>     pending missed ticks
>
>     Sorry for the very late followup on this but we finally were able
>     to get our testing set up again on stable 3.1 bits and have
>     seen some very bad results on 3.1.3-rc1, on the order of 1%.
>
>     Test enviroment was a 4-socket dual core machine with 24GB of
>     memory running six two-vcpu 2GB domains, four hvm plus two pv.
>     All six guests were running LTP simultaneously.  The four hvm
>     guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>     Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests.
>     All four hvm guests experienced skew around -1%, even the 32-bit
>     guest.  Less intensive testing didn''t exhibit much skew at
all.
>
>     A representative graph is attached.
>
>     Dave, I wonder if some portion of your patches didn''t end up
in
>     the xen trees?
>
>     (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>     14.318MHz HPET.)
>
>     Thanks,
>     Dan
>
>     P.S. Many thanks to Deepak and Akira for running tests.
>
>     > -----Original Message-----
>     > From: xen-devel-bounces@lists.xensource.com
>     > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>     > Dave Winchell
>     > Sent: Wednesday, January 09, 2008 9:53 AM
>     > To: Keir Fraser
>     > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com;
Dave
>     > Winchell
>     > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>     > disables pending
>     > missed ticks
>     >
>     >
>     > Hi Keir,
>     >
>     > The latest change, c/s 16690, looks fine.
>     > I agree that the code in c/s 16690 is equivalent to
>     > the code I submitted. Also, your version is more
>     > concise.
>     >
>     > The error tests confirm the equivalence. With overnight cpu loads,
>     > the checked in version was accurate to +.048% for sles
>     > and +.038% for red hat. My version was +.046% and +.032% in a
>     > 2 hour test.
>     > I don''t think the difference is significant.
>     >
>     > i/o loads produced errors of +.01%.
>     >
>     > Thanks for all your efforts on this issue.
>     >
>     > Regards,
>     > Dave
>     >
>     >
>     >
>     > Keir Fraser wrote:
>     >
>     > >Applied as c/s 16690, although the checked-in patch is
>     > smaller. I think the
>     > >only important fix is to pt_intr_post() and the only bit of
>     > the patch I
>     > >totally omitted was the change to pt_process_missed_ticks().
>     > I don''t think
>     > >that change can be important, but let''s see what
happens to the
>     error
>     > >percentage...
>     > >
>     > > -- Keir
>     > >
>     > >On 4/1/08 23:24, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>     > >
>     > >
>     > >
>     > >>Hi Dan and Keir,
>     > >>
>     > >>Attached is a patch that fixes some issues with the SYNC
policy
>     > >>(no_missed_ticks_pending).
>     > >>I have not tried to make the change the minimal one, but,
>     > rather, just
>     > >>ported into
>     > >>the new code what I know to work well. The error for
>     > >>no_missed_ticks_pending goes from
>     > >>over 3% to .03% with this change according to my testing.
>     > >>
>     > >>Regards,
>     > >>Dave
>     > >>
>     > >>Dan Magenheimer wrote:
>     > >>
>     > >>
>     > >>
>     > >>>Hi Dave --
>     > >>>
>     > >>>Did you get your correction ported?  If so, it would
be
>     > nice to see this get
>     > >>>into 3.1.3.
>     > >>>
>     > >>>Note that I just did some very limited testing with
>     > timer_mode=2(=SYNC=no
>     > >>>missed ticks pending)
>     > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and
the
>     > worst error I''ve
>     > >>>seen so far
>     > >>>is 0.012%.  But I haven''t tried any exotic
loads, just LTP.
>     > >>>
>     > >>>Thanks,
>     > >>>Dan
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>>-----Original Message-----
>     > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>     > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>     > >>>>To: dan.magenheimer@oracle.com
>     > >>>>Cc: Keir Fraser; Shan, Haitao;
>     > xen-devel@lists.xensource.com; Dong,
>     > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>     > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
>     > >>>>disables pending
>     > >>>>missed ticks
>     > >>>>
>     > >>>>
>     > >>>>Dan,
>     > >>>>
>     > >>>>I did some testing with the constant tsc offset
SYNC method
>     > >>>>(now called
>     > >>>>no_missed_ticks_pending)
>     > >>>>and found the error to be very high, much larger
than 1 %, as
>     > >>>>I recall.
>     > >>>>I have not had a chance to submit a correction. I
will try to
>     > >>>>do it later
>     > >>>>this week or the first week in January. My version
of
>     constant tsc
>     > >>>>offset SYNC method
>     > >>>>produces .02 % error, so I just need to port that
into the
>     > >>>>current code.
>     > >>>>
>     > >>>>The error you got for both of those kernels is
what I would
>     expect
>     > >>>>for the default mode, delay_for_missed_ticks.
>     > >>>>
>     > >>>>I''ll let Keir answer on how to set the
time mode.
>     > >>>>
>     > >>>>Regards,
>     > >>>>Dave
>     > >>>>
>     > >>>>Dan Magenheimer wrote:
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>Anyone make measurements on the final patch?
>     > >>>>>
>     > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw
a loss of
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>about 0.2% with no load.  This was xen-unstable
tip today
>     > >>>>with no options specified.  32-bit was about
0.01%.
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>I think I missed something... how do I run the
various
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>accounting choices and which ones are known to be
appropriate
>     > >>>>for which kernels?
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>Thanks,
>     > >>>>>Dan
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>>-----Original Message-----
>     > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>     >
>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf
Of
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>Keir Fraser
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>     > >>>>>>To: Dave Winchell
>     > >>>>>>Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
>     > Eddie; Jiang,
>     > >>>>>>Yunhong
>     > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that
>     > >>>>>>disables pending
>     > >>>>>>missed ticks
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>Please take a look at xen-unstable
changeset 16545.
>     > >>>>>>
>     > >>>>>>-- Keir
>     > >>>>>>
>     > >>>>>>On 26/11/07 20:57, "Dave
Winchell"
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>><dwinchell@virtualiron.com> wrote:
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>Keir,
>     > >>>>>>>
>     > >>>>>>>The accuracy data I''ve
collected for i/o loads for the
>     > >>>>>>>various time protocols follows. In
addition, the data
>     > >>>>>>>for cpu loads is shown.
>     > >>>>>>>
>     > >>>>>>>The loads labeled cpu and i/o-8 are on
an 8 processor AMD
>     box.
>     > >>>>>>>Two guests, red hat and sles 64 bit, 8
vcpu each.
>     > >>>>>>>The cpu load is usex -e36 on each
guest.
>     > >>>>>>>(usex is available at
>     http://people.redhat.com/anderson/usex.)
>     > >>>>>>>i/o load is 8 instances of dd
if=/dev/hda6 of=/dev/null.
>     > >>>>>>>
>     > >>>>>>>The loads labeled i/o-32 are 32
instances of dd.
>     > >>>>>>>Also, these are run on 4 cpu AMD box.
>     > >>>>>>>In addition, there is an idle rh-32bit
guest.
>     > >>>>>>>All three guests are 8vcpu.
>     > >>>>>>>
>     > >>>>>>>The loads labeled i/o-4/32 are the
same as i/o-32
>     > >>>>>>>except that the redhat-64 guest has 4
instances of dd.
>     > >>>>>>>
>     > >>>>>>>Date Duration Protocol sles, rhat
error load
>     > >>>>>>>
>     > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
+4.42 sec -.006%,
>     > +.005% cpu
>     > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec,
+1.44 sec, -.001%,
>     > +.012% cpu
>     > >>>>>>>
>     > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
sec, -.009%,
>     -.004% cpu
>     > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26
sec, -.005%, -.005% cpu
>     > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8
sec, -.008%, -.003% cpu
>     > >>>>>>>
>     > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec
-.045%, -.040% cpu
>     > >>>>>>>11/08 15 hrs 39 min MIXED -19.
sec,-17.4 sec, -.034%,
>     > -.031% cpu
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
sec,-55.7 sec, -.01%,
>     > -.09% i/o-8
>     > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
sec,-14.0 sec, -.015%
>     > -.14% i/o-8
>     > >>>>>>>
>     > >>>>>>>11/13 15 hrs 38 min SYNC -9.7
sec,-12.3 sec, -.017%,
>     > -.022% i/o-8
>     > >>>>>>>11/14 48 min SYNC - .46 sec, - .48
sec, -.017%, -.018% i/o-8
>     > >>>>>>>
>     > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec,
-4.15 sec, -.020%,
>     > -.029% i/o-8
>     > >>>>>>>11/20 16 hrs 2 min MIXED -13.4
sec,-18.1 sec, -.023%,
>     > -.031% i/o-8
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67
sec, -.12%, -.04% i/o-32
>     > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
sec, -.011%,
>     > -.005% i/o-32
>     > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77
sec -.10%, -.11% i/o-32
>     > >>>>>>>
>     > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
13. sec -.07%,
>     > .003% i/o-4/32
>     > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec,
1.44 sec, -.017%,
>     > .01% i/o-4/32
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>Overhead measurements:
>     > >>>>>>>
>     > >>>>>>>Progress in terms of number of passes
through a fixed
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>system workload
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>on an 8 vcpu red hat with an 8 vcpu
sles idle.
>     > >>>>>>>The workload was usex -b48.
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>ASYNC 167 min 145 passes .868
passes/min
>     > >>>>>>>SYNC 167 min 144 passes .862
passes/min
>     > >>>>>>>SYNC 1065 min 919 passes .863
passes/min
>     > >>>>>>>MIXED 221 min 196 passes .887
passes/min
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>Conclusions:
>     > >>>>>>>
>     > >>>>>>>The only protocol which meets the .05%
accuracy
>     > requirement for ntp
>     > >>>>>>>tracking under the loads
>     > >>>>>>>above is the SYNC protocol. The worst
case accuracies for
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>SYNC, MIXED,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>and ASYNC
>     > >>>>>>>are .022%, .12%, and .14%,
respectively.
>     > >>>>>>>
>     > >>>>>>>We could reduce the cost of the SYNC
method by only
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>scheduling the extra
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>wakeups if a certain number
>     > >>>>>>>of ticks are missed.
>     > >>>>>>>
>     > >>>>>>>Regards,
>     > >>>>>>>Dave
>     > >>>>>>>
>     > >>>>>>>Keir Fraser wrote:
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>
>     > >>>>>>>>On 9/11/07 19:22, "Dave
Winchell"
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>><dwinchell@virtualiron.com> wrote:
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>>Since I had a high error
(~.03%) for the ASYNC method a
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>couple of days ago,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>I ran another ASYNC test. I
think there may have
>     > been something
>     > >>>>>>>>>wrong with the code I used a
couple of days ago for
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>ASYNC. It may have been
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>missing the immediate delivery
of interrupt after context
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>switch in.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>My results indicate that
either SYNC or ASYNC give
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>acceptable accuracy,
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>each running consistently
around or under .01%. MIXED has
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>a fairly high
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>error of
>     > >>>>>>>>>greater than .03%. Probably
too close to .05% ntp
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>threshold for comfort.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>I don''t have an
overnight run with SYNC. I plan to leave
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>SYNC running
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>over the weekend. If
you''d rather I can leave MIXED
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>running instead.
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>It may be too early to pick
the protocol and I can run
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>more overnight tests
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>>next week.
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>>
>     > >>>>>>>>I''m a bit worried about
any unwanted side effects of the
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>SYNC+run_timer
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>approach -- e.g., whether timer
wakeups will cause higher
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>system-wide CPU
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>contention. I find it easier to
think through the
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>implications of ASYNC. I''m
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>surprised that MIXED loses time,
and is less accurate than
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>ASYNC. Perhaps it
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>delivers more timer interrupts
than the other approaches,
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>and each interrupt
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>event causes a small accumulated
error?
>     > >>>>>>>>
>     > >>>>>>>>Overall I would consider MIXED and
ASYNC as favourites and
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>if the latter is
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>actually more accurate then I can
simply revert the
>     > changeset that
>     > >>>>>>>>implemented MIXED.
>     > >>>>>>>>
>     > >>>>>>>>Perhaps rather than running more
of the same workloads you
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>could try idle
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>VCPUs and I/O bound VCPUs (e.g.,
repeated large disc reads
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>to /dev/null)? We
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>don''t have any data on
workloads that aren''t CPU bound, so
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>that''s really an
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>>>obvious place to put any further
effort imo.
>     > >>>>>>>>
>     > >>>>>>>>-- Keir
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     > >>>>>>>>
>     >
>>>>>>_______________________________________________
>     > >>>>>>Xen-devel mailing list
>     > >>>>>>Xen-devel@lists.xensource.com
>     > >>>>>>http://lists.xensource.com/xen-devel
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>>
>     > >>>
>     > >>>
>     > >>>
>     > >>>
>     > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>     > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007
+0000
>     > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008
-0500
>     > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru
>     > >>
>     > >>     missed_ticks = missed_ticks / (s_time_t)
pt->period + 1;
>     > >>     if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) )
>     > >>-        pt->do_not_freeze = !pt->pending_intr_nr;
>     > >>+        pt->do_not_freeze = 1;
>     > >>     else
>     > >>         pt->pending_intr_nr += missed_ticks;
>     > >>     pt->scheduled += missed_ticks * pt->period;
>     > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)
>     > >>
>     > >>     pt_lock(pt);
>     > >>
>     > >>-    pt->pending_intr_nr++;
>     > >>+    if ( mode_is(pt->vcpu->domain,
no_missed_ticks_pending) ) {
>     > >>+        pt->pending_intr_nr = 1;
>     > >>+ pt->do_not_freeze = 0;
>     > >>+    }
>     > >>+    else
>     > >>+ pt->pending_intr_nr++;
>     > >>
>     > >>     if ( !pt->one_shot )
>     > >>     {
>     > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v,
struct
>     > >>         return;
>     > >>     }
>     > >>
>     > >>-    pt->do_not_freeze = 0;
>     > >>-
>     > >>     if ( pt->one_shot )
>     > >>     {
>     > >>         pt->enabled = 0;
>     > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v,
struct
>     > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
>     > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
>     > missed ticks */
>     > >>         }
>     > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending)
) {
>     > >>+     pt->pending_intr_nr--;
>     > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
>     > >>+ }
>     > >>         else
>     > >>         {
>     > >>             pt->last_plt_gtime +=
pt->period_cycles;
>     > >>
>     > >>
>     > >
>     > >
>     > >
>     > >
>     >
>     >
>     > _______________________________________________
>     > Xen-devel mailing list
>     > Xen-devel@lists.xensource.com
>     > http://lists.xensource.com/xen-devel
>     >
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave (Keir, see suggestion below) --

Thanks!

Turning off vhpet certainly helps a lot (though see below).

I wonder if timekeeping with vhpet is so bad that it should be
turned off by default (in 3.1, 3.2, and unstable) until it is
fixed?  (I have a patch that defaults it off, can post it if
there is agreement on the above point.)  The whole point of an
HPET is to provide more precise timekeeping and if vhpet is
worse than vpit, it can only confuse users.  Comments?

In your testing, are you just measuring % skew over a long
period of time?   We are graphing the skew continuously and
seeing periodic behavior that is unsettling, even with pit.
See attached.  Though your algorithm recovers, the "cliffs"
could still cause real user problems.  I wonder if there is
anything that can be done to make the "recovery" more
responsive?

We are looking into what part(s) of LTP is causing the cliffs.

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Monday, January 28, 2008 8:21 AM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
> deepak.patel@oracle.com;
> akira.ijuin@oracle.com; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> I guess I''m a bit out of date calling for clock= usage.
> Looking at linux 2.6.20.4 sources, I think you should specify
> "clocksource=pit nohpet" on the linux guest bootline.
> 
> You can leave the xen and dom0 bootlines as they are.
> The xen and guest clocksources do not need to be the same.
> In my tests, xen is using the hpet for its timekeeping and
> that appears to be the default.
> 
> When you boot the guests you should see
>     time.c: Using PIT/TSC based timekeeping.
> on the rh4u5-64 guest, and something similar on the others.
> 
>  > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>  > 14.318MHz HPET.)
> 
> This appears to be the xen state, which is fine.
> I was wrongly assuming that this was the guest state.
> You might want to look in your guest logs and see what they 
> were picking
> for a clock source.
> 
> Regards,
> Dave
> 
> 
> 
> 
> Dan Magenheimer wrote:
> 
> > Thanks, I hadn''t realized that!  No wonder we didn''t
see the same
> > improvement you saw!
> >
> >> Try specifying clock=pit on the linux boot line...
> >
> > I''m confused... do you mean "clocksource=pit" on
the Xen
> command line or
> > "nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
> dom0?) command
> > line?  Or both places?  Since the tests take awhile, it 
> would be nice
> > to get this right the first time.  Do the Xen and guest 
> clocksources need
> > to be the same?
> >
> > Thanks,
> > Dan
> >
> >  -----Original Message-----
> > *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
> > *Sent:* Sunday, January 27, 2008 2:22 PM
> > *To:* dan.magenheimer@oracle.com; Keir Fraser
> > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> > akira.ijuin@oracle.com; Dave Winchell
> > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
> > pending missed ticks
> >
> >     Hi Dan,
> >
> >     Hpet timer does have a fairly large error, as I was trying this
> >     one recently.
> >     I don''t remember what I got for error, but 1% sounds 
> about right.
> >     The problem is that hpet is not built on top of vpt.c, 
> the module
> >     Keir and I did
> >     all the recent work in, for its periodic timer needs. Try
> >     specifying clock=pit
> >     on the linux boot line. If it still picks the hpet, which it
> >     might, let me know
> >     and I''ll tell you how to get around this.
> >
> >     Regards,
> >     Dave
> >
> >
> >
> >
> >     
> --------------------------------------------------------------
> ----------
> >     *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> >     *Sent:* Fri 1/25/2008 6:50 PM
> >     *To:* Dave Winchell; Keir Fraser
> >     *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
> >     akira.ijuin@oracle.com
> >     *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >     pending missed ticks
> >
> >     Sorry for the very late followup on this but we finally 
> were able
> >     to get our testing set up again on stable 3.1 bits and have
> >     seen some very bad results on 3.1.3-rc1, on the order of 1%.
> >
> >     Test enviroment was a 4-socket dual core machine with 24GB of
> >     memory running six two-vcpu 2GB domains, four hvm plus two pv.
> >     All six guests were running LTP simultaneously.  The four hvm
> >     guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
> >     Timer_mode was set to 2 for 64-bit guests and 0 for 
> 32-bit guests.
> >     All four hvm guests experienced skew around -1%, even the 32-bit
> >     guest.  Less intensive testing didn''t exhibit much skew
at all.
> >
> >     A representative graph is attached.
> >
> >     Dave, I wonder if some portion of your patches didn''t end
up in
> >     the xen trees?
> >
> >     (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
> >     14.318MHz HPET.)
> >
> >     Thanks,
> >     Dan
> >
> >     P.S. Many thanks to Deepak and Akira for running tests.
> >
> >     > -----Original Message-----
> >     > From: xen-devel-bounces@lists.xensource.com
> >     > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >     > Dave Winchell
> >     > Sent: Wednesday, January 09, 2008 9:53 AM
> >     > To: Keir Fraser
> >     > Cc: dan.magenheimer@oracle.com; 
> xen-devel@lists.xensource.com; Dave
> >     > Winchell
> >     > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >     > disables pending
> >     > missed ticks
> >     >
> >     >
> >     > Hi Keir,
> >     >
> >     > The latest change, c/s 16690, looks fine.
> >     > I agree that the code in c/s 16690 is equivalent to
> >     > the code I submitted. Also, your version is more
> >     > concise.
> >     >
> >     > The error tests confirm the equivalence. With 
> overnight cpu loads,
> >     > the checked in version was accurate to +.048% for sles
> >     > and +.038% for red hat. My version was +.046% and +.032% in a
> >     > 2 hour test.
> >     > I don''t think the difference is significant.
> >     >
> >     > i/o loads produced errors of +.01%.
> >     >
> >     > Thanks for all your efforts on this issue.
> >     >
> >     > Regards,
> >     > Dave
> >     >
> >     >
> >     >
> >     > Keir Fraser wrote:
> >     >
> >     > >Applied as c/s 16690, although the checked-in patch is
> >     > smaller. I think the
> >     > >only important fix is to pt_intr_post() and the only bit
of
> >     > the patch I
> >     > >totally omitted was the change to
pt_process_missed_ticks().
> >     > I don''t think
> >     > >that change can be important, but let''s see what
> happens to the
> >     error
> >     > >percentage...
> >     > >
> >     > > -- Keir
> >     > >
> >     > >On 4/1/08 23:24, "Dave Winchell" 
> <dwinchell@virtualiron.com> wrote:
> >     > >
> >     > >
> >     > >
> >     > >>Hi Dan and Keir,
> >     > >>
> >     > >>Attached is a patch that fixes some issues with the 
> SYNC policy
> >     > >>(no_missed_ticks_pending).
> >     > >>I have not tried to make the change the minimal one,
but,
> >     > rather, just
> >     > >>ported into
> >     > >>the new code what I know to work well. The error for
> >     > >>no_missed_ticks_pending goes from
> >     > >>over 3% to .03% with this change according to my
testing.
> >     > >>
> >     > >>Regards,
> >     > >>Dave
> >     > >>
> >     > >>Dan Magenheimer wrote:
> >     > >>
> >     > >>
> >     > >>
> >     > >>>Hi Dave --
> >     > >>>
> >     > >>>Did you get your correction ported?  If so, it
would be
> >     > nice to see this get
> >     > >>>into 3.1.3.
> >     > >>>
> >     > >>>Note that I just did some very limited testing
with
> >     > timer_mode=2(=SYNC=no
> >     > >>>missed ticks pending)
> >     > >>>on tip of xen-3.1-testing (64-bit Linux hv guest)
and the
> >     > worst error I''ve
> >     > >>>seen so far
> >     > >>>is 0.012%.  But I haven''t tried any
exotic loads, just LTP.
> >     > >>>
> >     > >>>Thanks,
> >     > >>>Dan
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>>-----Original Message-----
> >     > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >     > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >     > >>>>To: dan.magenheimer@oracle.com
> >     > >>>>Cc: Keir Fraser; Shan, Haitao;
> >     > xen-devel@lists.xensource.com; Dong,
> >     > >>>>Eddie; Jiang, Yunhong; Dave Winchell
> >     > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that
> >     > >>>>disables pending
> >     > >>>>missed ticks
> >     > >>>>
> >     > >>>>
> >     > >>>>Dan,
> >     > >>>>
> >     > >>>>I did some testing with the constant tsc
offset
> SYNC method
> >     > >>>>(now called
> >     > >>>>no_missed_ticks_pending)
> >     > >>>>and found the error to be very high, much
larger
> than 1 %, as
> >     > >>>>I recall.
> >     > >>>>I have not had a chance to submit a
correction. I
> will try to
> >     > >>>>do it later
> >     > >>>>this week or the first week in January. My
version of
> >     constant tsc
> >     > >>>>offset SYNC method
> >     > >>>>produces .02 % error, so I just need to port
that into the
> >     > >>>>current code.
> >     > >>>>
> >     > >>>>The error you got for both of those kernels
is
> what I would
> >     expect
> >     > >>>>for the default mode, delay_for_missed_ticks.
> >     > >>>>
> >     > >>>>I''ll let Keir answer on how to set
the time mode.
> >     > >>>>
> >     > >>>>Regards,
> >     > >>>>Dave
> >     > >>>>
> >     > >>>>Dan Magenheimer wrote:
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>Anyone make measurements on the final
patch?
> >     > >>>>>
> >     > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel
and saw a loss of
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>about 0.2% with no load.  This was
xen-unstable tip today
> >     > >>>>with no options specified.  32-bit was about
0.01%.
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>I think I missed something... how do I
run the various
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>accounting choices and which ones are known
to be
> appropriate
> >     > >>>>for which kernels?
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>Thanks,
> >     > >>>>>Dan
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>>-----Original Message-----
> >     > >>>>>>From:
xen-devel-bounces@lists.xensource.com
> >     > 
> >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>Keir Fraser
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>>Sent: Thursday, December 06, 2007
4:57 AM
> >     > >>>>>>To: Dave Winchell
> >     > >>>>>>Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
> >     > Eddie; Jiang,
> >     > >>>>>>Yunhong
> >     > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add
a timer mode that
> >     > >>>>>>disables pending
> >     > >>>>>>missed ticks
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>Please take a look at xen-unstable
changeset 16545.
> >     > >>>>>>
> >     > >>>>>>-- Keir
> >     > >>>>>>
> >     > >>>>>>On 26/11/07 20:57, "Dave
Winchell"
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>><dwinchell@virtualiron.com> wrote:
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>Keir,
> >     > >>>>>>>
> >     > >>>>>>>The accuracy data I''ve
collected for i/o loads for the
> >     > >>>>>>>various time protocols follows.
In addition, the data
> >     > >>>>>>>for cpu loads is shown.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled cpu and i/o-8
are on an 8
> processor AMD
> >     box.
> >     > >>>>>>>Two guests, red hat and sles 64
bit, 8 vcpu each.
> >     > >>>>>>>The cpu load is usex -e36 on each
guest.
> >     > >>>>>>>(usex is available at
> >     http://people.redhat.com/anderson/usex.)
> >     > >>>>>>>i/o load is 8 instances of dd
if=/dev/hda6
> of=/dev/null.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled i/o-32 are 32
instances of dd.
> >     > >>>>>>>Also, these are run on 4 cpu AMD
box.
> >     > >>>>>>>In addition, there is an idle
rh-32bit guest.
> >     > >>>>>>>All three guests are 8vcpu.
> >     > >>>>>>>
> >     > >>>>>>>The loads labeled i/o-4/32 are
the same as i/o-32
> >     > >>>>>>>except that the redhat-64 guest
has 4 instances of dd.
> >     > >>>>>>>
> >     > >>>>>>>Date Duration Protocol sles, rhat
error load
> >     > >>>>>>>
> >     > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96
sec, +4.42 sec -.006%,
> >     > +.005% cpu
> >     > >>>>>>>11/09 3 hrs 19 min ASYNC -.13
sec, +1.44 sec, -.001%,
> >     > +.012% cpu
> >     > >>>>>>>
> >     > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec,
-.34 sec, -.009%,
> >     -.004% cpu
> >     > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec,
-.26 sec,
> -.005%, -.005% cpu
> >     > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec,
-8 sec,
> -.008%, -.003% cpu
> >     > >>>>>>>
> >     > >>>>>>>11/08 28 min MIXED -.75 sec, -.67
sec -.045%,
> -.040% cpu
> >     > >>>>>>>11/08 15 hrs 39 min MIXED -19.
sec,-17.4 sec, -.034%,
> >     > -.031% cpu
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
sec,-55.7 sec, -.01%,
> >     > -.09% i/o-8
> >     > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
sec,-14.0 sec, -.015%
> >     > -.14% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>11/13 15 hrs 38 min SYNC -9.7
sec,-12.3 sec, -.017%,
> >     > -.022% i/o-8
> >     > >>>>>>>11/14 48 min SYNC - .46 sec, -
.48 sec,
> -.017%, -.018% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec,
-4.15 sec, -.020%,
> >     > -.029% i/o-8
> >     > >>>>>>>11/20 16 hrs 2 min MIXED -13.4
sec,-18.1 sec, -.023%,
> >     > -.031% i/o-8
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>11/21 28 min MIXED -2.01 sec,
-.67 sec, -.12%,
> -.04% i/o-32
> >     > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec,
-.43 sec, -.011%,
> >     > -.005% i/o-32
> >     > >>>>>>>11/21 40 min ASYNC -2.43 sec,
-2.77 sec -.10%,
> -.11% i/o-32
> >     > >>>>>>>
> >     > >>>>>>>11/26 113 hrs 46 min MIXED -297.
sec, 13. sec -.07%,
> >     > .003% i/o-4/32
> >     > >>>>>>>11/26 4 hrs 50 min SYNC -3.21
sec, 1.44 sec, -.017%,
> >     > .01% i/o-4/32
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>Overhead measurements:
> >     > >>>>>>>
> >     > >>>>>>>Progress in terms of number of
passes through a fixed
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>system workload
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>on an 8 vcpu red hat with an 8
vcpu sles idle.
> >     > >>>>>>>The workload was usex -b48.
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>ASYNC 167 min 145 passes .868
passes/min
> >     > >>>>>>>SYNC 167 min 144 passes .862
passes/min
> >     > >>>>>>>SYNC 1065 min 919 passes .863
passes/min
> >     > >>>>>>>MIXED 221 min 196 passes .887
passes/min
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>Conclusions:
> >     > >>>>>>>
> >     > >>>>>>>The only protocol which meets the
.05% accuracy
> >     > requirement for ntp
> >     > >>>>>>>tracking under the loads
> >     > >>>>>>>above is the SYNC protocol. The
worst case
> accuracies for
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>SYNC, MIXED,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>and ASYNC
> >     > >>>>>>>are .022%, .12%, and .14%,
respectively.
> >     > >>>>>>>
> >     > >>>>>>>We could reduce the cost of the
SYNC method by only
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>scheduling the extra
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>wakeups if a certain number
> >     > >>>>>>>of ticks are missed.
> >     > >>>>>>>
> >     > >>>>>>>Regards,
> >     > >>>>>>>Dave
> >     > >>>>>>>
> >     > >>>>>>>Keir Fraser wrote:
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>
> >     > >>>>>>>>On 9/11/07 19:22, "Dave
Winchell"
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>><dwinchell@virtualiron.com>
wrote:
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>>Since I had a high error
(~.03%) for the
> ASYNC method a
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>couple of days ago,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>I ran another ASYNC test.
I think there may have
> >     > been something
> >     > >>>>>>>>>wrong with the code I
used a couple of days ago for
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>ASYNC. It may have been
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>missing the immediate
delivery of interrupt
> after context
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>switch in.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>My results indicate that
either SYNC or ASYNC give
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>acceptable accuracy,
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>each running consistently
around or under
> .01%. MIXED has
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>a fairly high
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>error of
> >     > >>>>>>>>>greater than .03%.
Probably too close to .05% ntp
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>threshold for comfort.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>I don''t have an
overnight run with SYNC. I
> plan to leave
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>SYNC running
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>over the weekend. If
you''d rather I can leave MIXED
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>running instead.
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>It may be too early to
pick the protocol and
> I can run
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>more overnight tests
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>>next week.
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>>
> >     > >>>>>>>>I''m a bit worried
about any unwanted side
> effects of the
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>SYNC+run_timer
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>approach -- e.g., whether
timer wakeups will
> cause higher
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>system-wide CPU
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>contention. I find it easier
to think through the
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>implications of ASYNC. I''m
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>surprised that MIXED loses
time, and is less
> accurate than
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>ASYNC. Perhaps it
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>delivers more timer
interrupts than the other
> approaches,
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>and each interrupt
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>event causes a small
accumulated error?
> >     > >>>>>>>>
> >     > >>>>>>>>Overall I would consider
MIXED and ASYNC as
> favourites and
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>if the latter is
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>actually more accurate then I
can simply revert the
> >     > changeset that
> >     > >>>>>>>>implemented MIXED.
> >     > >>>>>>>>
> >     > >>>>>>>>Perhaps rather than running
more of the same
> workloads you
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>could try idle
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>VCPUs and I/O bound VCPUs
(e.g., repeated
> large disc reads
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>to /dev/null)? We
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>don''t have any data
on workloads that aren''t
> CPU bound, so
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>that''s really an
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>>>obvious place to put any
further effort imo.
> >     > >>>>>>>>
> >     > >>>>>>>>-- Keir
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     > >>>>>>>>
> >     >
>>>>>>_______________________________________________
> >     > >>>>>>Xen-devel mailing list
> >     > >>>>>>Xen-devel@lists.xensource.com
> >     > >>>>>>http://lists.xensource.com/xen-devel
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>>
> >     > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >     > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007
+0000
> >     > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008
-0500
> >     > >>@@ -58,7 +58,7 @@ static void
pt_process_missed_ticks(stru
> >     > >>
> >     > >>     missed_ticks = missed_ticks / (s_time_t) 
> pt->period + 1;
> >     > >>     if ( mode_is(pt->vcpu->domain, 
> no_missed_ticks_pending) )
> >     > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
> >     > >>+        pt->do_not_freeze = 1;
> >     > >>     else
> >     > >>         pt->pending_intr_nr += missed_ticks;
> >     > >>     pt->scheduled += missed_ticks *
pt->period;
> >     > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void
*data)
> >     > >>
> >     > >>     pt_lock(pt);
> >     > >>
> >     > >>-    pt->pending_intr_nr++;
> >     > >>+    if ( mode_is(pt->vcpu->domain, 
> no_missed_ticks_pending) ) {
> >     > >>+        pt->pending_intr_nr = 1;
> >     > >>+ pt->do_not_freeze = 0;
> >     > >>+    }
> >     > >>+    else
> >     > >>+ pt->pending_intr_nr++;
> >     > >>
> >     > >>     if ( !pt->one_shot )
> >     > >>     {
> >     > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v,
struct
> >     > >>         return;
> >     > >>     }
> >     > >>
> >     > >>-    pt->do_not_freeze = 0;
> >     > >>-
> >     > >>     if ( pt->one_shot )
> >     > >>     {
> >     > >>         pt->enabled = 0;
> >     > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu 
> *v, struct
> >     > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
> >     > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
> >     > missed ticks */
> >     > >>         }
> >     > >>+ else if ( mode_is(v->domain,
no_missed_ticks_pending) ) {
> >     > >>+     pt->pending_intr_nr--;
> >     > >>+     pt->last_plt_gtime = hvm_get_guest_time(v);
> >     > >>+ }
> >     > >>         else
> >     > >>         {
> >     > >>             pt->last_plt_gtime +=
pt->period_cycles;
> >     > >>
> >     > >>
> >     > >
> >     > >
> >     > >
> >     > >
> >     >
> >     >
> >     > _______________________________________________
> >     > Xen-devel mailing list
> >     > Xen-devel@lists.xensource.com
> >     > http://lists.xensource.com/xen-devel
> >     >
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

I agree with you on disabling and the need for fixing hpet.
If no one else wants to fix it, I could give it a try.
I think we could make the hpet work better than pit/tsc.

While I monitor the skew over time, I''m not as methodical as you.
Its possible that I missed this behaviour.

I''ve been thinking about what you''ve seen and have no clue as
to the cause.
Let me ask you a few questions:

Is the graph for RHEL5u1-64? (I''ve never tested this one.)
What was the behaviour of the other guests running?
If they had spikes, were they at the same wall time?
Were the other guests running ltp as well?

How are you measuring skew?

Are you running ntpd?

Anything that you can discover that would be in sync with
the spikes would be very helpful!

The code that I test with is our product code, which is based
on 3.1. So it is possible that something in 3.2 other than vpt.c
is the cause. I can test with 3.2, if necessary.

thanks,
Dave



Dan Magenheimer wrote:
>Hi Dave (Keir, see suggestion below) --
>
>Thanks!
>
>Turning off vhpet certainly helps a lot (though see below).
>
>I wonder if timekeeping with vhpet is so bad that it should be
>turned off by default (in 3.1, 3.2, and unstable) until it is
>fixed?  (I have a patch that defaults it off, can post it if
>there is agreement on the above point.)  The whole point of an
>HPET is to provide more precise timekeeping and if vhpet is
>worse than vpit, it can only confuse users.  Comments?
>  
>
>In your testing, are you just measuring % skew over a long
>period of time? 
>
>  We are graphing the skew continuously and
>seeing periodic behavior that is unsettling, even with pit.
>See attached.  Though your algorithm recovers, the "cliffs"
>could still cause real user problems.  I wonder if there is
>anything that can be done to make the "recovery" more
>responsive?
>
>We are looking into what part(s) of LTP is causing the cliffs.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Monday, January 28, 2008 8:21 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>deepak.patel@oracle.com;
>>akira.ijuin@oracle.com; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>I guess I''m a bit out of date calling for clock= usage.
>>Looking at linux 2.6.20.4 sources, I think you should specify
>>"clocksource=pit nohpet" on the linux guest bootline.
>>
>>You can leave the xen and dom0 bootlines as they are.
>>The xen and guest clocksources do not need to be the same.
>>In my tests, xen is using the hpet for its timekeeping and
>>that appears to be the default.
>>
>>When you boot the guests you should see
>>    time.c: Using PIT/TSC based timekeeping.
>>on the rh4u5-64 guest, and something similar on the others.
>>
>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>> > 14.318MHz HPET.)
>>
>>This appears to be the xen state, which is fine.
>>I was wrongly assuming that this was the guest state.
>>You might want to look in your guest logs and see what they 
>>were picking
>>for a clock source.
>>
>>Regards,
>>Dave
>>
>>
>>
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Thanks, I hadn''t realized that!  No wonder we
didn''t see the same
>>>improvement you saw!
>>>
>>>      
>>>
>>>>Try specifying clock=pit on the linux boot line...
>>>>        
>>>>
>>>I''m confused... do you mean "clocksource=pit" on
the Xen
>>>      
>>>
>>command line or
>>    
>>
>>>"nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
>>>      
>>>
>>dom0?) command
>>    
>>
>>>line?  Or both places?  Since the tests take awhile, it 
>>>      
>>>
>>would be nice
>>    
>>
>>>to get this right the first time.  Do the Xen and guest 
>>>      
>>>
>>clocksources need
>>    
>>
>>>to be the same?
>>>
>>>Thanks,
>>>Dan
>>>
>>> -----Original Message-----
>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>*Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>akira.ijuin@oracle.com; Dave Winchell
>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables
>>>pending missed ticks
>>>
>>>    Hi Dan,
>>>
>>>    Hpet timer does have a fairly large error, as I was trying this
>>>    one recently.
>>>    I don''t remember what I got for error, but 1% sounds 
>>>      
>>>
>>about right.
>>    
>>
>>>    The problem is that hpet is not built on top of vpt.c, 
>>>      
>>>
>>the module
>>    
>>
>>>    Keir and I did
>>>    all the recent work in, for its periodic timer needs. Try
>>>    specifying clock=pit
>>>    on the linux boot line. If it still picks the hpet, which it
>>>    might, let me know
>>>    and I''ll tell you how to get around this.
>>>
>>>    Regards,
>>>    Dave
>>>
>>>
>>>
>>>
>>>    
>>>      
>>>
>>--------------------------------------------------------------
>>----------
>>    
>>
>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>    *To:* Dave Winchell; Keir Fraser
>>>    *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>    akira.ijuin@oracle.com
>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
>>>      
>>>
>>that disables
>>    
>>
>>>    pending missed ticks
>>>
>>>    Sorry for the very late followup on this but we finally 
>>>      
>>>
>>were able
>>    
>>
>>>    to get our testing set up again on stable 3.1 bits and have
>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>
>>>    Test enviroment was a 4-socket dual core machine with 24GB of
>>>    memory running six two-vcpu 2GB domains, four hvm plus two pv.
>>>    All six guests were running LTP simultaneously.  The four hvm
>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64.
>>>    Timer_mode was set to 2 for 64-bit guests and 0 for 
>>>      
>>>
>>32-bit guests.
>>    
>>
>>>    All four hvm guests experienced skew around -1%, even the 32-bit
>>>    guest.  Less intensive testing didn''t exhibit much skew
at all.
>>>
>>>    A representative graph is attached.
>>>
>>>    Dave, I wonder if some portion of your patches didn''t
end up in
>>>    the xen trees?
>>>
>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>    14.318MHz HPET.)
>>>
>>>    Thanks,
>>>    Dan
>>>
>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>
>>>    > -----Original Message-----
>>>    > From: xen-devel-bounces@lists.xensource.com
>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>    > Dave Winchell
>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>    > To: Keir Fraser
>>>    > Cc: dan.magenheimer@oracle.com; 
>>>      
>>>
>>xen-devel@lists.xensource.com; Dave
>>    
>>
>>>    > Winchell
>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>    > disables pending
>>>    > missed ticks
>>>    >
>>>    >
>>>    > Hi Keir,
>>>    >
>>>    > The latest change, c/s 16690, looks fine.
>>>    > I agree that the code in c/s 16690 is equivalent to
>>>    > the code I submitted. Also, your version is more
>>>    > concise.
>>>    >
>>>    > The error tests confirm the equivalence. With 
>>>      
>>>
>>overnight cpu loads,
>>    
>>
>>>    > the checked in version was accurate to +.048% for sles
>>>    > and +.038% for red hat. My version was +.046% and +.032% in
a
>>>    > 2 hour test.
>>>    > I don''t think the difference is significant.
>>>    >
>>>    > i/o loads produced errors of +.01%.
>>>    >
>>>    > Thanks for all your efforts on this issue.
>>>    >
>>>    > Regards,
>>>    > Dave
>>>    >
>>>    >
>>>    >
>>>    > Keir Fraser wrote:
>>>    >
>>>    > >Applied as c/s 16690, although the checked-in patch is
>>>    > smaller. I think the
>>>    > >only important fix is to pt_intr_post() and the only
bit of
>>>    > the patch I
>>>    > >totally omitted was the change to
pt_process_missed_ticks().
>>>    > I don''t think
>>>    > >that change can be important, but let''s see
what
>>>      
>>>
>>happens to the
>>    
>>
>>>    error
>>>    > >percentage...
>>>    > >
>>>    > > -- Keir
>>>    > >
>>>    > >On 4/1/08 23:24, "Dave Winchell" 
>>>      
>>>
>><dwinchell@virtualiron.com> wrote:
>>    
>>
>>>    > >
>>>    > >
>>>    > >
>>>    > >>Hi Dan and Keir,
>>>    > >>
>>>    > >>Attached is a patch that fixes some issues with the
>>>      
>>>
>>SYNC policy
>>    
>>
>>>    > >>(no_missed_ticks_pending).
>>>    > >>I have not tried to make the change the minimal
one, but,
>>>    > rather, just
>>>    > >>ported into
>>>    > >>the new code what I know to work well. The error
for
>>>    > >>no_missed_ticks_pending goes from
>>>    > >>over 3% to .03% with this change according to my
testing.
>>>    > >>
>>>    > >>Regards,
>>>    > >>Dave
>>>    > >>
>>>    > >>Dan Magenheimer wrote:
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>>Hi Dave --
>>>    > >>>
>>>    > >>>Did you get your correction ported?  If so, it
would be
>>>    > nice to see this get
>>>    > >>>into 3.1.3.
>>>    > >>>
>>>    > >>>Note that I just did some very limited testing
with
>>>    > timer_mode=2(=SYNC=no
>>>    > >>>missed ticks pending)
>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv
guest) and the
>>>    > worst error I''ve
>>>    > >>>seen so far
>>>    > >>>is 0.012%.  But I haven''t tried any
exotic loads, just LTP.
>>>    > >>>
>>>    > >>>Thanks,
>>>    > >>>Dan
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>>-----Original Message-----
>>>    > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>    > >>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>    > >>>>To: dan.magenheimer@oracle.com
>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>    > xen-devel@lists.xensource.com; Dong,
>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that
>>>    > >>>>disables pending
>>>    > >>>>missed ticks
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>Dan,
>>>    > >>>>
>>>    > >>>>I did some testing with the constant tsc
offset
>>>      
>>>
>>SYNC method
>>    
>>
>>>    > >>>>(now called
>>>    > >>>>no_missed_ticks_pending)
>>>    > >>>>and found the error to be very high, much
larger
>>>      
>>>
>>than 1 %, as
>>    
>>
>>>    > >>>>I recall.
>>>    > >>>>I have not had a chance to submit a
correction. I
>>>      
>>>
>>will try to
>>    
>>
>>>    > >>>>do it later
>>>    > >>>>this week or the first week in January. My
version of
>>>    constant tsc
>>>    > >>>>offset SYNC method
>>>    > >>>>produces .02 % error, so I just need to
port that into the
>>>    > >>>>current code.
>>>    > >>>>
>>>    > >>>>The error you got for both of those kernels
is
>>>      
>>>
>>what I would
>>    
>>
>>>    expect
>>>    > >>>>for the default mode,
delay_for_missed_ticks.
>>>    > >>>>
>>>    > >>>>I''ll let Keir answer on how to set
the time mode.
>>>    > >>>>
>>>    > >>>>Regards,
>>>    > >>>>Dave
>>>    > >>>>
>>>    > >>>>Dan Magenheimer wrote:
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>Anyone make measurements on the final
patch?
>>>    > >>>>>
>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel
and saw a loss of
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>about 0.2% with no load.  This was
xen-unstable tip today
>>>    > >>>>with no options specified.  32-bit was
about 0.01%.
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>I think I missed something... how do I
run the various
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>accounting choices and which ones are known
to be
>>>      
>>>
>>appropriate
>>    
>>
>>>    > >>>>for which kernels?
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>Thanks,
>>>    > >>>>>Dan
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>>-----Original Message-----
>>>    > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>    > 
>>>      
>>>
>>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
>>>>>>>>                
>>>>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>Keir Fraser
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>>Sent: Thursday, December 06, 2007
4:57 AM
>>>    > >>>>>>To: Dave Winchell
>>>    > >>>>>>Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
>>>    > Eddie; Jiang,
>>>    > >>>>>>Yunhong
>>>    > >>>>>>Subject: Re: [Xen-devel] [PATCH]
Add a timer mode that
>>>    > >>>>>>disables pending
>>>    > >>>>>>missed ticks
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>Please take a look at xen-unstable
changeset 16545.
>>>    > >>>>>>
>>>    > >>>>>>-- Keir
>>>    > >>>>>>
>>>    > >>>>>>On 26/11/07 20:57, "Dave
Winchell"
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>><dwinchell@virtualiron.com> wrote:
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>Keir,
>>>    > >>>>>>>
>>>    > >>>>>>>The accuracy data I''ve
collected for i/o loads for the
>>>    > >>>>>>>various time protocols follows.
In addition, the data
>>>    > >>>>>>>for cpu loads is shown.
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled cpu and i/o-8
are on an 8
>>>      
>>>
>>processor AMD
>>    
>>
>>>    box.
>>>    > >>>>>>>Two guests, red hat and sles 64
bit, 8 vcpu each.
>>>    > >>>>>>>The cpu load is usex -e36 on
each guest.
>>>    > >>>>>>>(usex is available at
>>>    http://people.redhat.com/anderson/usex.)
>>>    > >>>>>>>i/o load is 8 instances of dd
if=/dev/hda6
>>>      
>>>
>>of=/dev/null.
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled i/o-32 are 32
instances of dd.
>>>    > >>>>>>>Also, these are run on 4 cpu
AMD box.
>>>    > >>>>>>>In addition, there is an idle
rh-32bit guest.
>>>    > >>>>>>>All three guests are 8vcpu.
>>>    > >>>>>>>
>>>    > >>>>>>>The loads labeled i/o-4/32 are
the same as i/o-32
>>>    > >>>>>>>except that the redhat-64 guest
has 4 instances of dd.
>>>    > >>>>>>>
>>>    > >>>>>>>Date Duration Protocol sles,
rhat error load
>>>    > >>>>>>>
>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96
sec, +4.42 sec -.006%,
>>>    > +.005% cpu
>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC -.13
sec, +1.44 sec, -.001%,
>>>    > +.012% cpu
>>>    > >>>>>>>
>>>    > >>>>>>>11/08 2 hrs 21 min SYNC -.80
sec, -.34 sec, -.009%,
>>>    -.004% cpu
>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24
sec, -.26 sec,
>>>      
>>>
>>-.005%, -.005% cpu
>>    
>>
>>>    > >>>>>>>11/12 65 hrs 40 min SYNC -18
sec, -8 sec,
>>>      
>>>
>>-.008%, -.003% cpu
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/08 28 min MIXED -.75 sec,
-.67 sec -.045%,
>>>      
>>>
>>-.040% cpu
>>    
>>
>>>    > >>>>>>>11/08 15 hrs 39 min MIXED -19.
sec,-17.4 sec, -.034%,
>>>    > -.031% cpu
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1
sec,-55.7 sec, -.01%,
>>>    > -.09% i/o-8
>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47
sec,-14.0 sec, -.015%
>>>    > -.14% i/o-8
>>>    > >>>>>>>
>>>    > >>>>>>>11/13 15 hrs 38 min SYNC -9.7
sec,-12.3 sec, -.017%,
>>>    > -.022% i/o-8
>>>    > >>>>>>>11/14 48 min SYNC - .46 sec, -
.48 sec,
>>>      
>>>
>>-.017%, -.018% i/o-8
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/14 4 hrs 2 min MIXED -2.9
sec, -4.15 sec, -.020%,
>>>    > -.029% i/o-8
>>>    > >>>>>>>11/20 16 hrs 2 min MIXED -13.4
sec,-18.1 sec, -.023%,
>>>    > -.031% i/o-8
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/21 28 min MIXED -2.01 sec,
-.67 sec, -.12%,
>>>      
>>>
>>-.04% i/o-32
>>    
>>
>>>    > >>>>>>>11/21 2 hrs 25 min SYNC -.96
sec, -.43 sec, -.011%,
>>>    > -.005% i/o-32
>>>    > >>>>>>>11/21 40 min ASYNC -2.43 sec,
-2.77 sec -.10%,
>>>      
>>>
>>-.11% i/o-32
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>11/26 113 hrs 46 min MIXED
-297. sec, 13. sec -.07%,
>>>    > .003% i/o-4/32
>>>    > >>>>>>>11/26 4 hrs 50 min SYNC -3.21
sec, 1.44 sec, -.017%,
>>>    > .01% i/o-4/32
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>Overhead measurements:
>>>    > >>>>>>>
>>>    > >>>>>>>Progress in terms of number of
passes through a fixed
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>system workload
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>on an 8 vcpu red hat with an 8
vcpu sles idle.
>>>    > >>>>>>>The workload was usex -b48.
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>ASYNC 167 min 145 passes .868
passes/min
>>>    > >>>>>>>SYNC 167 min 144 passes .862
passes/min
>>>    > >>>>>>>SYNC 1065 min 919 passes .863
passes/min
>>>    > >>>>>>>MIXED 221 min 196 passes .887
passes/min
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>Conclusions:
>>>    > >>>>>>>
>>>    > >>>>>>>The only protocol which meets
the .05% accuracy
>>>    > requirement for ntp
>>>    > >>>>>>>tracking under the loads
>>>    > >>>>>>>above is the SYNC protocol. The
worst case
>>>      
>>>
>>accuracies for
>>    
>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>SYNC, MIXED,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>and ASYNC
>>>    > >>>>>>>are .022%, .12%, and .14%,
respectively.
>>>    > >>>>>>>
>>>    > >>>>>>>We could reduce the cost of the
SYNC method by only
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>scheduling the extra
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>wakeups if a certain number
>>>    > >>>>>>>of ticks are missed.
>>>    > >>>>>>>
>>>    > >>>>>>>Regards,
>>>    > >>>>>>>Dave
>>>    > >>>>>>>
>>>    > >>>>>>>Keir Fraser wrote:
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>
>>>    > >>>>>>>>On 9/11/07 19:22,
"Dave Winchell"
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>><dwinchell@virtualiron.com>
wrote:
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>>Since I had a high
error (~.03%) for the
>>>      
>>>
>>ASYNC method a
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>couple of days ago,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>I ran another ASYNC
test. I think there may have
>>>    > been something
>>>    > >>>>>>>>>wrong with the code I
used a couple of days ago for
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>ASYNC. It may have been
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>missing the immediate
delivery of interrupt
>>>      
>>>
>>after context
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>switch in.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>My results indicate
that either SYNC or ASYNC give
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>acceptable accuracy,
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>each running
consistently around or under
>>>      
>>>
>>.01%. MIXED has
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>a fairly high
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>error of
>>>    > >>>>>>>>>greater than .03%.
Probably too close to .05% ntp
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>threshold for comfort.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>I don''t have
an overnight run with SYNC. I
>>>      
>>>
>>plan to leave
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>SYNC running
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>over the weekend. If
you''d rather I can leave MIXED
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>running instead.
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>It may be too early to
pick the protocol and
>>>      
>>>
>>I can run
>>    
>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>more overnight tests
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>>next week.
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>>
>>>    > >>>>>>>>I''m a bit worried
about any unwanted side
>>>      
>>>
>>effects of the
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>SYNC+run_timer
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>approach -- e.g., whether
timer wakeups will
>>>      
>>>
>>cause higher
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>system-wide CPU
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>contention. I find it
easier to think through the
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>implications of ASYNC. I''m
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>surprised that MIXED loses
time, and is less
>>>      
>>>
>>accurate than
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>ASYNC. Perhaps it
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>delivers more timer
interrupts than the other
>>>      
>>>
>>approaches,
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>and each interrupt
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>event causes a small
accumulated error?
>>>    > >>>>>>>>
>>>    > >>>>>>>>Overall I would consider
MIXED and ASYNC as
>>>      
>>>
>>favourites and
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>if the latter is
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>actually more accurate then
I can simply revert the
>>>    > changeset that
>>>    > >>>>>>>>implemented MIXED.
>>>    > >>>>>>>>
>>>    > >>>>>>>>Perhaps rather than running
more of the same
>>>      
>>>
>>workloads you
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>could try idle
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>VCPUs and I/O bound VCPUs
(e.g., repeated
>>>      
>>>
>>large disc reads
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>to /dev/null)? We
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>don''t have any
data on workloads that aren''t
>>>      
>>>
>>CPU bound, so
>>    
>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>that''s really an
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>>>obvious place to put any
further effort imo.
>>>    > >>>>>>>>
>>>    > >>>>>>>>-- Keir
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    > >>>>>>>>
>>>    >
>>>>>>_______________________________________________
>>>    > >>>>>>Xen-devel mailing list
>>>    > >>>>>>Xen-devel@lists.xensource.com
>>>    >
>>>>>>http://lists.xensource.com/xen-devel
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>>
>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
2007 +0000
>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
2008 -0500
>>>    > >>@@ -58,7 +58,7 @@ static void
pt_process_missed_ticks(stru
>>>    > >>
>>>    > >>     missed_ticks = missed_ticks / (s_time_t) 
>>>      
>>>
>>pt->period + 1;
>>    
>>
>>>    > >>     if ( mode_is(pt->vcpu->domain, 
>>>      
>>>
>>no_missed_ticks_pending) )
>>    
>>
>>>    > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
>>>    > >>+        pt->do_not_freeze = 1;
>>>    > >>     else
>>>    > >>         pt->pending_intr_nr += missed_ticks;
>>>    > >>     pt->scheduled += missed_ticks *
pt->period;
>>>    > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void
*data)
>>>    > >>
>>>    > >>     pt_lock(pt);
>>>    > >>
>>>    > >>-    pt->pending_intr_nr++;
>>>    > >>+    if ( mode_is(pt->vcpu->domain, 
>>>      
>>>
>>no_missed_ticks_pending) ) {
>>    
>>
>>>    > >>+        pt->pending_intr_nr = 1;
>>>    > >>+ pt->do_not_freeze = 0;
>>>    > >>+    }
>>>    > >>+    else
>>>    > >>+ pt->pending_intr_nr++;
>>>    > >>
>>>    > >>     if ( !pt->one_shot )
>>>    > >>     {
>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu
*v, struct
>>>    > >>         return;
>>>    > >>     }
>>>    > >>
>>>    > >>-    pt->do_not_freeze = 0;
>>>    > >>-
>>>    > >>     if ( pt->one_shot )
>>>    > >>     {
>>>    > >>         pt->enabled = 0;
>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu 
>>>      
>>>
>>*v, struct
>>    
>>
>>>    > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>    > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
>>>    > missed ticks */
>>>    > >>         }
>>>    > >>+ else if ( mode_is(v->domain,
no_missed_ticks_pending) ) {
>>>    > >>+     pt->pending_intr_nr--;
>>>    > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>    > >>+ }
>>>    > >>         else
>>>    > >>         {
>>>    > >>             pt->last_plt_gtime +=
pt->period_cycles;
>>>    > >>
>>>    > >>
>>>    > >
>>>    > >
>>>    > >
>>>    > >
>>>    >
>>>    >
>>>    > _______________________________________________
>>>    > Xen-devel mailing list
>>>    > Xen-devel@lists.xensource.com
>>>    > http://lists.xensource.com/xen-devel
>>>    >
>>>
>>>      
>>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>
> Is the graph for RHEL5u1-64? (I''ve never tested this one.)
I do not know which graph was attached with this. But I saw this 
behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I was 
running ltp tests continuously.
> What was the behaviour of the other guests running?
All pvm guests are fine. But behavior of most of the hvm guests were as 
described.
> If they had spikes, were they at the same wall time?
No. They are not at the same wall time.
> Were the other guests running ltp as well?
>
Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp continuously.
> How are you measuring skew?
I was collecting output of "ntpdate -q  <timeserver> every 300
seconds
(5 minutes) and have created graph based on that.
>
> Are you running ntpd?
>
Yes. ntp was running on all the guests.

I am investigating what causes this spikes and let everyone know what 
are my findings.

Thanks,
Deepak
> Anything that you can discover that would be in sync with
> the spikes would be very helpful!
>
> The code that I test with is our product code, which is based
> on 3.1. So it is possible that something in 3.2 other than vpt.c
> is the cause. I can test with 3.2, if necessary.
>
> thanks,
> Dave
>
>
>
> Dan Magenheimer wrote:
>
>> Hi Dave (Keir, see suggestion below) --
>>
>> Thanks!
>>
>> Turning off vhpet certainly helps a lot (though see below).
>>
>> I wonder if timekeeping with vhpet is so bad that it should be
>> turned off by default (in 3.1, 3.2, and unstable) until it is
>> fixed?  (I have a patch that defaults it off, can post it if
>> there is agreement on the above point.)  The whole point of an
>> HPET is to provide more precise timekeeping and if vhpet is
>> worse than vpit, it can only confuse users.  Comments?
>>  
>>
>> In your testing, are you just measuring % skew over a long
>> period of time?
>>  We are graphing the skew continuously and
>> seeing periodic behavior that is unsettling, even with pit.
>> See attached.  Though your algorithm recovers, the "cliffs"
>> could still cause real user problems.  I wonder if there is
>> anything that can be done to make the "recovery" more
>> responsive?
>>
>> We are looking into what part(s) of LTP is causing the cliffs.
>>
>> Thanks,
>> Dan
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Monday, January 28, 2008 8:21 AM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>> deepak.patel@oracle.com;
>>> akira.ijuin@oracle.com; Dave Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
pending
>>> missed ticks
>>>
>>>
>>> Dan,
>>>
>>> I guess I''m a bit out of date calling for clock= usage.
>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>
>>> You can leave the xen and dom0 bootlines as they are.
>>> The xen and guest clocksources do not need to be the same.
>>> In my tests, xen is using the hpet for its timekeeping and
>>> that appears to be the default.
>>>
>>> When you boot the guests you should see
>>>    time.c: Using PIT/TSC based timekeeping.
>>> on the rh4u5-64 guest, and something similar on the others.
>>>
>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>> > 14.318MHz HPET.)
>>>
>>> This appears to be the xen state, which is fine.
>>> I was wrongly assuming that this was the guest state.
>>> You might want to look in your guest logs and see what they were 
>>> picking
>>> for a clock source.
>>>
>>> Regards,
>>> Dave
>>>
>>>
>>>
>>>
>>> Dan Magenheimer wrote:
>>>
>>>   
>>>
>>>> Thanks, I hadn''t realized that!  No wonder we
didn''t see the same
>>>> improvement you saw!
>>>>
>>>>     
>>>>
>>>>> Try specifying clock=pit on the linux boot line...
>>>>>       
>>>>
>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
>>>
>>> command line or
>>>   
>>>
>>>> "nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
>>>
>>> dom0?) command
>>>   
>>>
>>>> line?  Or both places?  Since the tests take awhile, it     
>>>
>>> would be nice
>>>   
>>>
>>>> to get this right the first time.  Do the Xen and guest     
>>>
>>> clocksources need
>>>   
>>>
>>>> to be the same?
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>> -----Original Message-----
>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com;
>>>> akira.ijuin@oracle.com; Dave Winchell
>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that
disables
>>>> pending missed ticks
>>>>
>>>>    Hi Dan,
>>>>
>>>>    Hpet timer does have a fairly large error, as I was trying
this
>>>>    one recently.
>>>>    I don''t remember what I got for error, but 1%
sounds
>>>
>>> about right.
>>>   
>>>
>>>>    The problem is that hpet is not built on top of vpt.c,     
>>>
>>> the module
>>>   
>>>
>>>>    Keir and I did
>>>>    all the recent work in, for its periodic timer needs. Try
>>>>    specifying clock=pit
>>>>    on the linux boot line. If it still picks the hpet, which it
>>>>    might, let me know
>>>>    and I''ll tell you how to get around this.
>>>>
>>>>    Regards,
>>>>    Dave
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>
>>> --------------------------------------------------------------
>>> ----------
>>>   
>>>
>>>>    *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
>>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>>    *To:* Dave Winchell; Keir Fraser
>>>>    *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>    akira.ijuin@oracle.com
>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode     
>>>
>>> that disables
>>>   
>>>
>>>>    pending missed ticks
>>>>
>>>>    Sorry for the very late followup on this but we finally     
>>>
>>> were able
>>>   
>>>
>>>>    to get our testing set up again on stable 3.1 bits and have
>>>>    seen some very bad results on 3.1.3-rc1, on the order of 1%.
>>>>
>>>>    Test enviroment was a 4-socket dual core machine with 24GB
of
>>>>    memory running six two-vcpu 2GB domains, four hvm plus two
pv.
>>>>    All six guests were running LTP simultaneously.  The four
hvm
>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
RHEL4u5-64.
>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for     
>>>
>>> 32-bit guests.
>>>   
>>>
>>>>    All four hvm guests experienced skew around -1%, even the
32-bit
>>>>    guest.  Less intensive testing didn''t exhibit much
skew at all.
>>>>
>>>>    A representative graph is attached.
>>>>
>>>>    Dave, I wonder if some portion of your patches
didn''t end up in
>>>>    the xen trees?
>>>>
>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>>    14.318MHz HPET.)
>>>>
>>>>    Thanks,
>>>>    Dan
>>>>
>>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>>
>>>>    > -----Original Message-----
>>>>    > From: xen-devel-bounces@lists.xensource.com
>>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf
Of
>>>>    > Dave Winchell
>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>    > To: Keir Fraser
>>>>    > Cc: dan.magenheimer@oracle.com;     
>>>
>>> xen-devel@lists.xensource.com; Dave
>>>   
>>>
>>>>    > Winchell
>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>    > disables pending
>>>>    > missed ticks
>>>>    >
>>>>    >
>>>>    > Hi Keir,
>>>>    >
>>>>    > The latest change, c/s 16690, looks fine.
>>>>    > I agree that the code in c/s 16690 is equivalent to
>>>>    > the code I submitted. Also, your version is more
>>>>    > concise.
>>>>    >
>>>>    > The error tests confirm the equivalence. With     
>>>
>>> overnight cpu loads,
>>>   
>>>
>>>>    > the checked in version was accurate to +.048% for sles
>>>>    > and +.038% for red hat. My version was +.046% and
+.032% in a
>>>>    > 2 hour test.
>>>>    > I don''t think the difference is significant.
>>>>    >
>>>>    > i/o loads produced errors of +.01%.
>>>>    >
>>>>    > Thanks for all your efforts on this issue.
>>>>    >
>>>>    > Regards,
>>>>    > Dave
>>>>    >
>>>>    >
>>>>    >
>>>>    > Keir Fraser wrote:
>>>>    >
>>>>    > >Applied as c/s 16690, although the checked-in patch
is
>>>>    > smaller. I think the
>>>>    > >only important fix is to pt_intr_post() and the
only bit of
>>>>    > the patch I
>>>>    > >totally omitted was the change to
pt_process_missed_ticks().
>>>>    > I don''t think
>>>>    > >that change can be important, but let''s
see what
>>>
>>> happens to the
>>>   
>>>
>>>>    error
>>>>    > >percentage...
>>>>    > >
>>>>    > > -- Keir
>>>>    > >
>>>>    > >On 4/1/08 23:24, "Dave Winchell"     
>>>
>>> <dwinchell@virtualiron.com> wrote:
>>>   
>>>
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    > >>Hi Dan and Keir,
>>>>    > >>
>>>>    > >>Attached is a patch that fixes some issues with
the
>>>
>>> SYNC policy
>>>   
>>>
>>>>    > >>(no_missed_ticks_pending).
>>>>    > >>I have not tried to make the change the minimal
one, but,
>>>>    > rather, just
>>>>    > >>ported into
>>>>    > >>the new code what I know to work well. The
error for
>>>>    > >>no_missed_ticks_pending goes from
>>>>    > >>over 3% to .03% with this change according to
my testing.
>>>>    > >>
>>>>    > >>Regards,
>>>>    > >>Dave
>>>>    > >>
>>>>    > >>Dan Magenheimer wrote:
>>>>    > >>
>>>>    > >>
>>>>    > >>
>>>>    > >>>Hi Dave --
>>>>    > >>>
>>>>    > >>>Did you get your correction ported?  If so,
it would be
>>>>    > nice to see this get
>>>>    > >>>into 3.1.3.
>>>>    > >>>
>>>>    > >>>Note that I just did some very limited
testing with
>>>>    > timer_mode=2(=SYNC=no
>>>>    > >>>missed ticks pending)
>>>>    > >>>on tip of xen-3.1-testing (64-bit Linux hv
guest) and the
>>>>    > worst error I''ve
>>>>    > >>>seen so far
>>>>    > >>>is 0.012%.  But I haven''t tried
any exotic loads, just LTP.
>>>>    > >>>
>>>>    > >>>Thanks,
>>>>    > >>>Dan
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>>-----Original Message-----
>>>>    > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>    > >>>>Sent: Wednesday, December 19, 2007
12:33 PM
>>>>    > >>>>To: dan.magenheimer@oracle.com
>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>    > xen-devel@lists.xensource.com; Dong,
>>>>    > >>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>    > >>>>Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that
>>>>    > >>>>disables pending
>>>>    > >>>>missed ticks
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>Dan,
>>>>    > >>>>
>>>>    > >>>>I did some testing with the constant
tsc offset
>>>
>>> SYNC method
>>>   
>>>
>>>>    > >>>>(now called
>>>>    > >>>>no_missed_ticks_pending)
>>>>    > >>>>and found the error to be very high,
much larger
>>>
>>> than 1 %, as
>>>   
>>>
>>>>    > >>>>I recall.
>>>>    > >>>>I have not had a chance to submit a
correction. I
>>>
>>> will try to
>>>   
>>>
>>>>    > >>>>do it later
>>>>    > >>>>this week or the first week in January.
My version of
>>>>    constant tsc
>>>>    > >>>>offset SYNC method
>>>>    > >>>>produces .02 % error, so I just need to
port that into the
>>>>    > >>>>current code.
>>>>    > >>>>
>>>>    > >>>>The error you got for both of those
kernels is
>>>
>>> what I would
>>>   
>>>
>>>>    expect
>>>>    > >>>>for the default mode,
delay_for_missed_ticks.
>>>>    > >>>>
>>>>    > >>>>I''ll let Keir answer on how to
set the time mode.
>>>>    > >>>>
>>>>    > >>>>Regards,
>>>>    > >>>>Dave
>>>>    > >>>>
>>>>    > >>>>Dan Magenheimer wrote:
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>Anyone make measurements on the
final patch?
>>>>    > >>>>>
>>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm
kernel and saw a loss of
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>about 0.2% with no load.  This was
xen-unstable tip today
>>>>    > >>>>with no options specified.  32-bit was
about 0.01%.
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>I think I missed something... how
do I run the various
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>accounting choices and which ones are
known to be
>>>
>>> appropriate
>>>   
>>>
>>>>    > >>>>for which kernels?
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>Thanks,
>>>>    > >>>>>Dan
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>>-----Original Message-----
>>>>    > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>    >     
>>>>
>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>               
>>>>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>Keir Fraser
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>>Sent: Thursday, December 06,
2007 4:57 AM
>>>>    > >>>>>>To: Dave Winchell
>>>>    > >>>>>>Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
>>>>    > Eddie; Jiang,
>>>>    > >>>>>>Yunhong
>>>>    > >>>>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
>>>>    > >>>>>>disables pending
>>>>    > >>>>>>missed ticks
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>Please take a look at
xen-unstable changeset 16545.
>>>>    > >>>>>>
>>>>    > >>>>>>-- Keir
>>>>    > >>>>>>
>>>>    > >>>>>>On 26/11/07 20:57, "Dave
Winchell"
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>><dwinchell@virtualiron.com>
wrote:
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>Keir,
>>>>    > >>>>>>>
>>>>    > >>>>>>>The accuracy data
I''ve collected for i/o loads for the
>>>>    > >>>>>>>various time protocols
follows. In addition, the data
>>>>    > >>>>>>>for cpu loads is shown.
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled cpu and
i/o-8 are on an 8
>>>
>>> processor AMD
>>>   
>>>
>>>>    box.
>>>>    > >>>>>>>Two guests, red hat and
sles 64 bit, 8 vcpu each.
>>>>    > >>>>>>>The cpu load is usex -e36
on each guest.
>>>>    > >>>>>>>(usex is available at
>>>>    http://people.redhat.com/anderson/usex.)
>>>>    > >>>>>>>i/o load is 8 instances of
dd if=/dev/hda6
>>>
>>> of=/dev/null.
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled i/o-32
are 32 instances of dd.
>>>>    > >>>>>>>Also, these are run on 4
cpu AMD box.
>>>>    > >>>>>>>In addition, there is an
idle rh-32bit guest.
>>>>    > >>>>>>>All three guests are 8vcpu.
>>>>    > >>>>>>>
>>>>    > >>>>>>>The loads labeled i/o-4/32
are the same as i/o-32
>>>>    > >>>>>>>except that the redhat-64
guest has 4 instances of dd.
>>>>    > >>>>>>>
>>>>    > >>>>>>>Date Duration Protocol
sles, rhat error load
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/07 23 hrs 40 min ASYNC
-4.96 sec, +4.42 sec -.006%,
>>>>    > +.005% cpu
>>>>    > >>>>>>>11/09 3 hrs 19 min ASYNC
-.13 sec, +1.44 sec, -.001%,
>>>>    > +.012% cpu
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/08 2 hrs 21 min SYNC
-.80 sec, -.34 sec, -.009%,
>>>>    -.004% cpu
>>>>    > >>>>>>>11/08 1 hr 25 min SYNC -.24
sec, -.26 sec,
>>>
>>> -.005%, -.005% cpu
>>>   
>>>
>>>>    > >>>>>>>11/12 65 hrs 40 min SYNC
-18 sec, -8 sec,
>>>
>>> -.008%, -.003% cpu
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/08 28 min MIXED -.75
sec, -.67 sec -.045%,
>>>
>>> -.040% cpu
>>>   
>>>
>>>>    > >>>>>>>11/08 15 hrs 39 min MIXED
-19. sec,-17.4 sec, -.034%,
>>>>    > -.031% cpu
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/14 17 hrs 17 min ASYNC
-6.1 sec,-55.7 sec, -.01%,
>>>>    > -.09% i/o-8
>>>>    > >>>>>>>11/15 2 hrs 44 min ASYNC
-1.47 sec,-14.0 sec, -.015%
>>>>    > -.14% i/o-8
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/13 15 hrs 38 min SYNC
-9.7 sec,-12.3 sec, -.017%,
>>>>    > -.022% i/o-8
>>>>    > >>>>>>>11/14 48 min SYNC - .46
sec, - .48 sec,
>>>
>>> -.017%, -.018% i/o-8
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/14 4 hrs 2 min MIXED
-2.9 sec, -4.15 sec, -.020%,
>>>>    > -.029% i/o-8
>>>>    > >>>>>>>11/20 16 hrs 2 min MIXED
-13.4 sec,-18.1 sec, -.023%,
>>>>    > -.031% i/o-8
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/21 28 min MIXED -2.01
sec, -.67 sec, -.12%,
>>>
>>> -.04% i/o-32
>>>   
>>>
>>>>    > >>>>>>>11/21 2 hrs 25 min SYNC
-.96 sec, -.43 sec, -.011%,
>>>>    > -.005% i/o-32
>>>>    > >>>>>>>11/21 40 min ASYNC -2.43
sec, -2.77 sec -.10%,
>>>
>>> -.11% i/o-32
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>11/26 113 hrs 46 min MIXED
-297. sec, 13. sec -.07%,
>>>>    > .003% i/o-4/32
>>>>    > >>>>>>>11/26 4 hrs 50 min SYNC
-3.21 sec, 1.44 sec, -.017%,
>>>>    > .01% i/o-4/32
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>Overhead measurements:
>>>>    > >>>>>>>
>>>>    > >>>>>>>Progress in terms of number
of passes through a fixed
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>system workload
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>on an 8 vcpu red hat with
an 8 vcpu sles idle.
>>>>    > >>>>>>>The workload was usex -b48.
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>ASYNC 167 min 145 passes
.868 passes/min
>>>>    > >>>>>>>SYNC 167 min 144 passes
.862 passes/min
>>>>    > >>>>>>>SYNC 1065 min 919 passes
.863 passes/min
>>>>    > >>>>>>>MIXED 221 min 196 passes
.887 passes/min
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>Conclusions:
>>>>    > >>>>>>>
>>>>    > >>>>>>>The only protocol which
meets the .05% accuracy
>>>>    > requirement for ntp
>>>>    > >>>>>>>tracking under the loads
>>>>    > >>>>>>>above is the SYNC protocol.
The worst case
>>>
>>> accuracies for
>>>   
>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>SYNC, MIXED,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>and ASYNC
>>>>    > >>>>>>>are .022%, .12%, and .14%,
respectively.
>>>>    > >>>>>>>
>>>>    > >>>>>>>We could reduce the cost of
the SYNC method by only
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>scheduling the extra
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>wakeups if a certain number
>>>>    > >>>>>>>of ticks are missed.
>>>>    > >>>>>>>
>>>>    > >>>>>>>Regards,
>>>>    > >>>>>>>Dave
>>>>    > >>>>>>>
>>>>    > >>>>>>>Keir Fraser wrote:
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>
>>>>    > >>>>>>>>On 9/11/07 19:22,
"Dave Winchell"
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>>Since I had a high
error (~.03%) for the
>>>
>>> ASYNC method a
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>couple of days ago,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>I ran another ASYNC
test. I think there may have
>>>>    > been something
>>>>    > >>>>>>>>>wrong with the code
I used a couple of days ago for
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>ASYNC. It may have been
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>missing the
immediate delivery of interrupt
>>>
>>> after context
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>switch in.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>My results indicate
that either SYNC or ASYNC give
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>acceptable accuracy,
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>each running
consistently around or under
>>>
>>> .01%. MIXED has
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>a fairly high
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>error of
>>>>    > >>>>>>>>>greater than .03%.
Probably too close to .05% ntp
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>threshold for comfort.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>I don''t
have an overnight run with SYNC. I
>>>
>>> plan to leave
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>SYNC running
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>over the weekend.
If you''d rather I can leave MIXED
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>running instead.
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>It may be too early
to pick the protocol and
>>>
>>> I can run
>>>   
>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>more overnight tests
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>>next week.
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>>
>>>>    > >>>>>>>>I''m a bit
worried about any unwanted side
>>>
>>> effects of the
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>SYNC+run_timer
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>approach -- e.g.,
whether timer wakeups will
>>>
>>> cause higher
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>system-wide CPU
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>contention. I find it
easier to think through the
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>implications of ASYNC.
I''m
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>surprised that MIXED
loses time, and is less
>>>
>>> accurate than
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>ASYNC. Perhaps it
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>delivers more timer
interrupts than the other
>>>
>>> approaches,
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>and each interrupt
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>event causes a small
accumulated error?
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>Overall I would
consider MIXED and ASYNC as
>>>
>>> favourites and
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>if the latter is
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>actually more accurate
then I can simply revert the
>>>>    > changeset that
>>>>    > >>>>>>>>implemented MIXED.
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>Perhaps rather than
running more of the same
>>>
>>> workloads you
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>could try idle
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>VCPUs and I/O bound
VCPUs (e.g., repeated
>>>
>>> large disc reads
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>to /dev/null)? We
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>don''t have any
data on workloads that aren''t
>>>
>>> CPU bound, so
>>>   
>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>that''s really an
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>>>obvious place to put
any further effort imo.
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>-- Keir
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    > >>>>>>>>
>>>>    >
>>>>>>_______________________________________________
>>>>    > >>>>>>Xen-devel mailing list
>>>>    > >>>>>>Xen-devel@lists.xensource.com
>>>>    >
>>>>>>http://lists.xensource.com/xen-devel
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>>
>>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06
15:36:07 2007 +0000
>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04
17:58:16 2008 -0500
>>>>    > >>@@ -58,7 +58,7 @@ static void
pt_process_missed_ticks(stru
>>>>    > >>
>>>>    > >>     missed_ticks = missed_ticks / (s_time_t)
>>>
>>> pt->period + 1;
>>>   
>>>
>>>>    > >>     if ( mode_is(pt->vcpu->domain,     
>>>
>>> no_missed_ticks_pending) )
>>>   
>>>
>>>>    > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
>>>>    > >>+        pt->do_not_freeze = 1;
>>>>    > >>     else
>>>>    > >>         pt->pending_intr_nr +=
missed_ticks;
>>>>    > >>     pt->scheduled += missed_ticks *
pt->period;
>>>>    > >>@@ -127,7 +127,12 @@ static void
pt_timer_fn(void *data)
>>>>    > >>
>>>>    > >>     pt_lock(pt);
>>>>    > >>
>>>>    > >>-    pt->pending_intr_nr++;
>>>>    > >>+    if ( mode_is(pt->vcpu->domain,     
>>>
>>> no_missed_ticks_pending) ) {
>>>   
>>>
>>>>    > >>+        pt->pending_intr_nr = 1;
>>>>    > >>+ pt->do_not_freeze = 0;
>>>>    > >>+    }
>>>>    > >>+    else
>>>>    > >>+ pt->pending_intr_nr++;
>>>>    > >>
>>>>    > >>     if ( !pt->one_shot )
>>>>    > >>     {
>>>>    > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct
vcpu *v, struct
>>>>    > >>         return;
>>>>    > >>     }
>>>>    > >>
>>>>    > >>-    pt->do_not_freeze = 0;
>>>>    > >>-
>>>>    > >>     if ( pt->one_shot )
>>>>    > >>     {
>>>>    > >>         pt->enabled = 0;
>>>>    > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct
vcpu
>>>
>>> *v, struct
>>>   
>>>
>>>>    > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>    > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
>>>>    > missed ticks */
>>>>    > >>         }
>>>>    > >>+ else if ( mode_is(v->domain,
no_missed_ticks_pending) ) {
>>>>    > >>+     pt->pending_intr_nr--;
>>>>    > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>    > >>+ }
>>>>    > >>         else
>>>>    > >>         {
>>>>    > >>             pt->last_plt_gtime +=
pt->period_cycles;
>>>>    > >>
>>>>    > >>
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    > >
>>>>    >
>>>>    >
>>>>    > _______________________________________________
>>>>    > Xen-devel mailing list
>>>>    > Xen-devel@lists.xensource.com
>>>>    > http://lists.xensource.com/xen-devel
>>>>    >
>>>>
>>>>     
>>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan, Deeepak,

It may be that the underlying clock error is too great for ntp
to handle. It would be useful if you did not run ntpd
and, instead did ntpdate -b <timeserver> at the start of the test
for each guest. Then capture the data as you have been doing.
If the drift is greater than .05%, then we need to address that.

Another option is, when running ntpd, to enable loop statistics in 
/etc/ntp.conf
by adding this to the file:

statistics loopstats
statsdir /var/lib/ntp/

Then you will see loop data in that directory.
Correlating the data in the loopstats files with the
peaks in skew would be interesting. You will see entries of the form

54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10

Where the second to last column is the Allan Deviation. When that
gets over 1000, ntpd is working pretty hard. However, I have not seen ntpd
completely loose it like you have.

I''m on vacation until Monday, and won''t be reading
email.

Thanks for all your work on this!

-Dave

Deepak Patel wrote:
>
>>
>> Is the graph for RHEL5u1-64? (I''ve never tested this one.)
>
>
> I do not know which graph was attached with this. But I saw this 
> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I 
> was running ltp tests continuously.
>
>> What was the behaviour of the other guests running?
>
>
> All pvm guests are fine. But behavior of most of the hvm guests were 
> as described.
>
>> If they had spikes, were they at the same wall time?
>
>
> No. They are not at the same wall time.
>
>> Were the other guests running ltp as well?
>>
> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp 
> continuously.
>
>> How are you measuring skew?
>
>
> I was collecting output of "ntpdate -q  <timeserver> every 300
seconds
> (5 minutes) and have created graph based on that.
>
>>
>> Are you running ntpd?
>>
> Yes. ntp was running on all the guests.
>
> I am investigating what causes this spikes and let everyone know what 
> are my findings.
>
> Thanks,
> Deepak
>
>> Anything that you can discover that would be in sync with
>> the spikes would be very helpful!
>>
>> The code that I test with is our product code, which is based
>> on 3.1. So it is possible that something in 3.2 other than vpt.c
>> is the cause. I can test with 3.2, if necessary.
>>
>> thanks,
>> Dave
>>
>>
>>
>> Dan Magenheimer wrote:
>>
>>> Hi Dave (Keir, see suggestion below) --
>>>
>>> Thanks!
>>>
>>> Turning off vhpet certainly helps a lot (though see below).
>>>
>>> I wonder if timekeeping with vhpet is so bad that it should be
>>> turned off by default (in 3.1, 3.2, and unstable) until it is
>>> fixed?  (I have a patch that defaults it off, can post it if
>>> there is agreement on the above point.)  The whole point of an
>>> HPET is to provide more precise timekeeping and if vhpet is
>>> worse than vpit, it can only confuse users.  Comments?
>>>  
>>>
>>> In your testing, are you just measuring % skew over a long
>>> period of time?
>>>  We are graphing the skew continuously and
>>> seeing periodic behavior that is unsettling, even with pit.
>>> See attached.  Though your algorithm recovers, the
"cliffs"
>>> could still cause real user problems.  I wonder if there is
>>> anything that can be done to make the "recovery" more
>>> responsive?
>>>
>>> We are looking into what part(s) of LTP is causing the cliffs.
>>>
>>> Thanks,
>>> Dan
>>>
>>>  
>>>
>>>> -----Original Message-----
>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>> To: dan.magenheimer@oracle.com
>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; 
>>>> deepak.patel@oracle.com;
>>>> akira.ijuin@oracle.com; Dave Winchell
>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>> pending
>>>> missed ticks
>>>>
>>>>
>>>> Dan,
>>>>
>>>> I guess I''m a bit out of date calling for clock=
usage.
>>>> Looking at linux 2.6.20.4 sources, I think you should specify
>>>> "clocksource=pit nohpet" on the linux guest bootline.
>>>>
>>>> You can leave the xen and dom0 bootlines as they are.
>>>> The xen and guest clocksources do not need to be the same.
>>>> In my tests, xen is using the hpet for its timekeeping and
>>>> that appears to be the default.
>>>>
>>>> When you boot the guests you should see
>>>>    time.c: Using PIT/TSC based timekeeping.
>>>> on the rh4u5-64 guest, and something similar on the others.
>>>>
>>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer
>>>> > 14.318MHz HPET.)
>>>>
>>>> This appears to be the xen state, which is fine.
>>>> I was wrongly assuming that this was the guest state.
>>>> You might want to look in your guest logs and see what they
were
>>>> picking
>>>> for a clock source.
>>>>
>>>> Regards,
>>>> Dave
>>>>
>>>>
>>>>
>>>>
>>>> Dan Magenheimer wrote:
>>>>
>>>>  
>>>>
>>>>> Thanks, I hadn''t realized that!  No wonder we
didn''t see the same
>>>>> improvement you saw!
>>>>>
>>>>>    
>>>>>
>>>>>> Try specifying clock=pit on the linux boot line...
>>>>>>       
>>>>>
>>>>>
>>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
>>>>
>>>>
>>>> command line or
>>>>  
>>>>
>>>>> "nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
>>>>
>>>>
>>>> dom0?) command
>>>>  
>>>>
>>>>> line?  Or both places?  Since the tests take awhile, it
>>>>
>>>>
>>>> would be nice
>>>>  
>>>>
>>>>> to get this right the first time.  Do the Xen and guest
>>>>
>>>>
>>>> clocksources need
>>>>  
>>>>
>>>>> to be the same?
>>>>>
>>>>> Thanks,
>>>>> Dan
>>>>>
>>>>> -----Original Message-----
>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>> *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that
disables
>>>>> pending missed ticks
>>>>>
>>>>>    Hi Dan,
>>>>>
>>>>>    Hpet timer does have a fairly large error, as I was
trying this
>>>>>    one recently.
>>>>>    I don''t remember what I got for error, but 1%
sounds
>>>>
>>>>
>>>> about right.
>>>>  
>>>>
>>>>>    The problem is that hpet is not built on top of vpt.c,
>>>>
>>>>
>>>> the module
>>>>  
>>>>
>>>>>    Keir and I did
>>>>>    all the recent work in, for its periodic timer needs.
Try
>>>>>    specifying clock=pit
>>>>>    on the linux boot line. If it still picks the hpet,
which it
>>>>>    might, let me know
>>>>>    and I''ll tell you how to get around this.
>>>>>
>>>>>    Regards,
>>>>>    Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>
>>>>
>>>> --------------------------------------------------------------
>>>> ----------
>>>>  
>>>>
>>>>>    *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
>>>>>    *Sent:* Fri 1/25/2008 6:50 PM
>>>>>    *To:* Dave Winchell; Keir Fraser
>>>>>    *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>    akira.ijuin@oracle.com
>>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode     
>>>>
>>>>
>>>> that disables
>>>>  
>>>>
>>>>>    pending missed ticks
>>>>>
>>>>>    Sorry for the very late followup on this but we finally
>>>>
>>>>
>>>> were able
>>>>  
>>>>
>>>>>    to get our testing set up again on stable 3.1 bits and
have
>>>>>    seen some very bad results on 3.1.3-rc1, on the order of
1%.
>>>>>
>>>>>    Test enviroment was a 4-socket dual core machine with
24GB of
>>>>>    memory running six two-vcpu 2GB domains, four hvm plus
two pv.
>>>>>    All six guests were running LTP simultaneously.  The
four hvm
>>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
RHEL4u5-64.
>>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for     
>>>>
>>>>
>>>> 32-bit guests.
>>>>  
>>>>
>>>>>    All four hvm guests experienced skew around -1%, even
the 32-bit
>>>>>    guest.  Less intensive testing didn''t exhibit
much skew at all.
>>>>>
>>>>>    A representative graph is attached.
>>>>>
>>>>>    Dave, I wonder if some portion of your patches
didn''t end up in
>>>>>    the xen trees?
>>>>>
>>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform
timer
>>>>>    14.318MHz HPET.)
>>>>>
>>>>>    Thanks,
>>>>>    Dan
>>>>>
>>>>>    P.S. Many thanks to Deepak and Akira for running tests.
>>>>>
>>>>>    > -----Original Message-----
>>>>>    > From: xen-devel-bounces@lists.xensource.com
>>>>>    > [mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
>>>>>    > Dave Winchell
>>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>    > To: Keir Fraser
>>>>>    > Cc: dan.magenheimer@oracle.com;     
>>>>
>>>>
>>>> xen-devel@lists.xensource.com; Dave
>>>>  
>>>>
>>>>>    > Winchell
>>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
>>>>>    > disables pending
>>>>>    > missed ticks
>>>>>    >
>>>>>    >
>>>>>    > Hi Keir,
>>>>>    >
>>>>>    > The latest change, c/s 16690, looks fine.
>>>>>    > I agree that the code in c/s 16690 is equivalent to
>>>>>    > the code I submitted. Also, your version is more
>>>>>    > concise.
>>>>>    >
>>>>>    > The error tests confirm the equivalence. With     
>>>>
>>>>
>>>> overnight cpu loads,
>>>>  
>>>>
>>>>>    > the checked in version was accurate to +.048% for
sles
>>>>>    > and +.038% for red hat. My version was +.046% and
+.032% in a
>>>>>    > 2 hour test.
>>>>>    > I don''t think the difference is
significant.
>>>>>    >
>>>>>    > i/o loads produced errors of +.01%.
>>>>>    >
>>>>>    > Thanks for all your efforts on this issue.
>>>>>    >
>>>>>    > Regards,
>>>>>    > Dave
>>>>>    >
>>>>>    >
>>>>>    >
>>>>>    > Keir Fraser wrote:
>>>>>    >
>>>>>    > >Applied as c/s 16690, although the checked-in
patch is
>>>>>    > smaller. I think the
>>>>>    > >only important fix is to pt_intr_post() and the
only bit of
>>>>>    > the patch I
>>>>>    > >totally omitted was the change to
pt_process_missed_ticks().
>>>>>    > I don''t think
>>>>>    > >that change can be important, but
let''s see what
>>>>
>>>>
>>>> happens to the
>>>>  
>>>>
>>>>>    error
>>>>>    > >percentage...
>>>>>    > >
>>>>>    > > -- Keir
>>>>>    > >
>>>>>    > >On 4/1/08 23:24, "Dave Winchell"     
>>>>
>>>>
>>>> <dwinchell@virtualiron.com> wrote:
>>>>  
>>>>
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    > >>Hi Dan and Keir,
>>>>>    > >>
>>>>>    > >>Attached is a patch that fixes some issues
with the
>>>>
>>>>
>>>> SYNC policy
>>>>  
>>>>
>>>>>    > >>(no_missed_ticks_pending).
>>>>>    > >>I have not tried to make the change the
minimal one, but,
>>>>>    > rather, just
>>>>>    > >>ported into
>>>>>    > >>the new code what I know to work well. The
error for
>>>>>    > >>no_missed_ticks_pending goes from
>>>>>    > >>over 3% to .03% with this change according
to my testing.
>>>>>    > >>
>>>>>    > >>Regards,
>>>>>    > >>Dave
>>>>>    > >>
>>>>>    > >>Dan Magenheimer wrote:
>>>>>    > >>
>>>>>    > >>
>>>>>    > >>
>>>>>    > >>>Hi Dave --
>>>>>    > >>>
>>>>>    > >>>Did you get your correction ported?  If
so, it would be
>>>>>    > nice to see this get
>>>>>    > >>>into 3.1.3.
>>>>>    > >>>
>>>>>    > >>>Note that I just did some very limited
testing with
>>>>>    > timer_mode=2(=SYNC=no
>>>>>    > >>>missed ticks pending)
>>>>>    > >>>on tip of xen-3.1-testing (64-bit Linux
hv guest) and the
>>>>>    > worst error I''ve
>>>>>    > >>>seen so far
>>>>>    > >>>is 0.012%.  But I haven''t
tried any exotic loads, just LTP.
>>>>>    > >>>
>>>>>    > >>>Thanks,
>>>>>    > >>>Dan
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>>-----Original Message-----
>>>>>    > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>    > >>>>Sent: Wednesday, December 19, 2007
12:33 PM
>>>>>    > >>>>To: dan.magenheimer@oracle.com
>>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>    > xen-devel@lists.xensource.com; Dong,
>>>>>    > >>>>Eddie; Jiang, Yunhong; Dave
Winchell
>>>>>    > >>>>Subject: Re: [Xen-devel] [PATCH]
Add a timer mode that
>>>>>    > >>>>disables pending
>>>>>    > >>>>missed ticks
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>Dan,
>>>>>    > >>>>
>>>>>    > >>>>I did some testing with the
constant tsc offset
>>>>
>>>>
>>>> SYNC method
>>>>  
>>>>
>>>>>    > >>>>(now called
>>>>>    > >>>>no_missed_ticks_pending)
>>>>>    > >>>>and found the error to be very
high, much larger
>>>>
>>>>
>>>> than 1 %, as
>>>>  
>>>>
>>>>>    > >>>>I recall.
>>>>>    > >>>>I have not had a chance to submit a
correction. I
>>>>
>>>>
>>>> will try to
>>>>  
>>>>
>>>>>    > >>>>do it later
>>>>>    > >>>>this week or the first week in
January. My version of
>>>>>    constant tsc
>>>>>    > >>>>offset SYNC method
>>>>>    > >>>>produces .02 % error, so I just
need to port that into the
>>>>>    > >>>>current code.
>>>>>    > >>>>
>>>>>    > >>>>The error you got for both of those
kernels is
>>>>
>>>>
>>>> what I would
>>>>  
>>>>
>>>>>    expect
>>>>>    > >>>>for the default mode,
delay_for_missed_ticks.
>>>>>    > >>>>
>>>>>    > >>>>I''ll let Keir answer on
how to set the time mode.
>>>>>    > >>>>
>>>>>    > >>>>Regards,
>>>>>    > >>>>Dave
>>>>>    > >>>>
>>>>>    > >>>>Dan Magenheimer wrote:
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>Anyone make measurements on the
final patch?
>>>>>    > >>>>>
>>>>>    > >>>>>I just ran a 64-bit RHEL5.1 pvm
kernel and saw a loss of
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>about 0.2% with no load.  This was
xen-unstable tip today
>>>>>    > >>>>with no options specified.  32-bit
was about 0.01%.
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>I think I missed something...
how do I run the various
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>accounting choices and which ones
are known to be
>>>>
>>>>
>>>> appropriate
>>>>  
>>>>
>>>>>    > >>>>for which kernels?
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>Thanks,
>>>>>    > >>>>>Dan
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>>-----Original Message-----
>>>>>    > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>    >    
>>>>>
>>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>Keir Fraser
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>>Sent: Thursday, December
06, 2007 4:57 AM
>>>>>    > >>>>>>To: Dave Winchell
>>>>>    > >>>>>>Cc: Shan, Haitao;
xen-devel@lists.xensource.com; Dong,
>>>>>    > Eddie; Jiang,
>>>>>    > >>>>>>Yunhong
>>>>>    > >>>>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
>>>>>    > >>>>>>disables pending
>>>>>    > >>>>>>missed ticks
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>Please take a look at
xen-unstable changeset 16545.
>>>>>    > >>>>>>
>>>>>    > >>>>>>-- Keir
>>>>>    > >>>>>>
>>>>>    > >>>>>>On 26/11/07 20:57,
"Dave Winchell"
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>><dwinchell@virtualiron.com>
wrote:
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>Keir,
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The accuracy data
I''ve collected for i/o loads for the
>>>>>    > >>>>>>>various time protocols
follows. In addition, the data
>>>>>    > >>>>>>>for cpu loads is shown.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled cpu
and i/o-8 are on an 8
>>>>
>>>>
>>>> processor AMD
>>>>  
>>>>
>>>>>    box.
>>>>>    > >>>>>>>Two guests, red hat and
sles 64 bit, 8 vcpu each.
>>>>>    > >>>>>>>The cpu load is usex
-e36 on each guest.
>>>>>    > >>>>>>>(usex is available at
>>>>>    http://people.redhat.com/anderson/usex.)
>>>>>    > >>>>>>>i/o load is 8 instances
of dd if=/dev/hda6
>>>>
>>>>
>>>> of=/dev/null.
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled
i/o-32 are 32 instances of dd.
>>>>>    > >>>>>>>Also, these are run on
4 cpu AMD box.
>>>>>    > >>>>>>>In addition, there is
an idle rh-32bit guest.
>>>>>    > >>>>>>>All three guests are
8vcpu.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The loads labeled
i/o-4/32 are the same as i/o-32
>>>>>    > >>>>>>>except that the
redhat-64 guest has 4 instances of dd.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Date Duration Protocol
sles, rhat error load
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/07 23 hrs 40 min
ASYNC -4.96 sec, +4.42 sec -.006%,
>>>>>    > +.005% cpu
>>>>>    > >>>>>>>11/09 3 hrs 19 min
ASYNC -.13 sec, +1.44 sec, -.001%,
>>>>>    > +.012% cpu
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/08 2 hrs 21 min SYNC
-.80 sec, -.34 sec, -.009%,
>>>>>    -.004% cpu
>>>>>    > >>>>>>>11/08 1 hr 25 min SYNC
-.24 sec, -.26 sec,
>>>>
>>>>
>>>> -.005%, -.005% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>11/12 65 hrs 40 min
SYNC -18 sec, -8 sec,
>>>>
>>>>
>>>> -.008%, -.003% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/08 28 min MIXED -.75
sec, -.67 sec -.045%,
>>>>
>>>>
>>>> -.040% cpu
>>>>  
>>>>
>>>>>    > >>>>>>>11/08 15 hrs 39 min
MIXED -19. sec,-17.4 sec, -.034%,
>>>>>    > -.031% cpu
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/14 17 hrs 17 min
ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>    > -.09% i/o-8
>>>>>    > >>>>>>>11/15 2 hrs 44 min
ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>    > -.14% i/o-8
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/13 15 hrs 38 min
SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>    > -.022% i/o-8
>>>>>    > >>>>>>>11/14 48 min SYNC - .46
sec, - .48 sec,
>>>>
>>>>
>>>> -.017%, -.018% i/o-8
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/14 4 hrs 2 min MIXED
-2.9 sec, -4.15 sec, -.020%,
>>>>>    > -.029% i/o-8
>>>>>    > >>>>>>>11/20 16 hrs 2 min
MIXED -13.4 sec,-18.1 sec, -.023%,
>>>>>    > -.031% i/o-8
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/21 28 min MIXED
-2.01 sec, -.67 sec, -.12%,
>>>>
>>>>
>>>> -.04% i/o-32
>>>>  
>>>>
>>>>>    > >>>>>>>11/21 2 hrs 25 min SYNC
-.96 sec, -.43 sec, -.011%,
>>>>>    > -.005% i/o-32
>>>>>    > >>>>>>>11/21 40 min ASYNC
-2.43 sec, -2.77 sec -.10%,
>>>>
>>>>
>>>> -.11% i/o-32
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>11/26 113 hrs 46 min
MIXED -297. sec, 13. sec -.07%,
>>>>>    > .003% i/o-4/32
>>>>>    > >>>>>>>11/26 4 hrs 50 min SYNC
-3.21 sec, 1.44 sec, -.017%,
>>>>>    > .01% i/o-4/32
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Overhead measurements:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Progress in terms of
number of passes through a fixed
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>system workload
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>on an 8 vcpu red hat
with an 8 vcpu sles idle.
>>>>>    > >>>>>>>The workload was usex
-b48.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>ASYNC 167 min 145
passes .868 passes/min
>>>>>    > >>>>>>>SYNC 167 min 144 passes
.862 passes/min
>>>>>    > >>>>>>>SYNC 1065 min 919
passes .863 passes/min
>>>>>    > >>>>>>>MIXED 221 min 196
passes .887 passes/min
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Conclusions:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>The only protocol which
meets the .05% accuracy
>>>>>    > requirement for ntp
>>>>>    > >>>>>>>tracking under the
loads
>>>>>    > >>>>>>>above is the SYNC
protocol. The worst case
>>>>
>>>>
>>>> accuracies for
>>>>  
>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>SYNC, MIXED,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>and ASYNC
>>>>>    > >>>>>>>are .022%, .12%, and
.14%, respectively.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>We could reduce the
cost of the SYNC method by only
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>scheduling the extra
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>wakeups if a certain
number
>>>>>    > >>>>>>>of ticks are missed.
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Regards,
>>>>>    > >>>>>>>Dave
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>Keir Fraser wrote:
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>
>>>>>    > >>>>>>>>On 9/11/07 19:22,
"Dave Winchell"
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>>Since I had a
high error (~.03%) for the
>>>>
>>>>
>>>> ASYNC method a
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>couple of days ago,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>I ran another
ASYNC test. I think there may have
>>>>>    > been something
>>>>>    > >>>>>>>>>wrong with the
code I used a couple of days ago for
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>ASYNC. It may have been
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>missing the
immediate delivery of interrupt
>>>>
>>>>
>>>> after context
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>switch in.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>My results
indicate that either SYNC or ASYNC give
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>acceptable accuracy,
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>each running
consistently around or under
>>>>
>>>>
>>>> .01%. MIXED has
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>a fairly high
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>error of
>>>>>    > >>>>>>>>>greater than
.03%. Probably too close to .05% ntp
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>threshold for comfort.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>I
don''t have an overnight run with SYNC. I
>>>>
>>>>
>>>> plan to leave
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>SYNC running
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>over the
weekend. If you''d rather I can leave MIXED
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>running instead.
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>It may be too
early to pick the protocol and
>>>>
>>>>
>>>> I can run
>>>>  
>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>more overnight tests
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>>next week.
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>>
>>>>>    > >>>>>>>>I''m a bit
worried about any unwanted side
>>>>
>>>>
>>>> effects of the
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>SYNC+run_timer
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>approach -- e.g.,
whether timer wakeups will
>>>>
>>>>
>>>> cause higher
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>system-wide CPU
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>contention. I find
it easier to think through the
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>implications of ASYNC.
I''m
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>surprised that
MIXED loses time, and is less
>>>>
>>>>
>>>> accurate than
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>ASYNC. Perhaps it
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>delivers more timer
interrupts than the other
>>>>
>>>>
>>>> approaches,
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>and each interrupt
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>event causes a
small accumulated error?
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>Overall I would
consider MIXED and ASYNC as
>>>>
>>>>
>>>> favourites and
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>if the latter is
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>actually more
accurate then I can simply revert the
>>>>>    > changeset that
>>>>>    > >>>>>>>>implemented MIXED.
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>Perhaps rather than
running more of the same
>>>>
>>>>
>>>> workloads you
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>could try idle
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>VCPUs and I/O bound
VCPUs (e.g., repeated
>>>>
>>>>
>>>> large disc reads
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>to /dev/null)? We
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>don''t have
any data on workloads that aren''t
>>>>
>>>>
>>>> CPU bound, so
>>>>  
>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>that''s really an
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>>>obvious place to
put any further effort imo.
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>-- Keir
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    > >>>>>>>>
>>>>>    >
>>>>>>_______________________________________________
>>>>>    > >>>>>>Xen-devel mailing list
>>>>>    >
>>>>>>Xen-devel@lists.xensource.com
>>>>>    >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>>
>>>>>    > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06
15:36:07 2007 +0000
>>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04
17:58:16 2008 -0500
>>>>>    > >>@@ -58,7 +58,7 @@ static void
pt_process_missed_ticks(stru
>>>>>    > >>
>>>>>    > >>     missed_ticks = missed_ticks /
(s_time_t)
>>>>
>>>>
>>>> pt->period + 1;
>>>>  
>>>>
>>>>>    > >>     if ( mode_is(pt->vcpu->domain,
>>>>
>>>>
>>>> no_missed_ticks_pending) )
>>>>  
>>>>
>>>>>    > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
>>>>>    > >>+        pt->do_not_freeze = 1;
>>>>>    > >>     else
>>>>>    > >>         pt->pending_intr_nr +=
missed_ticks;
>>>>>    > >>     pt->scheduled += missed_ticks *
pt->period;
>>>>>    > >>@@ -127,7 +127,12 @@ static void
pt_timer_fn(void *data)
>>>>>    > >>
>>>>>    > >>     pt_lock(pt);
>>>>>    > >>
>>>>>    > >>-    pt->pending_intr_nr++;
>>>>>    > >>+    if ( mode_is(pt->vcpu->domain,
>>>>
>>>>
>>>> no_missed_ticks_pending) ) {
>>>>  
>>>>
>>>>>    > >>+        pt->pending_intr_nr = 1;
>>>>>    > >>+ pt->do_not_freeze = 0;
>>>>>    > >>+    }
>>>>>    > >>+    else
>>>>>    > >>+ pt->pending_intr_nr++;
>>>>>    > >>
>>>>>    > >>     if ( !pt->one_shot )
>>>>>    > >>     {
>>>>>    > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct vcpu *v, struct
>>>>>    > >>         return;
>>>>>    > >>     }
>>>>>    > >>
>>>>>    > >>-    pt->do_not_freeze = 0;
>>>>>    > >>-
>>>>>    > >>     if ( pt->one_shot )
>>>>>    > >>     {
>>>>>    > >>         pt->enabled = 0;
>>>>>    > >>@@ -235,6 +238,10 @@ void
pt_intr_post(struct vcpu
>>>>
>>>>
>>>> *v, struct
>>>>  
>>>>
>>>>>    > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>>    > >>             pt->pending_intr_nr = 0; /*
''collapse'' all
>>>>>    > missed ticks */
>>>>>    > >>         }
>>>>>    > >>+ else if ( mode_is(v->domain,
no_missed_ticks_pending) ) {
>>>>>    > >>+     pt->pending_intr_nr--;
>>>>>    > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>>    > >>+ }
>>>>>    > >>         else
>>>>>    > >>         {
>>>>>    > >>             pt->last_plt_gtime +=
pt->period_cycles;
>>>>>    > >>
>>>>>    > >>
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    > >
>>>>>    >
>>>>>    >
>>>>>    > _______________________________________________
>>>>>    > Xen-devel mailing list
>>>>>    > Xen-devel@lists.xensource.com
>>>>>    > http://lists.xensource.com/xen-devel
>>>>>    >
>>>>>
>>>>>     
>>>>
>>>>
>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

OK, Deepak repeated the test without ntpd and using ntpdate -b before
the test.

The attached graph shows his results: el5u1-64 (best=~0.07%),
el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).

We will continue to look at LTP to try to isolate.

Thanks,
Dan

P.S. elXuY is essentially RHEL XuY with some patches.
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Wednesday, January 30, 2008 2:45 PM
> To: Deepak Patel
> Cc: dan.magenheimer@oracle.com; Keir Fraser;
> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan, Deeepak,
> 
> It may be that the underlying clock error is too great for ntp
> to handle. It would be useful if you did not run ntpd
> and, instead did ntpdate -b <timeserver> at the start of the test
> for each guest. Then capture the data as you have been doing.
> If the drift is greater than .05%, then we need to address that.
> 
> Another option is, when running ntpd, to enable loop statistics in
> /etc/ntp.conf
> by adding this to the file:
> 
> statistics loopstats
> statsdir /var/lib/ntp/
> 
> Then you will see loop data in that directory.
> Correlating the data in the loopstats files with the
> peaks in skew would be interesting. You will see entries of the form
> 
> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
> 
> Where the second to last column is the Allan Deviation. When that
> gets over 1000, ntpd is working pretty hard. However, I have 
> not seen ntpd
> completely loose it like you have.
> 
> I''m on vacation until Monday, and won''t be reading
> email.
> 
> Thanks for all your work on this!
> 
> -Dave
> 
> Deepak Patel wrote:
> 
> >
> >>
> >> Is the graph for RHEL5u1-64? (I''ve never tested this
one.)
> >
> >
> > I do not know which graph was attached with this. But I saw this
> > behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
> > was running ltp tests continuously.
> >
> >> What was the behaviour of the other guests running?
> >
> >
> > All pvm guests are fine. But behavior of most of the hvm guests were
> > as described.
> >
> >> If they had spikes, were they at the same wall time?
> >
> >
> > No. They are not at the same wall time.
> >
> >> Were the other guests running ltp as well?
> >>
> > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
> > continuously.
> >
> >> How are you measuring skew?
> >
> >
> > I was collecting output of "ntpdate -q  <timeserver> every 
> 300 seconds
> > (5 minutes) and have created graph based on that.
> >
> >>
> >> Are you running ntpd?
> >>
> > Yes. ntp was running on all the guests.
> >
> > I am investigating what causes this spikes and let everyone 
> know what
> > are my findings.
> >
> > Thanks,
> > Deepak
> >
> >> Anything that you can discover that would be in sync with
> >> the spikes would be very helpful!
> >>
> >> The code that I test with is our product code, which is based
> >> on 3.1. So it is possible that something in 3.2 other than vpt.c
> >> is the cause. I can test with 3.2, if necessary.
> >>
> >> thanks,
> >> Dave
> >>
> >>
> >>
> >> Dan Magenheimer wrote:
> >>
> >>> Hi Dave (Keir, see suggestion below) --
> >>>
> >>> Thanks!
> >>>
> >>> Turning off vhpet certainly helps a lot (though see below).
> >>>
> >>> I wonder if timekeeping with vhpet is so bad that it should be
> >>> turned off by default (in 3.1, 3.2, and unstable) until it is
> >>> fixed?  (I have a patch that defaults it off, can post it if
> >>> there is agreement on the above point.)  The whole point of an
> >>> HPET is to provide more precise timekeeping and if vhpet is
> >>> worse than vpit, it can only confuse users.  Comments?
> >>>
> >>>
> >>> In your testing, are you just measuring % skew over a long
> >>> period of time?
> >>>  We are graphing the skew continuously and
> >>> seeing periodic behavior that is unsettling, even with pit.
> >>> See attached.  Though your algorithm recovers, the
"cliffs"
> >>> could still cause real user problems.  I wonder if there is
> >>> anything that can be done to make the "recovery"
more
> >>> responsive?
> >>>
> >>> We are looking into what part(s) of LTP is causing the cliffs.
> >>>
> >>> Thanks,
> >>> Dan
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>> Sent: Monday, January 28, 2008 8:21 AM
> >>>> To: dan.magenheimer@oracle.com
> >>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
> >>>> deepak.patel@oracle.com;
> >>>> akira.ijuin@oracle.com; Dave Winchell
> >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables
> >>>> pending
> >>>> missed ticks
> >>>>
> >>>>
> >>>> Dan,
> >>>>
> >>>> I guess I''m a bit out of date calling for clock=
usage.
> >>>> Looking at linux 2.6.20.4 sources, I think you should
specify
> >>>> "clocksource=pit nohpet" on the linux guest
bootline.
> >>>>
> >>>> You can leave the xen and dom0 bootlines as they are.
> >>>> The xen and guest clocksources do not need to be the same.
> >>>> In my tests, xen is using the hpet for its timekeeping and
> >>>> that appears to be the default.
> >>>>
> >>>> When you boot the guests you should see
> >>>>    time.c: Using PIT/TSC based timekeeping.
> >>>> on the rh4u5-64 guest, and something similar on the
others.
> >>>>
> >>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform
timer
> >>>> > 14.318MHz HPET.)
> >>>>
> >>>> This appears to be the xen state, which is fine.
> >>>> I was wrongly assuming that this was the guest state.
> >>>> You might want to look in your guest logs and see what
they were
> >>>> picking
> >>>> for a clock source.
> >>>>
> >>>> Regards,
> >>>> Dave
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Dan Magenheimer wrote:
> >>>>
> >>>>
> >>>>
> >>>>> Thanks, I hadn''t realized that!  No wonder we
didn''t
> see the same
> >>>>> improvement you saw!
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Try specifying clock=pit on the linux boot line...
> >>>>>>
> >>>>>
> >>>>>
> >>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
> >>>>
> >>>>
> >>>> command line or
> >>>>
> >>>>
> >>>>> "nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
> >>>>
> >>>>
> >>>> dom0?) command
> >>>>
> >>>>
> >>>>> line?  Or both places?  Since the tests take awhile,
it
> >>>>
> >>>>
> >>>> would be nice
> >>>>
> >>>>
> >>>>> to get this right the first time.  Do the Xen and
guest
> >>>>
> >>>>
> >>>> clocksources need
> >>>>
> >>>>
> >>>>> to be the same?
> >>>>>
> >>>>> Thanks,
> >>>>> Dan
> >>>>>
> >>>>> -----Original Message-----
> >>>>> *From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
> >>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
> >>>>> *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
> >>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
> that disables
> >>>>> pending missed ticks
> >>>>>
> >>>>>    Hi Dan,
> >>>>>
> >>>>>    Hpet timer does have a fairly large error, as I was
> trying this
> >>>>>    one recently.
> >>>>>    I don''t remember what I got for error, but
1% sounds
> >>>>
> >>>>
> >>>> about right.
> >>>>
> >>>>
> >>>>>    The problem is that hpet is not built on top of
vpt.c,
> >>>>
> >>>>
> >>>> the module
> >>>>
> >>>>
> >>>>>    Keir and I did
> >>>>>    all the recent work in, for its periodic timer
needs. Try
> >>>>>    specifying clock=pit
> >>>>>    on the linux boot line. If it still picks the hpet,
which it
> >>>>>    might, let me know
> >>>>>    and I''ll tell you how to get around this.
> >>>>>
> >>>>>    Regards,
> >>>>>    Dave
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
--------------------------------------------------------------
> >>>> ----------
> >>>>
> >>>>
> >>>>>    *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
> >>>>>    *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>    *To:* Dave Winchell; Keir Fraser
> >>>>>    *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
> >>>>>    akira.ijuin@oracle.com
> >>>>>    *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>
> >>>>
> >>>> that disables
> >>>>
> >>>>
> >>>>>    pending missed ticks
> >>>>>
> >>>>>    Sorry for the very late followup on this but we
finally
> >>>>
> >>>>
> >>>> were able
> >>>>
> >>>>
> >>>>>    to get our testing set up again on stable 3.1 bits
and have
> >>>>>    seen some very bad results on 3.1.3-rc1, on the
order of 1%.
> >>>>>
> >>>>>    Test enviroment was a 4-socket dual core machine
with 24GB of
> >>>>>    memory running six two-vcpu 2GB domains, four hvm 
> plus two pv.
> >>>>>    All six guests were running LTP simultaneously. 
The four hvm
> >>>>>    guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and 
> RHEL4u5-64.
> >>>>>    Timer_mode was set to 2 for 64-bit guests and 0 for
> >>>>
> >>>>
> >>>> 32-bit guests.
> >>>>
> >>>>
> >>>>>    All four hvm guests experienced skew around -1%, 
> even the 32-bit
> >>>>>    guest.  Less intensive testing didn''t
exhibit much
> skew at all.
> >>>>>
> >>>>>    A representative graph is attached.
> >>>>>
> >>>>>    Dave, I wonder if some portion of your patches 
> didn''t end up in
> >>>>>    the xen trees?
> >>>>>
> >>>>>    (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
Platform timer
> >>>>>    14.318MHz HPET.)
> >>>>>
> >>>>>    Thanks,
> >>>>>    Dan
> >>>>>
> >>>>>    P.S. Many thanks to Deepak and Akira for running
tests.
> >>>>>
> >>>>>    > -----Original Message-----
> >>>>>    > From: xen-devel-bounces@lists.xensource.com
> >>>>>    >
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>    > Dave Winchell
> >>>>>    > Sent: Wednesday, January 09, 2008 9:53 AM
> >>>>>    > To: Keir Fraser
> >>>>>    > Cc: dan.magenheimer@oracle.com;
> >>>>
> >>>>
> >>>> xen-devel@lists.xensource.com; Dave
> >>>>
> >>>>
> >>>>>    > Winchell
> >>>>>    > Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that
> >>>>>    > disables pending
> >>>>>    > missed ticks
> >>>>>    >
> >>>>>    >
> >>>>>    > Hi Keir,
> >>>>>    >
> >>>>>    > The latest change, c/s 16690, looks fine.
> >>>>>    > I agree that the code in c/s 16690 is
equivalent to
> >>>>>    > the code I submitted. Also, your version is
more
> >>>>>    > concise.
> >>>>>    >
> >>>>>    > The error tests confirm the equivalence. With
> >>>>
> >>>>
> >>>> overnight cpu loads,
> >>>>
> >>>>
> >>>>>    > the checked in version was accurate to +.048%
for sles
> >>>>>    > and +.038% for red hat. My version was +.046%
and
> +.032% in a
> >>>>>    > 2 hour test.
> >>>>>    > I don''t think the difference is
significant.
> >>>>>    >
> >>>>>    > i/o loads produced errors of +.01%.
> >>>>>    >
> >>>>>    > Thanks for all your efforts on this issue.
> >>>>>    >
> >>>>>    > Regards,
> >>>>>    > Dave
> >>>>>    >
> >>>>>    >
> >>>>>    >
> >>>>>    > Keir Fraser wrote:
> >>>>>    >
> >>>>>    > >Applied as c/s 16690, although the
checked-in patch is
> >>>>>    > smaller. I think the
> >>>>>    > >only important fix is to pt_intr_post()
and the
> only bit of
> >>>>>    > the patch I
> >>>>>    > >totally omitted was the change to 
> pt_process_missed_ticks().
> >>>>>    > I don''t think
> >>>>>    > >that change can be important, but
let''s see what
> >>>>
> >>>>
> >>>> happens to the
> >>>>
> >>>>
> >>>>>    error
> >>>>>    > >percentage...
> >>>>>    > >
> >>>>>    > > -- Keir
> >>>>>    > >
> >>>>>    > >On 4/1/08 23:24, "Dave Winchell"
> >>>>
> >>>>
> >>>> <dwinchell@virtualiron.com> wrote:
> >>>>
> >>>>
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >>Hi Dan and Keir,
> >>>>>    > >>
> >>>>>    > >>Attached is a patch that fixes some
issues with the
> >>>>
> >>>>
> >>>> SYNC policy
> >>>>
> >>>>
> >>>>>    > >>(no_missed_ticks_pending).
> >>>>>    > >>I have not tried to make the change
the minimal one, but,
> >>>>>    > rather, just
> >>>>>    > >>ported into
> >>>>>    > >>the new code what I know to work well.
The error for
> >>>>>    > >>no_missed_ticks_pending goes from
> >>>>>    > >>over 3% to .03% with this change
according to my testing.
> >>>>>    > >>
> >>>>>    > >>Regards,
> >>>>>    > >>Dave
> >>>>>    > >>
> >>>>>    > >>Dan Magenheimer wrote:
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >>>Hi Dave --
> >>>>>    > >>>
> >>>>>    > >>>Did you get your correction
ported?  If so, it would be
> >>>>>    > nice to see this get
> >>>>>    > >>>into 3.1.3.
> >>>>>    > >>>
> >>>>>    > >>>Note that I just did some very
limited testing with
> >>>>>    > timer_mode=2(=SYNC=no
> >>>>>    > >>>missed ticks pending)
> >>>>>    > >>>on tip of xen-3.1-testing (64-bit
Linux hv
> guest) and the
> >>>>>    > worst error I''ve
> >>>>>    > >>>seen so far
> >>>>>    > >>>is 0.012%.  But I haven''t
tried any exotic
> loads, just LTP.
> >>>>>    > >>>
> >>>>>    > >>>Thanks,
> >>>>>    > >>>Dan
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>>-----Original Message-----
> >>>>>    > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>    > >>>>Sent: Wednesday, December 19,
2007 12:33 PM
> >>>>>    > >>>>To: dan.magenheimer@oracle.com
> >>>>>    > >>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>    > xen-devel@lists.xensource.com; Dong,
> >>>>>    > >>>>Eddie; Jiang, Yunhong; Dave
Winchell
> >>>>>    > >>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
> >>>>>    > >>>>disables pending
> >>>>>    > >>>>missed ticks
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>Dan,
> >>>>>    > >>>>
> >>>>>    > >>>>I did some testing with the
constant tsc offset
> >>>>
> >>>>
> >>>> SYNC method
> >>>>
> >>>>
> >>>>>    > >>>>(now called
> >>>>>    > >>>>no_missed_ticks_pending)
> >>>>>    > >>>>and found the error to be very
high, much larger
> >>>>
> >>>>
> >>>> than 1 %, as
> >>>>
> >>>>
> >>>>>    > >>>>I recall.
> >>>>>    > >>>>I have not had a chance to
submit a correction. I
> >>>>
> >>>>
> >>>> will try to
> >>>>
> >>>>
> >>>>>    > >>>>do it later
> >>>>>    > >>>>this week or the first week in
January. My version of
> >>>>>    constant tsc
> >>>>>    > >>>>offset SYNC method
> >>>>>    > >>>>produces .02 % error, so I
just need to port
> that into the
> >>>>>    > >>>>current code.
> >>>>>    > >>>>
> >>>>>    > >>>>The error you got for both of
those kernels is
> >>>>
> >>>>
> >>>> what I would
> >>>>
> >>>>
> >>>>>    expect
> >>>>>    > >>>>for the default mode,
delay_for_missed_ticks.
> >>>>>    > >>>>
> >>>>>    > >>>>I''ll let Keir answer
on how to set the time mode.
> >>>>>    > >>>>
> >>>>>    > >>>>Regards,
> >>>>>    > >>>>Dave
> >>>>>    > >>>>
> >>>>>    > >>>>Dan Magenheimer wrote:
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>Anyone make measurements
on the final patch?
> >>>>>    > >>>>>
> >>>>>    > >>>>>I just ran a 64-bit
RHEL5.1 pvm kernel and
> saw a loss of
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>about 0.2% with no load.  This
was
> xen-unstable tip today
> >>>>>    > >>>>with no options specified. 
32-bit was about 0.01%.
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>I think I missed
something... how do I run the various
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>accounting choices and which
ones are known to be
> >>>>
> >>>>
> >>>> appropriate
> >>>>
> >>>>
> >>>>>    > >>>>for which kernels?
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>Thanks,
> >>>>>    > >>>>>Dan
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>>-----Original
Message-----
> >>>>>    > >>>>>>From:
xen-devel-bounces@lists.xensource.com
> >>>>>    >
> >>>>>
> >>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>Keir Fraser
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>>Sent: Thursday,
December 06, 2007 4:57 AM
> >>>>>    > >>>>>>To: Dave Winchell
> >>>>>    > >>>>>>Cc: Shan, Haitao; 
> xen-devel@lists.xensource.com; Dong,
> >>>>>    > Eddie; Jiang,
> >>>>>    > >>>>>>Yunhong
> >>>>>    > >>>>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer
> mode that
> >>>>>    > >>>>>>disables pending
> >>>>>    > >>>>>>missed ticks
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>Please take a look at
xen-unstable changeset 16545.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>-- Keir
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>On 26/11/07 20:57,
"Dave Winchell"
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    >
>>>><dwinchell@virtualiron.com> wrote:
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>Keir,
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The accuracy data
I''ve collected for i/o
> loads for the
> >>>>>    > >>>>>>>various time
protocols follows. In
> addition, the data
> >>>>>    > >>>>>>>for cpu loads is
shown.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled
cpu and i/o-8 are on an 8
> >>>>
> >>>>
> >>>> processor AMD
> >>>>
> >>>>
> >>>>>    box.
> >>>>>    > >>>>>>>Two guests, red
hat and sles 64 bit, 8 vcpu each.
> >>>>>    > >>>>>>>The cpu load is
usex -e36 on each guest.
> >>>>>    > >>>>>>>(usex is available
at
> >>>>>    http://people.redhat.com/anderson/usex.)
> >>>>>    > >>>>>>>i/o load is 8
instances of dd if=/dev/hda6
> >>>>
> >>>>
> >>>> of=/dev/null.
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled
i/o-32 are 32 instances of dd.
> >>>>>    > >>>>>>>Also, these are
run on 4 cpu AMD box.
> >>>>>    > >>>>>>>In addition, there
is an idle rh-32bit guest.
> >>>>>    > >>>>>>>All three guests
are 8vcpu.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The loads labeled
i/o-4/32 are the same as i/o-32
> >>>>>    > >>>>>>>except that the
redhat-64 guest has 4
> instances of dd.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Date Duration
Protocol sles, rhat error load
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/07 23 hrs 40
min ASYNC -4.96 sec, +4.42
> sec -.006%,
> >>>>>    > +.005% cpu
> >>>>>    > >>>>>>>11/09 3 hrs 19 min
ASYNC -.13 sec, +1.44
> sec, -.001%,
> >>>>>    > +.012% cpu
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/08 2 hrs 21 min
SYNC -.80 sec, -.34 sec, -.009%,
> >>>>>    -.004% cpu
> >>>>>    > >>>>>>>11/08 1 hr 25 min
SYNC -.24 sec, -.26 sec,
> >>>>
> >>>>
> >>>> -.005%, -.005% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/12 65 hrs 40
min SYNC -18 sec, -8 sec,
> >>>>
> >>>>
> >>>> -.008%, -.003% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/08 28 min MIXED
-.75 sec, -.67 sec -.045%,
> >>>>
> >>>>
> >>>> -.040% cpu
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/08 15 hrs 39
min MIXED -19. sec,-17.4
> sec, -.034%,
> >>>>>    > -.031% cpu
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/14 17 hrs 17
min ASYNC -6.1 sec,-55.7 sec, -.01%,
> >>>>>    > -.09% i/o-8
> >>>>>    > >>>>>>>11/15 2 hrs 44 min
ASYNC -1.47 sec,-14.0 sec, -.015%
> >>>>>    > -.14% i/o-8
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/13 15 hrs 38
min SYNC -9.7 sec,-12.3 sec, -.017%,
> >>>>>    > -.022% i/o-8
> >>>>>    > >>>>>>>11/14 48 min SYNC
- .46 sec, - .48 sec,
> >>>>
> >>>>
> >>>> -.017%, -.018% i/o-8
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/14 4 hrs 2 min
MIXED -2.9 sec, -4.15 sec, -.020%,
> >>>>>    > -.029% i/o-8
> >>>>>    > >>>>>>>11/20 16 hrs 2 min
MIXED -13.4 sec,-18.1
> sec, -.023%,
> >>>>>    > -.031% i/o-8
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/21 28 min MIXED
-2.01 sec, -.67 sec, -.12%,
> >>>>
> >>>>
> >>>> -.04% i/o-32
> >>>>
> >>>>
> >>>>>    > >>>>>>>11/21 2 hrs 25 min
SYNC -.96 sec, -.43 sec, -.011%,
> >>>>>    > -.005% i/o-32
> >>>>>    > >>>>>>>11/21 40 min ASYNC
-2.43 sec, -2.77 sec -.10%,
> >>>>
> >>>>
> >>>> -.11% i/o-32
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>11/26 113 hrs 46
min MIXED -297. sec, 13. sec -.07%,
> >>>>>    > .003% i/o-4/32
> >>>>>    > >>>>>>>11/26 4 hrs 50 min
SYNC -3.21 sec, 1.44 sec, -.017%,
> >>>>>    > .01% i/o-4/32
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Overhead
measurements:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Progress in terms
of number of passes
> through a fixed
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>system workload
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>on an 8 vcpu red
hat with an 8 vcpu sles idle.
> >>>>>    > >>>>>>>The workload was
usex -b48.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>ASYNC 167 min 145
passes .868 passes/min
> >>>>>    > >>>>>>>SYNC 167 min 144
passes .862 passes/min
> >>>>>    > >>>>>>>SYNC 1065 min 919
passes .863 passes/min
> >>>>>    > >>>>>>>MIXED 221 min 196
passes .887 passes/min
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Conclusions:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>The only protocol
which meets the .05% accuracy
> >>>>>    > requirement for ntp
> >>>>>    > >>>>>>>tracking under the
loads
> >>>>>    > >>>>>>>above is the SYNC
protocol. The worst case
> >>>>
> >>>>
> >>>> accuracies for
> >>>>
> >>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>SYNC, MIXED,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>and ASYNC
> >>>>>    > >>>>>>>are .022%, .12%,
and .14%, respectively.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>We could reduce
the cost of the SYNC method by only
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>scheduling the extra
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>wakeups if a
certain number
> >>>>>    > >>>>>>>of ticks are
missed.
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Regards,
> >>>>>    > >>>>>>>Dave
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>Keir Fraser wrote:
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>
> >>>>>    > >>>>>>>>On 9/11/07
19:22, "Dave Winchell"
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    >
>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>>Since I
had a high error (~.03%) for the
> >>>>
> >>>>
> >>>> ASYNC method a
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>couple of days ago,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>I ran
another ASYNC test. I think there may have
> >>>>>    > been something
> >>>>>    > >>>>>>>>>wrong with
the code I used a couple of
> days ago for
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>ASYNC. It may have
been
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>missing
the immediate delivery of interrupt
> >>>>
> >>>>
> >>>> after context
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>switch in.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>My results
indicate that either SYNC or ASYNC give
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>acceptable accuracy,
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>each
running consistently around or under
> >>>>
> >>>>
> >>>> .01%. MIXED has
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>a fairly high
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>error of
> >>>>>    > >>>>>>>>>greater
than .03%. Probably too close to .05% ntp
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>threshold for comfort.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>I
don''t have an overnight run with SYNC. I
> >>>>
> >>>>
> >>>> plan to leave
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>SYNC running
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>over the
weekend. If you''d rather I can
> leave MIXED
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>running instead.
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>It may be
too early to pick the protocol and
> >>>>
> >>>>
> >>>> I can run
> >>>>
> >>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>more overnight tests
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>>next week.
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>>
> >>>>>    > >>>>>>>>I''m a
bit worried about any unwanted side
> >>>>
> >>>>
> >>>> effects of the
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>SYNC+run_timer
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>approach --
e.g., whether timer wakeups will
> >>>>
> >>>>
> >>>> cause higher
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>system-wide CPU
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>contention. I
find it easier to think through the
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>implications of ASYNC.
I''m
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>surprised that
MIXED loses time, and is less
> >>>>
> >>>>
> >>>> accurate than
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>ASYNC. Perhaps it
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>delivers more
timer interrupts than the other
> >>>>
> >>>>
> >>>> approaches,
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>and each interrupt
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>event causes a
small accumulated error?
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>Overall I
would consider MIXED and ASYNC as
> >>>>
> >>>>
> >>>> favourites and
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>if the latter is
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>actually more
accurate then I can simply revert the
> >>>>>    > changeset that
> >>>>>    > >>>>>>>>implemented
MIXED.
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>Perhaps rather
than running more of the same
> >>>>
> >>>>
> >>>> workloads you
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>could try idle
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>VCPUs and I/O
bound VCPUs (e.g., repeated
> >>>>
> >>>>
> >>>> large disc reads
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>to /dev/null)? We
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>don''t
have any data on workloads that aren''t
> >>>>
> >>>>
> >>>> CPU bound, so
> >>>>
> >>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>that''s really
an
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>>>obvious place
to put any further effort imo.
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>-- Keir
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    > >>>>>>>>
> >>>>>    >
>>>>>>_______________________________________________
> >>>>>    > >>>>>>Xen-devel mailing list
> >>>>>    >
>>>>>>Xen-devel@lists.xensource.com
> >>>>>    >
>>>>>>http://lists.xensource.com/xen-devel
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>>
> >>>>>    > >>diff -r cfdbdca5b831
xen/arch/x86/hvm/vpt.c
> >>>>>    > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec
06 15:36:07
> 2007 +0000
> >>>>>    > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan
04 17:58:16
> 2008 -0500
> >>>>>    > >>@@ -58,7 +58,7 @@ static void 
> pt_process_missed_ticks(stru
> >>>>>    > >>
> >>>>>    > >>     missed_ticks = missed_ticks /
(s_time_t)
> >>>>
> >>>>
> >>>> pt->period + 1;
> >>>>
> >>>>
> >>>>>    > >>     if (
mode_is(pt->vcpu->domain,
> >>>>
> >>>>
> >>>> no_missed_ticks_pending) )
> >>>>
> >>>>
> >>>>>    > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
> >>>>>    > >>+        pt->do_not_freeze = 1;
> >>>>>    > >>     else
> >>>>>    > >>         pt->pending_intr_nr +=
missed_ticks;
> >>>>>    > >>     pt->scheduled += missed_ticks
* pt->period;
> >>>>>    > >>@@ -127,7 +127,12 @@ static void
pt_timer_fn(void *data)
> >>>>>    > >>
> >>>>>    > >>     pt_lock(pt);
> >>>>>    > >>
> >>>>>    > >>-    pt->pending_intr_nr++;
> >>>>>    > >>+    if (
mode_is(pt->vcpu->domain,
> >>>>
> >>>>
> >>>> no_missed_ticks_pending) ) {
> >>>>
> >>>>
> >>>>>    > >>+        pt->pending_intr_nr = 1;
> >>>>>    > >>+ pt->do_not_freeze = 0;
> >>>>>    > >>+    }
> >>>>>    > >>+    else
> >>>>>    > >>+ pt->pending_intr_nr++;
> >>>>>    > >>
> >>>>>    > >>     if ( !pt->one_shot )
> >>>>>    > >>     {
> >>>>>    > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct
> vcpu *v, struct
> >>>>>    > >>         return;
> >>>>>    > >>     }
> >>>>>    > >>
> >>>>>    > >>-    pt->do_not_freeze = 0;
> >>>>>    > >>-
> >>>>>    > >>     if ( pt->one_shot )
> >>>>>    > >>     {
> >>>>>    > >>         pt->enabled = 0;
> >>>>>    > >>@@ -235,6 +238,10 @@ void
pt_intr_post(struct vcpu
> >>>>
> >>>>
> >>>> *v, struct
> >>>>
> >>>>
> >>>>>    > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
> >>>>>    > >>             pt->pending_intr_nr =
0; /* ''collapse'' all
> >>>>>    > missed ticks */
> >>>>>    > >>         }
> >>>>>    > >>+ else if ( mode_is(v->domain, 
> no_missed_ticks_pending) ) {
> >>>>>    > >>+     pt->pending_intr_nr--;
> >>>>>    > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
> >>>>>    > >>+ }
> >>>>>    > >>         else
> >>>>>    > >>         {
> >>>>>    > >>             pt->last_plt_gtime +=
pt->period_cycles;
> >>>>>    > >>
> >>>>>    > >>
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    > >
> >>>>>    >
> >>>>>    >
> >>>>>    >
_______________________________________________
> >>>>>    > Xen-devel mailing list
> >>>>>    > Xen-devel@lists.xensource.com
> >>>>>    > http://lists.xensource.com/xen-devel
> >>>>>    >
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan, Deepak:

Thanks for the data. Those drifts are severe - no wonder ntp couldn''t
keep then in synch. I''ll try to reproduce that behaviour here, with my 
code base.
If I can''t reproduce it, I''ll try 3.2.

If you can isolate what ltp is doing during the cliffs, that would be very
helpful.

thanks,
Dave




Dan Magenheimer wrote:
>OK, Deepak repeated the test without ntpd and using ntpdate -b before
>the test.
>
>The attached graph shows his results: el5u1-64 (best=~0.07%),
>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>
>We will continue to look at LTP to try to isolate.
>
>Thanks,
>Dan
>
>P.S. elXuY is essentially RHEL XuY with some patches.
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Wednesday, January 30, 2008 2:45 PM
>>To: Deepak Patel
>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan, Deeepak,
>>
>>It may be that the underlying clock error is too great for ntp
>>to handle. It would be useful if you did not run ntpd
>>and, instead did ntpdate -b <timeserver> at the start of the test
>>for each guest. Then capture the data as you have been doing.
>>If the drift is greater than .05%, then we need to address that.
>>
>>Another option is, when running ntpd, to enable loop statistics in
>>/etc/ntp.conf
>>by adding this to the file:
>>
>>statistics loopstats
>>statsdir /var/lib/ntp/
>>
>>Then you will see loop data in that directory.
>>Correlating the data in the loopstats files with the
>>peaks in skew would be interesting. You will see entries of the form
>>
>>54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>
>>Where the second to last column is the Allan Deviation. When that
>>gets over 1000, ntpd is working pretty hard. However, I have 
>>not seen ntpd
>>completely loose it like you have.
>>
>>I''m on vacation until Monday, and won''t be reading
>>email.
>>
>>Thanks for all your work on this!
>>
>>-Dave
>>
>>Deepak Patel wrote:
>>
>>    
>>
>>>>Is the graph for RHEL5u1-64? (I''ve never tested this
one.)
>>>>        
>>>>
>>>I do not know which graph was attached with this. But I saw this
>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I
>>>was running ltp tests continuously.
>>>
>>>      
>>>
>>>>What was the behaviour of the other guests running?
>>>>        
>>>>
>>>All pvm guests are fine. But behavior of most of the hvm guests were
>>>as described.
>>>
>>>      
>>>
>>>>If they had spikes, were they at the same wall time?
>>>>        
>>>>
>>>No. They are not at the same wall time.
>>>
>>>      
>>>
>>>>Were the other guests running ltp as well?
>>>>
>>>>        
>>>>
>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>continuously.
>>>
>>>      
>>>
>>>>How are you measuring skew?
>>>>        
>>>>
>>>I was collecting output of "ntpdate -q  <timeserver>
every
>>>      
>>>
>>300 seconds
>>    
>>
>>>(5 minutes) and have created graph based on that.
>>>
>>>      
>>>
>>>>Are you running ntpd?
>>>>
>>>>        
>>>>
>>>Yes. ntp was running on all the guests.
>>>
>>>I am investigating what causes this spikes and let everyone 
>>>      
>>>
>>know what
>>    
>>
>>>are my findings.
>>>
>>>Thanks,
>>>Deepak
>>>
>>>      
>>>
>>>>Anything that you can discover that would be in sync with
>>>>the spikes would be very helpful!
>>>>
>>>>The code that I test with is our product code, which is based
>>>>on 3.1. So it is possible that something in 3.2 other than vpt.c
>>>>is the cause. I can test with 3.2, if necessary.
>>>>
>>>>thanks,
>>>>Dave
>>>>
>>>>
>>>>
>>>>Dan Magenheimer wrote:
>>>>
>>>>        
>>>>
>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>
>>>>>Thanks!
>>>>>
>>>>>Turning off vhpet certainly helps a lot (though see below).
>>>>>
>>>>>I wonder if timekeeping with vhpet is so bad that it should
be
>>>>>turned off by default (in 3.1, 3.2, and unstable) until it
is
>>>>>fixed?  (I have a patch that defaults it off, can post it if
>>>>>there is agreement on the above point.)  The whole point of
an
>>>>>HPET is to provide more precise timekeeping and if vhpet is
>>>>>worse than vpit, it can only confuse users.  Comments?
>>>>>
>>>>>
>>>>>In your testing, are you just measuring % skew over a long
>>>>>period of time?
>>>>> We are graphing the skew continuously and
>>>>>seeing periodic behavior that is unsettling, even with pit.
>>>>>See attached.  Though your algorithm recovers, the
"cliffs"
>>>>>could still cause real user problems.  I wonder if there is
>>>>>anything that can be done to make the "recovery"
more
>>>>>responsive?
>>>>>
>>>>>We are looking into what part(s) of LTP is causing the
cliffs.
>>>>>
>>>>>Thanks,
>>>>>Dan
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>Sent: Monday, January 28, 2008 8:21 AM
>>>>>>To: dan.magenheimer@oracle.com
>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>deepak.patel@oracle.com;
>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables
>>>>>>pending
>>>>>>missed ticks
>>>>>>
>>>>>>
>>>>>>Dan,
>>>>>>
>>>>>>I guess I''m a bit out of date calling for
clock= usage.
>>>>>>Looking at linux 2.6.20.4 sources, I think you should
specify
>>>>>>"clocksource=pit nohpet" on the linux guest
bootline.
>>>>>>
>>>>>>You can leave the xen and dom0 bootlines as they are.
>>>>>>The xen and guest clocksources do not need to be the
same.
>>>>>>In my tests, xen is using the hpet for its timekeeping
and
>>>>>>that appears to be the default.
>>>>>>
>>>>>>When you boot the guests you should see
>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>on the rh4u5-64 guest, and something similar on the
others.
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform
timer
>>>>>>>14.318MHz HPET.)
>>>>>>>              
>>>>>>>
>>>>>>This appears to be the xen state, which is fine.
>>>>>>I was wrongly assuming that this was the guest state.
>>>>>>You might want to look in your guest logs and see what
they were
>>>>>>picking
>>>>>>for a clock source.
>>>>>>
>>>>>>Regards,
>>>>>>Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>Dan Magenheimer wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Thanks, I hadn''t realized that!  No wonder
we didn''t
>>>>>>>              
>>>>>>>
>>see the same
>>    
>>
>>>>>>>improvement you saw!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>Try specifying clock=pit on the linux boot
line...
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>I''m confused... do you mean
"clocksource=pit" on the Xen
>>>>>>>              
>>>>>>>
>>>>>>command line or
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>"nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
>>>>>>>              
>>>>>>>
>>>>>>dom0?) command
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>line?  Or both places?  Since the tests take awhile,
it
>>>>>>>              
>>>>>>>
>>>>>>would be nice
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>to get this right the first time.  Do the Xen and
guest
>>>>>>>              
>>>>>>>
>>>>>>clocksources need
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>to be the same?
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Dan
>>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>*From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>*Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>akira.ijuin@oracle.com; Dave Winchell
>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode 
>>>>>>>              
>>>>>>>
>>that disables
>>    
>>
>>>>>>>pending missed ticks
>>>>>>>
>>>>>>>   Hi Dan,
>>>>>>>
>>>>>>>   Hpet timer does have a fairly large error, as I
was
>>>>>>>              
>>>>>>>
>>trying this
>>    
>>
>>>>>>>   one recently.
>>>>>>>   I don''t remember what I got for error,
but 1% sounds
>>>>>>>              
>>>>>>>
>>>>>>about right.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   The problem is that hpet is not built on top of
vpt.c,
>>>>>>>              
>>>>>>>
>>>>>>the module
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   Keir and I did
>>>>>>>   all the recent work in, for its periodic timer
needs. Try
>>>>>>>   specifying clock=pit
>>>>>>>   on the linux boot line. If it still picks the
hpet, which it
>>>>>>>   might, let me know
>>>>>>>   and I''ll tell you how to get around
this.
>>>>>>>
>>>>>>>   Regards,
>>>>>>>   Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>--------------------------------------------------------------
>>>>>>----------
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>   *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>   akira.ijuin@oracle.com
>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a timer
mode
>>>>>>>              
>>>>>>>
>>>>>>that disables
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   pending missed ticks
>>>>>>>
>>>>>>>   Sorry for the very late followup on this but we
finally
>>>>>>>              
>>>>>>>
>>>>>>were able
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   to get our testing set up again on stable 3.1
bits and have
>>>>>>>   seen some very bad results on 3.1.3-rc1, on the
order of 1%.
>>>>>>>
>>>>>>>   Test enviroment was a 4-socket dual core machine
with 24GB of
>>>>>>>   memory running six two-vcpu 2GB domains, four hvm
>>>>>>>              
>>>>>>>
>>plus two pv.
>>    
>>
>>>>>>>   All six guests were running LTP simultaneously. 
The four hvm
>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64,
and
>>>>>>>              
>>>>>>>
>>RHEL4u5-64.
>>    
>>
>>>>>>>   Timer_mode was set to 2 for 64-bit guests and 0
for
>>>>>>>              
>>>>>>>
>>>>>>32-bit guests.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   All four hvm guests experienced skew around -1%, 
>>>>>>>              
>>>>>>>
>>even the 32-bit
>>    
>>
>>>>>>>   guest.  Less intensive testing didn''t
exhibit much
>>>>>>>              
>>>>>>>
>>skew at all.
>>    
>>
>>>>>>>   A representative graph is attached.
>>>>>>>
>>>>>>>   Dave, I wonder if some portion of your patches 
>>>>>>>              
>>>>>>>
>>didn''t end up in
>>    
>>
>>>>>>>   the xen trees?
>>>>>>>
>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
Platform timer
>>>>>>>   14.318MHz HPET.)
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>   Dan
>>>>>>>
>>>>>>>   P.S. Many thanks to Deepak and Akira for running
tests.
>>>>>>>
>>>>>>>   > -----Original Message-----
>>>>>>>   > From: xen-devel-bounces@lists.xensource.com
>>>>>>>   >
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>   > Dave Winchell
>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53 AM
>>>>>>>   > To: Keir Fraser
>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>              
>>>>>>>
>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > Winchell
>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that
>>>>>>>   > disables pending
>>>>>>>   > missed ticks
>>>>>>>   >
>>>>>>>   >
>>>>>>>   > Hi Keir,
>>>>>>>   >
>>>>>>>   > The latest change, c/s 16690, looks fine.
>>>>>>>   > I agree that the code in c/s 16690 is
equivalent to
>>>>>>>   > the code I submitted. Also, your version is
more
>>>>>>>   > concise.
>>>>>>>   >
>>>>>>>   > The error tests confirm the equivalence.
With
>>>>>>>              
>>>>>>>
>>>>>>overnight cpu loads,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > the checked in version was accurate to
+.048% for sles
>>>>>>>   > and +.038% for red hat. My version was
+.046% and
>>>>>>>              
>>>>>>>
>>+.032% in a
>>    
>>
>>>>>>>   > 2 hour test.
>>>>>>>   > I don''t think the difference is
significant.
>>>>>>>   >
>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>   >
>>>>>>>   > Thanks for all your efforts on this issue.
>>>>>>>   >
>>>>>>>   > Regards,
>>>>>>>   > Dave
>>>>>>>   >
>>>>>>>   >
>>>>>>>   >
>>>>>>>   > Keir Fraser wrote:
>>>>>>>   >
>>>>>>>   > >Applied as c/s 16690, although the
checked-in patch is
>>>>>>>   > smaller. I think the
>>>>>>>   > >only important fix is to pt_intr_post()
and the
>>>>>>>              
>>>>>>>
>>only bit of
>>    
>>
>>>>>>>   > the patch I
>>>>>>>   > >totally omitted was the change to 
>>>>>>>              
>>>>>>>
>>pt_process_missed_ticks().
>>    
>>
>>>>>>>   > I don''t think
>>>>>>>   > >that change can be important, but
let''s see what
>>>>>>>              
>>>>>>>
>>>>>>happens to the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   error
>>>>>>>   > >percentage...
>>>>>>>   > >
>>>>>>>   > > -- Keir
>>>>>>>   > >
>>>>>>>   > >On 4/1/08 23:24, "Dave
Winchell"
>>>>>>>              
>>>>>>>
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>   > >>
>>>>>>>   > >>Attached is a patch that fixes some
issues with the
>>>>>>>              
>>>>>>>
>>>>>>SYNC policy
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>   > >>I have not tried to make the change
the minimal one, but,
>>>>>>>   > rather, just
>>>>>>>   > >>ported into
>>>>>>>   > >>the new code what I know to work
well. The error for
>>>>>>>   > >>no_missed_ticks_pending goes from
>>>>>>>   > >>over 3% to .03% with this change
according to my testing.
>>>>>>>   > >>
>>>>>>>   > >>Regards,
>>>>>>>   > >>Dave
>>>>>>>   > >>
>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >>>Hi Dave --
>>>>>>>   > >>>
>>>>>>>   > >>>Did you get your correction
ported?  If so, it would be
>>>>>>>   > nice to see this get
>>>>>>>   > >>>into 3.1.3.
>>>>>>>   > >>>
>>>>>>>   > >>>Note that I just did some very
limited testing with
>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>   > >>>missed ticks pending)
>>>>>>>   > >>>on tip of xen-3.1-testing
(64-bit Linux hv
>>>>>>>              
>>>>>>>
>>guest) and the
>>    
>>
>>>>>>>   > worst error I''ve
>>>>>>>   > >>>seen so far
>>>>>>>   > >>>is 0.012%.  But I
haven''t tried any exotic
>>>>>>>              
>>>>>>>
>>loads, just LTP.
>>    
>>
>>>>>>>   > >>>
>>>>>>>   > >>>Thanks,
>>>>>>>   > >>>Dan
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>>-----Original Message-----
>>>>>>>   > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>   > >>>>Sent: Wednesday, December
19, 2007 12:33 PM
>>>>>>>   > >>>>To:
dan.magenheimer@oracle.com
>>>>>>>   > >>>>Cc: Keir Fraser; Shan,
Haitao;
>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>   > >>>>Eddie; Jiang, Yunhong; Dave
Winchell
>>>>>>>   > >>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
>>>>>>>   > >>>>disables pending
>>>>>>>   > >>>>missed ticks
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>Dan,
>>>>>>>   > >>>>
>>>>>>>   > >>>>I did some testing with the
constant tsc offset
>>>>>>>              
>>>>>>>
>>>>>>SYNC method
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>(now called
>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>   > >>>>and found the error to be
very high, much larger
>>>>>>>              
>>>>>>>
>>>>>>than 1 %, as
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>I recall.
>>>>>>>   > >>>>I have not had a chance to
submit a correction. I
>>>>>>>              
>>>>>>>
>>>>>>will try to
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>do it later
>>>>>>>   > >>>>this week or the first week
in January. My version of
>>>>>>>   constant tsc
>>>>>>>   > >>>>offset SYNC method
>>>>>>>   > >>>>produces .02 % error, so I
just need to port
>>>>>>>              
>>>>>>>
>>that into the
>>    
>>
>>>>>>>   > >>>>current code.
>>>>>>>   > >>>>
>>>>>>>   > >>>>The error you got for both
of those kernels is
>>>>>>>              
>>>>>>>
>>>>>>what I would
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   expect
>>>>>>>   > >>>>for the default mode,
delay_for_missed_ticks.
>>>>>>>   > >>>>
>>>>>>>   > >>>>I''ll let Keir
answer on how to set the time mode.
>>>>>>>   > >>>>
>>>>>>>   > >>>>Regards,
>>>>>>>   > >>>>Dave
>>>>>>>   > >>>>
>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>Anyone make measurements
on the final patch?
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>I just ran a 64-bit
RHEL5.1 pvm kernel and
>>>>>>>              
>>>>>>>
>>saw a loss of
>>    
>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>about 0.2% with no load. 
This was
>>>>>>>              
>>>>>>>
>>xen-unstable tip today
>>    
>>
>>>>>>>   > >>>>with no options specified. 
32-bit was about 0.01%.
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>I think I missed
something... how do I run the various
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>accounting choices and which
ones are known to be
>>>>>>>              
>>>>>>>
>>>>>>appropriate
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>for which kernels?
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>Thanks,
>>>>>>>   > >>>>>Dan
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>>-----Original
Message-----
>>>>>>>   > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>   >
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>Keir Fraser
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>>Sent: Thursday,
December 06, 2007 4:57 AM
>>>>>>>   > >>>>>>To: Dave Winchell
>>>>>>>   > >>>>>>Cc: Shan, Haitao; 
>>>>>>>              
>>>>>>>
>>xen-devel@lists.xensource.com; Dong,
>>    
>>
>>>>>>>   > Eddie; Jiang,
>>>>>>>   > >>>>>>Yunhong
>>>>>>>   > >>>>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer
>>>>>>>              
>>>>>>>
>>mode that
>>    
>>
>>>>>>>   > >>>>>>disables pending
>>>>>>>   > >>>>>>missed ticks
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>Please take a look
at xen-unstable changeset 16545.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>-- Keir
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>On 26/11/07 20:57,
"Dave Winchell"
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>Keir,
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The accuracy
data I''ve collected for i/o
>>>>>>>              
>>>>>>>
>>loads for the
>>    
>>
>>>>>>>   > >>>>>>>various time
protocols follows. In
>>>>>>>              
>>>>>>>
>>addition, the data
>>    
>>
>>>>>>>   > >>>>>>>for cpu loads is
shown.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads
labeled cpu and i/o-8 are on an 8
>>>>>>>              
>>>>>>>
>>>>>>processor AMD
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   box.
>>>>>>>   > >>>>>>>Two guests, red
hat and sles 64 bit, 8 vcpu each.
>>>>>>>   > >>>>>>>The cpu load is
usex -e36 on each guest.
>>>>>>>   > >>>>>>>(usex is
available at
>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>   > >>>>>>>i/o load is 8
instances of dd if=/dev/hda6
>>>>>>>              
>>>>>>>
>>>>>>of=/dev/null.
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads
labeled i/o-32 are 32 instances of dd.
>>>>>>>   > >>>>>>>Also, these are
run on 4 cpu AMD box.
>>>>>>>   > >>>>>>>In addition,
there is an idle rh-32bit guest.
>>>>>>>   > >>>>>>>All three guests
are 8vcpu.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The loads
labeled i/o-4/32 are the same as i/o-32
>>>>>>>   > >>>>>>>except that the
redhat-64 guest has 4
>>>>>>>              
>>>>>>>
>>instances of dd.
>>    
>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Date Duration
Protocol sles, rhat error load
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/07 23 hrs 40
min ASYNC -4.96 sec, +4.42
>>>>>>>              
>>>>>>>
>>sec -.006%,
>>    
>>
>>>>>>>   > +.005% cpu
>>>>>>>   > >>>>>>>11/09 3 hrs 19
min ASYNC -.13 sec, +1.44
>>>>>>>              
>>>>>>>
>>sec, -.001%,
>>    
>>
>>>>>>>   > +.012% cpu
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/08 2 hrs 21
min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>   -.004% cpu
>>>>>>>   > >>>>>>>11/08 1 hr 25
min SYNC -.24 sec, -.26 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.005%, -.005% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/12 65 hrs 40
min SYNC -18 sec, -8 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.008%, -.003% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/08 28 min
MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>              
>>>>>>>
>>>>>>-.040% cpu
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/08 15 hrs 39
min MIXED -19. sec,-17.4
>>>>>>>              
>>>>>>>
>>sec, -.034%,
>>    
>>
>>>>>>>   > -.031% cpu
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/14 17 hrs 17
min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>   > -.09% i/o-8
>>>>>>>   > >>>>>>>11/15 2 hrs 44
min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>   > -.14% i/o-8
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/13 15 hrs 38
min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>   > -.022% i/o-8
>>>>>>>   > >>>>>>>11/14 48 min
SYNC - .46 sec, - .48 sec,
>>>>>>>              
>>>>>>>
>>>>>>-.017%, -.018% i/o-8
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/14 4 hrs 2
min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>   > -.029% i/o-8
>>>>>>>   > >>>>>>>11/20 16 hrs 2
min MIXED -13.4 sec,-18.1
>>>>>>>              
>>>>>>>
>>sec, -.023%,
>>    
>>
>>>>>>>   > -.031% i/o-8
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/21 28 min
MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>              
>>>>>>>
>>>>>>-.04% i/o-32
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>11/21 2 hrs 25
min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>   > -.005% i/o-32
>>>>>>>   > >>>>>>>11/21 40 min
ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>              
>>>>>>>
>>>>>>-.11% i/o-32
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>11/26 113 hrs 46
min MIXED -297. sec, 13. sec -.07%,
>>>>>>>   > .003% i/o-4/32
>>>>>>>   > >>>>>>>11/26 4 hrs 50
min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>   > .01% i/o-4/32
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Overhead
measurements:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Progress in
terms of number of passes
>>>>>>>              
>>>>>>>
>>through a fixed
>>    
>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>system workload
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>on an 8 vcpu red
hat with an 8 vcpu sles idle.
>>>>>>>   > >>>>>>>The workload was
usex -b48.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>ASYNC 167 min
145 passes .868 passes/min
>>>>>>>   > >>>>>>>SYNC 167 min 144
passes .862 passes/min
>>>>>>>   > >>>>>>>SYNC 1065 min
919 passes .863 passes/min
>>>>>>>   > >>>>>>>MIXED 221 min
196 passes .887 passes/min
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>The only
protocol which meets the .05% accuracy
>>>>>>>   > requirement for ntp
>>>>>>>   > >>>>>>>tracking under
the loads
>>>>>>>   > >>>>>>>above is the
SYNC protocol. The worst case
>>>>>>>              
>>>>>>>
>>>>>>accuracies for
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>   > >>>>>>>are .022%, .12%,
and .14%, respectively.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>We could reduce
the cost of the SYNC method by only
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>scheduling the extra
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>wakeups if a
certain number
>>>>>>>   > >>>>>>>of ticks are
missed.
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Regards,
>>>>>>>   > >>>>>>>Dave
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>Keir Fraser
wrote:
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>
>>>>>>>   > >>>>>>>>On 9/11/07
19:22, "Dave Winchell"
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>>Since I
had a high error (~.03%) for the
>>>>>>>              
>>>>>>>
>>>>>>ASYNC method a
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>couple of days ago,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>I ran
another ASYNC test. I think there may have
>>>>>>>   > been something
>>>>>>>   > >>>>>>>>>wrong
with the code I used a couple of
>>>>>>>              
>>>>>>>
>>days ago for
>>    
>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>ASYNC. It may have
been
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>missing
the immediate delivery of interrupt
>>>>>>>              
>>>>>>>
>>>>>>after context
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>switch in.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>My
results indicate that either SYNC or ASYNC give
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>acceptable accuracy,
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>each
running consistently around or under
>>>>>>>              
>>>>>>>
>>>>>>.01%. MIXED has
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>a fairly high
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>error of
>>>>>>>   > >>>>>>>>>greater
than .03%. Probably too close to .05% ntp
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>threshold for
comfort.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>I
don''t have an overnight run with SYNC. I
>>>>>>>              
>>>>>>>
>>>>>>plan to leave
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>SYNC running
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>over the
weekend. If you''d rather I can
>>>>>>>              
>>>>>>>
>>leave MIXED
>>    
>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>running instead.
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>It may
be too early to pick the protocol and
>>>>>>>              
>>>>>>>
>>>>>>I can run
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>more overnight tests
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>>next
week.
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>>
>>>>>>>   > >>>>>>>>I''m
a bit worried about any unwanted side
>>>>>>>              
>>>>>>>
>>>>>>effects of the
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>approach --
e.g., whether timer wakeups will
>>>>>>>              
>>>>>>>
>>>>>>cause higher
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>contention.
I find it easier to think through the
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>implications of
ASYNC. I''m
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>surprised
that MIXED loses time, and is less
>>>>>>>              
>>>>>>>
>>>>>>accurate than
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>ASYNC. Perhaps it
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>delivers
more timer interrupts than the other
>>>>>>>              
>>>>>>>
>>>>>>approaches,
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>and each interrupt
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>event causes
a small accumulated error?
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>Overall I
would consider MIXED and ASYNC as
>>>>>>>              
>>>>>>>
>>>>>>favourites and
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>if the latter is
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>actually
more accurate then I can simply revert the
>>>>>>>   > changeset that
>>>>>>>   > >>>>>>>>implemented
MIXED.
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>Perhaps
rather than running more of the same
>>>>>>>              
>>>>>>>
>>>>>>workloads you
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>could try idle
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>VCPUs and
I/O bound VCPUs (e.g., repeated
>>>>>>>              
>>>>>>>
>>>>>>large disc reads
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>to /dev/null)? We
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>              
>>>>>>>
>>>>>>CPU bound, so
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>that''s
really an
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>>>obvious
place to put any further effort imo.
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   > >>>>>>>>
>>>>>>>   >
>>>>>>_______________________________________________
>>>>>>>   > >>>>>>Xen-devel mailing
list
>>>>>>>   >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>   >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>>
>>>>>>>   > >>diff -r cfdbdca5b831
xen/arch/x86/hvm/vpt.c
>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec
06 15:36:07
>>>>>>>              
>>>>>>>
>>2007 +0000
>>    
>>
>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan
04 17:58:16
>>>>>>>              
>>>>>>>
>>2008 -0500
>>    
>>
>>>>>>>   > >>@@ -58,7 +58,7 @@ static void 
>>>>>>>              
>>>>>>>
>>pt_process_missed_ticks(stru
>>    
>>
>>>>>>>   > >>
>>>>>>>   > >>     missed_ticks = missed_ticks /
(s_time_t)
>>>>>>>              
>>>>>>>
>>>>>>pt->period + 1;
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>              
>>>>>>>
>>>>>>no_missed_ticks_pending) )
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
>>>>>>>   > >>+        pt->do_not_freeze = 1;
>>>>>>>   > >>     else
>>>>>>>   > >>         pt->pending_intr_nr +=
missed_ticks;
>>>>>>>   > >>     pt->scheduled +=
missed_ticks * pt->period;
>>>>>>>   > >>@@ -127,7 +127,12 @@ static void
pt_timer_fn(void *data)
>>>>>>>   > >>
>>>>>>>   > >>     pt_lock(pt);
>>>>>>>   > >>
>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>   > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>              
>>>>>>>
>>>>>>no_missed_ticks_pending) ) {
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>+        pt->pending_intr_nr = 1;
>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>   > >>+    }
>>>>>>>   > >>+    else
>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>   > >>
>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>   > >>     {
>>>>>>>   > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct
>>>>>>>              
>>>>>>>
>>vcpu *v, struct
>>    
>>
>>>>>>>   > >>         return;
>>>>>>>   > >>     }
>>>>>>>   > >>
>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>   > >>-
>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>   > >>     {
>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>   > >>@@ -235,6 +238,10 @@ void
pt_intr_post(struct vcpu
>>>>>>>              
>>>>>>>
>>>>>>*v, struct
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>   > >>             pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>>>>   > >>             pt->pending_intr_nr
= 0; /* ''collapse'' all
>>>>>>>   > missed ticks */
>>>>>>>   > >>         }
>>>>>>>   > >>+ else if ( mode_is(v->domain, 
>>>>>>>              
>>>>>>>
>>no_missed_ticks_pending) ) {
>>    
>>
>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>   > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>>>>   > >>+ }
>>>>>>>   > >>         else
>>>>>>>   > >>         {
>>>>>>>   > >>             pt->last_plt_gtime
+= pt->period_cycles;
>>>>>>>   > >>
>>>>>>>   > >>
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   > >
>>>>>>>   >
>>>>>>>   >
>>>>>>>   >
_______________________________________________
>>>>>>>   > Xen-devel mailing list
>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>   >
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>            
>>>>>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

Sorry it took me so long, but I finally ran an ltp test today.
Its on rh4u4-64. I''m using the defaults for ltp and using a script
called runltp. I had a usex load on rh4u5-64. No ntpd.
virtual processors / physical processors = 2.

The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
for -.005% and .008%.

I''m running a 3.1 based hypervisor with some clock related tweaks that
I haven''t submitted, because I''m still characterizing them.

I''m stopping the usex load on 4u5-64 now and replacing it with ltp
and will leave the two guests running ltp over the weekend.

Regards,
Dave


Dave Winchell wrote:
> Hi Dan, Deepak:
>
> Thanks for the data. Those drifts are severe - no wonder ntp
couldn''t
> keep then in synch. I''ll try to reproduce that behaviour here,
with my
> code base.
> If I can''t reproduce it, I''ll try 3.2.
>
> If you can isolate what ltp is doing during the cliffs, that would be 
> very
> helpful.
>
> thanks,
> Dave
>
>
>
>
> Dan Magenheimer wrote:
>
>> OK, Deepak repeated the test without ntpd and using ntpdate -b before
>> the test.
>>
>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>
>> We will continue to look at LTP to try to isolate.
>>
>> Thanks,
>> Dan
>>
>> P.S. elXuY is essentially RHEL XuY with some patches.
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>> To: Deepak Patel
>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave
Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
pending
>>> missed ticks
>>>
>>>
>>> Dan, Deeepak,
>>>
>>> It may be that the underlying clock error is too great for ntp
>>> to handle. It would be useful if you did not run ntpd
>>> and, instead did ntpdate -b <timeserver> at the start of the
test
>>> for each guest. Then capture the data as you have been doing.
>>> If the drift is greater than .05%, then we need to address that.
>>>
>>> Another option is, when running ntpd, to enable loop statistics in
>>> /etc/ntp.conf
>>> by adding this to the file:
>>>
>>> statistics loopstats
>>> statsdir /var/lib/ntp/
>>>
>>> Then you will see loop data in that directory.
>>> Correlating the data in the loopstats files with the
>>> peaks in skew would be interesting. You will see entries of the
form
>>>
>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10
>>>
>>> Where the second to last column is the Allan Deviation. When that
>>> gets over 1000, ntpd is working pretty hard. However, I have not 
>>> seen ntpd
>>> completely loose it like you have.
>>>
>>> I''m on vacation until Monday, and won''t be
reading
>>> email.
>>>
>>> Thanks for all your work on this!
>>>
>>> -Dave
>>>
>>> Deepak Patel wrote:
>>>
>>>   
>>>
>>>>> Is the graph for RHEL5u1-64? (I''ve never tested
this one.)
>>>>>       
>>>>
>>>> I do not know which graph was attached with this. But I saw
this
>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests
when I
>>>> was running ltp tests continuously.
>>>>
>>>>     
>>>>
>>>>> What was the behaviour of the other guests running?
>>>>>       
>>>>
>>>> All pvm guests are fine. But behavior of most of the hvm guests
were
>>>> as described.
>>>>
>>>>     
>>>>
>>>>> If they had spikes, were they at the same wall time?
>>>>>       
>>>>
>>>> No. They are not at the same wall time.
>>>>
>>>>     
>>>>
>>>>> Were the other guests running ltp as well?
>>>>>
>>>>>       
>>>>
>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp
>>>> continuously.
>>>>
>>>>     
>>>>
>>>>> How are you measuring skew?
>>>>>       
>>>>
>>>> I was collecting output of "ntpdate -q  <timeserver>
every
>>>
>>> 300 seconds
>>>   
>>>
>>>> (5 minutes) and have created graph based on that.
>>>>
>>>>     
>>>>
>>>>> Are you running ntpd?
>>>>>
>>>>>       
>>>>
>>>> Yes. ntp was running on all the guests.
>>>>
>>>> I am investigating what causes this spikes and let everyone
>>>
>>> know what
>>>   
>>>
>>>> are my findings.
>>>>
>>>> Thanks,
>>>> Deepak
>>>>
>>>>     
>>>>
>>>>> Anything that you can discover that would be in sync with
>>>>> the spikes would be very helpful!
>>>>>
>>>>> The code that I test with is our product code, which is
based
>>>>> on 3.1. So it is possible that something in 3.2 other than
vpt.c
>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>
>>>>> thanks,
>>>>> Dave
>>>>>
>>>>>
>>>>>
>>>>> Dan Magenheimer wrote:
>>>>>
>>>>>       
>>>>>
>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Turning off vhpet certainly helps a lot (though see
below).
>>>>>>
>>>>>> I wonder if timekeeping with vhpet is so bad that it
should be
>>>>>> turned off by default (in 3.1, 3.2, and unstable) until
it is
>>>>>> fixed?  (I have a patch that defaults it off, can post
it if
>>>>>> there is agreement on the above point.)  The whole
point of an
>>>>>> HPET is to provide more precise timekeeping and if
vhpet is
>>>>>> worse than vpit, it can only confuse users.  Comments?
>>>>>>
>>>>>>
>>>>>> In your testing, are you just measuring % skew over a
long
>>>>>> period of time?
>>>>>> We are graphing the skew continuously and
>>>>>> seeing periodic behavior that is unsettling, even with
pit.
>>>>>> See attached.  Though your algorithm recovers, the
"cliffs"
>>>>>> could still cause real user problems.  I wonder if
there is
>>>>>> anything that can be done to make the
"recovery" more
>>>>>> responsive?
>>>>>>
>>>>>> We are looking into what part(s) of LTP is causing the
cliffs.
>>>>>>
>>>>>> Thanks,
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>> deepak.patel@oracle.com;
>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that disables
>>>>>>> pending
>>>>>>> missed ticks
>>>>>>>
>>>>>>>
>>>>>>> Dan,
>>>>>>>
>>>>>>> I guess I''m a bit out of date calling for
clock= usage.
>>>>>>> Looking at linux 2.6.20.4 sources, I think you
should specify
>>>>>>> "clocksource=pit nohpet" on the linux
guest bootline.
>>>>>>>
>>>>>>> You can leave the xen and dom0 bootlines as they
are.
>>>>>>> The xen and guest clocksources do not need to be
the same.
>>>>>>> In my tests, xen is using the hpet for its
timekeeping and
>>>>>>> that appears to be the default.
>>>>>>>
>>>>>>> When you boot the guests you should see
>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>> on the rh4u5-64 guest, and something similar on the
others.
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
Platform timer
>>>>>>>> 14.318MHz HPET.)
>>>>>>>>             
>>>>>>>
>>>>>>> This appears to be the xen state, which is fine.
>>>>>>> I was wrongly assuming that this was the guest
state.
>>>>>>> You might want to look in your guest logs and see
what they were
>>>>>>> picking
>>>>>>> for a clock source.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan Magenheimer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> Thanks, I hadn''t realized that!  No
wonder we didn''t
>>>>>>>
>>> see the same
>>>   
>>>
>>>>>>>> improvement you saw!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>> Try specifying clock=pit on the linux boot
line...
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>
>>>>>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
>>>>>>>>             
>>>>>>>
>>>>>>> command line or
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> "nohpet" / "clock=pit" /
"clocksource=pit" on the guest (or
>>>>>>>>             
>>>>>>>
>>>>>>> dom0?) command
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> line?  Or both places?  Since the tests take
awhile, it
>>>>>>>>             
>>>>>>>
>>>>>>> would be nice
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> to get this right the first time.  Do the Xen
and guest
>>>>>>>>             
>>>>>>>
>>>>>>> clocksources need
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> to be the same?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Dan
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> *From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer
mode
>>>>>>>
>>> that disables
>>>   
>>>
>>>>>>>> pending missed ticks
>>>>>>>>
>>>>>>>>   Hi Dan,
>>>>>>>>
>>>>>>>>   Hpet timer does have a fairly large error, as
I was
>>>>>>>
>>> trying this
>>>   
>>>
>>>>>>>>   one recently.
>>>>>>>>   I don''t remember what I got for
error, but 1% sounds
>>>>>>>>             
>>>>>>>
>>>>>>> about right.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   The problem is that hpet is not built on top
of vpt.c,
>>>>>>>>             
>>>>>>>
>>>>>>> the module
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   Keir and I did
>>>>>>>>   all the recent work in, for its periodic
timer needs. Try
>>>>>>>>   specifying clock=pit
>>>>>>>>   on the linux boot line. If it still picks the
hpet, which it
>>>>>>>>   might, let me know
>>>>>>>>   and I''ll tell you how to get around
this.
>>>>>>>>
>>>>>>>>   Regards,
>>>>>>>>   Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>>>>
--------------------------------------------------------------
>>>>>>> ----------
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>   *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a
timer mode
>>>>>>>>             
>>>>>>>
>>>>>>> that disables
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   pending missed ticks
>>>>>>>>
>>>>>>>>   Sorry for the very late followup on this but
we finally
>>>>>>>>             
>>>>>>>
>>>>>>> were able
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   to get our testing set up again on stable 3.1
bits and have
>>>>>>>>   seen some very bad results on 3.1.3-rc1, on
the order of 1%.
>>>>>>>>
>>>>>>>>   Test enviroment was a 4-socket dual core
machine with 24GB of
>>>>>>>>   memory running six two-vcpu 2GB domains, four
hvm
>>>>>>>
>>> plus two pv.
>>>   
>>>
>>>>>>>>   All six guests were running LTP
simultaneously.  The four hvm
>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32,
RHEL5-64, and
>>>>>>>
>>> RHEL4u5-64.
>>>   
>>>
>>>>>>>>   Timer_mode was set to 2 for 64-bit guests and
0 for
>>>>>>>>             
>>>>>>>
>>>>>>> 32-bit guests.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   All four hvm guests experienced skew around
-1%,
>>>>>>>
>>> even the 32-bit
>>>   
>>>
>>>>>>>>   guest.  Less intensive testing
didn''t exhibit much
>>>>>>>
>>> skew at all.
>>>   
>>>
>>>>>>>>   A representative graph is attached.
>>>>>>>>
>>>>>>>>   Dave, I wonder if some portion of your
patches
>>>>>>>
>>> didn''t end up in
>>>   
>>>
>>>>>>>>   the xen trees?
>>>>>>>>
>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
Platform timer
>>>>>>>>   14.318MHz HPET.)
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>   Dan
>>>>>>>>
>>>>>>>>   P.S. Many thanks to Deepak and Akira for
running tests.
>>>>>>>>
>>>>>>>>   > -----Original Message-----
>>>>>>>>   > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>   >
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>   > Dave Winchell
>>>>>>>>   > Sent: Wednesday, January 09, 2008 9:53
AM
>>>>>>>>   > To: Keir Fraser
>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>             
>>>>>>>
>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > Winchell
>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that
>>>>>>>>   > disables pending
>>>>>>>>   > missed ticks
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   > Hi Keir,
>>>>>>>>   >
>>>>>>>>   > The latest change, c/s 16690, looks
fine.
>>>>>>>>   > I agree that the code in c/s 16690 is
equivalent to
>>>>>>>>   > the code I submitted. Also, your version
is more
>>>>>>>>   > concise.
>>>>>>>>   >
>>>>>>>>   > The error tests confirm the equivalence.
With
>>>>>>>>             
>>>>>>>
>>>>>>> overnight cpu loads,
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > the checked in version was accurate to
+.048% for sles
>>>>>>>>   > and +.038% for red hat. My version was
+.046% and
>>>>>>>
>>> +.032% in a
>>>   
>>>
>>>>>>>>   > 2 hour test.
>>>>>>>>   > I don''t think the difference is
significant.
>>>>>>>>   >
>>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>>   >
>>>>>>>>   > Thanks for all your efforts on this
issue.
>>>>>>>>   >
>>>>>>>>   > Regards,
>>>>>>>>   > Dave
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>   >
>>>>>>>>   > >Applied as c/s 16690, although the
checked-in patch is
>>>>>>>>   > smaller. I think the
>>>>>>>>   > >only important fix is to
pt_intr_post() and the
>>>>>>>
>>> only bit of
>>>   
>>>
>>>>>>>>   > the patch I
>>>>>>>>   > >totally omitted was the change to
>>>>>>>
>>> pt_process_missed_ticks().
>>>   
>>>
>>>>>>>>   > I don''t think
>>>>>>>>   > >that change can be important, but
let''s see what
>>>>>>>>             
>>>>>>>
>>>>>>> happens to the
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   error
>>>>>>>>   > >percentage...
>>>>>>>>   > >
>>>>>>>>   > > -- Keir
>>>>>>>>   > >
>>>>>>>>   > >On 4/1/08 23:24, "Dave
Winchell"
>>>>>>>>             
>>>>>>>
>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>   > >>
>>>>>>>>   > >>Attached is a patch that fixes
some issues with the
>>>>>>>>             
>>>>>>>
>>>>>>> SYNC policy
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>>   > >>I have not tried to make the
change the minimal one, but,
>>>>>>>>   > rather, just
>>>>>>>>   > >>ported into
>>>>>>>>   > >>the new code what I know to work
well. The error for
>>>>>>>>   > >>no_missed_ticks_pending goes
from
>>>>>>>>   > >>over 3% to .03% with this change
according to my testing.
>>>>>>>>   > >>
>>>>>>>>   > >>Regards,
>>>>>>>>   > >>Dave
>>>>>>>>   > >>
>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >>>Hi Dave --
>>>>>>>>   > >>>
>>>>>>>>   > >>>Did you get your correction
ported?  If so, it would be
>>>>>>>>   > nice to see this get
>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>   > >>>
>>>>>>>>   > >>>Note that I just did some
very limited testing with
>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>   > >>>missed ticks pending)
>>>>>>>>   > >>>on tip of xen-3.1-testing
(64-bit Linux hv
>>>>>>>
>>> guest) and the
>>>   
>>>
>>>>>>>>   > worst error I''ve
>>>>>>>>   > >>>seen so far
>>>>>>>>   > >>>is 0.012%.  But I
haven''t tried any exotic
>>>>>>>
>>> loads, just LTP.
>>>   
>>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>Thanks,
>>>>>>>>   > >>>Dan
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>>-----Original
Message-----
>>>>>>>>   > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>   > >>>>Sent: Wednesday,
December 19, 2007 12:33 PM
>>>>>>>>   > >>>>To:
dan.magenheimer@oracle.com
>>>>>>>>   > >>>>Cc: Keir Fraser; Shan,
Haitao;
>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>>   > >>>>Eddie; Jiang, Yunhong;
Dave Winchell
>>>>>>>>   > >>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
>>>>>>>>   > >>>>disables pending
>>>>>>>>   > >>>>missed ticks
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Dan,
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>I did some testing with
the constant tsc offset
>>>>>>>>             
>>>>>>>
>>>>>>> SYNC method
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>(now called
>>>>>>>>   > >>>>no_missed_ticks_pending)
>>>>>>>>   > >>>>and found the error to
be very high, much larger
>>>>>>>>             
>>>>>>>
>>>>>>> than 1 %, as
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>I recall.
>>>>>>>>   > >>>>I have not had a chance
to submit a correction. I
>>>>>>>>             
>>>>>>>
>>>>>>> will try to
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>do it later
>>>>>>>>   > >>>>this week or the first
week in January. My version of
>>>>>>>>   constant tsc
>>>>>>>>   > >>>>offset SYNC method
>>>>>>>>   > >>>>produces .02 % error, so
I just need to port
>>>>>>>
>>> that into the
>>>   
>>>
>>>>>>>>   > >>>>current code.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>The error you got for
both of those kernels is
>>>>>>>>             
>>>>>>>
>>>>>>> what I would
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   expect
>>>>>>>>   > >>>>for the default mode,
delay_for_missed_ticks.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>I''ll let Keir
answer on how to set the time mode.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Regards,
>>>>>>>>   > >>>>Dave
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>Dan Magenheimer wrote:
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>Anyone make
measurements on the final patch?
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>I just ran a 64-bit
RHEL5.1 pvm kernel and
>>>>>>>
>>> saw a loss of
>>>   
>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>about 0.2% with no load.
This was
>>>>>>>
>>> xen-unstable tip today
>>>   
>>>
>>>>>>>>   > >>>>with no options
specified.  32-bit was about 0.01%.
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>I think I missed
something... how do I run the various
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>accounting choices and
which ones are known to be
>>>>>>>>             
>>>>>>>
>>>>>>> appropriate
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>for which kernels?
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>Thanks,
>>>>>>>>   > >>>>>Dan
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>>-----Original
Message-----
>>>>>>>>   > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>   >
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>>Sent: Thursday,
December 06, 2007 4:57 AM
>>>>>>>>   > >>>>>>To: Dave
Winchell
>>>>>>>>   > >>>>>>Cc: Shan,
Haitao;
>>>>>>>
>>> xen-devel@lists.xensource.com; Dong,
>>>   
>>>
>>>>>>>>   > Eddie; Jiang,
>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>   > >>>>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer
>>>>>>>>             
>>>>>>>
>>> mode that
>>>   
>>>
>>>>>>>>   > >>>>>>disables pending
>>>>>>>>   > >>>>>>missed ticks
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>Please take a
look at xen-unstable changeset 16545.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>On 26/11/07
20:57, "Dave Winchell"
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>Keir,
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The accuracy
data I''ve collected for i/o
>>>>>>>
>>> loads for the
>>>   
>>>
>>>>>>>>   > >>>>>>>various time
protocols follows. In
>>>>>>>
>>> addition, the data
>>>   
>>>
>>>>>>>>   > >>>>>>>for cpu
loads is shown.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads
labeled cpu and i/o-8 are on an 8
>>>>>>>>             
>>>>>>>
>>>>>>> processor AMD
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   box.
>>>>>>>>   > >>>>>>>Two guests,
red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>   > >>>>>>>The cpu load
is usex -e36 on each guest.
>>>>>>>>   > >>>>>>>(usex is
available at
>>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>>   > >>>>>>>i/o load is
8 instances of dd if=/dev/hda6
>>>>>>>>             
>>>>>>>
>>>>>>> of=/dev/null.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads
labeled i/o-32 are 32 instances of dd.
>>>>>>>>   > >>>>>>>Also, these
are run on 4 cpu AMD box.
>>>>>>>>   > >>>>>>>In addition,
there is an idle rh-32bit guest.
>>>>>>>>   > >>>>>>>All three
guests are 8vcpu.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The loads
labeled i/o-4/32 are the same as i/o-32
>>>>>>>>   > >>>>>>>except that
the redhat-64 guest has 4
>>>>>>>
>>> instances of dd.
>>>   
>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Date
Duration Protocol sles, rhat error load
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/07 23 hrs
40 min ASYNC -4.96 sec, +4.42
>>>>>>>
>>> sec -.006%,
>>>   
>>>
>>>>>>>>   > +.005% cpu
>>>>>>>>   > >>>>>>>11/09 3 hrs
19 min ASYNC -.13 sec, +1.44
>>>>>>>
>>> sec, -.001%,
>>>   
>>>
>>>>>>>>   > +.012% cpu
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/08 2 hrs
21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>   -.004% cpu
>>>>>>>>   > >>>>>>>11/08 1 hr
25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.005%, -.005% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/12 65 hrs
40 min SYNC -18 sec, -8 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.008%, -.003% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/08 28 min
MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.040% cpu
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/08 15 hrs
39 min MIXED -19. sec,-17.4
>>>>>>>
>>> sec, -.034%,
>>>   
>>>
>>>>>>>>   > -.031% cpu
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/14 17 hrs
17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>   > -.09% i/o-8
>>>>>>>>   > >>>>>>>11/15 2 hrs
44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>   > -.14% i/o-8
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/13 15 hrs
38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>   > -.022% i/o-8
>>>>>>>>   > >>>>>>>11/14 48 min
SYNC - .46 sec, - .48 sec,
>>>>>>>>             
>>>>>>>
>>>>>>> -.017%, -.018% i/o-8
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/14 4 hrs
2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>   > -.029% i/o-8
>>>>>>>>   > >>>>>>>11/20 16 hrs
2 min MIXED -13.4 sec,-18.1
>>>>>>>
>>> sec, -.023%,
>>>   
>>>
>>>>>>>>   > -.031% i/o-8
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/21 28 min
MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.04% i/o-32
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>11/21 2 hrs
25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>   > -.005% i/o-32
>>>>>>>>   > >>>>>>>11/21 40 min
ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>             
>>>>>>>
>>>>>>> -.11% i/o-32
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>11/26 113
hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>   > .003% i/o-4/32
>>>>>>>>   > >>>>>>>11/26 4 hrs
50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>   > .01% i/o-4/32
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Overhead
measurements:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Progress in
terms of number of passes
>>>>>>>
>>> through a fixed
>>>   
>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>system workload
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>on an 8 vcpu
red hat with an 8 vcpu sles idle.
>>>>>>>>   > >>>>>>>The workload
was usex -b48.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>ASYNC 167
min 145 passes .868 passes/min
>>>>>>>>   > >>>>>>>SYNC 167 min
144 passes .862 passes/min
>>>>>>>>   > >>>>>>>SYNC 1065
min 919 passes .863 passes/min
>>>>>>>>   > >>>>>>>MIXED 221
min 196 passes .887 passes/min
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Conclusions:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>The only
protocol which meets the .05% accuracy
>>>>>>>>   > requirement for ntp
>>>>>>>>   > >>>>>>>tracking
under the loads
>>>>>>>>   > >>>>>>>above is the
SYNC protocol. The worst case
>>>>>>>>             
>>>>>>>
>>>>>>> accuracies for
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>and ASYNC
>>>>>>>>   > >>>>>>>are .022%,
.12%, and .14%, respectively.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>We could
reduce the cost of the SYNC method by only
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>scheduling the
extra
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>wakeups if a
certain number
>>>>>>>>   > >>>>>>>of ticks are
missed.
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Regards,
>>>>>>>>   > >>>>>>>Dave
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>Keir Fraser
wrote:
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>
>>>>>>>>   > >>>>>>>>On
9/11/07 19:22, "Dave Winchell"
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>             
>>>>>>>
>>>>>>> ASYNC method a
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>couple of days
ago,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>I
ran another ASYNC test. I think there may have
>>>>>>>>   > been something
>>>>>>>>   >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>
>>> days ago for
>>>   
>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>ASYNC. It may
have been
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>             
>>>>>>>
>>>>>>> after context
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>switch in.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>My
results indicate that either SYNC or ASYNC give
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>acceptable
accuracy,
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>each
running consistently around or under
>>>>>>>>             
>>>>>>>
>>>>>>> .01%. MIXED has
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>a fairly high
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>>>>>>>error of
>>>>>>>>   >
>>>>>>>>>greater than .03%. Probably too close to
.05% ntp
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>threshold for
comfort.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>I
don''t have an overnight run with SYNC. I
>>>>>>>>             
>>>>>>>
>>>>>>> plan to leave
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>SYNC running
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>over
the weekend. If you''d rather I can
>>>>>>>
>>> leave MIXED
>>>   
>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>running instead.
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>It
may be too early to pick the protocol and
>>>>>>>>             
>>>>>>>
>>>>>>> I can run
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>more overnight
tests
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>>next
week.
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   > >>>>>>>>>
>>>>>>>>   >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>             
>>>>>>>
>>>>>>> effects of the
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>SYNC+run_timer
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>approach
-- e.g., whether timer wakeups will
>>>>>>>>             
>>>>>>>
>>>>>>> cause higher
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>system-wide CPU
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>>>>>>contention. I find it easier to think through
the
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>implications of
ASYNC. I''m
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>             
>>>>>>>
>>>>>>> accurate than
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>ASYNC. Perhaps
it
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>delivers
more timer interrupts than the other
>>>>>>>>             
>>>>>>>
>>>>>>> approaches,
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>and each
interrupt
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>event
causes a small accumulated error?
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>Overall
I would consider MIXED and ASYNC as
>>>>>>>>             
>>>>>>>
>>>>>>> favourites and
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>if the latter is
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>actually
more accurate then I can simply revert the
>>>>>>>>   > changeset that
>>>>>>>>   >
>>>>>>>>implemented MIXED.
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>Perhaps
rather than running more of the same
>>>>>>>>             
>>>>>>>
>>>>>>> workloads you
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>could try idle
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>VCPUs
and I/O bound VCPUs (e.g., repeated
>>>>>>>>             
>>>>>>>
>>>>>>> large disc reads
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>to /dev/null)?
We
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>             
>>>>>>>
>>>>>>> CPU bound, so
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>that''s
really an
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>>>obvious
place to put any further effort imo.
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>-- Keir
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   > >>>>>>>>
>>>>>>>>   >
>>>>>>_______________________________________________
>>>>>>>>   > >>>>>>Xen-devel
mailing list
>>>>>>>>   >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>   >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>>
>>>>>>>>   > >>diff -r cfdbdca5b831
xen/arch/x86/hvm/vpt.c
>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c Thu
Dec 06 15:36:07
>>>>>>>>             
>>>>>>>
>>> 2007 +0000
>>>   
>>>
>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c Fri
Jan 04 17:58:16
>>>>>>>>             
>>>>>>>
>>> 2008 -0500
>>>   
>>>
>>>>>>>>   > >>@@ -58,7 +58,7 @@ static void
>>>>>>>
>>> pt_process_missed_ticks(stru
>>>   
>>>
>>>>>>>>   > >>
>>>>>>>>   > >>     missed_ticks = missed_ticks
/ (s_time_t)
>>>>>>>>             
>>>>>>>
>>>>>>> pt->period + 1;
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>>             
>>>>>>>
>>>>>>> no_missed_ticks_pending) )
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>-        pt->do_not_freeze =
!pt->pending_intr_nr;
>>>>>>>>   > >>+        pt->do_not_freeze =
1;
>>>>>>>>   > >>     else
>>>>>>>>   > >>         pt->pending_intr_nr
+= missed_ticks;
>>>>>>>>   > >>     pt->scheduled +=
missed_ticks * pt->period;
>>>>>>>>   > >>@@ -127,7 +127,12 @@ static void
pt_timer_fn(void *data)
>>>>>>>>   > >>
>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>   > >>
>>>>>>>>   > >>-    pt->pending_intr_nr++;
>>>>>>>>   > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>>             
>>>>>>>
>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>+        pt->pending_intr_nr
= 1;
>>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>>   > >>+    }
>>>>>>>>   > >>+    else
>>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>>   > >>
>>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>>   > >>     {
>>>>>>>>   > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct
>>>>>>>
>>> vcpu *v, struct
>>>   
>>>
>>>>>>>>   > >>         return;
>>>>>>>>   > >>     }
>>>>>>>>   > >>
>>>>>>>>   > >>-    pt->do_not_freeze = 0;
>>>>>>>>   > >>-
>>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>>   > >>     {
>>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>>   > >>@@ -235,6 +238,10 @@ void
pt_intr_post(struct vcpu
>>>>>>>>             
>>>>>>>
>>>>>>> *v, struct
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>>   > >>            
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>   > >>            
pt->pending_intr_nr = 0; /* ''collapse'' all
>>>>>>>>   > missed ticks */
>>>>>>>>   > >>         }
>>>>>>>>   > >>+ else if (
mode_is(v->domain,
>>>>>>>
>>> no_missed_ticks_pending) ) {
>>>   
>>>
>>>>>>>>   > >>+     pt->pending_intr_nr--;
>>>>>>>>   > >>+     pt->last_plt_gtime =
hvm_get_guest_time(v);
>>>>>>>>   > >>+ }
>>>>>>>>   > >>         else
>>>>>>>>   > >>         {
>>>>>>>>   > >>            
pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>   > >>
>>>>>>>>   > >>
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   > >
>>>>>>>>   >
>>>>>>>>   >
>>>>>>>>   >
_______________________________________________
>>>>>>>>   > Xen-devel mailing list
>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>>   >
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>>>>           
>>>>>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

Over the weekend the drift was +18 seconds for each guest (no ntp).
The duration was 3900 minutes, so the error for each was +.0077%.
Looking back through the data, it appears to drift linearly at
this rate. I''ve attached a plot for rh4u5-64.

This accuracy is better than what I''ve seen before (.03-.05%).
This may be due to the different load (ltp vs usex) or to one of the
changes I''ve made recently. I''ll do some experimentation to
see if there is
a fix I should propose.

This still doesn''t address the radical drift you saw.
The next step for me is to run 3.2 and see if I can reproduce it.

Regards,
Dave





Dave Winchell wrote:
> Hi Dan,
>
> Sorry it took me so long, but I finally ran an ltp test today.
> Its on rh4u4-64. I''m using the defaults for ltp and using a script
> called runltp. I had a usex load on rh4u5-64. No ntpd.
> virtual processors / physical processors = 2.
>
> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
> for -.005% and .008%.
>
> I''m running a 3.1 based hypervisor with some clock related tweaks
that
> I haven''t submitted, because I''m still characterizing
them.
>
> I''m stopping the usex load on 4u5-64 now and replacing it with ltp
> and will leave the two guests running ltp over the weekend.
>
> Regards,
> Dave
>
>
> Dave Winchell wrote:
>
>> Hi Dan, Deepak:
>>
>> Thanks for the data. Those drifts are severe - no wonder ntp
couldn''t
>> keep then in synch. I''ll try to reproduce that behaviour here,
with
>> my code base.
>> If I can''t reproduce it, I''ll try 3.2.
>>
>> If you can isolate what ltp is doing during the cliffs, that would be 
>> very
>> helpful.
>>
>> thanks,
>> Dave
>>
>>
>>
>>
>> Dan Magenheimer wrote:
>>
>>> OK, Deepak repeated the test without ntpd and using ntpdate -b
before
>>> the test.
>>>
>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>
>>> We will continue to look at LTP to try to isolate.
>>>
>>> Thanks,
>>> Dan
>>>
>>> P.S. elXuY is essentially RHEL XuY with some patches.
>>>
>>>  
>>>
>>>> -----Original Message-----
>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>>> To: Deepak Patel
>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave
Winchell
>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
>>>> pending
>>>> missed ticks
>>>>
>>>>
>>>> Dan, Deeepak,
>>>>
>>>> It may be that the underlying clock error is too great for ntp
>>>> to handle. It would be useful if you did not run ntpd
>>>> and, instead did ntpdate -b <timeserver> at the start of
the test
>>>> for each guest. Then capture the data as you have been doing.
>>>> If the drift is greater than .05%, then we need to address
that.
>>>>
>>>> Another option is, when running ntpd, to enable loop statistics
in
>>>> /etc/ntp.conf
>>>> by adding this to the file:
>>>>
>>>> statistics loopstats
>>>> statsdir /var/lib/ntp/
>>>>
>>>> Then you will see loop data in that directory.
>>>> Correlating the data in the loopstats files with the
>>>> peaks in skew would be interesting. You will see entries of the
form
>>>>
>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511
10
>>>>
>>>> Where the second to last column is the Allan Deviation. When
that
>>>> gets over 1000, ntpd is working pretty hard. However, I have
not
>>>> seen ntpd
>>>> completely loose it like you have.
>>>>
>>>> I''m on vacation until Monday, and won''t be
reading
>>>> email.
>>>>
>>>> Thanks for all your work on this!
>>>>
>>>> -Dave
>>>>
>>>> Deepak Patel wrote:
>>>>
>>>>  
>>>>
>>>>>> Is the graph for RHEL5u1-64? (I''ve never
tested this one.)
>>>>>>       
>>>>>
>>>>>
>>>>> I do not know which graph was attached with this. But I saw
this
>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
guests when I
>>>>> was running ltp tests continuously.
>>>>>
>>>>>    
>>>>>
>>>>>> What was the behaviour of the other guests running?
>>>>>>       
>>>>>
>>>>>
>>>>> All pvm guests are fine. But behavior of most of the hvm
guests were
>>>>> as described.
>>>>>
>>>>>    
>>>>>
>>>>>> If they had spikes, were they at the same wall time?
>>>>>>       
>>>>>
>>>>>
>>>>> No. They are not at the same wall time.
>>>>>
>>>>>    
>>>>>
>>>>>> Were the other guests running ltp as well?
>>>>>>
>>>>>>       
>>>>>
>>>>>
>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running
ltp
>>>>> continuously.
>>>>>
>>>>>    
>>>>>
>>>>>> How are you measuring skew?
>>>>>>       
>>>>>
>>>>>
>>>>> I was collecting output of "ntpdate -q 
<timeserver> every
>>>>
>>>>
>>>> 300 seconds
>>>>  
>>>>
>>>>> (5 minutes) and have created graph based on that.
>>>>>
>>>>>    
>>>>>
>>>>>> Are you running ntpd?
>>>>>>
>>>>>>       
>>>>>
>>>>>
>>>>> Yes. ntp was running on all the guests.
>>>>>
>>>>> I am investigating what causes this spikes and let everyone
>>>>
>>>>
>>>> know what
>>>>  
>>>>
>>>>> are my findings.
>>>>>
>>>>> Thanks,
>>>>> Deepak
>>>>>
>>>>>    
>>>>>
>>>>>> Anything that you can discover that would be in sync
with
>>>>>> the spikes would be very helpful!
>>>>>>
>>>>>> The code that I test with is our product code, which is
based
>>>>>> on 3.1. So it is possible that something in 3.2 other
than vpt.c
>>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>>
>>>>>> thanks,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dan Magenheimer wrote:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Turning off vhpet certainly helps a lot (though see
below).
>>>>>>>
>>>>>>> I wonder if timekeeping with vhpet is so bad that
it should be
>>>>>>> turned off by default (in 3.1, 3.2, and unstable)
until it is
>>>>>>> fixed?  (I have a patch that defaults it off, can
post it if
>>>>>>> there is agreement on the above point.)  The whole
point of an
>>>>>>> HPET is to provide more precise timekeeping and if
vhpet is
>>>>>>> worse than vpit, it can only confuse users. 
Comments?
>>>>>>>
>>>>>>>
>>>>>>> In your testing, are you just measuring % skew over
a long
>>>>>>> period of time?
>>>>>>> We are graphing the skew continuously and
>>>>>>> seeing periodic behavior that is unsettling, even
with pit.
>>>>>>> See attached.  Though your algorithm recovers, the
"cliffs"
>>>>>>> could still cause real user problems.  I wonder if
there is
>>>>>>> anything that can be done to make the
"recovery" more
>>>>>>> responsive?
>>>>>>>
>>>>>>> We are looking into what part(s) of LTP is causing
the cliffs.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com;
>>>>>>>> deepak.patel@oracle.com;
>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that disables
>>>>>>>> pending
>>>>>>>> missed ticks
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan,
>>>>>>>>
>>>>>>>> I guess I''m a bit out of date calling
for clock= usage.
>>>>>>>> Looking at linux 2.6.20.4 sources, I think you
should specify
>>>>>>>> "clocksource=pit nohpet" on the linux
guest bootline.
>>>>>>>>
>>>>>>>> You can leave the xen and dom0 bootlines as
they are.
>>>>>>>> The xen and guest clocksources do not need to
be the same.
>>>>>>>> In my tests, xen is using the hpet for its
timekeeping and
>>>>>>>> that appears to be the default.
>>>>>>>>
>>>>>>>> When you boot the guests you should see
>>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>>> on the rh4u5-64 guest, and something similar on
the others.
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04,
Platform timer
>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> This appears to be the xen state, which is
fine.
>>>>>>>> I was wrongly assuming that this was the guest
state.
>>>>>>>> You might want to look in your guest logs and
see what they were
>>>>>>>> picking
>>>>>>>> for a clock source.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> Thanks, I hadn''t realized that! 
No wonder we didn''t
>>>>>>>>
>>>>>>>>
>>>> see the same
>>>>  
>>>>
>>>>>>>>> improvement you saw!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>> Try specifying clock=pit on the linux
boot line...
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> command line or
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> "nohpet" / "clock=pit"
/ "clocksource=pit" on the guest (or
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> dom0?) command
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> line?  Or both places?  Since the tests
take awhile, it
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> would be nice
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> to get this right the first time.  Do the
Xen and guest
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> clocksources need
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>> to be the same?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> *From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM
>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir
Fraser
>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a
timer mode
>>>>>>>>
>>>>>>>>
>>>> that disables
>>>>  
>>>>
>>>>>>>>> pending missed ticks
>>>>>>>>>
>>>>>>>>>   Hi Dan,
>>>>>>>>>
>>>>>>>>>   Hpet timer does have a fairly large
error, as I was
>>>>>>>>
>>>>>>>>
>>>> trying this
>>>>  
>>>>
>>>>>>>>>   one recently.
>>>>>>>>>   I don''t remember what I got for
error, but 1% sounds
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> about right.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   The problem is that hpet is not built on
top of vpt.c,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> the module
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   Keir and I did
>>>>>>>>>   all the recent work in, for its periodic
timer needs. Try
>>>>>>>>>   specifying clock=pit
>>>>>>>>>   on the linux boot line. If it still picks
the hpet, which it
>>>>>>>>>   might, let me know
>>>>>>>>>   and I''ll tell you how to get
around this.
>>>>>>>>>
>>>>>>>>>   Regards,
>>>>>>>>>   Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>>
--------------------------------------------------------------
>>>>>>>> ----------
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>>   *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH] Add a
timer mode
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> that disables
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   pending missed ticks
>>>>>>>>>
>>>>>>>>>   Sorry for the very late followup on this
but we finally
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> were able
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   to get our testing set up again on stable
3.1 bits and have
>>>>>>>>>   seen some very bad results on 3.1.3-rc1,
on the order of 1%.
>>>>>>>>>
>>>>>>>>>   Test enviroment was a 4-socket dual core
machine with 24GB of
>>>>>>>>>   memory running six two-vcpu 2GB domains,
four hvm
>>>>>>>>
>>>>>>>>
>>>> plus two pv.
>>>>  
>>>>
>>>>>>>>>   All six guests were running LTP
simultaneously.  The four hvm
>>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32,
RHEL5-64, and
>>>>>>>>
>>>>>>>>
>>>> RHEL4u5-64.
>>>>  
>>>>
>>>>>>>>>   Timer_mode was set to 2 for 64-bit guests
and 0 for
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> 32-bit guests.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   All four hvm guests experienced skew
around -1%,
>>>>>>>>
>>>>>>>>
>>>> even the 32-bit
>>>>  
>>>>
>>>>>>>>>   guest.  Less intensive testing
didn''t exhibit much
>>>>>>>>
>>>>>>>>
>>>> skew at all.
>>>>  
>>>>
>>>>>>>>>   A representative graph is attached.
>>>>>>>>>
>>>>>>>>>   Dave, I wonder if some portion of your
patches
>>>>>>>>
>>>>>>>>
>>>> didn''t end up in
>>>>  
>>>>
>>>>>>>>>   the xen trees?
>>>>>>>>>
>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz stepping
04, Platform timer
>>>>>>>>>   14.318MHz HPET.)
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>   Dan
>>>>>>>>>
>>>>>>>>>   P.S. Many thanks to Deepak and Akira for
running tests.
>>>>>>>>>
>>>>>>>>>   > -----Original Message-----
>>>>>>>>>   > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>   >
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>   > Dave Winchell
>>>>>>>>>   > Sent: Wednesday, January 09, 2008
9:53 AM
>>>>>>>>>   > To: Keir Fraser
>>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > Winchell
>>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH] Add
a timer mode that
>>>>>>>>>   > disables pending
>>>>>>>>>   > missed ticks
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   > Hi Keir,
>>>>>>>>>   >
>>>>>>>>>   > The latest change, c/s 16690, looks
fine.
>>>>>>>>>   > I agree that the code in c/s 16690
is equivalent to
>>>>>>>>>   > the code I submitted. Also, your
version is more
>>>>>>>>>   > concise.
>>>>>>>>>   >
>>>>>>>>>   > The error tests confirm the
equivalence. With
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> overnight cpu loads,
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > the checked in version was accurate
to +.048% for sles
>>>>>>>>>   > and +.038% for red hat. My version
was +.046% and
>>>>>>>>
>>>>>>>>
>>>> +.032% in a
>>>>  
>>>>
>>>>>>>>>   > 2 hour test.
>>>>>>>>>   > I don''t think the
difference is significant.
>>>>>>>>>   >
>>>>>>>>>   > i/o loads produced errors of +.01%.
>>>>>>>>>   >
>>>>>>>>>   > Thanks for all your efforts on this
issue.
>>>>>>>>>   >
>>>>>>>>>   > Regards,
>>>>>>>>>   > Dave
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>>   >
>>>>>>>>>   > >Applied as c/s 16690, although
the checked-in patch is
>>>>>>>>>   > smaller. I think the
>>>>>>>>>   > >only important fix is to
pt_intr_post() and the
>>>>>>>>
>>>>>>>>
>>>> only bit of
>>>>  
>>>>
>>>>>>>>>   > the patch I
>>>>>>>>>   > >totally omitted was the change
to
>>>>>>>>
>>>>>>>>
>>>> pt_process_missed_ticks().
>>>>  
>>>>
>>>>>>>>>   > I don''t think
>>>>>>>>>   > >that change can be important,
but let''s see what
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> happens to the
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   error
>>>>>>>>>   > >percentage...
>>>>>>>>>   > >
>>>>>>>>>   > > -- Keir
>>>>>>>>>   > >
>>>>>>>>>   > >On 4/1/08 23:24, "Dave
Winchell"
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>>   > >>
>>>>>>>>>   > >>Attached is a patch that
fixes some issues with the
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> SYNC policy
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>(no_missed_ticks_pending).
>>>>>>>>>   > >>I have not tried to make the
change the minimal one, but,
>>>>>>>>>   > rather, just
>>>>>>>>>   > >>ported into
>>>>>>>>>   > >>the new code what I know to
work well. The error for
>>>>>>>>>   > >>no_missed_ticks_pending goes
from
>>>>>>>>>   > >>over 3% to .03% with this
change according to my testing.
>>>>>>>>>   > >>
>>>>>>>>>   > >>Regards,
>>>>>>>>>   > >>Dave
>>>>>>>>>   > >>
>>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >>>Hi Dave --
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Did you get your
correction ported?  If so, it would be
>>>>>>>>>   > nice to see this get
>>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Note that I just did
some very limited testing with
>>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>>   > >>>missed ticks pending)
>>>>>>>>>   > >>>on tip of
xen-3.1-testing (64-bit Linux hv
>>>>>>>>
>>>>>>>>
>>>> guest) and the
>>>>  
>>>>
>>>>>>>>>   > worst error I''ve
>>>>>>>>>   > >>>seen so far
>>>>>>>>>   > >>>is 0.012%.  But I
haven''t tried any exotic
>>>>>>>>
>>>>>>>>
>>>> loads, just LTP.
>>>>  
>>>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>Thanks,
>>>>>>>>>   > >>>Dan
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>>-----Original
Message-----
>>>>>>>>>   > >>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>   > >>>>Sent: Wednesday,
December 19, 2007 12:33 PM
>>>>>>>>>   > >>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>   > >>>>Cc: Keir Fraser;
Shan, Haitao;
>>>>>>>>>   > xen-devel@lists.xensource.com; Dong,
>>>>>>>>>   > >>>>Eddie; Jiang,
Yunhong; Dave Winchell
>>>>>>>>>   > >>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>   > >>>>disables pending
>>>>>>>>>   > >>>>missed ticks
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Dan,
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>I did some testing
with the constant tsc offset
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> SYNC method
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>(now called
>>>>>>>>>   >
>>>>no_missed_ticks_pending)
>>>>>>>>>   > >>>>and found the error
to be very high, much larger
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> than 1 %, as
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>I recall.
>>>>>>>>>   > >>>>I have not had a
chance to submit a correction. I
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> will try to
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>do it later
>>>>>>>>>   > >>>>this week or the
first week in January. My version of
>>>>>>>>>   constant tsc
>>>>>>>>>   > >>>>offset SYNC method
>>>>>>>>>   > >>>>produces .02 %
error, so I just need to port
>>>>>>>>
>>>>>>>>
>>>> that into the
>>>>  
>>>>
>>>>>>>>>   > >>>>current code.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>The error you got
for both of those kernels is
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> what I would
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   expect
>>>>>>>>>   > >>>>for the default
mode, delay_for_missed_ticks.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>I''ll let
Keir answer on how to set the time mode.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Regards,
>>>>>>>>>   > >>>>Dave
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>Dan Magenheimer
wrote:
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>Anyone make
measurements on the final patch?
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>I just ran a
64-bit RHEL5.1 pvm kernel and
>>>>>>>>
>>>>>>>>
>>>> saw a loss of
>>>>  
>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>about 0.2% with no
load.  This was
>>>>>>>>
>>>>>>>>
>>>> xen-unstable tip today
>>>>  
>>>>
>>>>>>>>>   > >>>>with no options
specified.  32-bit was about 0.01%.
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>I think I missed
something... how do I run the various
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>accounting choices
and which ones are known to be
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> appropriate
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>for which kernels?
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>Thanks,
>>>>>>>>>   > >>>>>Dan
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   >
>>>>>>-----Original Message-----
>>>>>>>>>   > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>            
>>>>>>>>>
>>>>>>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                       
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>>Sent:
Thursday, December 06, 2007 4:57 AM
>>>>>>>>>   > >>>>>>To: Dave
Winchell
>>>>>>>>>   > >>>>>>Cc: Shan,
Haitao;
>>>>>>>>
>>>>>>>>
>>>> xen-devel@lists.xensource.com; Dong,
>>>>  
>>>>
>>>>>>>>>   > Eddie; Jiang,
>>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>>   > >>>>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> mode that
>>>>  
>>>>
>>>>>>>>>   > >>>>>>disables
pending
>>>>>>>>>   > >>>>>>missed ticks
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>Please take
a look at xen-unstable changeset 16545.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>On 26/11/07
20:57, "Dave Winchell"
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>Keir,
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The
accuracy data I''ve collected for i/o
>>>>>>>>
>>>>>>>>
>>>> loads for the
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>various
time protocols follows. In
>>>>>>>>
>>>>>>>>
>>>> addition, the data
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>for cpu
loads is shown.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The
loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> processor AMD
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   box.
>>>>>>>>>   > >>>>>>>Two
guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>>   > >>>>>>>The cpu
load is usex -e36 on each guest.
>>>>>>>>>   > >>>>>>>(usex is
available at
>>>>>>>>>   http://people.redhat.com/anderson/usex.)
>>>>>>>>>   > >>>>>>>i/o load
is 8 instances of dd if=/dev/hda6
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> of=/dev/null.
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The
loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>   > >>>>>>>Also,
these are run on 4 cpu AMD box.
>>>>>>>>>   > >>>>>>>In
addition, there is an idle rh-32bit guest.
>>>>>>>>>   > >>>>>>>All
three guests are 8vcpu.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The
loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>>   > >>>>>>>except
that the redhat-64 guest has 4
>>>>>>>>
>>>>>>>>
>>>> instances of dd.
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Date
Duration Protocol sles, rhat error load
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/07 23
hrs 40 min ASYNC -4.96 sec, +4.42
>>>>>>>>
>>>>>>>>
>>>> sec -.006%,
>>>>  
>>>>
>>>>>>>>>   > +.005% cpu
>>>>>>>>>   > >>>>>>>11/09 3
hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>
>>>>>>>>
>>>> sec, -.001%,
>>>>  
>>>>
>>>>>>>>>   > +.012% cpu
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/08 2
hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>>   -.004% cpu
>>>>>>>>>   > >>>>>>>11/08 1
hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.005%, -.005% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/12 65
hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.008%, -.003% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/08 28
min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.040% cpu
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/08 15
hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>
>>>>>>>>
>>>> sec, -.034%,
>>>>  
>>>>
>>>>>>>>>   > -.031% cpu
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/14 17
hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>>   > -.09% i/o-8
>>>>>>>>>   > >>>>>>>11/15 2
hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>>   > -.14% i/o-8
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/13 15
hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>>   > -.022% i/o-8
>>>>>>>>>   > >>>>>>>11/14 48
min SYNC - .46 sec, - .48 sec,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/14 4
hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>>   > -.029% i/o-8
>>>>>>>>>   > >>>>>>>11/20 16
hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>
>>>>>>>>
>>>> sec, -.023%,
>>>>  
>>>>
>>>>>>>>>   > -.031% i/o-8
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/21 28
min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.04% i/o-32
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>11/21 2
hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>>   > -.005% i/o-32
>>>>>>>>>   > >>>>>>>11/21 40
min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> -.11% i/o-32
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>11/26
113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>>   > .003% i/o-4/32
>>>>>>>>>   > >>>>>>>11/26 4
hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>>   > .01% i/o-4/32
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Overhead
measurements:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Progress
in terms of number of passes
>>>>>>>>
>>>>>>>>
>>>> through a fixed
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>system
workload
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>on an 8
vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>   > >>>>>>>The
workload was usex -b48.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>ASYNC
167 min 145 passes .868 passes/min
>>>>>>>>>   > >>>>>>>SYNC 167
min 144 passes .862 passes/min
>>>>>>>>>   > >>>>>>>SYNC
1065 min 919 passes .863 passes/min
>>>>>>>>>   > >>>>>>>MIXED
221 min 196 passes .887 passes/min
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   >
>>>>>>>Conclusions:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>The only
protocol which meets the .05% accuracy
>>>>>>>>>   > requirement for ntp
>>>>>>>>>   > >>>>>>>tracking
under the loads
>>>>>>>>>   > >>>>>>>above is
the SYNC protocol. The worst case
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> accuracies for
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>SYNC, MIXED,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>and
ASYNC
>>>>>>>>>   > >>>>>>>are
.022%, .12%, and .14%, respectively.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>We could
reduce the cost of the SYNC method by only
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>scheduling
the extra
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>wakeups
if a certain number
>>>>>>>>>   > >>>>>>>of ticks
are missed.
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Regards,
>>>>>>>>>   > >>>>>>>Dave
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>Keir
Fraser wrote:
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>
>>>>>>>>>   > >>>>>>>>On
9/11/07 19:22, "Dave Winchell"
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> ASYNC method a
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>couple of
days ago,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>I ran another ASYNC test. I think there may
have
>>>>>>>>>   > been something
>>>>>>>>>   >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>
>>>>>>>>
>>>> days ago for
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>ASYNC. It
may have been
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> after context
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>switch in.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>My results indicate that either SYNC or
ASYNC give
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>acceptable
accuracy,
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>each running consistently around or under
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> .01%. MIXED has
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>a fairly
high
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>error of
>>>>>>>>>   >
>>>>>>>>>greater than .03%. Probably too close to
.05% ntp
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>threshold
for comfort.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> plan to leave
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>SYNC running
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>over the weekend. If you''d rather I
can
>>>>>>>>
>>>>>>>>
>>>> leave MIXED
>>>>  
>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>running
instead.
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> I can run
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>more
overnight tests
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>>next week.
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   > >>>>>>>>>
>>>>>>>>>   >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> effects of the
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>SYNC+run_timer
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> cause higher
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>system-wide
CPU
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>contention. I find it easier to think through
the
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>implications
of ASYNC. I''m
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> accurate than
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>ASYNC.
Perhaps it
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> approaches,
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>and each
interrupt
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>event causes a small accumulated error?
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> favourites and
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>if the
latter is
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>actually more accurate then I can simply revert
the
>>>>>>>>>   > changeset that
>>>>>>>>>   >
>>>>>>>>implemented MIXED.
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> workloads you
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>could try
idle
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> large disc reads
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>to
/dev/null)? We
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> CPU bound, so
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>that''s really an
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   >
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>--
Keir
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   > >>>>>>>>
>>>>>>>>>   >
>>>>>>_______________________________________________
>>>>>>>>>   > >>>>>>Xen-devel
mailing list
>>>>>>>>>   >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>   >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>>
>>>>>>>>>   > >>diff -r cfdbdca5b831
xen/arch/x86/hvm/vpt.c
>>>>>>>>>   > >>--- a/xen/arch/x86/hvm/vpt.c
Thu Dec 06 15:36:07
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> 2007 +0000
>>>>  
>>>>
>>>>>>>>>   > >>+++ b/xen/arch/x86/hvm/vpt.c
Fri Jan 04 17:58:16
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>> 2008 -0500
>>>>  
>>>>
>>>>>>>>>   > >>@@ -58,7 +58,7 @@ static
void
>>>>>>>>
>>>>>>>>
>>>> pt_process_missed_ticks(stru
>>>>  
>>>>
>>>>>>>>>   > >>
>>>>>>>>>   > >>     missed_ticks =
missed_ticks / (s_time_t)
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> pt->period + 1;
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> no_missed_ticks_pending) )
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>   > >>+       
pt->do_not_freeze = 1;
>>>>>>>>>   > >>     else
>>>>>>>>>   > >>        
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>   > >>     pt->scheduled +=
missed_ticks * pt->period;
>>>>>>>>>   > >>@@ -127,7 +127,12 @@ static
void pt_timer_fn(void *data)
>>>>>>>>>   > >>
>>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>>   > >>
>>>>>>>>>   > >>-   
pt->pending_intr_nr++;
>>>>>>>>>   > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>+       
pt->pending_intr_nr = 1;
>>>>>>>>>   > >>+ pt->do_not_freeze = 0;
>>>>>>>>>   > >>+    }
>>>>>>>>>   > >>+    else
>>>>>>>>>   > >>+ pt->pending_intr_nr++;
>>>>>>>>>   > >>
>>>>>>>>>   > >>     if ( !pt->one_shot )
>>>>>>>>>   > >>     {
>>>>>>>>>   > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct
>>>>>>>>
>>>>>>>>
>>>> vcpu *v, struct
>>>>  
>>>>
>>>>>>>>>   > >>         return;
>>>>>>>>>   > >>     }
>>>>>>>>>   > >>
>>>>>>>>>   > >>-    pt->do_not_freeze =
0;
>>>>>>>>>   > >>-
>>>>>>>>>   > >>     if ( pt->one_shot )
>>>>>>>>>   > >>     {
>>>>>>>>>   > >>         pt->enabled = 0;
>>>>>>>>>   > >>@@ -235,6 +238,10 @@ void
pt_intr_post(struct vcpu
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>> *v, struct
>>>>>>>>
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>>   > >>            
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>   > >>            
pt->pending_intr_nr = 0; /* ''collapse'' all
>>>>>>>>>   > missed ticks */
>>>>>>>>>   > >>         }
>>>>>>>>>   > >>+ else if (
mode_is(v->domain,
>>>>>>>>
>>>>>>>>
>>>> no_missed_ticks_pending) ) {
>>>>  
>>>>
>>>>>>>>>   > >>+    
pt->pending_intr_nr--;
>>>>>>>>>   > >>+     pt->last_plt_gtime
= hvm_get_guest_time(v);
>>>>>>>>>   > >>+ }
>>>>>>>>>   > >>         else
>>>>>>>>>   > >>         {
>>>>>>>>>   > >>            
pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>   > >>
>>>>>>>>>   > >>
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   > >
>>>>>>>>>   >
>>>>>>>>>   >
>>>>>>>>>   >
_______________________________________________
>>>>>>>>>   > Xen-devel mailing list
>>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>>   > http://lists.xensource.com/xen-devel
>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>
>>>>>>>>
>>>>>>>>           
>>>>>>>
>>>>>>>
>>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

I ran the ltp tests with 3.2 and found the errors
for a 16 hour run to be:

rh4u564 -9.9 sec (-.017%)
rh4u464 -7.3 sec (-.013%)

There were no cliffs and the drift was linear.

I think the problem you had may be due to the use of the
pm timer. If you still have the boot log, it would tell you.

When I first tried a guest on 3.2 with "clocksource=pit nohpet"
I noticed that it picked the pm timer. Adding "nopmtimer", the
guest will pick the pit.

The reason I didn''t have the problem with our 3.1 base is that
I had disabled the hpet and the pmtimer by not advertising them
in the acpi tables. I did this so long ago, I forgot that I had to
disable pmtimer as well as hpet.

So, can you re-run your test with "clocksource=pit nohpet nopmtimer"?
You should see this in the boot messages:

time.c: Using PIT/TSC based timekeeping.

Thanks,
Dave


Dave Winchell wrote:
> Hi Dan,
>
> Over the weekend the drift was +18 seconds for each guest (no ntp).
> The duration was 3900 minutes, so the error for each was +.0077%.
> Looking back through the data, it appears to drift linearly at
> this rate. I''ve attached a plot for rh4u5-64.
>
> This accuracy is better than what I''ve seen before (.03-.05%).
> This may be due to the different load (ltp vs usex) or to one of the
> changes I''ve made recently. I''ll do some experimentation
to see if
> there is
> a fix I should propose.
>
> This still doesn''t address the radical drift you saw.
> The next step for me is to run 3.2 and see if I can reproduce it.
>
> Regards,
> Dave
>
>
>
>
>
> Dave Winchell wrote:
>
>> Hi Dan,
>>
>> Sorry it took me so long, but I finally ran an ltp test today.
>> Its on rh4u4-64. I''m using the defaults for ltp and using a
script
>> called runltp. I had a usex load on rh4u5-64. No ntpd.
>> virtual processors / physical processors = 2.
>>
>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
>> for -.005% and .008%.
>>
>> I''m running a 3.1 based hypervisor with some clock related
tweaks that
>> I haven''t submitted, because I''m still characterizing
them.
>>
>> I''m stopping the usex load on 4u5-64 now and replacing it with
ltp
>> and will leave the two guests running ltp over the weekend.
>>
>> Regards,
>> Dave
>>
>>
>> Dave Winchell wrote:
>>
>>> Hi Dan, Deepak:
>>>
>>> Thanks for the data. Those drifts are severe - no wonder ntp
couldn''t
>>> keep then in synch. I''ll try to reproduce that behaviour
here, with
>>> my code base.
>>> If I can''t reproduce it, I''ll try 3.2.
>>>
>>> If you can isolate what ltp is doing during the cliffs, that would 
>>> be very
>>> helpful.
>>>
>>> thanks,
>>> Dave
>>>
>>>
>>>
>>>
>>> Dan Magenheimer wrote:
>>>
>>>> OK, Deepak repeated the test without ntpd and using ntpdate -b
before
>>>> the test.
>>>>
>>>> The attached graph shows his results: el5u1-64 (best=~0.07%),
>>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>
>>>> We will continue to look at LTP to try to isolate.
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>> P.S. elXuY is essentially RHEL XuY with some patches.
>>>>
>>>>  
>>>>
>>>>> -----Original Message-----
>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> Sent: Wednesday, January 30, 2008 2:45 PM
>>>>> To: Deepak Patel
>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave
Winchell
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables
>>>>> pending
>>>>> missed ticks
>>>>>
>>>>>
>>>>> Dan, Deeepak,
>>>>>
>>>>> It may be that the underlying clock error is too great for
ntp
>>>>> to handle. It would be useful if you did not run ntpd
>>>>> and, instead did ntpdate -b <timeserver> at the start
of the test
>>>>> for each guest. Then capture the data as you have been
doing.
>>>>> If the drift is greater than .05%, then we need to address
that.
>>>>>
>>>>> Another option is, when running ntpd, to enable loop
statistics in
>>>>> /etc/ntp.conf
>>>>> by adding this to the file:
>>>>>
>>>>> statistics loopstats
>>>>> statsdir /var/lib/ntp/
>>>>>
>>>>> Then you will see loop data in that directory.
>>>>> Correlating the data in the loopstats files with the
>>>>> peaks in skew would be interesting. You will see entries of
the form
>>>>>
>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776
239.735511 10
>>>>>
>>>>> Where the second to last column is the Allan Deviation.
When that
>>>>> gets over 1000, ntpd is working pretty hard. However, I
have not
>>>>> seen ntpd
>>>>> completely loose it like you have.
>>>>>
>>>>> I''m on vacation until Monday, and won''t
be reading
>>>>> email.
>>>>>
>>>>> Thanks for all your work on this!
>>>>>
>>>>> -Dave
>>>>>
>>>>> Deepak Patel wrote:
>>>>>
>>>>>  
>>>>>
>>>>>>> Is the graph for RHEL5u1-64? (I''ve never
tested this one.)
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> I do not know which graph was attached with this. But I
saw this
>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm
guests when I
>>>>>> was running ltp tests continuously.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> What was the behaviour of the other guests running?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> All pvm guests are fine. But behavior of most of the
hvm guests were
>>>>>> as described.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> If they had spikes, were they at the same wall
time?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> No. They are not at the same wall time.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Were the other guests running ltp as well?
>>>>>>>
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are
running ltp
>>>>>> continuously.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> How are you measuring skew?
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> I was collecting output of "ntpdate -q 
<timeserver> every
>>>>>
>>>>>
>>>>>
>>>>> 300 seconds
>>>>>  
>>>>>
>>>>>> (5 minutes) and have created graph based on that.
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Are you running ntpd?
>>>>>>>
>>>>>>>       
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes. ntp was running on all the guests.
>>>>>>
>>>>>> I am investigating what causes this spikes and let
everyone
>>>>>
>>>>>
>>>>>
>>>>> know what
>>>>>  
>>>>>
>>>>>> are my findings.
>>>>>>
>>>>>> Thanks,
>>>>>> Deepak
>>>>>>
>>>>>>   
>>>>>>
>>>>>>> Anything that you can discover that would be in
sync with
>>>>>>> the spikes would be very helpful!
>>>>>>>
>>>>>>> The code that I test with is our product code,
which is based
>>>>>>> on 3.1. So it is possible that something in 3.2
other than vpt.c
>>>>>>> is the cause. I can test with 3.2, if necessary.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan Magenheimer wrote:
>>>>>>>
>>>>>>>     
>>>>>>>
>>>>>>>> Hi Dave (Keir, see suggestion below) --
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Turning off vhpet certainly helps a lot (though
see below).
>>>>>>>>
>>>>>>>> I wonder if timekeeping with vhpet is so bad
that it should be
>>>>>>>> turned off by default (in 3.1, 3.2, and
unstable) until it is
>>>>>>>> fixed?  (I have a patch that defaults it off,
can post it if
>>>>>>>> there is agreement on the above point.)  The
whole point of an
>>>>>>>> HPET is to provide more precise timekeeping and
if vhpet is
>>>>>>>> worse than vpit, it can only confuse users. 
Comments?
>>>>>>>>
>>>>>>>>
>>>>>>>> In your testing, are you just measuring % skew
over a long
>>>>>>>> period of time?
>>>>>>>> We are graphing the skew continuously and
>>>>>>>> seeing periodic behavior that is unsettling,
even with pit.
>>>>>>>> See attached.  Though your algorithm recovers,
the "cliffs"
>>>>>>>> could still cause real user problems.  I wonder
if there is
>>>>>>>> anything that can be done to make the
"recovery" more
>>>>>>>> responsive?
>>>>>>>>
>>>>>>>> We are looking into what part(s) of LTP is
causing the cliffs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>       
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
>>>>>>>>> To: dan.magenheimer@oracle.com
>>>>>>>>> Cc: Keir Fraser;
xen-devel@lists.xensource.com;
>>>>>>>>> deepak.patel@oracle.com;
>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that disables
>>>>>>>>> pending
>>>>>>>>> missed ticks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dan,
>>>>>>>>>
>>>>>>>>> I guess I''m a bit out of date
calling for clock= usage.
>>>>>>>>> Looking at linux 2.6.20.4 sources, I think
you should specify
>>>>>>>>> "clocksource=pit nohpet" on the
linux guest bootline.
>>>>>>>>>
>>>>>>>>> You can leave the xen and dom0 bootlines as
they are.
>>>>>>>>> The xen and guest clocksources do not need
to be the same.
>>>>>>>>> In my tests, xen is using the hpet for its
timekeeping and
>>>>>>>>> that appears to be the default.
>>>>>>>>>
>>>>>>>>> When you boot the guests you should see
>>>>>>>>>   time.c: Using PIT/TSC based timekeeping.
>>>>>>>>> on the rh4u5-64 guest, and something
similar on the others.
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping
04, Platform timer
>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This appears to be the xen state, which is
fine.
>>>>>>>>> I was wrongly assuming that this was the
guest state.
>>>>>>>>> You might want to look in your guest logs
and see what they were
>>>>>>>>> picking
>>>>>>>>> for a clock source.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> Thanks, I hadn''t realized
that!  No wonder we didn''t
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> see the same
>>>>>  
>>>>>
>>>>>>>>>> improvement you saw!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>           
>>>>>>>>>>
>>>>>>>>>>> Try specifying clock=pit on the
linux boot line...
>>>>>>>>>>>
>>>>>>>>>>>               
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I''m confused... do you mean
"clocksource=pit" on the Xen
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> command line or
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> "nohpet" /
"clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> dom0?) command
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> line?  Or both places?  Since the tests
take awhile, it
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> would be nice
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> to get this right the first time.  Do
the Xen and guest
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> clocksources need
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>> to be the same?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Dan
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> *From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22
PM
>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir
Fraser
>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add
a timer mode
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> that disables
>>>>>  
>>>>>
>>>>>>>>>> pending missed ticks
>>>>>>>>>>
>>>>>>>>>>   Hi Dan,
>>>>>>>>>>
>>>>>>>>>>   Hpet timer does have a fairly large
error, as I
>>>>>>>>>> was             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> trying this
>>>>>  
>>>>>
>>>>>>>>>>   one recently.
>>>>>>>>>>   I don''t remember what I got
for error, but 1% sounds
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> about right.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   The problem is that hpet is not built
on top of vpt.c,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the module
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   Keir and I did
>>>>>>>>>>   all the recent work in, for its
periodic timer needs. Try
>>>>>>>>>>   specifying clock=pit
>>>>>>>>>>   on the linux boot line. If it still
picks the hpet, which it
>>>>>>>>>>   might, let me know
>>>>>>>>>>   and I''ll tell you how to get
around this.
>>>>>>>>>>
>>>>>>>>>>   Regards,
>>>>>>>>>>   Dave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
--------------------------------------------------------------
>>>>>>>>> ----------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   *From:* Dan Magenheimer
[mailto:dan.magenheimer@oracle.com]
>>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
>>>>>>>>>>   *Cc:* xen-devel@lists.xensource.com;
deepak.patel@oracle.com;
>>>>>>>>>>   akira.ijuin@oracle.com
>>>>>>>>>>   *Subject:* RE: [Xen-devel] [PATCH]
Add a timer mode
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> that disables
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   pending missed ticks
>>>>>>>>>>
>>>>>>>>>>   Sorry for the very late followup on
this but we finally
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> were able
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   to get our testing set up again on
stable 3.1 bits and have
>>>>>>>>>>   seen some very bad results on
3.1.3-rc1, on the order of 1%.
>>>>>>>>>>
>>>>>>>>>>   Test enviroment was a 4-socket dual
core machine with 24GB of
>>>>>>>>>>   memory running six two-vcpu 2GB
domains, four hvm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> plus two pv.
>>>>>  
>>>>>
>>>>>>>>>>   All six guests were running LTP
simultaneously.  The four hvm
>>>>>>>>>>   guests were: RHEL5u1-64, RHEL4u5-32,
RHEL5-64, and
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> RHEL4u5-64.
>>>>>  
>>>>>
>>>>>>>>>>   Timer_mode was set to 2 for 64-bit
guests and 0 for
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 32-bit guests.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   All four hvm guests experienced skew
around -1%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> even the 32-bit
>>>>>  
>>>>>
>>>>>>>>>>   guest.  Less intensive testing
didn''t exhibit much
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> skew at all.
>>>>>  
>>>>>
>>>>>>>>>>   A representative graph is attached.
>>>>>>>>>>
>>>>>>>>>>   Dave, I wonder if some portion of
your patches
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> didn''t end up in
>>>>>  
>>>>>
>>>>>>>>>>   the xen trees?
>>>>>>>>>>
>>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz
stepping 04, Platform timer
>>>>>>>>>>   14.318MHz HPET.)
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>   Dan
>>>>>>>>>>
>>>>>>>>>>   P.S. Many thanks to Deepak and Akira
for running tests.
>>>>>>>>>>
>>>>>>>>>>   > -----Original Message-----
>>>>>>>>>>   > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>   >
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>   > Dave Winchell
>>>>>>>>>>   > Sent: Wednesday, January 09,
2008 9:53 AM
>>>>>>>>>>   > To: Keir Fraser
>>>>>>>>>>   > Cc: dan.magenheimer@oracle.com;
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> xen-devel@lists.xensource.com; Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > Winchell
>>>>>>>>>>   > Subject: Re: [Xen-devel] [PATCH]
Add a timer mode that
>>>>>>>>>>   > disables pending
>>>>>>>>>>   > missed ticks
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   > Hi Keir,
>>>>>>>>>>   >
>>>>>>>>>>   > The latest change, c/s 16690,
looks fine.
>>>>>>>>>>   > I agree that the code in c/s
16690 is equivalent to
>>>>>>>>>>   > the code I submitted. Also, your
version is more
>>>>>>>>>>   > concise.
>>>>>>>>>>   >
>>>>>>>>>>   > The error tests confirm the
equivalence. With
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> overnight cpu loads,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > the checked in version was
accurate to +.048% for sles
>>>>>>>>>>   > and +.038% for red hat. My
version was +.046%
>>>>>>>>>> and             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> +.032% in a
>>>>>  
>>>>>
>>>>>>>>>>   > 2 hour test.
>>>>>>>>>>   > I don''t think the
difference is significant.
>>>>>>>>>>   >
>>>>>>>>>>   > i/o loads produced errors of
+.01%.
>>>>>>>>>>   >
>>>>>>>>>>   > Thanks for all your efforts on
this issue.
>>>>>>>>>>   >
>>>>>>>>>>   > Regards,
>>>>>>>>>>   > Dave
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   > Keir Fraser wrote:
>>>>>>>>>>   >
>>>>>>>>>>   > >Applied as c/s 16690,
although the checked-in patch is
>>>>>>>>>>   > smaller. I think the
>>>>>>>>>>   > >only important fix is to
pt_intr_post() and the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> only bit of
>>>>>  
>>>>>
>>>>>>>>>>   > the patch I
>>>>>>>>>>   > >totally omitted was the
change to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> pt_process_missed_ticks().
>>>>>  
>>>>>
>>>>>>>>>>   > I don''t think
>>>>>>>>>>   > >that change can be
important, but let''s see what
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> happens to the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   error
>>>>>>>>>>   > >percentage...
>>>>>>>>>>   > >
>>>>>>>>>>   > > -- Keir
>>>>>>>>>>   > >
>>>>>>>>>>   > >On 4/1/08 23:24, "Dave
Winchell"
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <dwinchell@virtualiron.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >>Hi Dan and Keir,
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Attached is a patch that
fixes some issues with the
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SYNC policy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>(no_missed_ticks_pending).
>>>>>>>>>>   > >>I have not tried to make
the change the minimal one, but,
>>>>>>>>>>   > rather, just
>>>>>>>>>>   > >>ported into
>>>>>>>>>>   > >>the new code what I know
to work well. The error for
>>>>>>>>>>   > >>no_missed_ticks_pending
goes from
>>>>>>>>>>   > >>over 3% to .03% with
this change according to my testing.
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Regards,
>>>>>>>>>>   > >>Dave
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>Dan Magenheimer wrote:
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>>Hi Dave --
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Did you get your
correction ported?  If so, it would be
>>>>>>>>>>   > nice to see this get
>>>>>>>>>>   > >>>into 3.1.3.
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Note that I just did
some very limited testing with
>>>>>>>>>>   > timer_mode=2(=SYNC=no
>>>>>>>>>>   > >>>missed ticks
pending)
>>>>>>>>>>   > >>>on tip of
xen-3.1-testing (64-bit Linux hv
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> guest) and the
>>>>>  
>>>>>
>>>>>>>>>>   > worst error I''ve
>>>>>>>>>>   > >>>seen so far
>>>>>>>>>>   > >>>is 0.012%.  But I
haven''t tried any exotic
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> loads, just LTP.
>>>>>  
>>>>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>Thanks,
>>>>>>>>>>   > >>>Dan
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>>-----Original
Message-----
>>>>>>>>>>   > >>>>From: Dave
Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>   > >>>>Sent: Wednesday,
December 19, 2007 12:33 PM
>>>>>>>>>>   > >>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>>   > >>>>Cc: Keir Fraser;
Shan, Haitao;
>>>>>>>>>>   > xen-devel@lists.xensource.com;
Dong,
>>>>>>>>>>   > >>>>Eddie; Jiang,
Yunhong; Dave Winchell
>>>>>>>>>>   > >>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>   > >>>>disables pending
>>>>>>>>>>   > >>>>missed ticks
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Dan,
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>I did some
testing with the constant tsc offset
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SYNC method
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>(now called
>>>>>>>>>>   >
>>>>no_missed_ticks_pending)
>>>>>>>>>>   > >>>>and found the
error to be very high, much larger
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> than 1 %, as
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>I recall.
>>>>>>>>>>   > >>>>I have not had a
chance to submit a correction. I
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> will try to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>do it later
>>>>>>>>>>   > >>>>this week or the
first week in January. My version of
>>>>>>>>>>   constant tsc
>>>>>>>>>>   > >>>>offset SYNC
method
>>>>>>>>>>   > >>>>produces .02 %
error, so I just need to port
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> that into the
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>current code.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>The error you
got for both of those kernels is
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> what I would
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   expect
>>>>>>>>>>   > >>>>for the default
mode, delay_for_missed_ticks.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>I''ll
let Keir answer on how to set the time mode.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Regards,
>>>>>>>>>>   > >>>>Dave
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>Dan Magenheimer
wrote:
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>Anyone make
measurements on the final patch?
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>I just ran a
64-bit RHEL5.1 pvm kernel and
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> saw a loss of
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>about 0.2% with
no load.  This was
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> xen-unstable tip today
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>with no options
specified.  32-bit was about 0.01%.
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>I think I
missed something... how do I run the various
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>accounting
choices and which ones are known to be
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> appropriate
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>for which
kernels?
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>Thanks,
>>>>>>>>>>   > >>>>>Dan
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   >
>>>>>>-----Original Message-----
>>>>>>>>>>   > >>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>   >
>>>>>>>>>>
>>>>>>>>>>           
>>>>>>>>>>
>>>>>>>>>>>>>>>
[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                     
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>Keir Fraser
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>>Sent:
Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>   > >>>>>>To: Dave
Winchell
>>>>>>>>>>   > >>>>>>Cc:
Shan, Haitao;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> xen-devel@lists.xensource.com; Dong,
>>>>>  
>>>>>
>>>>>>>>>>   > Eddie; Jiang,
>>>>>>>>>>   > >>>>>>Yunhong
>>>>>>>>>>   > >>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> mode that
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>disables
pending
>>>>>>>>>>   > >>>>>>missed
ticks
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>Please
take a look at xen-unstable changeset 16545.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>-- Keir
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>On
26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>Keir,
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The
accuracy data I''ve collected for i/o
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> loads for the
>>>>>  
>>>>>
>>>>>>>>>>   >
>>>>>>>various time protocols follows. In
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> addition, the data
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>for
cpu loads is shown.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The
loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> processor AMD
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   box.
>>>>>>>>>>   > >>>>>>>Two
guests, red hat and sles 64 bit, 8 vcpu each.
>>>>>>>>>>   > >>>>>>>The
cpu load is usex -e36 on each guest.
>>>>>>>>>>   >
>>>>>>>(usex is available at
>>>>>>>>>>  
http://people.redhat.com/anderson/usex.)
>>>>>>>>>>   > >>>>>>>i/o
load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> of=/dev/null.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The
loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>>   >
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>   > >>>>>>>In
addition, there is an idle rh-32bit guest.
>>>>>>>>>>   > >>>>>>>All
three guests are 8vcpu.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The
loads labeled i/o-4/32 are the same as i/o-32
>>>>>>>>>>   >
>>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> instances of dd.
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Date
Duration Protocol sles, rhat error load
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>> +4.42             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec -.006%,
>>>>>  
>>>>>
>>>>>>>>>>   > +.005% cpu
>>>>>>>>>>   >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.001%,
>>>>>  
>>>>>
>>>>>>>>>>   > +.012% cpu
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%,
>>>>>>>>>>   -.004% cpu
>>>>>>>>>>   >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.005%, -.005% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.008%, -.003% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.040% cpu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.034%,
>>>>>  
>>>>>
>>>>>>>>>>   > -.031% cpu
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
>>>>>>>>>>   > -.09% i/o-8
>>>>>>>>>>   >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015%
>>>>>>>>>>   > -.14% i/o-8
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
>>>>>>>>>>   > -.022% i/o-8
>>>>>>>>>>   >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
>>>>>>>>>>   > -.029% i/o-8
>>>>>>>>>>   >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> sec, -.023%,
>>>>>  
>>>>>
>>>>>>>>>>   > -.031% i/o-8
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.04% i/o-32
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%,
>>>>>>>>>>   > -.005% i/o-32
>>>>>>>>>>   >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -.11% i/o-32
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
>>>>>>>>>>   > .003% i/o-4/32
>>>>>>>>>>   >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%,
>>>>>>>>>>   > .01% i/o-4/32
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>Overhead measurements:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>Progress in terms of number of passes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> through a fixed
>>>>>  
>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>system
workload
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>on
an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>   > >>>>>>>The
workload was usex -b48.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>   > >>>>>>>SYNC
167 min 144 passes .862 passes/min
>>>>>>>>>>   > >>>>>>>SYNC
1065 min 919 passes .863 passes/min
>>>>>>>>>>   >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>Conclusions:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>The
only protocol which meets the .05% accuracy
>>>>>>>>>>   > requirement for ntp
>>>>>>>>>>   >
>>>>>>>tracking under the loads
>>>>>>>>>>   >
>>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> accuracies for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>SYNC,
MIXED,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>and
ASYNC
>>>>>>>>>>   > >>>>>>>are
.022%, .12%, and .14%, respectively.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>We
could reduce the cost of the SYNC method by only
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>scheduling the extra
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>wakeups if a certain number
>>>>>>>>>>   > >>>>>>>of
ticks are missed.
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>Regards,
>>>>>>>>>>   > >>>>>>>Dave
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>Keir
Fraser wrote:
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   > >>>>>>>
>>>>>>>>>>   >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ASYNC method a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>couple
of days ago,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>I ran another ASYNC test. I think there may
have
>>>>>>>>>>   > been something
>>>>>>>>>>   >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> days ago for
>>>>>  
>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>ASYNC.
It may have been
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> after context
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>switch
in.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>My results indicate that either SYNC or
ASYNC give
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>acceptable accuracy,
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>each running consistently around or under
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> .01%. MIXED has
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>a fairly
high
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>error of
>>>>>>>>>>   >
>>>>>>>>>greater than .03%. Probably too close to
.05% ntp
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>threshold for comfort.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> plan to leave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>SYNC
running
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>over the weekend. If you''d rather I
can
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> leave MIXED
>>>>>  
>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>running
instead.
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can run
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   > >>>>>>more
overnight tests
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>>next week.
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>>
>>>>>>>>>>   >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> effects of the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>SYNC+run_timer
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cause higher
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>system-wide CPU
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>contention. I find it easier to think through
the
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>implications of ASYNC. I''m
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> accurate than
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>ASYNC.
Perhaps it
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> approaches,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>and each
interrupt
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>event causes a small accumulated error?
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> favourites and
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>if the
latter is
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>actually more accurate then I can simply revert
the
>>>>>>>>>>   > changeset that
>>>>>>>>>>   >
>>>>>>>>implemented MIXED.
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> workloads you
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>could
try idle
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> large disc reads
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>to
/dev/null)? We
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> CPU bound, so
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>that''s really an
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   >
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>>>-- Keir
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   > >>>>>>>>
>>>>>>>>>>   >
>>>>>>_______________________________________________
>>>>>>>>>>   >
>>>>>>Xen-devel mailing list
>>>>>>>>>>   >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>   >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>>
>>>>>>>>>>   > >>diff -r cfdbdca5b831
xen/arch/x86/hvm/vpt.c
>>>>>>>>>>   > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> 2007 +0000
>>>>>  
>>>>>
>>>>>>>>>>   > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> 2008 -0500
>>>>>  
>>>>>
>>>>>>>>>>   > >>@@ -58,7 +58,7 @@ static
void
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> pt_process_missed_ticks(stru
>>>>>  
>>>>>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     missed_ticks =
missed_ticks / (s_time_t)
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> pt->period + 1;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> no_missed_ticks_pending) )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>   > >>+       
pt->do_not_freeze = 1;
>>>>>>>>>>   > >>     else
>>>>>>>>>>   > >>        
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>   > >>     pt->scheduled +=
missed_ticks * pt->period;
>>>>>>>>>>   > >>@@ -127,7 +127,12 @@
static void pt_timer_fn(void *data)
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     pt_lock(pt);
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>-   
pt->pending_intr_nr++;
>>>>>>>>>>   > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>+       
pt->pending_intr_nr = 1;
>>>>>>>>>>   > >>+ pt->do_not_freeze =
0;
>>>>>>>>>>   > >>+    }
>>>>>>>>>>   > >>+    else
>>>>>>>>>>   > >>+
pt->pending_intr_nr++;
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>     if (
!pt->one_shot )
>>>>>>>>>>   > >>     {
>>>>>>>>>>   > >>@@ -221,8 +226,6 @@ void
pt_intr_post(struct
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> vcpu *v, struct
>>>>>  
>>>>>
>>>>>>>>>>   > >>         return;
>>>>>>>>>>   > >>     }
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>-   
pt->do_not_freeze = 0;
>>>>>>>>>>   > >>-
>>>>>>>>>>   > >>     if (
pt->one_shot )
>>>>>>>>>>   > >>     {
>>>>>>>>>>   > >>         pt->enabled
= 0;
>>>>>>>>>>   > >>@@ -235,6 +238,10 @@
void pt_intr_post(struct vcpu
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *v, struct
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         
>>>>>>>>>
>>>>>>>>>>   > >>            
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>   > >>            
pt->pending_intr_nr = 0; /* ''collapse'' all
>>>>>>>>>>   > missed ticks */
>>>>>>>>>>   > >>         }
>>>>>>>>>>   > >>+ else if (
mode_is(v->domain,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> no_missed_ticks_pending) ) {
>>>>>  
>>>>>
>>>>>>>>>>   > >>+    
pt->pending_intr_nr--;
>>>>>>>>>>   > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>   > >>+ }
>>>>>>>>>>   > >>         else
>>>>>>>>>>   > >>         {
>>>>>>>>>>   > >>            
pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>>   > >>
>>>>>>>>>>   > >>
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   > >
>>>>>>>>>>   >
>>>>>>>>>>   >
>>>>>>>>>>   >
_______________________________________________
>>>>>>>>>>   > Xen-devel mailing list
>>>>>>>>>>   > Xen-devel@lists.xensource.com
>>>>>>>>>>   >
http://lists.xensource.com/xen-devel
>>>>>>>>>>   >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>           
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>
>>
>
>
> ------------------------------------------------------------------------
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave --

Thanks for continuing to run tests!

Hmmm... I thought I had noticed that even though Linux will acknowledge
the existence of the pmtimer, it still prints:

time.c: Using PIT/TSC based timekeeping.

I will check again, but assuming the clocksource for our tests is
indeed pit, the huge difference in the results (yours vs ours) is
baffling. I wonder if the difference may be the underlying hardware.
Maybe we will try to ensure we can duplicate the results on a different
box.

So your testing was with stock 3.2.0 xen bits (what cset?) without
any of your [quote from below] "clock related tweaks that I
haven''t
submitted, because I''m still characterizing them"?

Could you also send detail on the rhel4u4-64 kernel you
are testing with, just to ensure we are not comparing apples
and oranges?  (Perhaps there''s some way we can even share the
identical disk image and vm.cfg file?)

And if our problem is indeed the pmtimer, I will need to submit
another patch to Keir to add an hvm pmtimer platform variable.
(Hmmm... I don''t think he''s even accepted the hpet variable
patch
yet.  I''ll have to check.)

Thanks,
Dan

> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Thursday, February 14, 2008 9:00 AM
> To: dan.magenheimer@oracle.com
> Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
> Patel
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dan,
> 
> I ran the ltp tests with 3.2 and found the errors
> for a 16 hour run to be:
> 
> rh4u564 -9.9 sec (-.017%)
> rh4u464 -7.3 sec (-.013%)
> 
> There were no cliffs and the drift was linear.
> 
> I think the problem you had may be due to the use of the
> pm timer. If you still have the boot log, it would tell you.
> 
> When I first tried a guest on 3.2 with "clocksource=pit nohpet"
> I noticed that it picked the pm timer. Adding "nopmtimer", the
> guest will pick the pit.
> 
> The reason I didn''t have the problem with our 3.1 base is that
> I had disabled the hpet and the pmtimer by not advertising them
> in the acpi tables. I did this so long ago, I forgot that I had to
> disable pmtimer as well as hpet.
> 
> So, can you re-run your test with "clocksource=pit nohpet
nopmtimer"?
> You should see this in the boot messages:
> 
> time.c: Using PIT/TSC based timekeeping.
> 
> Thanks,
> Dave
> 
> 
> Dave Winchell wrote:
> 
> > Hi Dan,
> >
> > Over the weekend the drift was +18 seconds for each guest (no ntp).
> > The duration was 3900 minutes, so the error for each was +.0077%.
> > Looking back through the data, it appears to drift linearly at
> > this rate. I''ve attached a plot for rh4u5-64.
> >
> > This accuracy is better than what I''ve seen before
(.03-.05%).
> > This may be due to the different load (ltp vs usex) or to one of the
> > changes I''ve made recently. I''ll do some
experimentation to see if
> > there is
> > a fix I should propose.
> >
> > This still doesn''t address the radical drift you saw.
> > The next step for me is to run 3.2 and see if I can reproduce it.
> >
> > Regards,
> > Dave
> >
> >
> >
> >
> >
> > Dave Winchell wrote:
> >
> >> Hi Dan,
> >>
> >> Sorry it took me so long, but I finally ran an ltp test today.
> >> Its on rh4u4-64. I''m using the defaults for ltp and using
a script
> >> called runltp. I had a usex load on rh4u5-64. No ntpd.
> >> virtual processors / physical processors = 2.
> >>
> >> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes
> >> for -.005% and .008%.
> >>
> >> I''m running a 3.1 based hypervisor with some clock
related
> tweaks that
> >> I haven''t submitted, because I''m still
characterizing them.
> >>
> >> I''m stopping the usex load on 4u5-64 now and replacing it
with ltp
> >> and will leave the two guests running ltp over the weekend.
> >>
> >> Regards,
> >> Dave
> >>
> >>
> >> Dave Winchell wrote:
> >>
> >>> Hi Dan, Deepak:
> >>>
> >>> Thanks for the data. Those drifts are severe - no wonder 
> ntp couldn''t
> >>> keep then in synch. I''ll try to reproduce that
behaviour
> here, with
> >>> my code base.
> >>> If I can''t reproduce it, I''ll try 3.2.
> >>>
> >>> If you can isolate what ltp is doing during the cliffs, that
would
> >>> be very
> >>> helpful.
> >>>
> >>> thanks,
> >>> Dave
> >>>
> >>>
> >>>
> >>>
> >>> Dan Magenheimer wrote:
> >>>
> >>>> OK, Deepak repeated the test without ntpd and using 
> ntpdate -b before
> >>>> the test.
> >>>>
> >>>> The attached graph shows his results: el5u1-64
(best=~0.07%),
> >>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
> >>>>
> >>>> We will continue to look at LTP to try to isolate.
> >>>>
> >>>> Thanks,
> >>>> Dan
> >>>>
> >>>> P.S. elXuY is essentially RHEL XuY with some patches.
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>> Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>> To: Deepak Patel
> >>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com;
> Dave Winchell
> >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables
> >>>>> pending
> >>>>> missed ticks
> >>>>>
> >>>>>
> >>>>> Dan, Deeepak,
> >>>>>
> >>>>> It may be that the underlying clock error is too great
for ntp
> >>>>> to handle. It would be useful if you did not run ntpd
> >>>>> and, instead did ntpdate -b <timeserver> at the
start
> of the test
> >>>>> for each guest. Then capture the data as you have been
doing.
> >>>>> If the drift is greater than .05%, then we need to
address that.
> >>>>>
> >>>>> Another option is, when running ntpd, to enable loop 
> statistics in
> >>>>> /etc/ntp.conf
> >>>>> by adding this to the file:
> >>>>>
> >>>>> statistics loopstats
> >>>>> statsdir /var/lib/ntp/
> >>>>>
> >>>>> Then you will see loop data in that directory.
> >>>>> Correlating the data in the loopstats files with the
> >>>>> peaks in skew would be interesting. You will see 
> entries of the form
> >>>>>
> >>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 
> 239.735511 10
> >>>>>
> >>>>> Where the second to last column is the Allan
Deviation.
> When that
> >>>>> gets over 1000, ntpd is working pretty hard. However,
I have not
> >>>>> seen ntpd
> >>>>> completely loose it like you have.
> >>>>>
> >>>>> I''m on vacation until Monday, and
won''t be reading
> >>>>> email.
> >>>>>
> >>>>> Thanks for all your work on this!
> >>>>>
> >>>>> -Dave
> >>>>>
> >>>>> Deepak Patel wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> Is the graph for RHEL5u1-64? (I''ve
never tested this one.)
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I do not know which graph was attached with this.
But
> I saw this
> >>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64
hvm
> guests when I
> >>>>>> was running ltp tests continuously.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> What was the behaviour of the other guests
running?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> All pvm guests are fine. But behavior of most of
the
> hvm guests were
> >>>>>> as described.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> If they had spikes, were they at the same wall
time?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> No. They are not at the same wall time.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Were the other guests running ltp as well?
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are
running ltp
> >>>>>> continuously.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> How are you measuring skew?
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I was collecting output of "ntpdate -q 
<timeserver> every
> >>>>>
> >>>>>
> >>>>>
> >>>>> 300 seconds
> >>>>>
> >>>>>
> >>>>>> (5 minutes) and have created graph based on that.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Are you running ntpd?
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Yes. ntp was running on all the guests.
> >>>>>>
> >>>>>> I am investigating what causes this spikes and let
everyone
> >>>>>
> >>>>>
> >>>>>
> >>>>> know what
> >>>>>
> >>>>>
> >>>>>> are my findings.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Deepak
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Anything that you can discover that would be
in sync with
> >>>>>>> the spikes would be very helpful!
> >>>>>>>
> >>>>>>> The code that I test with is our product code,
which is based
> >>>>>>> on 3.1. So it is possible that something in
3.2 other
> than vpt.c
> >>>>>>> is the cause. I can test with 3.2, if
necessary.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Dave
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Dan Magenheimer wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Dave (Keir, see suggestion below) --
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Turning off vhpet certainly helps a lot
(though see below).
> >>>>>>>>
> >>>>>>>> I wonder if timekeeping with vhpet is so
bad that it
> should be
> >>>>>>>> turned off by default (in 3.1, 3.2, and
unstable) until it is
> >>>>>>>> fixed?  (I have a patch that defaults it
off, can post it if
> >>>>>>>> there is agreement on the above point.) 
The whole
> point of an
> >>>>>>>> HPET is to provide more precise
timekeeping and if vhpet is
> >>>>>>>> worse than vpit, it can only confuse
users.  Comments?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In your testing, are you just measuring %
skew over a long
> >>>>>>>> period of time?
> >>>>>>>> We are graphing the skew continuously and
> >>>>>>>> seeing periodic behavior that is
unsettling, even with pit.
> >>>>>>>> See attached.  Though your algorithm
recovers, the "cliffs"
> >>>>>>>> could still cause real user problems.  I
wonder if there is
> >>>>>>>> anything that can be done to make the
"recovery" more
> >>>>>>>> responsive?
> >>>>>>>>
> >>>>>>>> We are looking into what part(s) of LTP is
causing
> the cliffs.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Dan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM
> >>>>>>>>> To: dan.magenheimer@oracle.com
> >>>>>>>>> Cc: Keir Fraser;
xen-devel@lists.xensource.com;
> >>>>>>>>> deepak.patel@oracle.com;
> >>>>>>>>> akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a
timer mode
> that disables
> >>>>>>>>> pending
> >>>>>>>>> missed ticks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Dan,
> >>>>>>>>>
> >>>>>>>>> I guess I''m a bit out of date
calling for clock= usage.
> >>>>>>>>> Looking at linux 2.6.20.4 sources, I
think you
> should specify
> >>>>>>>>> "clocksource=pit nohpet" on
the linux guest bootline.
> >>>>>>>>>
> >>>>>>>>> You can leave the xen and dom0
bootlines as they are.
> >>>>>>>>> The xen and guest clocksources do not
need to be the same.
> >>>>>>>>> In my tests, xen is using the hpet for
its timekeeping and
> >>>>>>>>> that appears to be the default.
> >>>>>>>>>
> >>>>>>>>> When you boot the guests you should
see
> >>>>>>>>>   time.c: Using PIT/TSC based
timekeeping.
> >>>>>>>>> on the rh4u5-64 guest, and something
similar on the others.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz
stepping 04, Platform timer
> >>>>>>>>>> 14.318MHz HPET.)
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This appears to be the xen state,
which is fine.
> >>>>>>>>> I was wrongly assuming that this was
the guest state.
> >>>>>>>>> You might want to look in your guest
logs and see
> what they were
> >>>>>>>>> picking
> >>>>>>>>> for a clock source.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Thanks, I hadn''t realized
that!  No wonder we didn''t
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> see the same
> >>>>>
> >>>>>
> >>>>>>>>>> improvement you saw!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Try specifying clock=pit on
the linux boot line...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I''m confused... do you
mean "clocksource=pit" on the Xen
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> command line or
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> "nohpet" /
"clock=pit" / "clocksource=pit" on the guest (or
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> dom0?) command
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> line?  Or both places?  Since the
tests take awhile, it
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> would be nice
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> to get this right the first time. 
Do the Xen and guest
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> clocksources need
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> to be the same?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Dan
> >>>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> *From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>>> *Sent:* Sunday, January 27, 2008
2:22 PM
> >>>>>>>>>> *To:* dan.magenheimer@oracle.com;
Keir Fraser
> >>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
> deepak.patel@oracle.com;
> >>>>>>>>>> akira.ijuin@oracle.com; Dave
Winchell
> >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH]
Add a timer mode
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> that disables
> >>>>>
> >>>>>
> >>>>>>>>>> pending missed ticks
> >>>>>>>>>>
> >>>>>>>>>>   Hi Dan,
> >>>>>>>>>>
> >>>>>>>>>>   Hpet timer does have a fairly
large error, as I
> >>>>>>>>>> was
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> trying this
> >>>>>
> >>>>>
> >>>>>>>>>>   one recently.
> >>>>>>>>>>   I don''t remember what I
got for error, but 1% sounds
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> about right.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   The problem is that hpet is not
built on top of vpt.c,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> the module
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   Keir and I did
> >>>>>>>>>>   all the recent work in, for its
periodic timer needs. Try
> >>>>>>>>>>   specifying clock=pit
> >>>>>>>>>>   on the linux boot line. If it
still picks the
> hpet, which it
> >>>>>>>>>>   might, let me know
> >>>>>>>>>>   and I''ll tell you how
to get around this.
> >>>>>>>>>>
> >>>>>>>>>>   Regards,
> >>>>>>>>>>   Dave
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 
> --------------------------------------------------------------
> >>>>>>>>> ----------
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   *From:* Dan Magenheimer 
> [mailto:dan.magenheimer@oracle.com]
> >>>>>>>>>>   *Sent:* Fri 1/25/2008 6:50 PM
> >>>>>>>>>>   *To:* Dave Winchell; Keir Fraser
> >>>>>>>>>>   *Cc:*
xen-devel@lists.xensource.com;
> deepak.patel@oracle.com;
> >>>>>>>>>>   akira.ijuin@oracle.com
> >>>>>>>>>>   *Subject:* RE: [Xen-devel]
[PATCH] Add a timer mode
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> that disables
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   pending missed ticks
> >>>>>>>>>>
> >>>>>>>>>>   Sorry for the very late followup
on this but we finally
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> were able
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   to get our testing set up again
on stable 3.1
> bits and have
> >>>>>>>>>>   seen some very bad results on
3.1.3-rc1, on the
> order of 1%.
> >>>>>>>>>>
> >>>>>>>>>>   Test enviroment was a 4-socket
dual core machine
> with 24GB of
> >>>>>>>>>>   memory running six two-vcpu 2GB
domains, four hvm
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> plus two pv.
> >>>>>
> >>>>>
> >>>>>>>>>>   All six guests were running LTP
simultaneously.
> The four hvm
> >>>>>>>>>>   guests were: RHEL5u1-64,
RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> RHEL4u5-64.
> >>>>>
> >>>>>
> >>>>>>>>>>   Timer_mode was set to 2 for
64-bit guests and 0 for
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 32-bit guests.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   All four hvm guests experienced
skew around -1%,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> even the 32-bit
> >>>>>
> >>>>>
> >>>>>>>>>>   guest.  Less intensive testing
didn''t exhibit much
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> skew at all.
> >>>>>
> >>>>>
> >>>>>>>>>>   A representative graph is
attached.
> >>>>>>>>>>
> >>>>>>>>>>   Dave, I wonder if some portion
of your patches
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> didn''t end up in
> >>>>>
> >>>>>
> >>>>>>>>>>   the xen trees?
> >>>>>>>>>>
> >>>>>>>>>>   (xm dmesg shows 8x Xeon 3.2GHz
stepping 04,
> Platform timer
> >>>>>>>>>>   14.318MHz HPET.)
> >>>>>>>>>>
> >>>>>>>>>>   Thanks,
> >>>>>>>>>>   Dan
> >>>>>>>>>>
> >>>>>>>>>>   P.S. Many thanks to Deepak and
Akira for running tests.
> >>>>>>>>>>
> >>>>>>>>>>   > -----Original Message-----
> >>>>>>>>>>   > From:
xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>   > 
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>   > Dave Winchell
> >>>>>>>>>>   > Sent: Wednesday, January
09, 2008 9:53 AM
> >>>>>>>>>>   > To: Keir Fraser
> >>>>>>>>>>   > Cc:
dan.magenheimer@oracle.com;
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> xen-devel@lists.xensource.com; Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > Winchell
> >>>>>>>>>>   > Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
> >>>>>>>>>>   > disables pending
> >>>>>>>>>>   > missed ticks
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   > Hi Keir,
> >>>>>>>>>>   >
> >>>>>>>>>>   > The latest change, c/s
16690, looks fine.
> >>>>>>>>>>   > I agree that the code in
c/s 16690 is equivalent to
> >>>>>>>>>>   > the code I submitted. Also,
your version is more
> >>>>>>>>>>   > concise.
> >>>>>>>>>>   >
> >>>>>>>>>>   > The error tests confirm the
equivalence. With
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> overnight cpu loads,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > the checked in version was
accurate to +.048% for sles
> >>>>>>>>>>   > and +.038% for red hat. My
version was +.046%
> >>>>>>>>>> and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> +.032% in a
> >>>>>
> >>>>>
> >>>>>>>>>>   > 2 hour test.
> >>>>>>>>>>   > I don''t think the
difference is significant.
> >>>>>>>>>>   >
> >>>>>>>>>>   > i/o loads produced errors
of +.01%.
> >>>>>>>>>>   >
> >>>>>>>>>>   > Thanks for all your efforts
on this issue.
> >>>>>>>>>>   >
> >>>>>>>>>>   > Regards,
> >>>>>>>>>>   > Dave
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   > Keir Fraser wrote:
> >>>>>>>>>>   >
> >>>>>>>>>>   > >Applied as c/s 16690,
although the checked-in patch is
> >>>>>>>>>>   > smaller. I think the
> >>>>>>>>>>   > >only important fix is
to pt_intr_post() and the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> only bit of
> >>>>>
> >>>>>
> >>>>>>>>>>   > the patch I
> >>>>>>>>>>   > >totally omitted was the
change to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> pt_process_missed_ticks().
> >>>>>
> >>>>>
> >>>>>>>>>>   > I don''t think
> >>>>>>>>>>   > >that change can be
important, but let''s see what
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> happens to the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   error
> >>>>>>>>>>   > >percentage...
> >>>>>>>>>>   > >
> >>>>>>>>>>   > > -- Keir
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >On 4/1/08 23:24,
"Dave Winchell"
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> <dwinchell@virtualiron.com>
wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >>Hi Dan and Keir,
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Attached is a patch
that fixes some issues with the
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> SYNC policy
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>(no_missed_ticks_pending).
> >>>>>>>>>>   > >>I have not tried to
make the change the
> minimal one, but,
> >>>>>>>>>>   > rather, just
> >>>>>>>>>>   > >>ported into
> >>>>>>>>>>   > >>the new code what I
know to work well. The error for
> >>>>>>>>>>   >
>>no_missed_ticks_pending goes from
> >>>>>>>>>>   > >>over 3% to .03%
with this change according
> to my testing.
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Regards,
> >>>>>>>>>>   > >>Dave
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>Dan Magenheimer
wrote:
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>>Hi Dave --
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Did you get
your correction ported?  If so,
> it would be
> >>>>>>>>>>   > nice to see this get
> >>>>>>>>>>   > >>>into 3.1.3.
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Note that I
just did some very limited testing with
> >>>>>>>>>>   > timer_mode=2(=SYNC=no
> >>>>>>>>>>   > >>>missed ticks
pending)
> >>>>>>>>>>   > >>>on tip of
xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> guest) and the
> >>>>>
> >>>>>
> >>>>>>>>>>   > worst error I''ve
> >>>>>>>>>>   > >>>seen so far
> >>>>>>>>>>   > >>>is 0.012%.  But
I haven''t tried any exotic
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> loads, just LTP.
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>Thanks,
> >>>>>>>>>>   > >>>Dan
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   >
>>>>-----Original Message-----
> >>>>>>>>>>   > >>>>From: Dave
Winchell
> [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>   > >>>>Sent:
Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>   > >>>>To:
dan.magenheimer@oracle.com
> >>>>>>>>>>   > >>>>Cc: Keir
Fraser; Shan, Haitao;
> >>>>>>>>>>   >
xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>   > >>>>Eddie;
Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>   > >>>>Subject:
Re: [Xen-devel] [PATCH] Add a
> timer mode that
> >>>>>>>>>>   > >>>>disables
pending
> >>>>>>>>>>   > >>>>missed
ticks
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Dan,
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>I did some
testing with the constant tsc offset
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> SYNC method
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>(now called
> >>>>>>>>>>   >
>>>>no_missed_ticks_pending)
> >>>>>>>>>>   > >>>>and found
the error to be very high, much larger
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> than 1 %, as
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>I recall.
> >>>>>>>>>>   > >>>>I have not
had a chance to submit a correction. I
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> will try to
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>do it later
> >>>>>>>>>>   > >>>>this week
or the first week in January. My
> version of
> >>>>>>>>>>   constant tsc
> >>>>>>>>>>   > >>>>offset SYNC
method
> >>>>>>>>>>   > >>>>produces
.02 % error, so I just need to port
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> that into the
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>current
code.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>The error
you got for both of those kernels is
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> what I would
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   expect
> >>>>>>>>>>   > >>>>for the
default mode, delay_for_missed_ticks.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   >
>>>>I''ll let Keir answer on how to set the time mode.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Regards,
> >>>>>>>>>>   > >>>>Dave
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>Dan
Magenheimer wrote:
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>Anyone
make measurements on the final patch?
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>I just
ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> saw a loss of
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>about 0.2%
with no load.  This was
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> xen-unstable tip today
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>>>with no
options specified.  32-bit was about 0.01%.
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>I think
I missed something... how do I
> run the various
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>accounting
choices and which ones are known to be
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> appropriate
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>>>for which
kernels?
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>Thanks,
> >>>>>>>>>>   > >>>>>Dan
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   >
>>>>>>-----Original Message-----
> >>>>>>>>>>   >
>>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>   >
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> 
> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>Keir Fraser
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>   > >>>>>>To:
Dave Winchell
> >>>>>>>>>>   > >>>>>>Cc:
Shan, Haitao;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> xen-devel@lists.xensource.com; Dong,
> >>>>>
> >>>>>
> >>>>>>>>>>   > Eddie; Jiang,
> >>>>>>>>>>   >
>>>>>>Yunhong
> >>>>>>>>>>   >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> mode that
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>disables pending
> >>>>>>>>>>   >
>>>>>>missed ticks
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>Please take a look at xen-unstable
> changeset 16545.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>--
Keir
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>On
26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>Keir,
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>The accuracy data I''ve collected for i/o
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> loads for the
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>various time protocols follows. In
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> addition, the data
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>for cpu loads is shown.
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> processor AMD
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   box.
> >>>>>>>>>>   >
>>>>>>>Two guests, red hat and sles 64 bit, 8
> vcpu each.
> >>>>>>>>>>   >
>>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>   >
>>>>>>>(usex is available at
> >>>>>>>>>>  
http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>   >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> of=/dev/null.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>The loads labeled i/o-32 are 32 instances of dd.
> >>>>>>>>>>   >
>>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>   >
>>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>   >
>>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>The loads labeled i/o-4/32 are the same
> as i/o-32
> >>>>>>>>>>   >
>>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> instances of dd.
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>> +4.42
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec -.006%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > +.005% cpu
> >>>>>>>>>>   >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.001%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > +.012% cpu
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> sec, -.009%,
> >>>>>>>>>>   -.004% cpu
> >>>>>>>>>>   >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.005%, -.005% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.008%, -.003% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.040% cpu
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.034%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > -.031% cpu
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> sec,-55.7 sec, -.01%,
> >>>>>>>>>>   > -.09% i/o-8
> >>>>>>>>>>   >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> sec,-14.0 sec, -.015%
> >>>>>>>>>>   > -.14% i/o-8
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> sec, -.017%,
> >>>>>>>>>>   > -.022% i/o-8
> >>>>>>>>>>   >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.017%, -.018% i/o-8
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> sec, -.020%,
> >>>>>>>>>>   > -.029% i/o-8
> >>>>>>>>>>   >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> sec, -.023%,
> >>>>>
> >>>>>
> >>>>>>>>>>   > -.031% i/o-8
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.04% i/o-32
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> sec, -.011%,
> >>>>>>>>>>   > -.005% i/o-32
> >>>>>>>>>>   >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -.11% i/o-32
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> 13. sec -.07%,
> >>>>>>>>>>   > .003% i/o-4/32
> >>>>>>>>>>   >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> sec, -.017%,
> >>>>>>>>>>   > .01% i/o-4/32
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Overhead measurements:
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Progress in terms of number of passes
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> through a fixed
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>system workload
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>>>>   >
>>>>>>>The workload was usex -b48.
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>   >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>   >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>   >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Conclusions:
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>The only protocol which meets the .05% accuracy
> >>>>>>>>>>   > requirement for ntp
> >>>>>>>>>>   >
>>>>>>>tracking under the loads
> >>>>>>>>>>   >
>>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> accuracies for
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>SYNC, MIXED,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>and ASYNC
> >>>>>>>>>>   >
>>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>We could reduce the cost of the SYNC
> method by only
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>scheduling the extra
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>wakeups if a certain number
> >>>>>>>>>>   >
>>>>>>>of ticks are missed.
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Regards,
> >>>>>>>>>>   >
>>>>>>>Dave
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>Keir Fraser wrote:
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ASYNC method a
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>couple of days ago,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>I ran another ASYNC test. I think
> there may have
> >>>>>>>>>>   > been something
> >>>>>>>>>>   >
>>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> days ago for
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>ASYNC. It may have been
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> after context
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>switch in.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>My results indicate that either SYNC
> or ASYNC give
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>acceptable accuracy,
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>each running consistently around or under
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> .01%. MIXED has
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   > >>>>>>a
fairly high
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>error of
> >>>>>>>>>>   >
>>>>>>>>>greater than .03%. Probably too close
> to .05% ntp
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>threshold for comfort.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> plan to leave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>SYNC running
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>over the weekend. If you''d rather I
can
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> leave MIXED
> >>>>>
> >>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>running instead.
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>It may be too early to pick the protocol and
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I can run
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>more overnight tests
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>>next week.
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>I''m a bit worried about any unwanted
side
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> effects of the
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>SYNC+run_timer
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> cause higher
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>system-wide CPU
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>contention. I find it easier to think
> through the
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>implications of ASYNC. I''m
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> accurate than
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>ASYNC. Perhaps it
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>delivers more timer interrupts than the other
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> approaches,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   > >>>>>>and
each interrupt
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> favourites and
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   > >>>>>>if
the latter is
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>actually more accurate then I can
> simply revert the
> >>>>>>>>>>   > changeset that
> >>>>>>>>>>   >
>>>>>>>>implemented MIXED.
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> workloads you
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>could try idle
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> large disc reads
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   > >>>>>>to
/dev/null)? We
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>don''t have any data on workloads that
aren''t
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> CPU bound, so
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>that''s really an
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   >
>>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>-- Keir
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>>>
> >>>>>>>>>>   >
>>>>>>_______________________________________________
> >>>>>>>>>>   >
>>>>>>Xen-devel mailing list
> >>>>>>>>>>   >
>>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>   >
>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>>
> >>>>>>>>>>   > >>diff -r
cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>   > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> 2007 +0000
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> 2008 -0500
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>@@ -58,7 +58,7 @@
static void
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> pt_process_missed_ticks(stru
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     missed_ticks =
missed_ticks / (s_time_t)
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> pt->period + 1;
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>     if (
mode_is(pt->vcpu->domain,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> no_missed_ticks_pending) )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>   > >>+       
pt->do_not_freeze = 1;
> >>>>>>>>>>   > >>     else
> >>>>>>>>>>   > >>        
pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>   > >>    
pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>   > >>@@ -127,7 +127,12
@@ static void
> pt_timer_fn(void *data)
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     pt_lock(pt);
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>-   
pt->pending_intr_nr++;
> >>>>>>>>>>   > >>+    if (
mode_is(pt->vcpu->domain,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> no_missed_ticks_pending) ) {
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>+       
pt->pending_intr_nr = 1;
> >>>>>>>>>>   > >>+
pt->do_not_freeze = 0;
> >>>>>>>>>>   > >>+    }
> >>>>>>>>>>   > >>+    else
> >>>>>>>>>>   > >>+
pt->pending_intr_nr++;
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>     if (
!pt->one_shot )
> >>>>>>>>>>   > >>     {
> >>>>>>>>>>   > >>@@ -221,8 +226,6 @@
void pt_intr_post(struct
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> vcpu *v, struct
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>         return;
> >>>>>>>>>>   > >>     }
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>-   
pt->do_not_freeze = 0;
> >>>>>>>>>>   > >>-
> >>>>>>>>>>   > >>     if (
pt->one_shot )
> >>>>>>>>>>   > >>     {
> >>>>>>>>>>   > >>        
pt->enabled = 0;
> >>>>>>>>>>   > >>@@ -235,6 +238,10
@@ void pt_intr_post(struct vcpu
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> *v, struct
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>   > >>            
pt->last_plt_gtime =
> hvm_get_guest_time(v);
> >>>>>>>>>>   > >>            
pt->pending_intr_nr = 0; /*
> ''collapse'' all
> >>>>>>>>>>   > missed ticks */
> >>>>>>>>>>   > >>         }
> >>>>>>>>>>   > >>+ else if (
mode_is(v->domain,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> no_missed_ticks_pending) ) {
> >>>>>
> >>>>>
> >>>>>>>>>>   > >>+    
pt->pending_intr_nr--;
> >>>>>>>>>>   > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>   > >>+ }
> >>>>>>>>>>   > >>         else
> >>>>>>>>>>   > >>         {
> >>>>>>>>>>   > >>            
pt->last_plt_gtime += pt->period_cycles;
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >>
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   > >
> >>>>>>>>>>   >
> >>>>>>>>>>   >
> >>>>>>>>>>   >
_______________________________________________
> >>>>>>>>>>   > Xen-devel mailing list
> >>>>>>>>>>   >
Xen-devel@lists.xensource.com
> >>>>>>>>>>   >
http://lists.xensource.com/xen-devel
> >>>>>>>>>>   >
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>
> >>
> >
> >
> > 
> --------------------------------------------------------------
> ----------
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan,

Here are some boot snipets for rh4u564 on xen 3.2.


#1:

Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro 
root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp 
(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 
(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
...
Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ 
console=ttyS0 clocksource=pit nohpet
Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, 
65536 bytes)
Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
...
Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 
CPUs: passed.
Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM timer
Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.



#2:

Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro 
root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp 
(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 
(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
...
Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ 
console=ttyS0 clocksource=pit nohpet nopmtimer
Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, 
65536 bytes)
Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
...
Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 
CPUs: passed.
Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.


As you can see, I only get the pit if I specify nopmtimer.

Dan Magenheimer wrote:
>Hi Dave --
>
>Thanks for continuing to run tests!
>
>Hmmm... I thought I had noticed that even though Linux will acknowledge
>the existence of the pmtimer, it still prints:
>
>time.c: Using PIT/TSC based timekeeping.
>
>I will check again, but assuming the clocksource for our tests is
>indeed pit, the huge difference in the results (yours vs ours) is
>baffling. I wonder if the difference may be the underlying hardware.
>Maybe we will try to ensure we can duplicate the results on a different
>box.
>  
>
>So your testing was with stock 3.2.0 xen bits (what cset?) without
>any of your [quote from below] "clock related tweaks that I
haven''t
>submitted, because I''m still characterizing them"?
>  
>
None of the tweaks I mentioned are in this test.
It was stock with some patches. However, none of the patches are time 
related to
my knowledge and I checked vpt.c to make sure that it is the same as 
what''s in unstable.
The only difference is in pt_intr_post, where I set the timer mode.
I don''t have timer mode tied into our config process yet, which
is different than official xen method.


(In pt_intr_post)
      else
      {
+       if(v->arch.paging.mode->guest_levels == 4)
+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = 
HVMPTM_no_missed_ticks_pending;
+       else
+           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = 
HVMPTM_delay_for_missed_ticks;
          if ( mode_is(v->domain, one_missed_tick_pending) ||
               mode_is(v->domain, no_missed_ticks_pending) )
          {
>Could you also send detail on the rhel4u4-64 kernel you
>are testing with, just to ensure we are not comparing apples
>and oranges?  (Perhaps there''s some way we can even share the
>identical disk image and vm.cfg file?)
>
>And if our problem is indeed the pmtimer, I will need to submit
>another patch to Keir to add an hvm pmtimer platform variable.
>(Hmmm... I don''t think he''s even accepted the hpet
variable patch
>yet.  I''ll have to check.)
>
>Thanks,
>Dan
>
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Thursday, February 14, 2008 9:00 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
>>Patel
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Hi Dan,
>>
>>I ran the ltp tests with 3.2 and found the errors
>>for a 16 hour run to be:
>>
>>rh4u564 -9.9 sec (-.017%)
>>rh4u464 -7.3 sec (-.013%)
>>
>>There were no cliffs and the drift was linear.
>>
>>I think the problem you had may be due to the use of the
>>pm timer. If you still have the boot log, it would tell you.
>>
>>When I first tried a guest on 3.2 with "clocksource=pit
nohpet"
>>I noticed that it picked the pm timer. Adding "nopmtimer", the
>>guest will pick the pit.
>>
>>The reason I didn''t have the problem with our 3.1 base is that
>>I had disabled the hpet and the pmtimer by not advertising them
>>in the acpi tables. I did this so long ago, I forgot that I had to
>>disable pmtimer as well as hpet.
>>
>>So, can you re-run your test with "clocksource=pit nohpet
nopmtimer"?
>>You should see this in the boot messages:
>>
>>time.c: Using PIT/TSC based timekeeping.
>>
>>Thanks,
>>Dave
>>
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Hi Dan,
>>>
>>>Over the weekend the drift was +18 seconds for each guest (no ntp).
>>>The duration was 3900 minutes, so the error for each was +.0077%.
>>>Looking back through the data, it appears to drift linearly at
>>>this rate. I''ve attached a plot for rh4u5-64.
>>>
>>>This accuracy is better than what I''ve seen before
(.03-.05%).
>>>This may be due to the different load (ltp vs usex) or to one of the
>>>changes I''ve made recently. I''ll do some
experimentation to see if
>>>there is
>>>a fix I should propose.
>>>
>>>This still doesn''t address the radical drift you saw.
>>>The next step for me is to run 3.2 and see if I can reproduce it.
>>>
>>>Regards,
>>>Dave
>>>
>>>
>>>
>>>
>>>
>>>Dave Winchell wrote:
>>>
>>>      
>>>
>>>>Hi Dan,
>>>>
>>>>Sorry it took me so long, but I finally ran an ltp test today.
>>>>Its on rh4u4-64. I''m using the defaults for ltp and
using a script
>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>virtual processors / physical processors = 2.
>>>>
>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300
minutes
>>>>for -.005% and .008%.
>>>>
>>>>I''m running a 3.1 based hypervisor with some clock
related
>>>>        
>>>>
>>tweaks that
>>    
>>
>>>>I haven''t submitted, because I''m still
characterizing them.
>>>>
>>>>I''m stopping the usex load on 4u5-64 now and replacing
it with ltp
>>>>and will leave the two guests running ltp over the weekend.
>>>>
>>>>Regards,
>>>>Dave
>>>>
>>>>
>>>>Dave Winchell wrote:
>>>>
>>>>        
>>>>
>>>>>Hi Dan, Deepak:
>>>>>
>>>>>Thanks for the data. Those drifts are severe - no wonder 
>>>>>          
>>>>>
>>ntp couldn''t
>>    
>>
>>>>>keep then in synch. I''ll try to reproduce that
behaviour
>>>>>          
>>>>>
>>here, with
>>    
>>
>>>>>my code base.
>>>>>If I can''t reproduce it, I''ll try 3.2.
>>>>>
>>>>>If you can isolate what ltp is doing during the cliffs, that
would
>>>>>be very
>>>>>helpful.
>>>>>
>>>>>thanks,
>>>>>Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Dan Magenheimer wrote:
>>>>>
>>>>>          
>>>>>
>>>>>>OK, Deepak repeated the test without ntpd and using 
>>>>>>            
>>>>>>
>>ntpdate -b before
>>    
>>
>>>>>>the test.
>>>>>>
>>>>>>The attached graph shows his results: el5u1-64
(best=~0.07%),
>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%).
>>>>>>
>>>>>>We will continue to look at LTP to try to isolate.
>>>>>>
>>>>>>Thanks,
>>>>>>Dan
>>>>>>
>>>>>>P.S. elXuY is essentially RHEL XuY with some patches.
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>To: Deepak Patel
>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
>>>>>>>              
>>>>>>>
>>Dave Winchell
>>    
>>
>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that disables
>>>>>>>pending
>>>>>>>missed ticks
>>>>>>>
>>>>>>>
>>>>>>>Dan, Deeepak,
>>>>>>>
>>>>>>>It may be that the underlying clock error is too
great for ntp
>>>>>>>to handle. It would be useful if you did not run
ntpd
>>>>>>>and, instead did ntpdate -b <timeserver> at
the start
>>>>>>>              
>>>>>>>
>>of the test
>>    
>>
>>>>>>>for each guest. Then capture the data as you have
been doing.
>>>>>>>If the drift is greater than .05%, then we need to
address that.
>>>>>>>
>>>>>>>Another option is, when running ntpd, to enable loop
>>>>>>>              
>>>>>>>
>>statistics in
>>    
>>
>>>>>>>/etc/ntp.conf
>>>>>>>by adding this to the file:
>>>>>>>
>>>>>>>statistics loopstats
>>>>>>>statsdir /var/lib/ntp/
>>>>>>>
>>>>>>>Then you will see loop data in that directory.
>>>>>>>Correlating the data in the loopstats files with the
>>>>>>>peaks in skew would be interesting. You will see 
>>>>>>>              
>>>>>>>
>>entries of the form
>>    
>>
>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776
>>>>>>>              
>>>>>>>
>>239.735511 10
>>    
>>
>>>>>>>Where the second to last column is the Allan
Deviation.
>>>>>>>              
>>>>>>>
>>When that
>>    
>>
>>>>>>>gets over 1000, ntpd is working pretty hard.
However, I have not
>>>>>>>seen ntpd
>>>>>>>completely loose it like you have.
>>>>>>>
>>>>>>>I''m on vacation until Monday, and
won''t be reading
>>>>>>>email.
>>>>>>>
>>>>>>>Thanks for all your work on this!
>>>>>>>
>>>>>>>-Dave
>>>>>>>
>>>>>>>Deepak Patel wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>Is the graph for RHEL5u1-64? (I''ve
never tested this one.)
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>I do not know which graph was attached with
this. But
>>>>>>>>                
>>>>>>>>
>>I saw this
>>    
>>
>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 -
64 hvm
>>>>>>>>                
>>>>>>>>
>>guests when I
>>    
>>
>>>>>>>>was running ltp tests continuously.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>What was the behaviour of the other guests
running?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>All pvm guests are fine. But behavior of most of
the
>>>>>>>>                
>>>>>>>>
>>hvm guests were
>>    
>>
>>>>>>>>as described.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>If they had spikes, were they at the same
wall time?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>No. They are not at the same wall time.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Were the other guests running ltp as well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests
are running ltp
>>>>>>>>continuously.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>How are you measuring skew?
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>I was collecting output of "ntpdate -q 
<timeserver> every
>>>>>>>>                
>>>>>>>>
>>>>>>>
>>>>>>>300 seconds
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>(5 minutes) and have created graph based on
that.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Are you running ntpd?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>
>>>>>>>>Yes. ntp was running on all the guests.
>>>>>>>>
>>>>>>>>I am investigating what causes this spikes and
let everyone
>>>>>>>>                
>>>>>>>>
>>>>>>>
>>>>>>>know what
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>are my findings.
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>Deepak
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Anything that you can discover that would be
in sync with
>>>>>>>>>the spikes would be very helpful!
>>>>>>>>>
>>>>>>>>>The code that I test with is our product
code, which is based
>>>>>>>>>on 3.1. So it is possible that something in
3.2 other
>>>>>>>>>                  
>>>>>>>>>
>>than vpt.c
>>    
>>
>>>>>>>>>is the cause. I can test with 3.2, if
necessary.
>>>>>>>>>
>>>>>>>>>thanks,
>>>>>>>>>Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>Hi Dave (Keir, see suggestion below) --
>>>>>>>>>>
>>>>>>>>>>Thanks!
>>>>>>>>>>
>>>>>>>>>>Turning off vhpet certainly helps a lot
(though see below).
>>>>>>>>>>
>>>>>>>>>>I wonder if timekeeping with vhpet is so
bad that it
>>>>>>>>>>                    
>>>>>>>>>>
>>should be
>>    
>>
>>>>>>>>>>turned off by default (in 3.1, 3.2, and
unstable) until it is
>>>>>>>>>>fixed?  (I have a patch that defaults it
off, can post it if
>>>>>>>>>>there is agreement on the above point.) 
The whole
>>>>>>>>>>                    
>>>>>>>>>>
>>point of an
>>    
>>
>>>>>>>>>>HPET is to provide more precise
timekeeping and if vhpet is
>>>>>>>>>>worse than vpit, it can only confuse
users.  Comments?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>In your testing, are you just measuring
% skew over a long
>>>>>>>>>>period of time?
>>>>>>>>>>We are graphing the skew continuously
and
>>>>>>>>>>seeing periodic behavior that is
unsettling, even with pit.
>>>>>>>>>>See attached.  Though your algorithm
recovers, the "cliffs"
>>>>>>>>>>could still cause real user problems.  I
wonder if there is
>>>>>>>>>>anything that can be done to make the
"recovery" more
>>>>>>>>>>responsive?
>>>>>>>>>>
>>>>>>>>>>We are looking into what part(s) of LTP
is causing
>>>>>>>>>>                    
>>>>>>>>>>
>>the cliffs.
>>    
>>
>>>>>>>>>>Thanks,
>>>>>>>>>>Dan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21
AM
>>>>>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>Cc: Keir Fraser;
xen-devel@lists.xensource.com;
>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>akira.ijuin@oracle.com; Dave
Winchell
>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add
a timer mode
>>>>>>>>>>>                      
>>>>>>>>>>>
>>that disables
>>    
>>
>>>>>>>>>>>pending
>>>>>>>>>>>missed ticks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan,
>>>>>>>>>>>
>>>>>>>>>>>I guess I''m a bit out of
date calling for clock= usage.
>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I
think you
>>>>>>>>>>>                      
>>>>>>>>>>>
>>should specify
>>    
>>
>>>>>>>>>>>"clocksource=pit nohpet"
on the linux guest bootline.
>>>>>>>>>>>
>>>>>>>>>>>You can leave the xen and dom0
bootlines as they are.
>>>>>>>>>>>The xen and guest clocksources do
not need to be the same.
>>>>>>>>>>>In my tests, xen is using the hpet
for its timekeeping and
>>>>>>>>>>>that appears to be the default.
>>>>>>>>>>>
>>>>>>>>>>>When you boot the guests you should
see
>>>>>>>>>>>  time.c: Using PIT/TSC based
timekeeping.
>>>>>>>>>>>on the rh4u5-64 guest, and something
similar on the others.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz
stepping 04, Platform timer
>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>This appears to be the xen state,
which is fine.
>>>>>>>>>>>I was wrongly assuming that this was
the guest state.
>>>>>>>>>>>You might want to look in your guest
logs and see
>>>>>>>>>>>                      
>>>>>>>>>>>
>>what they were
>>    
>>
>>>>>>>>>>>picking
>>>>>>>>>>>for a clock source.
>>>>>>>>>>>
>>>>>>>>>>>Regards,
>>>>>>>>>>>Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>Thanks, I hadn''t
realized that!  No wonder we didn''t
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>see the same
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>improvement you saw!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>Try specifying clock=pit on
the linux boot line...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>I''m confused... do you
mean "clocksource=pit" on the Xen
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>command line or
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>"nohpet" /
"clock=pit" / "clocksource=pit" on the guest (or
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>dom0?) command
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>line?  Or both places?  Since
the tests take awhile, it
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>would be nice
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>to get this right the first
time.  Do the Xen and guest
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>clocksources need
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>to be the same?
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>Dan
>>>>>>>>>>>>
>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>*From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008
2:22 PM
>>>>>>>>>>>>*To:*
dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>deepak.patel@oracle.com;
>>    
>>
>>>>>>>>>>>>akira.ijuin@oracle.com; Dave
Winchell
>>>>>>>>>>>>*Subject:* RE: [Xen-devel]
[PATCH] Add a timer mode
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>that disables
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>  Hi Dan,
>>>>>>>>>>>>
>>>>>>>>>>>>  Hpet timer does have a fairly
large error, as I
>>>>>>>>>>>>was
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>trying this
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  one recently.
>>>>>>>>>>>>  I don''t remember what
I got for error, but 1% sounds
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>about right.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  The problem is that hpet is
not built on top of vpt.c,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>the module
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  Keir and I did
>>>>>>>>>>>>  all the recent work in, for
its periodic timer needs. Try
>>>>>>>>>>>>  specifying clock=pit
>>>>>>>>>>>>  on the linux boot line. If it
still picks the
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>hpet, which it
>>    
>>
>>>>>>>>>>>>  might, let me know
>>>>>>>>>>>>  and I''ll tell you how
to get around this.
>>>>>>>>>>>>
>>>>>>>>>>>>  Regards,
>>>>>>>>>>>>  Dave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>--------------------------------------------------------------
>>    
>>
>>>>>>>>>>>----------
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  *From:* Dan Magenheimer 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:dan.magenheimer@oracle.com]
>>    
>>
>>>>>>>>>>>>  *Sent:* Fri 1/25/2008 6:50 PM
>>>>>>>>>>>>  *To:* Dave Winchell; Keir
Fraser
>>>>>>>>>>>>  *Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>deepak.patel@oracle.com;
>>    
>>
>>>>>>>>>>>>  akira.ijuin@oracle.com
>>>>>>>>>>>>  *Subject:* RE: [Xen-devel]
[PATCH] Add a timer mode
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>that disables
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  pending missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>  Sorry for the very late
followup on this but we finally
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>were able
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  to get our testing set up
again on stable 3.1
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>bits and have
>>    
>>
>>>>>>>>>>>>  seen some very bad results on
3.1.3-rc1, on the
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>order of 1%.
>>    
>>
>>>>>>>>>>>>  Test enviroment was a 4-socket
dual core machine
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>with 24GB of
>>    
>>
>>>>>>>>>>>>  memory running six two-vcpu
2GB domains, four hvm
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>plus two pv.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  All six guests were running
LTP simultaneously.
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>The four hvm
>>    
>>
>>>>>>>>>>>>  guests were: RHEL5u1-64,
RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>RHEL4u5-64.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  Timer_mode was set to 2 for
64-bit guests and 0 for
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  All four hvm guests
experienced skew around -1%,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>even the 32-bit
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  guest.  Less intensive testing
didn''t exhibit much
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>skew at all.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  A representative graph is
attached.
>>>>>>>>>>>>
>>>>>>>>>>>>  Dave, I wonder if some portion
of your patches
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>didn''t end up in
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  the xen trees?
>>>>>>>>>>>>
>>>>>>>>>>>>  (xm dmesg shows 8x Xeon 3.2GHz
stepping 04,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>Platform timer
>>    
>>
>>>>>>>>>>>>  14.318MHz HPET.)
>>>>>>>>>>>>
>>>>>>>>>>>>  Thanks,
>>>>>>>>>>>>  Dan
>>>>>>>>>>>>
>>>>>>>>>>>>  P.S. Many thanks to Deepak and
Akira for running tests.
>>>>>>>>>>>>
>>>>>>>>>>>>  > -----Original
Message-----
>>>>>>>>>>>>  > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>  > 
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>    
>>
>>>>>>>>>>>>  > Dave Winchell
>>>>>>>>>>>>  > Sent: Wednesday, January
09, 2008 9:53 AM
>>>>>>>>>>>>  > To: Keir Fraser
>>>>>>>>>>>>  > Cc:
dan.magenheimer@oracle.com;
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > Winchell
>>>>>>>>>>>>  > Subject: Re: [Xen-devel]
[PATCH] Add a timer mode that
>>>>>>>>>>>>  > disables pending
>>>>>>>>>>>>  > missed ticks
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Hi Keir,
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > The latest change, c/s
16690, looks fine.
>>>>>>>>>>>>  > I agree that the code in
c/s 16690 is equivalent to
>>>>>>>>>>>>  > the code I submitted.
Also, your version is more
>>>>>>>>>>>>  > concise.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > The error tests confirm
the equivalence. With
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>overnight cpu loads,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > the checked in version
was accurate to +.048% for sles
>>>>>>>>>>>>  > and +.038% for red hat.
My version was +.046%
>>>>>>>>>>>>and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>+.032% in a
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > 2 hour test.
>>>>>>>>>>>>  > I don''t think
the difference is significant.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > i/o loads produced errors
of +.01%.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Thanks for all your
efforts on this issue.
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Regards,
>>>>>>>>>>>>  > Dave
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > Keir Fraser wrote:
>>>>>>>>>>>>  >
>>>>>>>>>>>>  > >Applied as c/s 16690,
although the checked-in patch is
>>>>>>>>>>>>  > smaller. I think the
>>>>>>>>>>>>  > >only important fix is
to pt_intr_post() and the
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>only bit of
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > the patch I
>>>>>>>>>>>>  > >totally omitted was
the change to
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>pt_process_missed_ticks().
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > I don''t think
>>>>>>>>>>>>  > >that change can be
important, but let''s see what
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>happens to the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  error
>>>>>>>>>>>>  > >percentage...
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > > -- Keir
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >On 4/1/08 23:24,
"Dave Winchell"
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >>Hi Dan and Keir,
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Attached is a
patch that fixes some issues with the
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>SYNC policy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>(no_missed_ticks_pending).
>>>>>>>>>>>>  > >>I have not tried
to make the change the
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>minimal one, but,
>>    
>>
>>>>>>>>>>>>  > rather, just
>>>>>>>>>>>>  > >>ported into
>>>>>>>>>>>>  > >>the new code what
I know to work well. The error for
>>>>>>>>>>>>  >
>>no_missed_ticks_pending goes from
>>>>>>>>>>>>  > >>over 3% to .03%
with this change according
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>to my testing.
>>    
>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Regards,
>>>>>>>>>>>>  > >>Dave
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>Dan Magenheimer
wrote:
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>>Hi Dave --
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Did you get
your correction ported?  If so,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>it would be
>>    
>>
>>>>>>>>>>>>  > nice to see this get
>>>>>>>>>>>>  > >>>into 3.1.3.
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Note that I
just did some very limited testing with
>>>>>>>>>>>>  > timer_mode=2(=SYNC=no
>>>>>>>>>>>>  > >>>missed ticks
pending)
>>>>>>>>>>>>  > >>>on tip of
xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>guest) and the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > worst error I''ve
>>>>>>>>>>>>  > >>>seen so far
>>>>>>>>>>>>  > >>>is 0.012%. 
But I haven''t tried any exotic
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>loads, just LTP.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>Thanks,
>>>>>>>>>>>>  > >>>Dan
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  >
>>>>-----Original Message-----
>>>>>>>>>>>>  > >>>>From:
Dave Winchell
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:dwinchell@virtualiron.com]
>>    
>>
>>>>>>>>>>>>  > >>>>Sent:
Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>  > >>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>>>>  > >>>>Cc: Keir
Fraser; Shan, Haitao;
>>>>>>>>>>>>  >
xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>  > >>>>Eddie;
Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>  > >>>>Subject:
Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>timer mode that
>>    
>>
>>>>>>>>>>>>  > >>>>disables
pending
>>>>>>>>>>>>  > >>>>missed
ticks
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Dan,
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>I did
some testing with the constant tsc offset
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>SYNC method
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>(now
called
>>>>>>>>>>>>  >
>>>>no_missed_ticks_pending)
>>>>>>>>>>>>  > >>>>and found
the error to be very high, much larger
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>I recall.
>>>>>>>>>>>>  > >>>>I have
not had a chance to submit a correction. I
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>will try to
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>do it
later
>>>>>>>>>>>>  > >>>>this week
or the first week in January. My
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>version of
>>    
>>
>>>>>>>>>>>>  constant tsc
>>>>>>>>>>>>  > >>>>offset
SYNC method
>>>>>>>>>>>>  > >>>>produces
.02 % error, so I just need to port
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>that into the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>current
code.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>The error
you got for both of those kernels is
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>what I would
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  expect
>>>>>>>>>>>>  > >>>>for the
default mode, delay_for_missed_ticks.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  >
>>>>I''ll let Keir answer on how to set the time mode.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Regards,
>>>>>>>>>>>>  > >>>>Dave
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>Dan
Magenheimer wrote:
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  >
>>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>I
just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>saw a loss of
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>about
0.2% with no load.  This was
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>xen-unstable tip today
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>>>with no
options specified.  32-bit was about 0.01%.
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>I
think I missed something... how do I
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>run the various
>>    
>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  >
>>>>accounting choices and which ones are known to be
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>appropriate
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>for which
kernels?
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  >
>>>>>Thanks,
>>>>>>>>>>>>  > >>>>>Dan
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  >
>>>>>>-----Original Message-----
>>>>>>>>>>>>  >
>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>  >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>Keir
Fraser
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>  >
>>>>>>To: Dave Winchell
>>>>>>>>>>>>  >
>>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > Eddie; Jiang,
>>>>>>>>>>>>  >
>>>>>>Yunhong
>>>>>>>>>>>>  >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>mode that
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>disables pending
>>>>>>>>>>>>  >
>>>>>>missed ticks
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>changeset 16545.
>>    
>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>-- Keir
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>Keir,
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>The accuracy data I''ve collected for i/o
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>loads for the
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>various time protocols follows. In
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>addition, the data
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>for cpu loads is shown.
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>processor AMD
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  box.
>>>>>>>>>>>>  >
>>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>vcpu each.
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>  >
>>>>>>>(usex is available at
>>>>>>>>>>>> 
http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>  >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>The loads labeled i/o-32 are 32 instances of dd.
>>>>>>>>>>>>  >
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>  >
>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>  >
>>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>as i/o-32
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>instances of dd.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>+4.42
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec -.006%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > +.005% cpu
>>>>>>>>>>>>  >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.001%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > +.012% cpu
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.009%,
>>    
>>
>>>>>>>>>>>>  -.004% cpu
>>>>>>>>>>>>  >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.005%, -.005% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.008%, -.003% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.034%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > -.031% cpu
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec,-55.7 sec, -.01%,
>>    
>>
>>>>>>>>>>>>  > -.09% i/o-8
>>>>>>>>>>>>  >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec,-14.0 sec, -.015%
>>    
>>
>>>>>>>>>>>>  > -.14% i/o-8
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.017%,
>>    
>>
>>>>>>>>>>>>  > -.022% i/o-8
>>>>>>>>>>>>  >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.017%, -.018% i/o-8
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.020%,
>>    
>>
>>>>>>>>>>>>  > -.029% i/o-8
>>>>>>>>>>>>  >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>sec, -.023%,
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > -.031% i/o-8
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.011%,
>>    
>>
>>>>>>>>>>>>  > -.005% i/o-32
>>>>>>>>>>>>  >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>13. sec -.07%,
>>    
>>
>>>>>>>>>>>>  > .003% i/o-4/32
>>>>>>>>>>>>  >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>sec, -.017%,
>>    
>>
>>>>>>>>>>>>  > .01% i/o-4/32
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Overhead measurements:
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>through a fixed
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>system workload
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>  >
>>>>>>>The workload was usex -b48.
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>  >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>  >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>  >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Conclusions:
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>  > requirement for ntp
>>>>>>>>>>>>  >
>>>>>>>tracking under the loads
>>>>>>>>>>>>  >
>>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>accuracies for
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>SYNC, MIXED,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>and ASYNC
>>>>>>>>>>>>  >
>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>method by only
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>scheduling the extra
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>wakeups if a certain number
>>>>>>>>>>>>  >
>>>>>>>of ticks are missed.
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Regards,
>>>>>>>>>>>>  >
>>>>>>>Dave
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>couple of days ago,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>there may have
>>    
>>
>>>>>>>>>>>>  > been something
>>>>>>>>>>>>  >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>days ago for
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>ASYNC. It may have been
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>after context
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>switch in.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>or ASYNC give
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>acceptable accuracy,
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>each running consistently around or under
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  > >>>>>>a
fairly high
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>error of
>>>>>>>>>>>>  >
>>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>to .05% ntp
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>threshold for comfort.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>plan to leave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>SYNC running
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>over the weekend. If you''d rather I
can
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>leave MIXED
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>running instead.
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>I can run
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>more overnight tests
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>next week.
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>effects of the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>SYNC+run_timer
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>cause higher
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>system-wide CPU
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>through the
>>    
>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>implications of ASYNC. I''m
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>accurate than
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>approaches,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>and each interrupt
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>favourites and
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>if the latter is
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>actually more accurate then I can
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>simply revert the
>>    
>>
>>>>>>>>>>>>  > changeset that
>>>>>>>>>>>>  >
>>>>>>>>implemented MIXED.
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>workloads you
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>could try idle
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>large disc reads
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>to /dev/null)? We
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>that''s really an
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  >
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>-- Keir
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>>>
>>>>>>>>>>>>  >
>>>>>>_______________________________________________
>>>>>>>>>>>>  >
>>>>>>Xen-devel mailing list
>>>>>>>>>>>>  >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>  >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>>
>>>>>>>>>>>>  > >>diff -r
cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>  > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>2007 +0000
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>2008 -0500
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>@@ -58,7 +58,7 @@
static void
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>pt_process_missed_ticks(stru
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     missed_ticks
= missed_ticks / (s_time_t)
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>pt->period + 1;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>no_missed_ticks_pending) )
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>  > >>+       
pt->do_not_freeze = 1;
>>>>>>>>>>>>  > >>     else
>>>>>>>>>>>>  > >>        
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>  > >>    
pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>  > >>@@ -127,7 +127,12
@@ static void
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>pt_timer_fn(void *data)
>>    
>>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     pt_lock(pt);
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>-   
pt->pending_intr_nr++;
>>>>>>>>>>>>  > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>+       
pt->pending_intr_nr = 1;
>>>>>>>>>>>>  > >>+
pt->do_not_freeze = 0;
>>>>>>>>>>>>  > >>+    }
>>>>>>>>>>>>  > >>+    else
>>>>>>>>>>>>  > >>+
pt->pending_intr_nr++;
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>     if (
!pt->one_shot )
>>>>>>>>>>>>  > >>     {
>>>>>>>>>>>>  > >>@@ -221,8 +226,6
@@ void pt_intr_post(struct
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>vcpu *v, struct
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>         return;
>>>>>>>>>>>>  > >>     }
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>-   
pt->do_not_freeze = 0;
>>>>>>>>>>>>  > >>-
>>>>>>>>>>>>  > >>     if (
pt->one_shot )
>>>>>>>>>>>>  > >>     {
>>>>>>>>>>>>  > >>        
pt->enabled = 0;
>>>>>>>>>>>>  > >>@@ -235,6 +238,10
@@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>*v, struct
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>  > >>            
pt->last_plt_gtime =
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>hvm_get_guest_time(v);
>>    
>>
>>>>>>>>>>>>  > >>            
pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>''collapse'' all
>>    
>>
>>>>>>>>>>>>  > missed ticks */
>>>>>>>>>>>>  > >>         }
>>>>>>>>>>>>  > >>+ else if (
mode_is(v->domain,
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>  > >>+    
pt->pending_intr_nr--;
>>>>>>>>>>>>  > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>  > >>+ }
>>>>>>>>>>>>  > >>         else
>>>>>>>>>>>>  > >>         {
>>>>>>>>>>>>  > >>            
pt->last_plt_gtime += pt->period_cycles;
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >>
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  > >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
>>>>>>>>>>>>  >
_______________________________________________
>>>>>>>>>>>>  > Xen-devel mailing list
>>>>>>>>>>>>  >
Xen-devel@lists.xensource.com
>>>>>>>>>>>>  >
http://lists.xensource.com/xen-devel
>>>>>>>>>>>>  >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>
>>>      
>>>
>>--------------------------------------------------------------
>>----------
>>    
>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave --

No new results yet but one other question:

The problems we''ve seen with our testing have been with a heavily
oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
running LTP simultaneously.

Was your LTP testing oversubscribed or just a single guest?

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Thursday, February 14, 2008 10:56 AM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Dan,
> 
> Here are some boot snipets for rh4u564 on xen 3.2.
> 
> 
> #1:
> 
> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> ...
> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
> console=ttyS0 clocksource=pit nohpet
> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11,
> 65536 bytes)
> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
> ...
> Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
> CPUs: passed.
> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use 
> of PM timer
> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
> 
> 
> 
> #2:
> 
> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> ...
> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
> console=ttyS0 clocksource=pit nohpet nopmtimer
> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11,
> 65536 bytes)
> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
> ...
> Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
> CPUs: passed.
> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.
> 
> 
> As you can see, I only get the pit if I specify nopmtimer.
> 
> Dan Magenheimer wrote:
> 
> >Hi Dave --
> >
> >Thanks for continuing to run tests!
> >
> >Hmmm... I thought I had noticed that even though Linux will 
> acknowledge
> >the existence of the pmtimer, it still prints:
> >
> >time.c: Using PIT/TSC based timekeeping.
> >
> >I will check again, but assuming the clocksource for our tests is
> >indeed pit, the huge difference in the results (yours vs ours) is
> >baffling. I wonder if the difference may be the underlying hardware.
> >Maybe we will try to ensure we can duplicate the results on 
> a different
> >box.
> >
> >
> >So your testing was with stock 3.2.0 xen bits (what cset?) without
> >any of your [quote from below] "clock related tweaks that I
haven''t
> >submitted, because I''m still characterizing them"?
> >
> >
> None of the tweaks I mentioned are in this test.
> It was stock with some patches. However, none of the patches are time
> related to
> my knowledge and I checked vpt.c to make sure that it is the same as
> what''s in unstable.
> The only difference is in pt_intr_post, where I set the timer mode.
> I don''t have timer mode tied into our config process yet, which
> is different than official xen method.
> 
> 
> (In pt_intr_post)
>       else
>       {
> +       if(v->arch.paging.mode->guest_levels == 4)
> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE]
> HVMPTM_no_missed_ticks_pending;
> +       else
> +           v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE]
> HVMPTM_delay_for_missed_ticks;
>           if ( mode_is(v->domain, one_missed_tick_pending) ||
>                mode_is(v->domain, no_missed_ticks_pending) )
>           {
> 
> >Could you also send detail on the rhel4u4-64 kernel you
> >are testing with, just to ensure we are not comparing apples
> >and oranges?  (Perhaps there''s some way we can even share the
> >identical disk image and vm.cfg file?)
> >
> >And if our problem is indeed the pmtimer, I will need to submit
> >another patch to Keir to add an hvm pmtimer platform variable.
> >(Hmmm... I don''t think he''s even accepted the hpet
variable patch
> >yet.  I''ll have to check.)
> >
> >Thanks,
> >Dan
> >
> >
> >
> >
> >>-----Original Message-----
> >>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>Sent: Thursday, February 14, 2008 9:00 AM
> >>To: dan.magenheimer@oracle.com
> >>Cc: Dave Winchell; Keir Fraser; 
> xen-devel@lists.xensource.com; Deepak
> >>Patel
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>Hi Dan,
> >>
> >>I ran the ltp tests with 3.2 and found the errors
> >>for a 16 hour run to be:
> >>
> >>rh4u564 -9.9 sec (-.017%)
> >>rh4u464 -7.3 sec (-.013%)
> >>
> >>There were no cliffs and the drift was linear.
> >>
> >>I think the problem you had may be due to the use of the
> >>pm timer. If you still have the boot log, it would tell you.
> >>
> >>When I first tried a guest on 3.2 with "clocksource=pit
nohpet"
> >>I noticed that it picked the pm timer. Adding
"nopmtimer", the
> >>guest will pick the pit.
> >>
> >>The reason I didn''t have the problem with our 3.1 base is
that
> >>I had disabled the hpet and the pmtimer by not advertising them
> >>in the acpi tables. I did this so long ago, I forgot that I had to
> >>disable pmtimer as well as hpet.
> >>
> >>So, can you re-run your test with "clocksource=pit nohpet 
> nopmtimer"?
> >>You should see this in the boot messages:
> >>
> >>time.c: Using PIT/TSC based timekeeping.
> >>
> >>Thanks,
> >>Dave
> >>
> >>
> >>Dave Winchell wrote:
> >>
> >>
> >>
> >>>Hi Dan,
> >>>
> >>>Over the weekend the drift was +18 seconds for each guest (no
ntp).
> >>>The duration was 3900 minutes, so the error for each was
+.0077%.
> >>>Looking back through the data, it appears to drift linearly at
> >>>this rate. I''ve attached a plot for rh4u5-64.
> >>>
> >>>This accuracy is better than what I''ve seen before
(.03-.05%).
> >>>This may be due to the different load (ltp vs usex) or to 
> one of the
> >>>changes I''ve made recently. I''ll do some
experimentation to see if
> >>>there is
> >>>a fix I should propose.
> >>>
> >>>This still doesn''t address the radical drift you saw.
> >>>The next step for me is to run 3.2 and see if I can reproduce
it.
> >>>
> >>>Regards,
> >>>Dave
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>Dave Winchell wrote:
> >>>
> >>>
> >>>
> >>>>Hi Dan,
> >>>>
> >>>>Sorry it took me so long, but I finally ran an ltp test
today.
> >>>>Its on rh4u4-64. I''m using the defaults for ltp
and using a script
> >>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
> >>>>virtual processors / physical processors = 2.
> >>>>
> >>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300
minutes
> >>>>for -.005% and .008%.
> >>>>
> >>>>I''m running a 3.1 based hypervisor with some clock
related
> >>>>
> >>>>
> >>tweaks that
> >>
> >>
> >>>>I haven''t submitted, because I''m still
characterizing them.
> >>>>
> >>>>I''m stopping the usex load on 4u5-64 now and
replacing it with ltp
> >>>>and will leave the two guests running ltp over the weekend.
> >>>>
> >>>>Regards,
> >>>>Dave
> >>>>
> >>>>
> >>>>Dave Winchell wrote:
> >>>>
> >>>>
> >>>>
> >>>>>Hi Dan, Deepak:
> >>>>>
> >>>>>Thanks for the data. Those drifts are severe - no
wonder
> >>>>>
> >>>>>
> >>ntp couldn''t
> >>
> >>
> >>>>>keep then in synch. I''ll try to reproduce that
behaviour
> >>>>>
> >>>>>
> >>here, with
> >>
> >>
> >>>>>my code base.
> >>>>>If I can''t reproduce it, I''ll try
3.2.
> >>>>>
> >>>>>If you can isolate what ltp is doing during the cliffs,
> that would
> >>>>>be very
> >>>>>helpful.
> >>>>>
> >>>>>thanks,
> >>>>>Dave
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>Dan Magenheimer wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>OK, Deepak repeated the test without ntpd and using
> >>>>>>
> >>>>>>
> >>ntpdate -b before
> >>
> >>
> >>>>>>the test.
> >>>>>>
> >>>>>>The attached graph shows his results: el5u1-64
(best=~0.07%),
> >>>>>>el4u5-64 (middle=~0.2%), and el4u5-32
(worst=~0.3%).
> >>>>>>
> >>>>>>We will continue to look at LTP to try to isolate.
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Dan
> >>>>>>
> >>>>>>P.S. elXuY is essentially RHEL XuY with some
patches.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>-----Original Message-----
> >>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
> >>>>>>>To: Deepak Patel
> >>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
> >>>>>>>xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
> >>>>>>>
> >>>>>>>
> >>Dave Winchell
> >>
> >>
> >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that disables
> >>>>>>>pending
> >>>>>>>missed ticks
> >>>>>>>
> >>>>>>>
> >>>>>>>Dan, Deeepak,
> >>>>>>>
> >>>>>>>It may be that the underlying clock error is
too great for ntp
> >>>>>>>to handle. It would be useful if you did not
run ntpd
> >>>>>>>and, instead did ntpdate -b <timeserver>
at the start
> >>>>>>>
> >>>>>>>
> >>of the test
> >>
> >>
> >>>>>>>for each guest. Then capture the data as you
have been doing.
> >>>>>>>If the drift is greater than .05%, then we need
to
> address that.
> >>>>>>>
> >>>>>>>Another option is, when running ntpd, to enable
loop
> >>>>>>>
> >>>>>>>
> >>statistics in
> >>
> >>
> >>>>>>>/etc/ntp.conf
> >>>>>>>by adding this to the file:
> >>>>>>>
> >>>>>>>statistics loopstats
> >>>>>>>statsdir /var/lib/ntp/
> >>>>>>>
> >>>>>>>Then you will see loop data in that directory.
> >>>>>>>Correlating the data in the loopstats files
with the
> >>>>>>>peaks in skew would be interesting. You will
see
> >>>>>>>
> >>>>>>>
> >>entries of the form
> >>
> >>
> >>>>>>>54495 76787.701 -0.045153303 -132.569229
0.020806776
> >>>>>>>
> >>>>>>>
> >>239.735511 10
> >>
> >>
> >>>>>>>Where the second to last column is the Allan
Deviation.
> >>>>>>>
> >>>>>>>
> >>When that
> >>
> >>
> >>>>>>>gets over 1000, ntpd is working pretty hard.
However,
> I have not
> >>>>>>>seen ntpd
> >>>>>>>completely loose it like you have.
> >>>>>>>
> >>>>>>>I''m on vacation until Monday, and
won''t be reading
> >>>>>>>email.
> >>>>>>>
> >>>>>>>Thanks for all your work on this!
> >>>>>>>
> >>>>>>>-Dave
> >>>>>>>
> >>>>>>>Deepak Patel wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>Is the graph for RHEL5u1-64?
(I''ve never tested this one.)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>I do not know which graph was attached with
this. But
> >>>>>>>>
> >>>>>>>>
> >>I saw this
> >>
> >>
> >>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and
EL5U1 - 64 hvm
> >>>>>>>>
> >>>>>>>>
> >>guests when I
> >>
> >>
> >>>>>>>>was running ltp tests continuously.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>What was the behaviour of the other
guests running?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>All pvm guests are fine. But behavior of
most of the
> >>>>>>>>
> >>>>>>>>
> >>hvm guests were
> >>
> >>
> >>>>>>>>as described.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>If they had spikes, were they at the
same wall time?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>No. They are not at the same wall time.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Were the other guests running ltp as
well?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the
guests are running ltp
> >>>>>>>>continuously.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>How are you measuring skew?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>I was collecting output of "ntpdate -q
<timeserver> every
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>300 seconds
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>(5 minutes) and have created graph based on
that.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Are you running ntpd?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>Yes. ntp was running on all the guests.
> >>>>>>>>
> >>>>>>>>I am investigating what causes this spikes
and let everyone
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>know what
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>are my findings.
> >>>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>Deepak
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Anything that you can discover that
would be in sync with
> >>>>>>>>>the spikes would be very helpful!
> >>>>>>>>>
> >>>>>>>>>The code that I test with is our
product code, which is based
> >>>>>>>>>on 3.1. So it is possible that
something in 3.2 other
> >>>>>>>>>
> >>>>>>>>>
> >>than vpt.c
> >>
> >>
> >>>>>>>>>is the cause. I can test with 3.2, if
necessary.
> >>>>>>>>>
> >>>>>>>>>thanks,
> >>>>>>>>>Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>Hi Dave (Keir, see suggestion
below) --
> >>>>>>>>>>
> >>>>>>>>>>Thanks!
> >>>>>>>>>>
> >>>>>>>>>>Turning off vhpet certainly helps a
lot (though see below).
> >>>>>>>>>>
> >>>>>>>>>>I wonder if timekeeping with vhpet
is so bad that it
> >>>>>>>>>>
> >>>>>>>>>>
> >>should be
> >>
> >>
> >>>>>>>>>>turned off by default (in 3.1, 3.2,
and unstable)
> until it is
> >>>>>>>>>>fixed?  (I have a patch that
defaults it off, can post it if
> >>>>>>>>>>there is agreement on the above
point.)  The whole
> >>>>>>>>>>
> >>>>>>>>>>
> >>point of an
> >>
> >>
> >>>>>>>>>>HPET is to provide more precise
timekeeping and if vhpet is
> >>>>>>>>>>worse than vpit, it can only
confuse users.  Comments?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>In your testing, are you just
measuring % skew over a long
> >>>>>>>>>>period of time?
> >>>>>>>>>>We are graphing the skew
continuously and
> >>>>>>>>>>seeing periodic behavior that is
unsettling, even with pit.
> >>>>>>>>>>See attached.  Though your
algorithm recovers, the "cliffs"
> >>>>>>>>>>could still cause real user
problems.  I wonder if there is
> >>>>>>>>>>anything that can be done to make
the "recovery" more
> >>>>>>>>>>responsive?
> >>>>>>>>>>
> >>>>>>>>>>We are looking into what part(s) of
LTP is causing
> >>>>>>>>>>
> >>>>>>>>>>
> >>the cliffs.
> >>
> >>
> >>>>>>>>>>Thanks,
> >>>>>>>>>>Dan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>Sent: Monday, January 28, 2008
8:21 AM
> >>>>>>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>Cc: Keir Fraser;
xen-devel@lists.xensource.com;
> >>>>>>>>>>>deepak.patel@oracle.com;
> >>>>>>>>>>>akira.ijuin@oracle.com; Dave
Winchell
> >>>>>>>>>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>that disables
> >>
> >>
> >>>>>>>>>>>pending
> >>>>>>>>>>>missed ticks
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>Dan,
> >>>>>>>>>>>
> >>>>>>>>>>>I guess I''m a bit out
of date calling for clock= usage.
> >>>>>>>>>>>Looking at linux 2.6.20.4
sources, I think you
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>should specify
> >>
> >>
> >>>>>>>>>>>"clocksource=pit
nohpet" on the linux guest bootline.
> >>>>>>>>>>>
> >>>>>>>>>>>You can leave the xen and dom0
bootlines as they are.
> >>>>>>>>>>>The xen and guest clocksources
do not need to be the same.
> >>>>>>>>>>>In my tests, xen is using the
hpet for its timekeeping and
> >>>>>>>>>>>that appears to be the default.
> >>>>>>>>>>>
> >>>>>>>>>>>When you boot the guests you
should see
> >>>>>>>>>>>  time.c: Using PIT/TSC based
timekeeping.
> >>>>>>>>>>>on the rh4u5-64 guest, and
something similar on the others.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>(xm dmesg shows 8x Xeon
3.2GHz stepping 04, Platform timer
> >>>>>>>>>>>>14.318MHz HPET.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>This appears to be the xen
state, which is fine.
> >>>>>>>>>>>I was wrongly assuming that
this was the guest state.
> >>>>>>>>>>>You might want to look in your
guest logs and see
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>what they were
> >>
> >>
> >>>>>>>>>>>picking
> >>>>>>>>>>>for a clock source.
> >>>>>>>>>>>
> >>>>>>>>>>>Regards,
> >>>>>>>>>>>Dave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>Thanks, I hadn''t
realized that!  No wonder we didn''t
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>see the same
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>improvement you saw!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>Try specifying
clock=pit on the linux boot line...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>I''m confused... do
you mean "clocksource=pit" on the Xen
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>command line or
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>"nohpet" /
"clock=pit" / "clocksource=pit" on the
> guest (or
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>dom0?) command
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>line?  Or both places? 
Since the tests take awhile, it
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>would be nice
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>to get this right the first
time.  Do the Xen and guest
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>clocksources need
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>to be the same?
> >>>>>>>>>>>>
> >>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>Dan
> >>>>>>>>>>>>
> >>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>*From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>*Sent:* Sunday, January 27,
2008 2:22 PM
> >>>>>>>>>>>>*To:*
dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>deepak.patel@oracle.com;
> >>
> >>
> >>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
> >>>>>>>>>>>>*Subject:* RE: [Xen-devel]
[PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>that disables
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>pending missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Hi Dan,
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Hpet timer does have a
fairly large error, as I
> >>>>>>>>>>>>was
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>trying this
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  one recently.
> >>>>>>>>>>>>  I don''t remember
what I got for error, but 1% sounds
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>about right.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  The problem is that hpet
is not built on top of vpt.c,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>the module
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  Keir and I did
> >>>>>>>>>>>>  all the recent work in,
for its periodic timer
> needs. Try
> >>>>>>>>>>>>  specifying clock=pit
> >>>>>>>>>>>>  on the linux boot line.
If it still picks the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>hpet, which it
> >>
> >>
> >>>>>>>>>>>>  might, let me know
> >>>>>>>>>>>>  and I''ll tell
you how to get around this.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Regards,
> >>>>>>>>>>>>  Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>--------------------------------------------------------------
> >>
> >>
> >>>>>>>>>>>----------
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  *From:* Dan Magenheimer
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:dan.magenheimer@oracle.com]
> >>
> >>
> >>>>>>>>>>>>  *Sent:* Fri 1/25/2008
6:50 PM
> >>>>>>>>>>>>  *To:* Dave Winchell; Keir
Fraser
> >>>>>>>>>>>>  *Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>deepak.patel@oracle.com;
> >>
> >>
> >>>>>>>>>>>>  akira.ijuin@oracle.com
> >>>>>>>>>>>>  *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>that disables
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  pending missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Sorry for the very late
followup on this but we finally
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>were able
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  to get our testing set up
again on stable 3.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>bits and have
> >>
> >>
> >>>>>>>>>>>>  seen some very bad
results on 3.1.3-rc1, on the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>order of 1%.
> >>
> >>
> >>>>>>>>>>>>  Test enviroment was a
4-socket dual core machine
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>with 24GB of
> >>
> >>
> >>>>>>>>>>>>  memory running six
two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>plus two pv.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  All six guests were
running LTP simultaneously.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>The four hvm
> >>
> >>
> >>>>>>>>>>>>  guests were: RHEL5u1-64,
RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>RHEL4u5-64.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  Timer_mode was set to 2
for 64-bit guests and 0 for
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>32-bit guests.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  All four hvm guests
experienced skew around -1%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>even the 32-bit
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  guest.  Less intensive
testing didn''t exhibit much
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>skew at all.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  A representative graph is
attached.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Dave, I wonder if some
portion of your patches
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>didn''t end up in
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  the xen trees?
> >>>>>>>>>>>>
> >>>>>>>>>>>>  (xm dmesg shows 8x Xeon
3.2GHz stepping 04,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>Platform timer
> >>
> >>
> >>>>>>>>>>>>  14.318MHz HPET.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Thanks,
> >>>>>>>>>>>>  Dan
> >>>>>>>>>>>>
> >>>>>>>>>>>>  P.S. Many thanks to
Deepak and Akira for running tests.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  > -----Original
Message-----
> >>>>>>>>>>>>  > From:
xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>
> >>
> >>>>>>>>>>>>  > Dave Winchell
> >>>>>>>>>>>>  > Sent: Wednesday,
January 09, 2008 9:53 AM
> >>>>>>>>>>>>  > To: Keir Fraser
> >>>>>>>>>>>>  > Cc:
dan.magenheimer@oracle.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>xen-devel@lists.xensource.com;
Dave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > Winchell
> >>>>>>>>>>>>  > Subject: Re:
[Xen-devel] [PATCH] Add a timer mode that
> >>>>>>>>>>>>  > disables pending
> >>>>>>>>>>>>  > missed ticks
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Hi Keir,
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > The latest change,
c/s 16690, looks fine.
> >>>>>>>>>>>>  > I agree that the
code in c/s 16690 is equivalent to
> >>>>>>>>>>>>  > the code I
submitted. Also, your version is more
> >>>>>>>>>>>>  > concise.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > The error tests
confirm the equivalence. With
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>overnight cpu loads,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > the checked in
version was accurate to +.048% for sles
> >>>>>>>>>>>>  > and +.038% for red
hat. My version was +.046%
> >>>>>>>>>>>>and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>+.032% in a
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > 2 hour test.
> >>>>>>>>>>>>  > I don''t
think the difference is significant.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > i/o loads produced
errors of +.01%.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Thanks for all your
efforts on this issue.
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Regards,
> >>>>>>>>>>>>  > Dave
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > Keir Fraser wrote:
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  > >Applied as c/s
16690, although the
> checked-in patch is
> >>>>>>>>>>>>  > smaller. I think the
> >>>>>>>>>>>>  > >only important
fix is to pt_intr_post() and the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>only bit of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > the patch I
> >>>>>>>>>>>>  > >totally omitted
was the change to
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>pt_process_missed_ticks().
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > I don''t
think
> >>>>>>>>>>>>  > >that change can
be important, but let''s see what
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>happens to the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  error
> >>>>>>>>>>>>  > >percentage...
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > > -- Keir
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >On 4/1/08 23:24,
"Dave Winchell"
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
>
>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >>Hi Dan and
Keir,
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Attached is
a patch that fixes some issues with the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>SYNC policy
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>(no_missed_ticks_pending).
> >>>>>>>>>>>>  > >>I have not
tried to make the change the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>minimal one, but,
> >>
> >>
> >>>>>>>>>>>>  > rather, just
> >>>>>>>>>>>>  > >>ported into
> >>>>>>>>>>>>  > >>the new code
what I know to work well. The error for
> >>>>>>>>>>>>  >
>>no_missed_ticks_pending goes from
> >>>>>>>>>>>>  > >>over 3% to
.03% with this change according
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>to my testing.
> >>
> >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Regards,
> >>>>>>>>>>>>  > >>Dave
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>Dan
Magenheimer wrote:
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>>Hi Dave
--
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Did you
get your correction ported?  If so,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>it would be
> >>
> >>
> >>>>>>>>>>>>  > nice to see this get
> >>>>>>>>>>>>  > >>>into
3.1.3.
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Note
that I just did some very limited testing with
> >>>>>>>>>>>>  >
timer_mode=2(=SYNC=no
> >>>>>>>>>>>>  > >>>missed
ticks pending)
> >>>>>>>>>>>>  > >>>on tip
of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>guest) and the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > worst error
I''ve
> >>>>>>>>>>>>  > >>>seen so
far
> >>>>>>>>>>>>  > >>>is
0.012%.  But I haven''t tried any exotic
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>loads, just LTP.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>Thanks,
> >>>>>>>>>>>>  > >>>Dan
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  >
>>>>-----Original Message-----
> >>>>>>>>>>>>  >
>>>>From: Dave Winchell
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:dwinchell@virtualiron.com]
> >>
> >>
> >>>>>>>>>>>>  >
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>>>  > >>>>To:
dan.magenheimer@oracle.com
> >>>>>>>>>>>>  > >>>>Cc:
Keir Fraser; Shan, Haitao;
> >>>>>>>>>>>>  >
xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>  >
>>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>>>  >
>>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>timer mode that
> >>
> >>
> >>>>>>>>>>>>  >
>>>>disables pending
> >>>>>>>>>>>>  >
>>>>missed ticks
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>Dan,
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>I
did some testing with the constant tsc offset
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>SYNC method
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>(now
called
> >>>>>>>>>>>>  >
>>>>no_missed_ticks_pending)
> >>>>>>>>>>>>  > >>>>and
found the error to be very high, much larger
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>than 1 %, as
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>I
recall.
> >>>>>>>>>>>>  > >>>>I
have not had a chance to submit a correction. I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>will try to
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>do
it later
> >>>>>>>>>>>>  > >>>>this
week or the first week in January. My
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>version of
> >>
> >>
> >>>>>>>>>>>>  constant tsc
> >>>>>>>>>>>>  >
>>>>offset SYNC method
> >>>>>>>>>>>>  >
>>>>produces .02 % error, so I just need to port
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>that into the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>current code.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>The
error you got for both of those kernels is
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>what I would
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  expect
> >>>>>>>>>>>>  > >>>>for
the default mode, delay_for_missed_ticks.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>I''ll let Keir answer on how to set the time mode.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>Regards,
> >>>>>>>>>>>>  > >>>>Dave
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>Dan
Magenheimer wrote:
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  >
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>saw a loss of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  >
>>>>about 0.2% with no load.  This was
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>xen-unstable tip today
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>>>with
no options specified.  32-bit was
> about 0.01%.
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>>I think I missed something... how do I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>run the various
> >>
> >>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  >
>>>>accounting choices and which ones are known to be
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>appropriate
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>>>for
which kernels?
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>>Thanks,
> >>>>>>>>>>>>  >
>>>>>Dan
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  >
>>>>>>-----Original Message-----
> >>>>>>>>>>>>  >
>>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  > >>>>Keir
Fraser
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>>>  >
>>>>>>To: Dave Winchell
> >>>>>>>>>>>>  >
>>>>>>Cc: Shan, Haitao;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>xen-devel@lists.xensource.com; Dong,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > Eddie; Jiang,
> >>>>>>>>>>>>  >
>>>>>>Yunhong
> >>>>>>>>>>>>  >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>mode that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>disables pending
> >>>>>>>>>>>>  >
>>>>>>missed ticks
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>Please take a look at xen-unstable
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>changeset 16545.
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>-- Keir
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Keir,
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>The accuracy data I''ve collected for i/o
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>loads for the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>various time protocols follows. In
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>addition, the data
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>for cpu loads is shown.
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>processor AMD
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  box.
> >>>>>>>>>>>>  >
>>>>>>>Two guests, red hat and sles 64 bit, 8
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>vcpu each.
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>>>  >
>>>>>>>(usex is available at
> >>>>>>>>>>>> 
http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>  >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>of=/dev/null.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>The loads labeled i/o-32 are 32
> instances of dd.
> >>>>>>>>>>>>  >
>>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>>>  >
>>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>>>  >
>>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>The loads labeled i/o-4/32 are the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>as i/o-32
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>instances of dd.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>>>>+4.42
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec -.006%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > +.005% cpu
> >>>>>>>>>>>>  >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.001%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > +.012% cpu
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.009%,
> >>
> >>
> >>>>>>>>>>>>  -.004% cpu
> >>>>>>>>>>>>  >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.005%, -.005% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.008%, -.003% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.040% cpu
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.034%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > -.031% cpu
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec,-55.7 sec, -.01%,
> >>
> >>
> >>>>>>>>>>>>  > -.09% i/o-8
> >>>>>>>>>>>>  >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec,-14.0 sec, -.015%
> >>
> >>
> >>>>>>>>>>>>  > -.14% i/o-8
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.017%,
> >>
> >>
> >>>>>>>>>>>>  > -.022% i/o-8
> >>>>>>>>>>>>  >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.017%, -.018% i/o-8
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.020%,
> >>
> >>
> >>>>>>>>>>>>  > -.029% i/o-8
> >>>>>>>>>>>>  >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>sec, -.023%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > -.031% i/o-8
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.04% i/o-32
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.011%,
> >>
> >>
> >>>>>>>>>>>>  > -.005% i/o-32
> >>>>>>>>>>>>  >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>-.11% i/o-32
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>13. sec -.07%,
> >>
> >>
> >>>>>>>>>>>>  > .003% i/o-4/32
> >>>>>>>>>>>>  >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>sec, -.017%,
> >>
> >>
> >>>>>>>>>>>>  > .01% i/o-4/32
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Overhead measurements:
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Progress in terms of number of passes
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>through a fixed
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>system workload
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
> >>>>>>>>>>>>  >
>>>>>>>The workload was usex -b48.
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>>>  >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>>>  >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>>>  >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Conclusions:
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>The only protocol which meets the .05% accuracy
> >>>>>>>>>>>>  > requirement for ntp
> >>>>>>>>>>>>  >
>>>>>>>tracking under the loads
> >>>>>>>>>>>>  >
>>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>accuracies for
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>SYNC, MIXED,
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>and ASYNC
> >>>>>>>>>>>>  >
>>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>We could reduce the cost of the SYNC
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>method by only
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>scheduling the extra
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>wakeups if a certain number
> >>>>>>>>>>>>  >
>>>>>>>of ticks are missed.
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Regards,
> >>>>>>>>>>>>  >
>>>>>>>Dave
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>ASYNC method a
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>couple of days ago,
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>I ran another ASYNC test. I think
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>there may have
> >>
> >>
> >>>>>>>>>>>>  > been something
> >>>>>>>>>>>>  >
>>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>days ago for
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>ASYNC. It may have been
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>after context
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>switch in.
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>My results indicate that either SYNC
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>or ASYNC give
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>acceptable accuracy,
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>each running consistently around or under
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>.01%. MIXED has
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>a fairly high
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>error of
> >>>>>>>>>>>>  >
>>>>>>>>>greater than .03%. Probably too close
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>to .05% ntp
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>threshold for comfort.
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>plan to leave
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>SYNC running
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>over the weekend. If you''d rather I
can
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>leave MIXED
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>running instead.
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>It may be too early to pick the protocol and
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>I can run
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>more overnight tests
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>next week.
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>I''m a bit worried about any unwanted
side
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>effects of the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>SYNC+run_timer
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>cause higher
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>system-wide CPU
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>contention. I find it easier to think
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>through the
> >>
> >>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>implications of ASYNC. I''m
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>accurate than
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>ASYNC. Perhaps it
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>delivers more timer interrupts than the other
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>approaches,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>and each interrupt
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>favourites and
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>if the latter is
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>actually more accurate then I can
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>simply revert the
> >>
> >>
> >>>>>>>>>>>>  > changeset that
> >>>>>>>>>>>>  >
>>>>>>>>implemented MIXED.
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>workloads you
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>could try idle
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>large disc reads
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>to /dev/null)? We
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>don''t have any data on workloads that
aren''t
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>CPU bound, so
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>that''s really an
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>-- Keir
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>>>
> >>>>>>>>>>>>  >
>>>>>>_______________________________________________
> >>>>>>>>>>>>  >
>>>>>>Xen-devel mailing list
> >>>>>>>>>>>>  >
>>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>  >
>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  >
>>>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>>
> >>>>>>>>>>>>  > >>diff -r
cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>>>  > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>2007 +0000
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>2008 -0500
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>@@ -58,7
+58,7 @@ static void
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>pt_process_missed_ticks(stru
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>    
missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>pt->period + 1;
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>     if (
mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>no_missed_ticks_pending) )
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>>>  > >>+       
pt->do_not_freeze = 1;
> >>>>>>>>>>>>  > >>     else
> >>>>>>>>>>>>  > >>        
pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>>>  > >>    
pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>>>  > >>@@ -127,7
+127,12 @@ static void
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>pt_timer_fn(void *data)
> >>
> >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>    
pt_lock(pt);
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>-   
pt->pending_intr_nr++;
> >>>>>>>>>>>>  > >>+    if (
mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>+       
pt->pending_intr_nr = 1;
> >>>>>>>>>>>>  > >>+
pt->do_not_freeze = 0;
> >>>>>>>>>>>>  > >>+    }
> >>>>>>>>>>>>  > >>+    else
> >>>>>>>>>>>>  > >>+
pt->pending_intr_nr++;
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>     if (
!pt->one_shot )
> >>>>>>>>>>>>  > >>     {
> >>>>>>>>>>>>  > >>@@ -221,8
+226,6 @@ void pt_intr_post(struct
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>vcpu *v, struct
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>        
return;
> >>>>>>>>>>>>  > >>     }
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>-   
pt->do_not_freeze = 0;
> >>>>>>>>>>>>  > >>-
> >>>>>>>>>>>>  > >>     if (
pt->one_shot )
> >>>>>>>>>>>>  > >>     {
> >>>>>>>>>>>>  > >>        
pt->enabled = 0;
> >>>>>>>>>>>>  > >>@@ -235,6
+238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>*v, struct
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>  > >>            
pt->last_plt_gtime > >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>hvm_get_guest_time(v);
> >>
> >>
> >>>>>>>>>>>>  > >>            
pt->pending_intr_nr = 0; /*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>''collapse'' all
> >>
> >>
> >>>>>>>>>>>>  > missed ticks */
> >>>>>>>>>>>>  > >>         }
> >>>>>>>>>>>>  > >>+ else if (
mode_is(v->domain,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>no_missed_ticks_pending) ) {
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>  > >>+    
pt->pending_intr_nr--;
> >>>>>>>>>>>>  > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>>>  > >>+ }
> >>>>>>>>>>>>  > >>        
else
> >>>>>>>>>>>>  > >>         {
> >>>>>>>>>>>>  > >>            
pt->last_plt_gtime +=
> pt->period_cycles;
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >>
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  > >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>  >
_______________________________________________
> >>>>>>>>>>>>  > Xen-devel mailing
list
> >>>>>>>>>>>>  >
Xen-devel@lists.xensource.com
> >>>>>>>>>>>>  >
http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>  >
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> >>>
> >>>
> >>--------------------------------------------------------------
> >>----------
> >>
> >>
> >>
> >>
> >
> >
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

Mine was oversubscribed.
8 physical cpu, 2 guests, each with 8 vcpu.
I ran one instance of ltp on each guest, continuously. I hope ltp
loaded up all the vcpus. I seem to recall that it did, but I
could be wrong. If it didn''t, that would be a major difference
between our tests. I''ll verify this afternoon and run multiple
instances,
if necessary.

Thanks,
Dave



Dan Magenheimer wrote:
>Hi Dave --
>
>No new results yet but one other question:
>
>The problems we''ve seen with our testing have been with a heavily
>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>running LTP simultaneously.
>
>Was your LTP testing oversubscribed or just a single guest?
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Thursday, February 14, 2008 10:56 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
>>Winchell
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>Dan,
>>
>>Here are some boot snipets for rh4u564 on xen 3.2.
>>
>>
>>#1:
>>
>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>...
>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
>>console=ttyS0 clocksource=pit nohpet
>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>65536 bytes)
>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor.
>>...
>>Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
>>CPUs: passed.
>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use 
>>of PM timer
>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
>>
>>
>>
>>#2:
>>
>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404
>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>...
>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
>>console=ttyS0 clocksource=pit nohpet nopmtimer
>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11,
>>65536 bytes)
>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor.
>>...
>>Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
>>CPUs: passed.
>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping.
>>
>>
>>As you can see, I only get the pit if I specify nopmtimer.
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>Thanks for continuing to run tests!
>>>
>>>Hmmm... I thought I had noticed that even though Linux will 
>>>      
>>>
>>acknowledge
>>    
>>
>>>the existence of the pmtimer, it still prints:
>>>
>>>time.c: Using PIT/TSC based timekeeping.
>>>
>>>I will check again, but assuming the clocksource for our tests is
>>>indeed pit, the huge difference in the results (yours vs ours) is
>>>baffling. I wonder if the difference may be the underlying hardware.
>>>Maybe we will try to ensure we can duplicate the results on 
>>>      
>>>
>>a different
>>    
>>
>>>box.
>>>
>>>
>>>So your testing was with stock 3.2.0 xen bits (what cset?) without
>>>any of your [quote from below] "clock related tweaks that I
haven''t
>>>submitted, because I''m still characterizing them"?
>>>
>>>
>>>      
>>>
>>None of the tweaks I mentioned are in this test.
>>It was stock with some patches. However, none of the patches are time
>>related to
>>my knowledge and I checked vpt.c to make sure that it is the same as
>>what''s in unstable.
>>The only difference is in pt_intr_post, where I set the timer mode.
>>I don''t have timer mode tied into our config process yet, which
>>is different than official xen method.
>>
>>
>>(In pt_intr_post)
>>      else
>>      {
>>+       if(v->arch.paging.mode->guest_levels == 4)
>>+          
v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE]
>>HVMPTM_no_missed_ticks_pending;
>>+       else
>>+          
v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE]
>>HVMPTM_delay_for_missed_ticks;
>>          if ( mode_is(v->domain, one_missed_tick_pending) ||
>>               mode_is(v->domain, no_missed_ticks_pending) )
>>          {
>>
>>    
>>
>>>Could you also send detail on the rhel4u4-64 kernel you
>>>are testing with, just to ensure we are not comparing apples
>>>and oranges?  (Perhaps there''s some way we can even share
the
>>>identical disk image and vm.cfg file?)
>>>
>>>And if our problem is indeed the pmtimer, I will need to submit
>>>another patch to Keir to add an hvm pmtimer platform variable.
>>>(Hmmm... I don''t think he''s even accepted the hpet
variable patch
>>>yet.  I''ll have to check.)
>>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>Sent: Thursday, February 14, 2008 9:00 AM
>>>>To: dan.magenheimer@oracle.com
>>>>Cc: Dave Winchell; Keir Fraser; 
>>>>        
>>>>
>>xen-devel@lists.xensource.com; Deepak
>>    
>>
>>>>Patel
>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>disables pending
>>>>missed ticks
>>>>
>>>>
>>>>Hi Dan,
>>>>
>>>>I ran the ltp tests with 3.2 and found the errors
>>>>for a 16 hour run to be:
>>>>
>>>>rh4u564 -9.9 sec (-.017%)
>>>>rh4u464 -7.3 sec (-.013%)
>>>>
>>>>There were no cliffs and the drift was linear.
>>>>
>>>>I think the problem you had may be due to the use of the
>>>>pm timer. If you still have the boot log, it would tell you.
>>>>
>>>>When I first tried a guest on 3.2 with "clocksource=pit
nohpet"
>>>>I noticed that it picked the pm timer. Adding
"nopmtimer", the
>>>>guest will pick the pit.
>>>>
>>>>The reason I didn''t have the problem with our 3.1 base
is that
>>>>I had disabled the hpet and the pmtimer by not advertising them
>>>>in the acpi tables. I did this so long ago, I forgot that I had
to
>>>>disable pmtimer as well as hpet.
>>>>
>>>>So, can you re-run your test with "clocksource=pit nohpet 
>>>>        
>>>>
>>nopmtimer"?
>>    
>>
>>>>You should see this in the boot messages:
>>>>
>>>>time.c: Using PIT/TSC based timekeeping.
>>>>
>>>>Thanks,
>>>>Dave
>>>>
>>>>
>>>>Dave Winchell wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Hi Dan,
>>>>>
>>>>>Over the weekend the drift was +18 seconds for each guest
(no ntp).
>>>>>The duration was 3900 minutes, so the error for each was
+.0077%.
>>>>>Looking back through the data, it appears to drift linearly
at
>>>>>this rate. I''ve attached a plot for rh4u5-64.
>>>>>
>>>>>This accuracy is better than what I''ve seen before
(.03-.05%).
>>>>>This may be due to the different load (ltp vs usex) or to 
>>>>>          
>>>>>
>>one of the
>>    
>>
>>>>>changes I''ve made recently. I''ll do some
experimentation to see if
>>>>>there is
>>>>>a fix I should propose.
>>>>>
>>>>>This still doesn''t address the radical drift you
saw.
>>>>>The next step for me is to run 3.2 and see if I can
reproduce it.
>>>>>
>>>>>Regards,
>>>>>Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Dave Winchell wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Hi Dan,
>>>>>>
>>>>>>Sorry it took me so long, but I finally ran an ltp test
today.
>>>>>>Its on rh4u4-64. I''m using the defaults for ltp
and using a script
>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd.
>>>>>>virtual processors / physical processors = 2.
>>>>>>
>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in
300 minutes
>>>>>>for -.005% and .008%.
>>>>>>
>>>>>>I''m running a 3.1 based hypervisor with some
clock related
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>tweaks that
>>>>
>>>>
>>>>        
>>>>
>>>>>>I haven''t submitted, because I''m still
characterizing them.
>>>>>>
>>>>>>I''m stopping the usex load on 4u5-64 now and
replacing it with ltp
>>>>>>and will leave the two guests running ltp over the
weekend.
>>>>>>
>>>>>>Regards,
>>>>>>Dave
>>>>>>
>>>>>>
>>>>>>Dave Winchell wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>Hi Dan, Deepak:
>>>>>>>
>>>>>>>Thanks for the data. Those drifts are severe - no
wonder
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>ntp couldn''t
>>>>
>>>>
>>>>        
>>>>
>>>>>>>keep then in synch. I''ll try to reproduce
that behaviour
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>here, with
>>>>
>>>>
>>>>        
>>>>
>>>>>>>my code base.
>>>>>>>If I can''t reproduce it, I''ll try
3.2.
>>>>>>>
>>>>>>>If you can isolate what ltp is doing during the
cliffs,
>>>>>>>              
>>>>>>>
>>that would
>>    
>>
>>>>>>>be very
>>>>>>>helpful.
>>>>>>>
>>>>>>>thanks,
>>>>>>>Dave
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Dan Magenheimer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>OK, Deepak repeated the test without ntpd and
using
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>ntpdate -b before
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>the test.
>>>>>>>>
>>>>>>>>The attached graph shows his results: el5u1-64
(best=~0.07%),
>>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32
(worst=~0.3%).
>>>>>>>>
>>>>>>>>We will continue to look at LTP to try to
isolate.
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>Dan
>>>>>>>>
>>>>>>>>P.S. elXuY is essentially RHEL XuY with some
patches.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>-----Original Message-----
>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM
>>>>>>>>>To: Deepak Patel
>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser;
>>>>>>>>>xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>Dave Winchell
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that disables
>>>>>>>>>pending
>>>>>>>>>missed ticks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dan, Deeepak,
>>>>>>>>>
>>>>>>>>>It may be that the underlying clock error is
too great for ntp
>>>>>>>>>to handle. It would be useful if you did not
run ntpd
>>>>>>>>>and, instead did ntpdate -b
<timeserver> at the start
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>of the test
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>for each guest. Then capture the data as you
have been doing.
>>>>>>>>>If the drift is greater than .05%, then we
need to
>>>>>>>>>                  
>>>>>>>>>
>>address that.
>>    
>>
>>>>>>>>>Another option is, when running ntpd, to
enable loop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>statistics in
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>/etc/ntp.conf
>>>>>>>>>by adding this to the file:
>>>>>>>>>
>>>>>>>>>statistics loopstats
>>>>>>>>>statsdir /var/lib/ntp/
>>>>>>>>>
>>>>>>>>>Then you will see loop data in that
directory.
>>>>>>>>>Correlating the data in the loopstats files
with the
>>>>>>>>>peaks in skew would be interesting. You will
see
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>entries of the form
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>54495 76787.701 -0.045153303 -132.569229
0.020806776
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>239.735511 10
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>Where the second to last column is the Allan
Deviation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>When that
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>gets over 1000, ntpd is working pretty hard.
However,
>>>>>>>>>                  
>>>>>>>>>
>>I have not
>>    
>>
>>>>>>>>>seen ntpd
>>>>>>>>>completely loose it like you have.
>>>>>>>>>
>>>>>>>>>I''m on vacation until Monday, and
won''t be reading
>>>>>>>>>email.
>>>>>>>>>
>>>>>>>>>Thanks for all your work on this!
>>>>>>>>>
>>>>>>>>>-Dave
>>>>>>>>>
>>>>>>>>>Deepak Patel wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>Is the graph for RHEL5u1-64?
(I''ve never tested this one.)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>I do not know which graph was attached
with this. But
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>I saw this
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and
EL5U1 - 64 hvm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>guests when I
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>was running ltp tests continuously.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>What was the behaviour of the other
guests running?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>All pvm guests are fine. But behavior of
most of the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>hvm guests were
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>as described.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>If they had spikes, were they at the
same wall time?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>No. They are not at the same wall time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Were the other guests running ltp as
well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the
guests are running ltp
>>>>>>>>>>continuously.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>How are you measuring skew?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>I was collecting output of "ntpdate
-q  <timeserver> every
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>300 seconds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>(5 minutes) and have created graph based
on that.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Are you running ntpd?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>Yes. ntp was running on all the guests.
>>>>>>>>>>
>>>>>>>>>>I am investigating what causes this
spikes and let everyone
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>know what
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>are my findings.
>>>>>>>>>>
>>>>>>>>>>Thanks,
>>>>>>>>>>Deepak
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>Anything that you can discover that
would be in sync with
>>>>>>>>>>>the spikes would be very helpful!
>>>>>>>>>>>
>>>>>>>>>>>The code that I test with is our
product code, which is based
>>>>>>>>>>>on 3.1. So it is possible that
something in 3.2 other
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>than vpt.c
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>is the cause. I can test with 3.2,
if necessary.
>>>>>>>>>>>
>>>>>>>>>>>thanks,
>>>>>>>>>>>Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>Hi Dave (Keir, see suggestion
below) --
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>>Turning off vhpet certainly
helps a lot (though see below).
>>>>>>>>>>>>
>>>>>>>>>>>>I wonder if timekeeping with
vhpet is so bad that it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>should be
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>turned off by default (in 3.1,
3.2, and unstable)
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>until it is
>>    
>>
>>>>>>>>>>>>fixed?  (I have a patch that
defaults it off, can post it if
>>>>>>>>>>>>there is agreement on the above
point.)  The whole
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>point of an
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>HPET is to provide more precise
timekeeping and if vhpet is
>>>>>>>>>>>>worse than vpit, it can only
confuse users.  Comments?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>In your testing, are you just
measuring % skew over a long
>>>>>>>>>>>>period of time?
>>>>>>>>>>>>We are graphing the skew
continuously and
>>>>>>>>>>>>seeing periodic behavior that is
unsettling, even with pit.
>>>>>>>>>>>>See attached.  Though your
algorithm recovers, the "cliffs"
>>>>>>>>>>>>could still cause real user
problems.  I wonder if there is
>>>>>>>>>>>>anything that can be done to
make the "recovery" more
>>>>>>>>>>>>responsive?
>>>>>>>>>>>>
>>>>>>>>>>>>We are looking into what part(s)
of LTP is causing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>the cliffs.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>Dan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>Sent: Monday, January 28,
2008 8:21 AM
>>>>>>>>>>>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>>>>>Cc: Keir Fraser;
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave
Winchell
>>>>>>>>>>>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>that disables
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>pending
>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>
>>>>>>>>>>>>>I guess I''m a bit
out of date calling for clock= usage.
>>>>>>>>>>>>>Looking at linux 2.6.20.4
sources, I think you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>should specify
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>"clocksource=pit
nohpet" on the linux guest bootline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>You can leave the xen and
dom0 bootlines as they are.
>>>>>>>>>>>>>The xen and guest
clocksources do not need to be the same.
>>>>>>>>>>>>>In my tests, xen is using
the hpet for its timekeeping and
>>>>>>>>>>>>>that appears to be the
default.
>>>>>>>>>>>>>
>>>>>>>>>>>>>When you boot the guests you
should see
>>>>>>>>>>>>> time.c: Using PIT/TSC based
timekeeping.
>>>>>>>>>>>>>on the rh4u5-64 guest, and
something similar on the others.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon
3.2GHz stepping 04, Platform timer
>>>>>>>>>>>>>>14.318MHz HPET.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>This appears to be the xen
state, which is fine.
>>>>>>>>>>>>>I was wrongly assuming that
this was the guest state.
>>>>>>>>>>>>>You might want to look in
your guest logs and see
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>what they were
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>picking
>>>>>>>>>>>>>for a clock source.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks, I
hadn''t realized that!  No wonder we didn''t
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>see the same
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>>improvement you saw!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Try specifying
clock=pit on the linux boot line...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>I''m confused...
do you mean "clocksource=pit" on the Xen
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>command line or
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>"nohpet" /
"clock=pit" / "clocksource=pit" on the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>guest (or
>>    
>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>dom0?) command
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>line?  Or both places? 
Since the tests take awhile, it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>would be nice
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>to get this right the
first time.  Do the Xen and guest
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>clocksources need
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>to be the same?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>*From:* Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>*Sent:* Sunday, January
27, 2008 2:22 PM
>>>>>>>>>>>>>>*To:*
dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>deepak.patel@oracle.com;
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
>>>>>>>>>>>>>>*Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>that disables
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>>pending missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hpet timer does have a
fairly large error, as I
>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>trying this
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> one recently.
>>>>>>>>>>>>>> I don''t
remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>about right.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The problem is that
hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>the module
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Keir and I did
>>>>>>>>>>>>>> all the recent work in,
for its periodic timer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>needs. Try
>>    
>>
>>>>>>>>>>>>>> specifying clock=pit
>>>>>>>>>>>>>> on the linux boot line.
If it still picks the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>hpet, which it
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> might, let me know
>>>>>>>>>>>>>> and I''ll tell
you how to get around this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>--------------------------------------------------------------
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>----------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* Dan Magenheimer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>[mailto:dan.magenheimer@oracle.com]
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008
6:50 PM
>>>>>>>>>>>>>> *To:* Dave Winchell;
Keir Fraser
>>>>>>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>deepak.patel@oracle.com;
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> akira.ijuin@oracle.com
>>>>>>>>>>>>>> *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>that disables
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> pending missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for the very late
followup on this but we finally
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>were able
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to get our testing set
up again on stable 3.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>bits and have
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> seen some very bad
results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>order of 1%.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> Test enviroment was a
4-socket dual core machine
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>with 24GB of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> memory running six
two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>plus two pv.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> All six guests were
running LTP simultaneously.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>The four hvm
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> guests were:
RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>RHEL4u5-64.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> Timer_mode was set to 2
for 64-bit guests and 0 for
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> All four hvm guests
experienced skew around -1%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>even the 32-bit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> guest.  Less intensive
testing didn''t exhibit much
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>skew at all.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> A representative graph
is attached.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dave, I wonder if some
portion of your patches
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>didn''t end up in
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> the xen trees?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon
3.2GHz stepping 04,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>Platform timer
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. Many thanks to
Deepak and Akira for running tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > -----Original
Message-----
>>>>>>>>>>>>>> > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > Dave Winchell
>>>>>>>>>>>>>> > Sent: Wednesday,
January 09, 2008 9:53 AM
>>>>>>>>>>>>>> > To: Keir Fraser
>>>>>>>>>>>>>> > Cc:
dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > Winchell
>>>>>>>>>>>>>> > Subject: Re:
[Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>>>>> > disables pending
>>>>>>>>>>>>>> > missed ticks
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Hi Keir,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The latest change,
c/s 16690, looks fine.
>>>>>>>>>>>>>> > I agree that the
code in c/s 16690 is equivalent to
>>>>>>>>>>>>>> > the code I
submitted. Also, your version is more
>>>>>>>>>>>>>> > concise.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The error tests
confirm the equivalence. With
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>overnight cpu loads,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > the checked in
version was accurate to +.048% for sles
>>>>>>>>>>>>>> > and +.038% for red
hat. My version was +.046%
>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>+.032% in a
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > 2 hour test.
>>>>>>>>>>>>>> > I don''t
think the difference is significant.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > i/o loads produced
errors of +.01%.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks for all
your efforts on this issue.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Regards,
>>>>>>>>>>>>>> > Dave
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Keir Fraser wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > >Applied as c/s
16690, although the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>checked-in patch is
>>    
>>
>>>>>>>>>>>>>> > smaller. I think
the
>>>>>>>>>>>>>> > >only important
fix is to pt_intr_post() and the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>only bit of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > the patch I
>>>>>>>>>>>>>> > >totally
omitted was the change to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>pt_process_missed_ticks().
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > I don''t
think
>>>>>>>>>>>>>> > >that change
can be important, but let''s see what
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>happens to the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> error
>>>>>>>>>>>>>> > >percentage...
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > -- Keir
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >On 4/1/08
23:24, "Dave Winchell"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >>Hi Dan and
Keir,
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Attached
is a patch that fixes some issues with the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>SYNC policy
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>(no_missed_ticks_pending).
>>>>>>>>>>>>>> > >>I have not
tried to make the change the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>minimal one, but,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > rather, just
>>>>>>>>>>>>>> > >>ported
into
>>>>>>>>>>>>>> > >>the new
code what I know to work well. The error for
>>>>>>>>>>>>>> >
>>no_missed_ticks_pending goes from
>>>>>>>>>>>>>> > >>over 3% to
.03% with this change according
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>to my testing.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Regards,
>>>>>>>>>>>>>> > >>Dave
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>Dan
Magenheimer wrote:
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>>Hi
Dave --
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>Did
you get your correction ported?  If so,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>it would be
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > nice to see this
get
>>>>>>>>>>>>>> > >>>into
3.1.3.
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>Note
that I just did some very limited testing with
>>>>>>>>>>>>>> >
timer_mode=2(=SYNC=no
>>>>>>>>>>>>>> > >>>missed
ticks pending)
>>>>>>>>>>>>>> > >>>on tip
of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>guest) and the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > worst error
I''ve
>>>>>>>>>>>>>> > >>>seen
so far
>>>>>>>>>>>>>> > >>>is
0.012%.  But I haven''t tried any exotic
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>loads, just LTP.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> >
>>>Thanks,
>>>>>>>>>>>>>> > >>>Dan
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> >
>>>>-----Original Message-----
>>>>>>>>>>>>>> >
>>>>From: Dave Winchell
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>[mailto:dwinchell@virtualiron.com]
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>> >
>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>> >
>>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>> >
xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>>> >
>>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>> >
>>>>Subject: Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>timer mode that
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>disables pending
>>>>>>>>>>>>>> >
>>>>missed ticks
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>Dan,
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>I
did some testing with the constant tsc offset
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>SYNC method
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>(now called
>>>>>>>>>>>>>> >
>>>>no_missed_ticks_pending)
>>>>>>>>>>>>>> >
>>>>and found the error to be very high, much larger
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>I
recall.
>>>>>>>>>>>>>> > >>>>I
have not had a chance to submit a correction. I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>will try to
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>>>do
it later
>>>>>>>>>>>>>> >
>>>>this week or the first week in January. My
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>version of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> constant tsc
>>>>>>>>>>>>>> >
>>>>offset SYNC method
>>>>>>>>>>>>>> >
>>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>that into the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>current code.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>The error you got for both of those kernels is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>what I would
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>> >
>>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>I''ll let Keir answer on how to set the time mode.
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>Regards,
>>>>>>>>>>>>>> >
>>>>Dave
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>saw a loss of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>about 0.2% with no load.  This was
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>xen-unstable tip today
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>with no options specified.  32-bit was
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>about 0.01%.
>>    
>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>>I think I missed something... how do I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>run the various
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>accounting choices and which ones are known to be
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>appropriate
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>for which kernels?
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>>Thanks,
>>>>>>>>>>>>>> >
>>>>>Dan
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>>-----Original Message-----
>>>>>>>>>>>>>> >
>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>Keir Fraser
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>> >
>>>>>>To: Dave Winchell
>>>>>>>>>>>>>> >
>>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>xen-devel@lists.xensource.com; Dong,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > Eddie; Jiang,
>>>>>>>>>>>>>> >
>>>>>>Yunhong
>>>>>>>>>>>>>> >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>mode that
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>disables pending
>>>>>>>>>>>>>> >
>>>>>>missed ticks
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>changeset 16545.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>-- Keir
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Keir,
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>The accuracy data I''ve collected for i/o
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>loads for the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>various time protocols follows. In
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>addition, the data
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>for cpu loads is shown.
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>processor AMD
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> box.
>>>>>>>>>>>>>> >
>>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>vcpu each.
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>> >
>>>>>>>(usex is available at
>>>>>>>>>>>>>>
http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>> >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-32 are 32
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>instances of dd.
>>    
>>
>>>>>>>>>>>>>> >
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>>> >
>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>> >
>>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>as i/o-32
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>instances of dd.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>+4.42
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec -.006%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > +.005% cpu
>>>>>>>>>>>>>> >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.001%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > +.012% cpu
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec, -.009%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> -.004% cpu
>>>>>>>>>>>>>> >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.005%, -.005% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.008%, -.003% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.034%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > -.031% cpu
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec,-55.7 sec, -.01%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.09% i/o-8
>>>>>>>>>>>>>> >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec,-14.0 sec, -.015%
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.14% i/o-8
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec, -.017%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.022% i/o-8
>>>>>>>>>>>>>> >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.017%, -.018% i/o-8
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec, -.020%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.029% i/o-8
>>>>>>>>>>>>>> >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>sec, -.023%,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > -.031% i/o-8
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec, -.011%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > -.005% i/o-32
>>>>>>>>>>>>>> >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>13. sec -.07%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > .003% i/o-4/32
>>>>>>>>>>>>>> >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>sec, -.017%,
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > .01% i/o-4/32
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Overhead measurements:
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>through a fixed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>system workload
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>>> >
>>>>>>>The workload was usex -b48.
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>>> >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>>> >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>> >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Conclusions:
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>>> > requirement for
ntp
>>>>>>>>>>>>>> >
>>>>>>>tracking under the loads
>>>>>>>>>>>>>> >
>>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>accuracies for
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>SYNC, MIXED,
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>and ASYNC
>>>>>>>>>>>>>> >
>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>method by only
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>scheduling the extra
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>wakeups if a certain number
>>>>>>>>>>>>>> >
>>>>>>>of ticks are missed.
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Regards,
>>>>>>>>>>>>>> >
>>>>>>>Dave
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>couple of days ago,
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>there may have
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > been something
>>>>>>>>>>>>>> >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>days ago for
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>ASYNC. It may have been
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>after context
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>switch in.
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>or ASYNC give
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>acceptable accuracy,
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>each running consistently around or under
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>a fairly high
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>error of
>>>>>>>>>>>>>> >
>>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>to .05% ntp
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>threshold for comfort.
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>plan to leave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>SYNC running
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>over the weekend. If you''d rather I
can
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>leave MIXED
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>running instead.
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I can run
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>more overnight tests
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>next week.
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>effects of the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>SYNC+run_timer
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>cause higher
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>system-wide CPU
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>through the
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>implications of ASYNC. I''m
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>accurate than
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>approaches,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>and each interrupt
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>favourites and
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>if the latter is
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>actually more accurate then I can
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>simply revert the
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > changeset that
>>>>>>>>>>>>>> >
>>>>>>>>implemented MIXED.
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>workloads you
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>could try idle
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>large disc reads
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>to /dev/null)? We
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>that''s really an
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>-- Keir
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>_______________________________________________
>>>>>>>>>>>>>> >
>>>>>>Xen-devel mailing list
>>>>>>>>>>>>>> >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>> >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>diff -r
cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>> > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>2007 +0000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>2008 -0500
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>@@ -58,7
+58,7 @@ static void
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>pt_process_missed_ticks(stru
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>    
missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>pt->period + 1;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>     if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>no_missed_ticks_pending) )
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>-       
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>> > >>+       
pt->do_not_freeze = 1;
>>>>>>>>>>>>>> > >>     else
>>>>>>>>>>>>>> > >>        
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>> > >>    
pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>> > >>@@ -127,7
+127,12 @@ static void
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>pt_timer_fn(void *data)
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>    
pt_lock(pt);
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>-   
pt->pending_intr_nr++;
>>>>>>>>>>>>>> > >>+    if (
mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>+       
pt->pending_intr_nr = 1;
>>>>>>>>>>>>>> > >>+
pt->do_not_freeze = 0;
>>>>>>>>>>>>>> > >>+    }
>>>>>>>>>>>>>> > >>+    else
>>>>>>>>>>>>>> > >>+
pt->pending_intr_nr++;
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>     if (
!pt->one_shot )
>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>> > >>@@ -221,8
+226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>vcpu *v, struct
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>        
return;
>>>>>>>>>>>>>> > >>     }
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>-   
pt->do_not_freeze = 0;
>>>>>>>>>>>>>> > >>-
>>>>>>>>>>>>>> > >>     if (
pt->one_shot )
>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>> > >>        
pt->enabled = 0;
>>>>>>>>>>>>>> > >>@@ -235,6
+238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>*v, struct
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>> > >>          
pt->last_plt_gtime >>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>hvm_get_guest_time(v);
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > >>          
pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>''collapse'' all
>>>>
>>>>
>>>>        
>>>>
>>>>>>>>>>>>>> > missed ticks */
>>>>>>>>>>>>>> > >>         }
>>>>>>>>>>>>>> > >>+ else if
( mode_is(v->domain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>>>>> > >>+    
pt->pending_intr_nr--;
>>>>>>>>>>>>>> > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>> > >>+ }
>>>>>>>>>>>>>> > >>        
else
>>>>>>>>>>>>>> > >>         {
>>>>>>>>>>>>>> > >>          
pt->last_plt_gtime +=
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>pt->period_cycles;
>>    
>>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
_______________________________________________
>>>>>>>>>>>>>> > Xen-devel mailing
list
>>>>>>>>>>>>>> >
Xen-devel@lists.xensource.com
>>>>>>>>>>>>>> >
http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>
>>>>>          
>>>>>
>>>>--------------------------------------------------------------
>>>>----------
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

ltp runs by default loading up only one vcpu.
The -x option can be used to run multiple instances, though
in this mode you will get test failures.
I ran 8 instances on each guest for 16 hours, 25 min
and the time error was -11 sec (-.019%) on each guest.

Regards,
Dave

Dave Winchell wrote:
> Hi Dan,
>
> Mine was oversubscribed.
> 8 physical cpu, 2 guests, each with 8 vcpu.
> I ran one instance of ltp on each guest, continuously. I hope ltp
> loaded up all the vcpus. I seem to recall that it did, but I
> could be wrong. If it didn''t, that would be a major difference
> between our tests. I''ll verify this afternoon and run multiple
instances,
> if necessary.
>
> Thanks,
> Dave
>
>
>
> Dan Magenheimer wrote:
>
>> Hi Dave --
>>
>> No new results yet but one other question:
>>
>> The problems we''ve seen with our testing have been with a
heavily
>> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>> running LTP simultaneously.
>>
>> Was your LTP testing oversubscribed or just a single guest?
>>
>> Thanks,
>> Dan
>>
>>  
>>
>>> -----Original Message-----
>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>> Sent: Thursday, February 14, 2008 10:56 AM
>>> To: dan.magenheimer@oracle.com
>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
>>> Winchell
>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables
pending
>>> missed ticks
>>>
>>>
>>> Dan,
>>>
>>> Here are some boot snipets for rh4u564 on xen 3.2.
>>>
>>>
>>> #1:
>>>
>>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
>>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6
20060404
>>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>> ...
>>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/
>>> console=ttyS0 clocksource=pit nohpet
>>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order:
11,
>>> 65536 bytes)
>>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer.
>>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz
processor.
>>> ...
>>> Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8
>>> CPUs: passed.
>>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM
timer
>>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping.
>>>
>>>
>>>
>>> #2:
>>>
>>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
>>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6
20060404
>>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>> ...
>>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/
>>> console=ttyS0 clocksource=pit nohpet nopmtimer
>>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order:
11,
>>> 65536 bytes)
>>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer.
>>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz
processor.
>>> ...
>>> Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8
>>> CPUs: passed.
>>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based
timekeeping.
>>>
>>>
>>> As you can see, I only get the pit if I specify nopmtimer.
>>>
>>> Dan Magenheimer wrote:
>>>
>>>   
>>>
>>>> Hi Dave --
>>>>
>>>> Thanks for continuing to run tests!
>>>>
>>>> Hmmm... I thought I had noticed that even though Linux will
>>>
>>> acknowledge
>>>   
>>>
>>>> the existence of the pmtimer, it still prints:
>>>>
>>>> time.c: Using PIT/TSC based timekeeping.
>>>>
>>>> I will check again, but assuming the clocksource for our tests
is
>>>> indeed pit, the huge difference in the results (yours vs ours)
is
>>>> baffling. I wonder if the difference may be the underlying
hardware.
>>>> Maybe we will try to ensure we can duplicate the results on
>>>
>>> a different
>>>   
>>>
>>>> box.
>>>>
>>>>
>>>> So your testing was with stock 3.2.0 xen bits (what cset?)
without
>>>> any of your [quote from below] "clock related tweaks that
I haven''t
>>>> submitted, because I''m still characterizing
them"?
>>>>
>>>>
>>>>     
>>>
>>> None of the tweaks I mentioned are in this test.
>>> It was stock with some patches. However, none of the patches are
time
>>> related to
>>> my knowledge and I checked vpt.c to make sure that it is the same
as
>>> what''s in unstable.
>>> The only difference is in pt_intr_post, where I set the timer mode.
>>> I don''t have timer mode tied into our config process yet,
which
>>> is different than official xen method.
>>>
>>>
>>> (In pt_intr_post)
>>>      else
>>>      {
>>> +       if(v->arch.paging.mode->guest_levels == 4)
>>> +          
v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >>>
HVMPTM_no_missed_ticks_pending;
>>> +       else
>>> +          
v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >>>
HVMPTM_delay_for_missed_ticks;
>>>          if ( mode_is(v->domain, one_missed_tick_pending) ||
>>>               mode_is(v->domain, no_missed_ticks_pending) )
>>>          {
>>>
>>>   
>>>
>>>> Could you also send detail on the rhel4u4-64 kernel you
>>>> are testing with, just to ensure we are not comparing apples
>>>> and oranges?  (Perhaps there''s some way we can even
share the
>>>> identical disk image and vm.cfg file?)
>>>>
>>>> And if our problem is indeed the pmtimer, I will need to submit
>>>> another patch to Keir to add an hvm pmtimer platform variable.
>>>> (Hmmm... I don''t think he''s even accepted the
hpet variable patch
>>>> yet.  I''ll have to check.)
>>>>
>>>> Thanks,
>>>> Dan
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>> -----Original Message-----
>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>> Sent: Thursday, February 14, 2008 9:00 AM
>>>>> To: dan.magenheimer@oracle.com
>>>>> Cc: Dave Winchell; Keir Fraser;       
>>>>
>>> xen-devel@lists.xensource.com; Deepak
>>>   
>>>
>>>>> Patel
>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
>>>>> disables pending
>>>>> missed ticks
>>>>>
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> I ran the ltp tests with 3.2 and found the errors
>>>>> for a 16 hour run to be:
>>>>>
>>>>> rh4u564 -9.9 sec (-.017%)
>>>>> rh4u464 -7.3 sec (-.013%)
>>>>>
>>>>> There were no cliffs and the drift was linear.
>>>>>
>>>>> I think the problem you had may be due to the use of the
>>>>> pm timer. If you still have the boot log, it would tell
you.
>>>>>
>>>>> When I first tried a guest on 3.2 with
"clocksource=pit nohpet"
>>>>> I noticed that it picked the pm timer. Adding
"nopmtimer", the
>>>>> guest will pick the pit.
>>>>>
>>>>> The reason I didn''t have the problem with our 3.1
base is that
>>>>> I had disabled the hpet and the pmtimer by not advertising
them
>>>>> in the acpi tables. I did this so long ago, I forgot that I
had to
>>>>> disable pmtimer as well as hpet.
>>>>>
>>>>> So, can you re-run your test with "clocksource=pit
nohpet
>>>>
>>> nopmtimer"?
>>>   
>>>
>>>>> You should see this in the boot messages:
>>>>>
>>>>> time.c: Using PIT/TSC based timekeeping.
>>>>>
>>>>> Thanks,
>>>>> Dave
>>>>>
>>>>>
>>>>> Dave Winchell wrote:
>>>>>
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> Over the weekend the drift was +18 seconds for each
guest (no ntp).
>>>>>> The duration was 3900 minutes, so the error for each
was +.0077%.
>>>>>> Looking back through the data, it appears to drift
linearly at
>>>>>> this rate. I''ve attached a plot for rh4u5-64.
>>>>>>
>>>>>> This accuracy is better than what I''ve seen
before (.03-.05%).
>>>>>> This may be due to the different load (ltp vs usex) or
to
>>>>>
>>> one of the
>>>   
>>>
>>>>>> changes I''ve made recently. I''ll do
some experimentation to see if
>>>>>> there is
>>>>>> a fix I should propose.
>>>>>>
>>>>>> This still doesn''t address the radical drift
you saw.
>>>>>> The next step for me is to run 3.2 and see if I can
reproduce it.
>>>>>>
>>>>>> Regards,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dave Winchell wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> Sorry it took me so long, but I finally ran an ltp
test today.
>>>>>>> Its on rh4u4-64. I''m using the defaults
for ltp and using a script
>>>>>>> called runltp. I had a usex load on rh4u5-64. No
ntpd.
>>>>>>> virtual processors / physical processors = 2.
>>>>>>>
>>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4)
in 300 minutes
>>>>>>> for -.005% and .008%.
>>>>>>>
>>>>>>> I''m running a 3.1 based hypervisor with
some clock related
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>
>>>>> tweaks that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>> I haven''t submitted, because I''m
still characterizing them.
>>>>>>>
>>>>>>> I''m stopping the usex load on 4u5-64 now
and replacing it with ltp
>>>>>>> and will leave the two guests running ltp over the
weekend.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>> Dave Winchell wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>
>>>>>>>> Hi Dan, Deepak:
>>>>>>>>
>>>>>>>> Thanks for the data. Those drifts are severe -
no wonder
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>> ntp couldn''t
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>> keep then in synch. I''ll try to
reproduce that behaviour
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>
>>>>> here, with
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>> my code base.
>>>>>>>> If I can''t reproduce it, I''ll
try 3.2.
>>>>>>>>
>>>>>>>> If you can isolate what ltp is doing during the
cliffs,
>>>>>>>>             
>>>>>>>
>>> that would
>>>   
>>>
>>>>>>>> be very
>>>>>>>> helpful.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>             
>>>>>>>>
>>>>>>>>> OK, Deepak repeated the test without ntpd
and using
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>
>>>>> ntpdate -b before
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>> the test.
>>>>>>>>>
>>>>>>>>> The attached graph shows his results:
el5u1-64 (best=~0.07%),
>>>>>>>>> el4u5-64 (middle=~0.2%), and el4u5-32
(worst=~0.3%).
>>>>>>>>>
>>>>>>>>> We will continue to look at LTP to try to
isolate.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>> P.S. elXuY is essentially RHEL XuY with
some patches.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>               
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>> Sent: Wednesday, January 30, 2008 2:45
PM
>>>>>>>>>> To: Deepak Patel
>>>>>>>>>> Cc: dan.magenheimer@oracle.com; Keir
Fraser;
>>>>>>>>>> xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> Dave Winchell
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a
timer mode that disables
>>>>>>>>>> pending
>>>>>>>>>> missed ticks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dan, Deeepak,
>>>>>>>>>>
>>>>>>>>>> It may be that the underlying clock
error is too great for ntp
>>>>>>>>>> to handle. It would be useful if you
did not run ntpd
>>>>>>>>>> and, instead did ntpdate -b
<timeserver> at the start
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> of the test
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> for each guest. Then capture the data
as you have been doing.
>>>>>>>>>> If the drift is greater than .05%, then
we need to
>>>>>>>>>>                 
>>>>>>>>>
>>> address that.
>>>   
>>>
>>>>>>>>>> Another option is, when running ntpd,
to enable loop
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> statistics in
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> /etc/ntp.conf
>>>>>>>>>> by adding this to the file:
>>>>>>>>>>
>>>>>>>>>> statistics loopstats
>>>>>>>>>> statsdir /var/lib/ntp/
>>>>>>>>>>
>>>>>>>>>> Then you will see loop data in that
directory.
>>>>>>>>>> Correlating the data in the loopstats
files with the
>>>>>>>>>> peaks in skew would be interesting. You
will see
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> entries of the form
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> 54495 76787.701 -0.045153303
-132.569229 0.020806776
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> 239.735511 10
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> Where the second to last column is the
Allan Deviation.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>
>>>>> When that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>> gets over 1000, ntpd is working pretty
hard. However,
>>>>>>>>>>                 
>>>>>>>>>
>>> I have not
>>>   
>>>
>>>>>>>>>> seen ntpd
>>>>>>>>>> completely loose it like you have.
>>>>>>>>>>
>>>>>>>>>> I''m on vacation until Monday,
and won''t be reading
>>>>>>>>>> email.
>>>>>>>>>>
>>>>>>>>>> Thanks for all your work on this!
>>>>>>>>>>
>>>>>>>>>> -Dave
>>>>>>>>>>
>>>>>>>>>> Deepak Patel wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>> Is the graph for RHEL5u1-64?
(I''ve never tested this one.)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> I do not know which graph was
attached with this. But
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> I saw this
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> behavior in EL4u5 - 32, EL4U5 - 64
and EL5U1 - 64 hvm
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> guests when I
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> was running ltp tests continuously.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> What was the behaviour of the
other guests running?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> All pvm guests are fine. But
behavior of most of the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>> hvm guests were
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>> as described.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> If they had spikes, were they
at the same wall time?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> No. They are not at the same wall
time.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Were the other guests running
ltp as well?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> Yes all 6 guests (4 hvm and 2 pvm)
the guests are running ltp
>>>>>>>>>>> continuously.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> How are you measuring skew?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> I was collecting output of
"ntpdate -q  <timeserver> every
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>>>>>>> 300 seconds
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>> (5 minutes) and have created graph
based on that.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Are you running ntpd?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>>>>>>>> Yes. ntp was running on all the
guests.
>>>>>>>>>>>
>>>>>>>>>>> I am investigating what causes this
spikes and let everyone
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>
>>>>>>>>>> know what
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>> are my findings.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Deepak
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                   
>>>>>>>>>>>
>>>>>>>>>>>> Anything that you can discover
that would be in sync with
>>>>>>>>>>>> the spikes would be very
helpful!
>>>>>>>>>>>>
>>>>>>>>>>>> The code that I test with is
our product code, which is based
>>>>>>>>>>>> on 3.1. So it is possible that
something in 3.2 other
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>
>>>>> than vpt.c
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>> is the cause. I can test with
3.2, if necessary.
>>>>>>>>>>>>
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> Dave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                     
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dave (Keir, see
suggestion below) --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Turning off vhpet certainly
helps a lot (though see below).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I wonder if timekeeping
with vhpet is so bad that it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> should be
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> turned off by default (in
3.1, 3.2, and unstable)
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>> until it is
>>>   
>>>
>>>>>>>>>>>>> fixed?  (I have a patch
that defaults it off, can post it if
>>>>>>>>>>>>> there is agreement on the
above point.)  The whole
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> point of an
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> HPET is to provide more
precise timekeeping and if vhpet is
>>>>>>>>>>>>> worse than vpit, it can
only confuse users.  Comments?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In your testing, are you
just measuring % skew over a long
>>>>>>>>>>>>> period of time?
>>>>>>>>>>>>> We are graphing the skew
continuously and
>>>>>>>>>>>>> seeing periodic behavior
that is unsettling, even with pit.
>>>>>>>>>>>>> See attached.  Though your
algorithm recovers, the "cliffs"
>>>>>>>>>>>>> could still cause real user
problems.  I wonder if there is
>>>>>>>>>>>>> anything that can be done
to make the "recovery" more
>>>>>>>>>>>>> responsive?
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are looking into what
part(s) of LTP is causing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>> the cliffs.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original
Message-----
>>>>>>>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>> Sent: Monday, January
28, 2008 8:21 AM
>>>>>>>>>>>>>> To:
dan.magenheimer@oracle.com
>>>>>>>>>>>>>> Cc: Keir Fraser;
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>
deepak.patel@oracle.com;
>>>>>>>>>>>>>> akira.ijuin@oracle.com;
Dave Winchell
>>>>>>>>>>>>>> Subject: Re:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>> that disables
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> pending
>>>>>>>>>>>>>> missed ticks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I guess I''m a
bit out of date calling for clock= usage.
>>>>>>>>>>>>>> Looking at linux
2.6.20.4 sources, I think you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>> should specify
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> "clocksource=pit
nohpet" on the linux guest bootline.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can leave the xen
and dom0 bootlines as they are.
>>>>>>>>>>>>>> The xen and guest
clocksources do not need to be the same.
>>>>>>>>>>>>>> In my tests, xen is
using the hpet for its timekeeping and
>>>>>>>>>>>>>> that appears to be the
default.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When you boot the
guests you should see
>>>>>>>>>>>>>> time.c: Using PIT/TSC
based timekeeping.
>>>>>>>>>>>>>> on the rh4u5-64 guest,
and something similar on the others.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (xm dmesg shows 8x
Xeon 3.2GHz stepping 04, Platform timer
>>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This appears to be the
xen state, which is fine.
>>>>>>>>>>>>>> I was wrongly assuming
that this was the guest state.
>>>>>>>>>>>>>> You might want to look
in your guest logs and see
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>> what they were
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> picking
>>>>>>>>>>>>>> for a clock source.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dan Magenheimer wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, I
hadn''t realized that!  No wonder we didn''t
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> see the same
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> improvement you
saw!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Try specifying
clock=pit on the linux boot line...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I''m
confused... do you mean "clocksource=pit" on the Xen
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> command line or
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "nohpet"
/ "clock=pit" / "clocksource=pit" on the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> guest (or
>>>   
>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dom0?) command
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> line?  Or both
places?  Since the tests take awhile, it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> would be nice
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to get this right
the first time.  Do the Xen and guest
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> clocksources need
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to be the same?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----Original
Message-----
>>>>>>>>>>>>>>> *From:* Dave
Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>> *Sent:* Sunday,
January 27, 2008 2:22 PM
>>>>>>>>>>>>>>> *To:*
dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> deepak.patel@oracle.com;
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>>
akira.ijuin@oracle.com; Dave Winchell
>>>>>>>>>>>>>>> *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> that disables
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> pending missed
ticks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hpet timer does
have a fairly large error, as I
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> trying this
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> one recently.
>>>>>>>>>>>>>>> I don''t
remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> about right.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem is that
hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the module
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Keir and I did
>>>>>>>>>>>>>>> all the recent work
in, for its periodic timer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> needs. Try
>>>   
>>>
>>>>>>>>>>>>>>> specifying
clock=pit
>>>>>>>>>>>>>>> on the linux boot
line. If it still picks the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> hpet, which it
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> might, let me know
>>>>>>>>>>>>>>> and I''ll
tell you how to get around this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>
--------------------------------------------------------------
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *From:* Dan
Magenheimer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> [mailto:dan.magenheimer@oracle.com]
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> *Sent:* Fri
1/25/2008 6:50 PM
>>>>>>>>>>>>>>> *To:* Dave
Winchell; Keir Fraser
>>>>>>>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> deepak.patel@oracle.com;
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>>
akira.ijuin@oracle.com
>>>>>>>>>>>>>>> *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that disables
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pending missed
ticks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry for the very
late followup on this but we finally
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> were able
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to get our testing
set up again on stable 3.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> bits and have
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> seen some very bad
results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> order of 1%.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> Test enviroment was
a 4-socket dual core machine
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> with 24GB of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> memory running six
two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> plus two pv.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> All six guests were
running LTP simultaneously.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> The four hvm
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> guests were:
RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> RHEL4u5-64.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> Timer_mode was set
to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 32-bit guests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All four hvm guests
experienced skew around -1%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> even the 32-bit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> guest.  Less
intensive testing didn''t exhibit much
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> skew at all.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> A representative
graph is attached.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dave, I wonder if
some portion of your patches
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> didn''t end up in
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> the xen trees?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (xm dmesg shows 8x
Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> Platform timer
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> 14.318MHz HPET.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. Many thanks to
Deepak and Akira for running tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > -----Original
Message-----
>>>>>>>>>>>>>>> > From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > Dave Winchell
>>>>>>>>>>>>>>> > Sent:
Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>>>> > To: Keir
Fraser
>>>>>>>>>>>>>>> > Cc:
dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
xen-devel@lists.xensource.com; Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > Winchell
>>>>>>>>>>>>>>> > Subject: Re:
[Xen-devel] [PATCH] Add a timer mode that
>>>>>>>>>>>>>>> > disables
pending
>>>>>>>>>>>>>>> > missed ticks
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Hi Keir,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > The latest
change, c/s 16690, looks fine.
>>>>>>>>>>>>>>> > I agree that
the code in c/s 16690 is equivalent to
>>>>>>>>>>>>>>> > the code I
submitted. Also, your version is more
>>>>>>>>>>>>>>> > concise.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > The error
tests confirm the equivalence. With
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> overnight cpu loads,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > the checked in
version was accurate to +.048% for sles
>>>>>>>>>>>>>>> > and +.038% for
red hat. My version was +.046%
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> +.032% in a
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > 2 hour test.
>>>>>>>>>>>>>>> > I
don''t think the difference is significant.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > i/o loads
produced errors of +.01%.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thanks for all
your efforts on this issue.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Regards,
>>>>>>>>>>>>>>> > Dave
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Keir Fraser
wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > >Applied as
c/s 16690, although the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> checked-in patch is
>>>   
>>>
>>>>>>>>>>>>>>> > smaller. I
think the
>>>>>>>>>>>>>>> > >only
important fix is to pt_intr_post() and the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> only bit of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > the patch I
>>>>>>>>>>>>>>> > >totally
omitted was the change to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> pt_process_missed_ticks().
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > I
don''t think
>>>>>>>>>>>>>>> > >that
change can be important, but let''s see what
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> happens to the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>> >
>percentage...
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > -- Keir
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >On 4/1/08
23:24, "Dave Winchell"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
<dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >>Hi Dan
and Keir,
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> >
>>Attached is a patch that fixes some issues with the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SYNC policy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>(no_missed_ticks_pending).
>>>>>>>>>>>>>>> > >>I have
not tried to make the change the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> minimal one, but,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > rather, just
>>>>>>>>>>>>>>> > >>ported
into
>>>>>>>>>>>>>>> > >>the
new code what I know to work well. The error for
>>>>>>>>>>>>>>> >
>>no_missed_ticks_pending goes from
>>>>>>>>>>>>>>> > >>over
3% to .03% with this change according
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> to my testing.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> >
>>Regards,
>>>>>>>>>>>>>>> > >>Dave
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>Dan
Magenheimer wrote:
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>>Hi
Dave --
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> >
>>>Did you get your correction ported?  If so,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> it would be
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > nice to see
this get
>>>>>>>>>>>>>>> >
>>>into 3.1.3.
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> >
>>>Note that I just did some very limited testing with
>>>>>>>>>>>>>>> >
timer_mode=2(=SYNC=no
>>>>>>>>>>>>>>> >
>>>missed ticks pending)
>>>>>>>>>>>>>>> > >>>on
tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> guest) and the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > worst error
I''ve
>>>>>>>>>>>>>>> >
>>>seen so far
>>>>>>>>>>>>>>> > >>>is
0.012%.  But I haven''t tried any exotic
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> loads, just LTP.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> >
>>>Thanks,
>>>>>>>>>>>>>>> >
>>>Dan
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> >
>>>>-----Original Message-----
>>>>>>>>>>>>>>> >
>>>>From: Dave Winchell
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> [mailto:dwinchell@virtualiron.com]
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>>> >
>>>>To: dan.magenheimer@oracle.com
>>>>>>>>>>>>>>> >
>>>>Cc: Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>>> >
xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>>>>>> >
>>>>Eddie; Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>>> >
>>>>Subject: Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> timer mode that
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>disables pending
>>>>>>>>>>>>>>> >
>>>>missed ticks
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>Dan,
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>I did some testing with the constant tsc offset
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SYNC method
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>(now called
>>>>>>>>>>>>>>> >
>>>>no_missed_ticks_pending)
>>>>>>>>>>>>>>> >
>>>>and found the error to be very high, much larger
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> than 1 %, as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>I recall.
>>>>>>>>>>>>>>> >
>>>>I have not had a chance to submit a correction. I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> will try to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>do it later
>>>>>>>>>>>>>>> >
>>>>this week or the first week in January. My
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> version of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> constant tsc
>>>>>>>>>>>>>>> >
>>>>offset SYNC method
>>>>>>>>>>>>>>> >
>>>>produces .02 % error, so I just need to port
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> that into the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>current code.
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>The error you got for both of those kernels is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> what I would
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>>> >
>>>>for the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>I''ll let Keir answer on how to set the time mode.
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>Regards,
>>>>>>>>>>>>>>> >
>>>>Dave
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>>Anyone make measurements on the final patch?
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> saw a loss of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>about 0.2% with no load.  This was
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> xen-unstable tip today
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>with no options specified.  32-bit was
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> about 0.01%.
>>>   
>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>>I think I missed something... how do I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> run the various
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>accounting choices and which ones are known to be
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>for which kernels?
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>>Thanks,
>>>>>>>>>>>>>>> >
>>>>>Dan
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>-----Original Message-----
>>>>>>>>>>>>>>> >
>>>>>>From: xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>Keir Fraser
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>>> >
>>>>>>To: Dave Winchell
>>>>>>>>>>>>>>> >
>>>>>>Cc: Shan, Haitao;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> xen-devel@lists.xensource.com; Dong,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > Eddie; Jiang,
>>>>>>>>>>>>>>> >
>>>>>>Yunhong
>>>>>>>>>>>>>>> >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> mode that
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>disables pending
>>>>>>>>>>>>>>> >
>>>>>>missed ticks
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>Please take a look at xen-unstable
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> changeset 16545.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>-- Keir
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>On 26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Keir,
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The accuracy data I''ve collected for i/o
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> loads for the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>various time protocols follows. In
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> addition, the data
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>for cpu loads is shown.
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> processor AMD
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> box.
>>>>>>>>>>>>>>> >
>>>>>>>Two guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> vcpu each.
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>>> >
>>>>>>>(usex is available at
>>>>>>>>>>>>>>>
http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>>> >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> of=/dev/null.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-32 are 32
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> instances of dd.
>>>   
>>>
>>>>>>>>>>>>>>> >
>>>>>>>Also, these are run on 4 cpu AMD box.
>>>>>>>>>>>>>>> >
>>>>>>>In addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>>> >
>>>>>>>All three guests are 8vcpu.
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> as i/o-32
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>except that the redhat-64 guest has 4
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> instances of dd.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Date Duration Protocol sles, rhat error load
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>> +4.42
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> sec -.006%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > +.005% cpu
>>>>>>>>>>>>>>> >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.001%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > +.012% cpu
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec, -.009%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> -.004% cpu
>>>>>>>>>>>>>>> >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.005%, -.005% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.008%, -.003% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.040% cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.034%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > -.031% cpu
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec,-55.7 sec, -.01%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.09% i/o-8
>>>>>>>>>>>>>>> >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec,-14.0 sec, -.015%
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.14% i/o-8
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec, -.017%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.022% i/o-8
>>>>>>>>>>>>>>> >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.017%, -.018% i/o-8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec, -.020%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.029% i/o-8
>>>>>>>>>>>>>>> >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> sec, -.023%,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > -.031% i/o-8
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.04% i/o-32
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec, -.011%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > -.005% i/o-32
>>>>>>>>>>>>>>> >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -.11% i/o-32
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> 13. sec -.07%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > .003% i/o-4/32
>>>>>>>>>>>>>>> >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> sec, -.017%,
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > .01% i/o-4/32
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Overhead measurements:
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Progress in terms of number of passes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> through a fixed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>system workload
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle.
>>>>>>>>>>>>>>> >
>>>>>>>The workload was usex -b48.
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
>>>>>>>>>>>>>>> >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
>>>>>>>>>>>>>>> >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>>> >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Conclusions:
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>The only protocol which meets the .05% accuracy
>>>>>>>>>>>>>>> > requirement
for ntp
>>>>>>>>>>>>>>> >
>>>>>>>tracking under the loads
>>>>>>>>>>>>>>> >
>>>>>>>above is the SYNC protocol. The worst case
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> accuracies for
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>SYNC, MIXED,
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>and ASYNC
>>>>>>>>>>>>>>> >
>>>>>>>are .022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>We could reduce the cost of the SYNC
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> method by only
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>scheduling the extra
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>wakeups if a certain number
>>>>>>>>>>>>>>> >
>>>>>>>of ticks are missed.
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Regards,
>>>>>>>>>>>>>>> >
>>>>>>>Dave
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>Keir Fraser wrote:
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>><dwinchell@virtualiron.com> wrote:
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>Since I had a high error (~.03%) for the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ASYNC method a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>couple of days ago,
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>I ran another ASYNC test. I think
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> there may have
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > been something
>>>>>>>>>>>>>>> >
>>>>>>>>>wrong with the code I used a couple of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> days ago for
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>ASYNC. It may have been
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>missing the immediate delivery of interrupt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> after context
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>switch in.
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>My results indicate that either SYNC
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> or ASYNC give
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>acceptable accuracy,
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>each running consistently around or under
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> .01%. MIXED has
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>a fairly high
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>error of
>>>>>>>>>>>>>>> >
>>>>>>>>>greater than .03%. Probably too close
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> to .05% ntp
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>threshold for comfort.
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> plan to leave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>SYNC running
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>over the weekend. If you''d rather I
can
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> leave MIXED
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>running instead.
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>It may be too early to pick the protocol and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I can run
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>more overnight tests
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>next week.
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>I''m a bit worried about any unwanted
side
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> effects of the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>SYNC+run_timer
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>approach -- e.g., whether timer wakeups will
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cause higher
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>system-wide CPU
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>contention. I find it easier to think
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> through the
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>implications of ASYNC. I''m
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>surprised that MIXED loses time, and is less
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> accurate than
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>ASYNC. Perhaps it
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>delivers more timer interrupts than the other
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> approaches,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>and each interrupt
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>event causes a small accumulated error?
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>Overall I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> favourites and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>if the latter is
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>actually more accurate then I can
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> simply revert the
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > changeset that
>>>>>>>>>>>>>>> >
>>>>>>>>implemented MIXED.
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>Perhaps rather than running more of the same
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> workloads you
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>could try idle
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> large disc reads
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>to /dev/null)? We
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>don''t have any data on workloads that
aren''t
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CPU bound, so
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>that''s really an
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>obvious place to put any further effort imo.
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>-- Keir
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>_______________________________________________
>>>>>>>>>>>>>>> >
>>>>>>Xen-devel mailing list
>>>>>>>>>>>>>>> >
>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>> >
>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> >
>>>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>diff
-r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>>> > >>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> 2007 +0000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> 2008 -0500
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>@@
-58,7 +58,7 @@ static void
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> pt_process_missed_ticks(stru
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>    
missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pt->period + 1;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>    
if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
no_missed_ticks_pending) )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>-     
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>>> > >>+     
pt->do_not_freeze = 1;
>>>>>>>>>>>>>>> > >>    
else
>>>>>>>>>>>>>>> > >>      
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>>> > >>    
pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>>> > >>@@
-127,7 +127,12 @@ static void
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> pt_timer_fn(void *data)
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>    
pt_lock(pt);
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>-   
pt->pending_intr_nr++;
>>>>>>>>>>>>>>> > >>+   
if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
no_missed_ticks_pending) ) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+     
pt->pending_intr_nr = 1;
>>>>>>>>>>>>>>> > >>+
pt->do_not_freeze = 0;
>>>>>>>>>>>>>>> > >>+    }
>>>>>>>>>>>>>>> > >>+   
else
>>>>>>>>>>>>>>> > >>+
pt->pending_intr_nr++;
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>    
if ( !pt->one_shot )
>>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>>> > >>@@
-221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> vcpu *v, struct
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>      
return;
>>>>>>>>>>>>>>> > >>     }
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>-   
pt->do_not_freeze = 0;
>>>>>>>>>>>>>>> > >>-
>>>>>>>>>>>>>>> > >>    
if ( pt->one_shot )
>>>>>>>>>>>>>>> > >>     {
>>>>>>>>>>>>>>> > >>      
pt->enabled = 0;
>>>>>>>>>>>>>>> > >>@@
-235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *v, struct
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > >>      
pt->last_plt_gtime
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> hvm_get_guest_time(v);
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > >>      
pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>> ''collapse'' all
>>>>>
>>>>>
>>>>>       
>>>>>
>>>>>>>>>>>>>>> > missed ticks
*/
>>>>>>>>>>>>>>> > >>      
}
>>>>>>>>>>>>>>> > >>+ else
if ( mode_is(v->domain,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> no_missed_ticks_pending) ) {
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                 
>>>>>>>>>>
>>>>>>>>>>>>>>> > >>+    
pt->pending_intr_nr--;
>>>>>>>>>>>>>>> > >>+    
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>>> > >>+ }
>>>>>>>>>>>>>>> > >>      
else
>>>>>>>>>>>>>>> > >>      
{
>>>>>>>>>>>>>>> > >>      
pt->last_plt_gtime +=
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> pt->period_cycles;
>>>   
>>>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
_______________________________________________
>>>>>>>>>>>>>>> > Xen-devel
mailing list
>>>>>>>>>>>>>>> >
Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>> >
http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                       
>>>>>>>>>>>>
>>>>>>
>>>>>>         
>>>>>
>>>>>
--------------------------------------------------------------
>>>>> ----------
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>       
>>>>
>>>>
>>>>     
>>>
>>>   
>>
>>
>>  
>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave --

Thanks for that observation on ltp running on one vcpu!

With "clocksource=pit nohpet nopmtimer" our clock skew
problems seem to have been reduced to a reasonable
percentage.  However, our 32-bit clock skew seems to
show a measureable problem now.  As a result,
I''ve been doing some digging into kernel sources and
have observed the following relative to RHEL4 (2.6.9-based)
kernels and RHEL5 (2.6.18-based) kernels and thought I
would document them for posterity.  Some of
our confusion arises from the fact that invalid command
line parameters are silently ignored.

RHEL4:
- clock= is a valid parameter for RHEL4-32
- clocksource= is not a valid parameter for RHEL4-xx
- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
- nopmtimer is not a valid parameter for RHEL4-xx
- notsc is a valid parameter for RHEL4-32, not RHEL4-64
- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently

For Xen RHEL4 HVM guests:
- I *think* clock=pit is sufficient for RHEL4-32 [1]
- I *think* nohpet is sufficient for RHEL4-64 [1]

RHEL5:
- there are two kinds of timekeeping, WALL and gtod
- clocksource= is a valid parameter for RHEL5-xx
- clock= is a valid but deprecated parameter for RHEL5-xx
- clock= and clocksource= are essentially equivalent
- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
- clock=pit changes the gtod source but not the WALL source[2]
- nohpet nopmtimer changes the WALL source to PIT
- /sys/devices/system/clocksource/clocksource0/...
  available_clocksource lists the possible clock sources
  current_clocksource lists the chosen clock source
  ..but neither of these works in a RHEL5 guest!

For Xen RHEL5 HVM guests:
- I *think* clock=pit is sufficient for RHEL5-32
- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64

Other info:
- As of 2.6.24.2, clock= is still valid (though still deprecated)

So, some open questions:
[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
    (I *think* not as it has never come up in any email.)
[2] In RHEL5, I *think* it is the WALL source that we care about?

And finally, since invalid command line parameters are ignored.
I think specifying:
	clock=pit nohpet nopmtimer
will force the guest clock sources into the optimal state for
all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
question above on tsc).  And we should keep an eye on
kernel/time/clocksource.c to ensure the __setup("clock="...)
line doesn''t go away before RHEL6.

Note that if hpet=0 and pmtimer=0 were the default hvm platform
parameters for all xen hvm guests (on all versions of xen),
specifying kernel command line parameters would be unnecessary,
but c''est la vie.

Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
and timer_mode=2 for 64-bit RHEL guests.

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Tuesday, February 19, 2008 8:27 AM
> To: dan.magenheimer@oracle.com
> Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
> Patel
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> 
> Hi Dan,
> 
> ltp runs by default loading up only one vcpu.
> The -x option can be used to run multiple instances, though
> in this mode you will get test failures.
> I ran 8 instances on each guest for 16 hours, 25 min
> and the time error was -11 sec (-.019%) on each guest.
> 
> Regards,
> Dave
> 
> Dave Winchell wrote:
> 
> > Hi Dan,
> >
> > Mine was oversubscribed.
> > 8 physical cpu, 2 guests, each with 8 vcpu.
> > I ran one instance of ltp on each guest, continuously. I hope ltp
> > loaded up all the vcpus. I seem to recall that it did, but I
> > could be wrong. If it didn''t, that would be a major
difference
> > between our tests. I''ll verify this afternoon and run 
> multiple instances,
> > if necessary.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > Dan Magenheimer wrote:
> >
> >> Hi Dave --
> >>
> >> No new results yet but one other question:
> >>
> >> The problems we''ve seen with our testing have been with a
heavily
> >> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
> >> running LTP simultaneously.
> >>
> >> Was your LTP testing oversubscribed or just a single guest?
> >>
> >> Thanks,
> >> Dan
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>> Sent: Thursday, February 14, 2008 10:56 AM
> >>> To: dan.magenheimer@oracle.com
> >>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel;
Dave
> >>> Winchell
> >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> >>> missed ticks
> >>>
> >>>
> >>> Dan,
> >>>
> >>> Here are some boot snipets for rh4u564 on xen 3.2.
> >>>
> >>>
> >>> #1:
> >>>
> >>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro
> >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> >>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
> 3.4.6 20060404
> >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>> ...
> >>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro
root=LABEL=/
> >>> console=ttyS0 clocksource=pit nohpet
> >>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> >>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 
> 2048 (order: 11,
> >>> 65536 bytes)
> >>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM
timer.
> >>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 
> MHz processor.
> >>> ...
> >>> Feb 14 10:45:00 vs076 kernel: checking TSC 
> synchronization across 8
> >>> CPUs: passed.
> >>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> >>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to 
> use of PM timer
> >>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based
timekeeping.
> >>>
> >>>
> >>>
> >>> #2:
> >>>
> >>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro
> >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
> >>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
> >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
> 3.4.6 20060404
> >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>> ...
> >>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro
root=LABEL=/
> >>> console=ttyS0 clocksource=pit nohpet nopmtimer
> >>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> >>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 
> 2048 (order: 11,
> >>> 65536 bytes)
> >>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz 
> PIT timer.
> >>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 
> MHz processor.
> >>> ...
> >>> Feb 14 10:47:59 vs076 kernel: checking TSC 
> synchronization across 8
> >>> CPUs: passed.
> >>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> >>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based 
> timekeeping.
> >>>
> >>>
> >>> As you can see, I only get the pit if I specify nopmtimer.
> >>>
> >>> Dan Magenheimer wrote:
> >>>
> >>>
> >>>
> >>>> Hi Dave --
> >>>>
> >>>> Thanks for continuing to run tests!
> >>>>
> >>>> Hmmm... I thought I had noticed that even though Linux
will
> >>>
> >>> acknowledge
> >>>
> >>>
> >>>> the existence of the pmtimer, it still prints:
> >>>>
> >>>> time.c: Using PIT/TSC based timekeeping.
> >>>>
> >>>> I will check again, but assuming the clocksource for our
tests is
> >>>> indeed pit, the huge difference in the results (yours vs
ours) is
> >>>> baffling. I wonder if the difference may be the 
> underlying hardware.
> >>>> Maybe we will try to ensure we can duplicate the results
on
> >>>
> >>> a different
> >>>
> >>>
> >>>> box.
> >>>>
> >>>>
> >>>> So your testing was with stock 3.2.0 xen bits (what 
> cset?) without
> >>>> any of your [quote from below] "clock related tweaks 
> that I haven''t
> >>>> submitted, because I''m still characterizing
them"?
> >>>>
> >>>>
> >>>>
> >>>
> >>> None of the tweaks I mentioned are in this test.
> >>> It was stock with some patches. However, none of the 
> patches are time
> >>> related to
> >>> my knowledge and I checked vpt.c to make sure that it is 
> the same as
> >>> what''s in unstable.
> >>> The only difference is in pt_intr_post, where I set the 
> timer mode.
> >>> I don''t have timer mode tied into our config process
yet, which
> >>> is different than official xen method.
> >>>
> >>>
> >>> (In pt_intr_post)
> >>>      else
> >>>      {
> >>> +       if(v->arch.paging.mode->guest_levels == 4)
> >>> +           
> v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >
>>> HVMPTM_no_missed_ticks_pending;
> >>> +       else
> >>> +           
> v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >
>>> HVMPTM_delay_for_missed_ticks;
> >>>          if ( mode_is(v->domain, one_missed_tick_pending)
||
> >>>               mode_is(v->domain, no_missed_ticks_pending) )
> >>>          {
> >>>
> >>>
> >>>
> >>>> Could you also send detail on the rhel4u4-64 kernel you
> >>>> are testing with, just to ensure we are not comparing
apples
> >>>> and oranges?  (Perhaps there''s some way we can
even share the
> >>>> identical disk image and vm.cfg file?)
> >>>>
> >>>> And if our problem is indeed the pmtimer, I will need to
submit
> >>>> another patch to Keir to add an hvm pmtimer platform
variable.
> >>>> (Hmmm... I don''t think he''s even
accepted the hpet variable patch
> >>>> yet.  I''ll have to check.)
> >>>>
> >>>> Thanks,
> >>>> Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>> Sent: Thursday, February 14, 2008 9:00 AM
> >>>>> To: dan.magenheimer@oracle.com
> >>>>> Cc: Dave Winchell; Keir Fraser;
> >>>>
> >>> xen-devel@lists.xensource.com; Deepak
> >>>
> >>>
> >>>>> Patel
> >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>> disables pending
> >>>>> missed ticks
> >>>>>
> >>>>>
> >>>>> Hi Dan,
> >>>>>
> >>>>> I ran the ltp tests with 3.2 and found the errors
> >>>>> for a 16 hour run to be:
> >>>>>
> >>>>> rh4u564 -9.9 sec (-.017%)
> >>>>> rh4u464 -7.3 sec (-.013%)
> >>>>>
> >>>>> There were no cliffs and the drift was linear.
> >>>>>
> >>>>> I think the problem you had may be due to the use of
the
> >>>>> pm timer. If you still have the boot log, it would
tell you.
> >>>>>
> >>>>> When I first tried a guest on 3.2 with
"clocksource=pit nohpet"
> >>>>> I noticed that it picked the pm timer. Adding
"nopmtimer", the
> >>>>> guest will pick the pit.
> >>>>>
> >>>>> The reason I didn''t have the problem with our
3.1 base is that
> >>>>> I had disabled the hpet and the pmtimer by not
advertising them
> >>>>> in the acpi tables. I did this so long ago, I forgot 
> that I had to
> >>>>> disable pmtimer as well as hpet.
> >>>>>
> >>>>> So, can you re-run your test with
"clocksource=pit nohpet
> >>>>
> >>> nopmtimer"?
> >>>
> >>>
> >>>>> You should see this in the boot messages:
> >>>>>
> >>>>> time.c: Using PIT/TSC based timekeeping.
> >>>>>
> >>>>> Thanks,
> >>>>> Dave
> >>>>>
> >>>>>
> >>>>> Dave Winchell wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Hi Dan,
> >>>>>>
> >>>>>> Over the weekend the drift was +18 seconds for
each
> guest (no ntp).
> >>>>>> The duration was 3900 minutes, so the error for
each
> was +.0077%.
> >>>>>> Looking back through the data, it appears to drift
linearly at
> >>>>>> this rate. I''ve attached a plot for
rh4u5-64.
> >>>>>>
> >>>>>> This accuracy is better than what I''ve
seen before (.03-.05%).
> >>>>>> This may be due to the different load (ltp vs
usex) or to
> >>>>>
> >>> one of the
> >>>
> >>>
> >>>>>> changes I''ve made recently. I''ll
do some
> experimentation to see if
> >>>>>> there is
> >>>>>> a fix I should propose.
> >>>>>>
> >>>>>> This still doesn''t address the radical
drift you saw.
> >>>>>> The next step for me is to run 3.2 and see if I
can
> reproduce it.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Dave
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Dave Winchell wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Hi Dan,
> >>>>>>>
> >>>>>>> Sorry it took me so long, but I finally ran an
ltp test today.
> >>>>>>> Its on rh4u4-64. I''m using the
defaults for ltp and
> using a script
> >>>>>>> called runltp. I had a usex load on rh4u5-64.
No ntpd.
> >>>>>>> virtual processors / physical processors = 2.
> >>>>>>>
> >>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec
(4u4) in
> 300 minutes
> >>>>>>> for -.005% and .008%.
> >>>>>>>
> >>>>>>> I''m running a 3.1 based hypervisor
with some clock related
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>> tweaks that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> I haven''t submitted, because
I''m still characterizing them.
> >>>>>>>
> >>>>>>> I''m stopping the usex load on 4u5-64
now and
> replacing it with ltp
> >>>>>>> and will leave the two guests running ltp over
the weekend.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Dave
> >>>>>>>
> >>>>>>>
> >>>>>>> Dave Winchell wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hi Dan, Deepak:
> >>>>>>>>
> >>>>>>>> Thanks for the data. Those drifts are
severe - no wonder
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>> ntp couldn''t
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>> keep then in synch. I''ll try to
reproduce that behaviour
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>> here, with
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>> my code base.
> >>>>>>>> If I can''t reproduce it,
I''ll try 3.2.
> >>>>>>>>
> >>>>>>>> If you can isolate what ltp is doing
during the cliffs,
> >>>>>>>>
> >>>>>>>
> >>> that would
> >>>
> >>>
> >>>>>>>> be very
> >>>>>>>> helpful.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Dave
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> OK, Deepak repeated the test without
ntpd and using
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>> ntpdate -b before
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>> the test.
> >>>>>>>>>
> >>>>>>>>> The attached graph shows his results:
el5u1-64
> (best=~0.07%),
> >>>>>>>>> el4u5-64 (middle=~0.2%), and el4u5-32
(worst=~0.3%).
> >>>>>>>>>
> >>>>>>>>> We will continue to look at LTP to try
to isolate.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Dan
> >>>>>>>>>
> >>>>>>>>> P.S. elXuY is essentially RHEL XuY
with some patches.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>>> Sent: Wednesday, January 30, 2008
2:45 PM
> >>>>>>>>>> To: Deepak Patel
> >>>>>>>>>> Cc: dan.magenheimer@oracle.com;
Keir Fraser;
> >>>>>>>>>> xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> Dave Winchell
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> Subject: Re: [Xen-devel] [PATCH]
Add a timer mode
> that disables
> >>>>>>>>>> pending
> >>>>>>>>>> missed ticks
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Dan, Deeepak,
> >>>>>>>>>>
> >>>>>>>>>> It may be that the underlying
clock error is too
> great for ntp
> >>>>>>>>>> to handle. It would be useful if
you did not run ntpd
> >>>>>>>>>> and, instead did ntpdate -b
<timeserver> at the start
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> of the test
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> for each guest. Then capture the
data as you have
> been doing.
> >>>>>>>>>> If the drift is greater than .05%,
then we need to
> >>>>>>>>>>
> >>>>>>>>>
> >>> address that.
> >>>
> >>>
> >>>>>>>>>> Another option is, when running
ntpd, to enable loop
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> statistics in
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> /etc/ntp.conf
> >>>>>>>>>> by adding this to the file:
> >>>>>>>>>>
> >>>>>>>>>> statistics loopstats
> >>>>>>>>>> statsdir /var/lib/ntp/
> >>>>>>>>>>
> >>>>>>>>>> Then you will see loop data in
that directory.
> >>>>>>>>>> Correlating the data in the
loopstats files with the
> >>>>>>>>>> peaks in skew would be
interesting. You will see
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> entries of the form
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> 54495 76787.701 -0.045153303
-132.569229 0.020806776
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> 239.735511 10
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> Where the second to last column is
the Allan Deviation.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>> When that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>> gets over 1000, ntpd is working
pretty hard. However,
> >>>>>>>>>>
> >>>>>>>>>
> >>> I have not
> >>>
> >>>
> >>>>>>>>>> seen ntpd
> >>>>>>>>>> completely loose it like you have.
> >>>>>>>>>>
> >>>>>>>>>> I''m on vacation until
Monday, and won''t be reading
> >>>>>>>>>> email.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for all your work on this!
> >>>>>>>>>>
> >>>>>>>>>> -Dave
> >>>>>>>>>>
> >>>>>>>>>> Deepak Patel wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> Is the graph for
RHEL5u1-64? (I''ve never tested
> this one.)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I do not know which graph was
attached with this. But
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> I saw this
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> behavior in EL4u5 - 32, EL4U5
- 64 and EL5U1 - 64 hvm
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> guests when I
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> was running ltp tests
continuously.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> What was the behaviour of
the other guests running?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> All pvm guests are fine. But
behavior of most of the
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>> hvm guests were
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>> as described.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> If they had spikes, were
they at the same wall time?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> No. They are not at the same
wall time.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Were the other guests
running ltp as well?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Yes all 6 guests (4 hvm and 2
pvm) the guests are
> running ltp
> >>>>>>>>>>> continuously.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> How are you measuring
skew?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I was collecting output of
"ntpdate -q  <timeserver> every
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 300 seconds
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> (5 minutes) and have created
graph based on that.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Are you running ntpd?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Yes. ntp was running on all
the guests.
> >>>>>>>>>>>
> >>>>>>>>>>> I am investigating what causes
this spikes and
> let everyone
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> know what
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> are my findings.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Deepak
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Anything that you can
discover that would be in sync with
> >>>>>>>>>>>> the spikes would be very
helpful!
> >>>>>>>>>>>>
> >>>>>>>>>>>> The code that I test with
is our product code,
> which is based
> >>>>>>>>>>>> on 3.1. So it is possible
that something in 3.2 other
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>> than vpt.c
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>> is the cause. I can test
with 3.2, if necessary.
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks,
> >>>>>>>>>>>> Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Dan Magenheimer wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Dave (Keir, see
suggestion below) --
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Turning off vhpet
certainly helps a lot (though
> see below).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wonder if
timekeeping with vhpet is so bad that it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> should be
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> turned off by default
(in 3.1, 3.2, and unstable)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> until it is
> >>>
> >>>
> >>>>>>>>>>>>> fixed?  (I have a
patch that defaults it off,
> can post it if
> >>>>>>>>>>>>> there is agreement on
the above point.)  The whole
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> point of an
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> HPET is to provide
more precise timekeeping and
> if vhpet is
> >>>>>>>>>>>>> worse than vpit, it
can only confuse users.  Comments?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In your testing, are
you just measuring % skew
> over a long
> >>>>>>>>>>>>> period of time?
> >>>>>>>>>>>>> We are graphing the
skew continuously and
> >>>>>>>>>>>>> seeing periodic
behavior that is unsettling,
> even with pit.
> >>>>>>>>>>>>> See attached.  Though
your algorithm recovers,
> the "cliffs"
> >>>>>>>>>>>>> could still cause real
user problems.  I wonder
> if there is
> >>>>>>>>>>>>> anything that can be
done to make the "recovery" more
> >>>>>>>>>>>>> responsive?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We are looking into
what part(s) of LTP is causing
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> the cliffs.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> -----Original
Message-----
> >>>>>>>>>>>>>> From: Dave
Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>> Sent: Monday,
January 28, 2008 8:21 AM
> >>>>>>>>>>>>>> To:
dan.magenheimer@oracle.com
> >>>>>>>>>>>>>> Cc: Keir Fraser;
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>
deepak.patel@oracle.com;
> >>>>>>>>>>>>>>
akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>> Subject: Re:
[Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> that disables
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> pending
> >>>>>>>>>>>>>> missed ticks
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dan,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I guess
I''m a bit out of date calling for clock= usage.
> >>>>>>>>>>>>>> Looking at linux
2.6.20.4 sources, I think you
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> should specify
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>
"clocksource=pit nohpet" on the linux guest bootline.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You can leave the
xen and dom0 bootlines as they are.
> >>>>>>>>>>>>>> The xen and guest
clocksources do not need to
> be the same.
> >>>>>>>>>>>>>> In my tests, xen
is using the hpet for its
> timekeeping and
> >>>>>>>>>>>>>> that appears to be
the default.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When you boot the
guests you should see
> >>>>>>>>>>>>>> time.c: Using
PIT/TSC based timekeeping.
> >>>>>>>>>>>>>> on the rh4u5-64
guest, and something similar
> on the others.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> (xm dmesg
shows 8x Xeon 3.2GHz stepping 04,
> Platform timer
> >>>>>>>>>>>>>>> 14.318MHz
HPET.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This appears to be
the xen state, which is fine.
> >>>>>>>>>>>>>> I was wrongly
assuming that this was the guest state.
> >>>>>>>>>>>>>> You might want to
look in your guest logs and see
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>> what they were
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> picking
> >>>>>>>>>>>>>> for a clock
source.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dan Magenheimer
wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, I
hadn''t realized that!  No wonder we didn''t
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> see the same
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> improvement
you saw!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Try
specifying clock=pit on the linux boot line...
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I''m
confused... do you mean "clocksource=pit"
> on the Xen
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> command line or
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
"nohpet" / "clock=pit" / "clocksource=pit" on the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> guest (or
> >>>
> >>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> dom0?) command
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> line?  Or both
places?  Since the tests take
> awhile, it
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> would be nice
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to get this
right the first time.  Do the Xen
> and guest
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> clocksources need
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to be the
same?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -----Original
Message-----
> >>>>>>>>>>>>>>> *From:* Dave
Winchell
> [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>>> *Sent:*
Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>>>>>>> *To:*
dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> deepak.patel@oracle.com;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>
akira.ijuin@oracle.com; Dave Winchell
> >>>>>>>>>>>>>>> *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> that disables
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> pending missed
ticks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Dan,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hpet timer
does have a fairly large error, as I
> >>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> trying this
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> one recently.
> >>>>>>>>>>>>>>> I
don''t remember what I got for error, but 1% sounds
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> about right.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The problem is
that hpet is not built on top of vpt.c,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> the module
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Keir and I did
> >>>>>>>>>>>>>>> all the recent
work in, for its periodic timer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> needs. Try
> >>>
> >>>
> >>>>>>>>>>>>>>> specifying
clock=pit
> >>>>>>>>>>>>>>> on the linux
boot line. If it still picks the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> hpet, which it
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> might, let me
know
> >>>>>>>>>>>>>>> and
I''ll tell you how to get around this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>> Dave
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>
--------------------------------------------------------------
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>> ----------
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *From:* Dan
Magenheimer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:dan.magenheimer@oracle.com]
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> *Sent:* Fri
1/25/2008 6:50 PM
> >>>>>>>>>>>>>>> *To:* Dave
Winchell; Keir Fraser
> >>>>>>>>>>>>>>> *Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> deepak.patel@oracle.com;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>
akira.ijuin@oracle.com
> >>>>>>>>>>>>>>> *Subject:* RE:
[Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> that disables
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> pending missed
ticks
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry for the
very late followup on this but
> we finally
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> were able
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to get our
testing set up again on stable 3.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> bits and have
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> seen some very
bad results on 3.1.3-rc1, on the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> order of 1%.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> Test
enviroment was a 4-socket dual core machine
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> with 24GB of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> memory running
six two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> plus two pv.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> All six guests
were running LTP simultaneously.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> The four hvm
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> guests were:
RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> RHEL4u5-64.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> Timer_mode was
set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 32-bit guests.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> All four hvm
guests experienced skew around -1%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> even the 32-bit
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> guest.  Less
intensive testing didn''t exhibit much
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> skew at all.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> A
representative graph is attached.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Dave, I wonder
if some portion of your patches
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> didn''t end up in
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> the xen trees?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> (xm dmesg
shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> Platform timer
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> 14.318MHz
HPET.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> P.S. Many
thanks to Deepak and Akira for
> running tests.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
-----Original Message-----
> >>>>>>>>>>>>>>> > From:
xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > Dave
Winchell
> >>>>>>>>>>>>>>> > Sent:
Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>>>>>> > To: Keir
Fraser
> >>>>>>>>>>>>>>> > Cc:
dan.magenheimer@oracle.com;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
xen-devel@lists.xensource.com; Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > Winchell
> >>>>>>>>>>>>>>> > Subject:
Re: [Xen-devel] [PATCH] Add a
> timer mode that
> >>>>>>>>>>>>>>> > disables
pending
> >>>>>>>>>>>>>>> > missed
ticks
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Hi Keir,
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > The
latest change, c/s 16690, looks fine.
> >>>>>>>>>>>>>>> > I agree
that the code in c/s 16690 is equivalent to
> >>>>>>>>>>>>>>> > the code
I submitted. Also, your version is more
> >>>>>>>>>>>>>>> > concise.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > The error
tests confirm the equivalence. With
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> overnight cpu
loads,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > the
checked in version was accurate to
> +.048% for sles
> >>>>>>>>>>>>>>> > and
+.038% for red hat. My version was +.046%
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> +.032% in a
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > 2 hour
test.
> >>>>>>>>>>>>>>> > I
don''t think the difference is significant.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > i/o loads
produced errors of +.01%.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Thanks
for all your efforts on this issue.
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Regards,
> >>>>>>>>>>>>>>> > Dave
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> > Keir
Fraser wrote:
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
>Applied as c/s 16690, although the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> checked-in patch is
> >>>
> >>>
> >>>>>>>>>>>>>>> > smaller.
I think the
> >>>>>>>>>>>>>>> > >only
important fix is to pt_intr_post() and the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> only bit of
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > the patch
I
> >>>>>>>>>>>>>>> >
>totally omitted was the change to
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> pt_process_missed_ticks().
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > I
don''t think
> >>>>>>>>>>>>>>> > >that
change can be important, but let''s see what
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> happens to the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>> >
>percentage...
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > > --
Keir
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >On
4/1/08 23:24, "Dave Winchell"
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
<dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> >
>>Hi Dan and Keir,
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> >
>>Attached is a patch that fixes some
> issues with the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> SYNC policy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>(no_missed_ticks_pending).
> >>>>>>>>>>>>>>> > >>I
have not tried to make the change the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> minimal one, but,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > rather,
just
> >>>>>>>>>>>>>>> >
>>ported into
> >>>>>>>>>>>>>>> >
>>the new code what I know to work well.
> The error for
> >>>>>>>>>>>>>>> >
>>no_missed_ticks_pending goes from
> >>>>>>>>>>>>>>> >
>>over 3% to .03% with this change according
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> to my testing.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> >
>>Regards,
> >>>>>>>>>>>>>>> >
>>Dave
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> >
>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> >
>>>Hi Dave --
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>Did you get your correction ported?  If so,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> it would be
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > nice to
see this get
> >>>>>>>>>>>>>>> >
>>>into 3.1.3.
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>Note that I just did some very limited
> testing with
> >>>>>>>>>>>>>>> >
timer_mode=2(=SYNC=no
> >>>>>>>>>>>>>>> >
>>>missed ticks pending)
> >>>>>>>>>>>>>>> >
>>>on tip of xen-3.1-testing (64-bit Linux hv
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> guest) and the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > worst
error I''ve
> >>>>>>>>>>>>>>> >
>>>seen so far
> >>>>>>>>>>>>>>> >
>>>is 0.012%.  But I haven''t tried any exotic
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> loads, just LTP.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>Thanks,
> >>>>>>>>>>>>>>> >
>>>Dan
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>>-----Original Message-----
> >>>>>>>>>>>>>>> >
>>>>From: Dave Winchell
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:dwinchell@virtualiron.com]
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>Sent: Wednesday, December 19, 2007 12:33 PM
> >>>>>>>>>>>>>>> >
>>>>To: dan.magenheimer@oracle.com
> >>>>>>>>>>>>>>> >
>>>>Cc: Keir Fraser; Shan, Haitao;
> >>>>>>>>>>>>>>> >
xen-devel@lists.xensource.com; Dong,
> >>>>>>>>>>>>>>> >
>>>>Eddie; Jiang, Yunhong; Dave Winchell
> >>>>>>>>>>>>>>> >
>>>>Subject: Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> timer mode that
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>disables pending
> >>>>>>>>>>>>>>> >
>>>>missed ticks
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>Dan,
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>I did some testing with the constant tsc offset
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> SYNC method
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>(now called
> >>>>>>>>>>>>>>> >
>>>>no_missed_ticks_pending)
> >>>>>>>>>>>>>>> >
>>>>and found the error to be very high, much larger
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> than 1 %, as
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>I recall.
> >>>>>>>>>>>>>>> >
>>>>I have not had a chance to submit a
> correction. I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> will try to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>do it later
> >>>>>>>>>>>>>>> >
>>>>this week or the first week in January. My
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> version of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> constant tsc
> >>>>>>>>>>>>>>> >
>>>>offset SYNC method
> >>>>>>>>>>>>>>> >
>>>>produces .02 % error, so I just need to port
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> that into the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>current code.
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>The error you got for both of those kernels is
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> what I would
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>> >
>>>>for the default mode, delay_for_missed_ticks.
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>I''ll let Keir answer on how to set the
> time mode.
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>Regards,
> >>>>>>>>>>>>>>> >
>>>>Dave
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>Dan Magenheimer wrote:
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>>Anyone make measurements on the final patch?
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> saw a loss of
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>about 0.2% with no load.  This was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> xen-unstable tip today
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>with no options specified.  32-bit was
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> about 0.01%.
> >>>
> >>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>>I think I missed something... how do I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> run the various
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>accounting choices and which ones are
> known to be
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>for which kernels?
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>>Thanks,
> >>>>>>>>>>>>>>> >
>>>>>Dan
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>-----Original Message-----
> >>>>>>>>>>>>>>> >
>>>>>>From: xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>Keir Fraser
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>>>Sent: Thursday, December 06, 2007 4:57 AM
> >>>>>>>>>>>>>>> >
>>>>>>To: Dave Winchell
> >>>>>>>>>>>>>>> >
>>>>>>Cc: Shan, Haitao;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> xen-devel@lists.xensource.com;
Dong,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > Eddie;
Jiang,
> >>>>>>>>>>>>>>> >
>>>>>>Yunhong
> >>>>>>>>>>>>>>> >
>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> mode that
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>disables pending
> >>>>>>>>>>>>>>> >
>>>>>>missed ticks
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>Please take a look at xen-unstable
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> changeset 16545.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>-- Keir
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>On 26/11/07 20:57, "Dave Winchell"
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Keir,
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The accuracy data I''ve collected for i/o
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> loads for the
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>various time protocols follows. In
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> addition, the data
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>for cpu loads is shown.
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The loads labeled cpu and i/o-8 are on an 8
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> processor AMD
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> box.
> >>>>>>>>>>>>>>> >
>>>>>>>Two guests, red hat and sles 64 bit, 8
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> vcpu each.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The cpu load is usex -e36 on each guest.
> >>>>>>>>>>>>>>> >
>>>>>>>(usex is available at
> >>>>>>>>>>>>>>>
http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>>>> >
>>>>>>>i/o load is 8 instances of dd if=/dev/hda6
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> of=/dev/null.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-32 are 32
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> instances of dd.
> >>>
> >>>
> >>>>>>>>>>>>>>> >
>>>>>>>Also, these are run on 4 cpu AMD box.
> >>>>>>>>>>>>>>> >
>>>>>>>In addition, there is an idle rh-32bit guest.
> >>>>>>>>>>>>>>> >
>>>>>>>All three guests are 8vcpu.
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The loads labeled i/o-4/32 are the same
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> as i/o-32
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>except that the redhat-64 guest has 4
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> instances of dd.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Date Duration Protocol sles, rhat error load
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec,
> >>>>>>>>>>>>>>> +4.42
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec -.006%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > +.005%
cpu
> >>>>>>>>>>>>>>> >
>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.001%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > +.012%
cpu
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.009%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> -.004% cpu
> >>>>>>>>>>>>>>> >
>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.005%, -.005% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.008%, -.003% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.040% cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.034%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > -.031%
cpu
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/14 17 hrs 17 min ASYNC -6.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec,-55.7 sec, -.01%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.09%
i/o-8
> >>>>>>>>>>>>>>> >
>>>>>>>11/15 2 hrs 44 min ASYNC -1.47
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec,-14.0 sec, -.015%
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.14%
i/o-8
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.017%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.022%
i/o-8
> >>>>>>>>>>>>>>> >
>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.017%, -.018%
i/o-8
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.020%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.029%
i/o-8
> >>>>>>>>>>>>>>> >
>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> sec, -.023%,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > -.031%
i/o-8
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/21 28 min MIXED -2.01 sec, -.67
> sec, -.12%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.04% i/o-32
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.011%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > -.005%
i/o-32
> >>>>>>>>>>>>>>> >
>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77
> sec -.10%,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -.11% i/o-32
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>11/26 113 hrs 46 min MIXED -297. sec,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> 13. sec -.07%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > .003%
i/o-4/32
> >>>>>>>>>>>>>>> >
>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> sec, -.017%,
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > .01%
i/o-4/32
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Overhead measurements:
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Progress in terms of number of passes
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> through a fixed
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>system workload
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>on an 8 vcpu red hat with an 8 vcpu
> sles idle.
> >>>>>>>>>>>>>>> >
>>>>>>>The workload was usex -b48.
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>ASYNC 167 min 145 passes .868 passes/min
> >>>>>>>>>>>>>>> >
>>>>>>>SYNC 167 min 144 passes .862 passes/min
> >>>>>>>>>>>>>>> >
>>>>>>>SYNC 1065 min 919 passes .863 passes/min
> >>>>>>>>>>>>>>> >
>>>>>>>MIXED 221 min 196 passes .887 passes/min
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Conclusions:
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>The only protocol which meets the
> .05% accuracy
> >>>>>>>>>>>>>>> >
requirement for ntp
> >>>>>>>>>>>>>>> >
>>>>>>>tracking under the loads
> >>>>>>>>>>>>>>> >
>>>>>>>above is the SYNC protocol. The worst case
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> accuracies for
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>SYNC, MIXED,
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>and ASYNC
> >>>>>>>>>>>>>>> >
>>>>>>>are .022%, .12%, and .14%, respectively.
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>We could reduce the cost of the SYNC
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> method by only
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>scheduling the extra
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>wakeups if a certain number
> >>>>>>>>>>>>>>> >
>>>>>>>of ticks are missed.
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Regards,
> >>>>>>>>>>>>>>> >
>>>>>>>Dave
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>Keir Fraser wrote:
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>On 9/11/07 19:22, "Dave Winchell"
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>><dwinchell@virtualiron.com> wrote:
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>Since I had a high error (~.03%) for the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ASYNC method a
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>couple of days ago,
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>I ran another ASYNC test. I think
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> there may have
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > been
something
> >>>>>>>>>>>>>>> >
>>>>>>>>>wrong with the code I used a couple of
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> days ago for
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>ASYNC. It may have been
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>missing the immediate delivery of interrupt
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> after context
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>switch in.
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>My results indicate that either SYNC
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> or ASYNC give
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>acceptable accuracy,
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>each running consistently around or under
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> .01%. MIXED has
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>a fairly high
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>error of
> >>>>>>>>>>>>>>> >
>>>>>>>>>greater than .03%. Probably too close
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> to .05% ntp
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>threshold for comfort.
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>I don''t have an overnight run with
SYNC. I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> plan to leave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>SYNC running
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>over the weekend. If you''d rather I
can
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> leave MIXED
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>running instead.
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>It may be too early to pick the
> protocol and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I can run
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>more overnight tests
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>next week.
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>I''m a bit worried about any unwanted
side
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> effects of the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>SYNC+run_timer
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>approach -- e.g., whether timer wakeups will
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> cause higher
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>system-wide CPU
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>contention. I find it easier to think
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> through the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>implications of ASYNC. I''m
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>surprised that MIXED loses time, and is less
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> accurate than
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>ASYNC. Perhaps it
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>delivers more timer interrupts than
> the other
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> approaches,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>and each interrupt
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>event causes a small accumulated error?
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>Overall I would consider MIXED and ASYNC as
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> favourites and
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>if the latter is
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>actually more accurate then I can
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> simply revert the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > changeset
that
> >>>>>>>>>>>>>>> >
>>>>>>>>implemented MIXED.
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>Perhaps rather than running more of the same
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> workloads you
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>could try idle
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> large disc reads
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>to /dev/null)? We
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>don''t have any data on workloads that
aren''t
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CPU bound, so
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>that''s really an
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>obvious place to put any further effort imo.
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>-- Keir
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>>>
> >>>>>>>>>>>>>>> > 
> >>>>>>_______________________________________________
> >>>>>>>>>>>>>>> >
>>>>>>Xen-devel mailing list
> >>>>>>>>>>>>>>> >
>>>>>>Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>> >
>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>>
> >>>>>>>>>>>>>>> >
>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
> >>>>>>>>>>>>>>> >
>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> 2007 +0000
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> 2008 -0500
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> >
>>@@ -58,7 +58,7 @@ static void
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> pt_process_missed_ticks(stru
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >> 
missed_ticks = missed_ticks / (s_time_t)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> pt->period + 1;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >> 
if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
no_missed_ticks_pending) )
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>-
pt->do_not_freeze = !pt->pending_intr_nr;
> >>>>>>>>>>>>>>> > >>+
pt->do_not_freeze = 1;
> >>>>>>>>>>>>>>> > >> 
else
> >>>>>>>>>>>>>>> > >> 
pt->pending_intr_nr += missed_ticks;
> >>>>>>>>>>>>>>> > >> 
pt->scheduled += missed_ticks * pt->period;
> >>>>>>>>>>>>>>> >
>>@@ -127,7 +127,12 @@ static void
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> pt_timer_fn(void *data)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >> 
pt_lock(pt);
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>-
pt->pending_intr_nr++;
> >>>>>>>>>>>>>>> > >>+
if ( mode_is(pt->vcpu->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
no_missed_ticks_pending) ) {
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >>+
pt->pending_intr_nr = 1;
> >>>>>>>>>>>>>>> > >>+
pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>> > >>+
}
> >>>>>>>>>>>>>>> > >>+
else
> >>>>>>>>>>>>>>> > >>+
pt->pending_intr_nr++;
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >> 
if ( !pt->one_shot )
> >>>>>>>>>>>>>>> > >> 
{
> >>>>>>>>>>>>>>> >
>>@@ -221,8 +226,6 @@ void pt_intr_post(struct
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> vcpu *v, struct
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >> 
return;
> >>>>>>>>>>>>>>> > >> 
}
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>-
pt->do_not_freeze = 0;
> >>>>>>>>>>>>>>> > >>-
> >>>>>>>>>>>>>>> > >> 
if ( pt->one_shot )
> >>>>>>>>>>>>>>> > >> 
{
> >>>>>>>>>>>>>>> > >> 
pt->enabled = 0;
> >>>>>>>>>>>>>>> >
>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *v, struct
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > >> 
pt->last_plt_gtime >
>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> hvm_get_guest_time(v);
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > >> 
pt->pending_intr_nr = 0; /*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> ''collapse'' all
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>> > missed
ticks */
> >>>>>>>>>>>>>>> > >> 
}
> >>>>>>>>>>>>>>> > >>+
else if ( mode_is(v->domain,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>> no_missed_ticks_pending) ) {
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>> > >>+
pt->pending_intr_nr--;
> >>>>>>>>>>>>>>> > >>+
pt->last_plt_gtime = hvm_get_guest_time(v);
> >>>>>>>>>>>>>>> > >>+
}
> >>>>>>>>>>>>>>> > >> 
else
> >>>>>>>>>>>>>>> > >> 
{
> >>>>>>>>>>>>>>> > >> 
pt->last_plt_gtime +>
>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>> pt->period_cycles;
> >>>
> >>>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >>
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> > >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>> >
_______________________________________________
> >>>>>>>>>>>>>>> > Xen-devel
mailing list
> >>>>>>>>>>>>>>> >
Xen-devel@lists.xensource.com
> >>>>>>>>>>>>>>> >
http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
--------------------------------------------------------------
> >>>>> ----------
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/2/08 17:55, "Dan Magenheimer" <dan.magenheimer@oracle.com>
wrote:
> Note that if hpet=0 and pmtimer=0 were the default hvm platform
> parameters for all xen hvm guests (on all versions of xen),
> specifying kernel command line parameters would be unnecessary,
> but c''est la vie.
I will get round to pmtimer at some point. I haven''t put the hpet patch
into
3.2 or 3.1 because it did not trivially backport. Someone else will have to
do that work if they want it (including 3.2, if you want 3.1!).

 -- Keir
 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

Thanks for all the investigation you''ve done!

-Dave


Dan Magenheimer wrote:
>Hi Dave --
>
>Thanks for that observation on ltp running on one vcpu!
>
>With "clocksource=pit nohpet nopmtimer" our clock skew
>problems seem to have been reduced to a reasonable
>percentage.  However, our 32-bit clock skew seems to
>show a measureable problem now.
>
For the 32 bit guest, which timesource did it pick?
>  As a result,
>I''ve been doing some digging into kernel sources and
>have observed the following relative to RHEL4 (2.6.9-based)
>kernels and RHEL5 (2.6.18-based) kernels and thought I
>would document them for posterity.  Some of
>our confusion arises from the fact that invalid command
>line parameters are silently ignored.
>
>RHEL4:
>- clock= is a valid parameter for RHEL4-32
>- clocksource= is not a valid parameter for RHEL4-xx
>- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
>- nopmtimer is not a valid parameter for RHEL4-xx
>- notsc is a valid parameter for RHEL4-32, not RHEL4-64
>- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently
>
>For Xen RHEL4 HVM guests:
>- I *think* clock=pit is sufficient for RHEL4-32 [1]
>- I *think* nohpet is sufficient for RHEL4-64 [1]
>
>RHEL5:
>- there are two kinds of timekeeping, WALL and gtod
>- clocksource= is a valid parameter for RHEL5-xx
>- clock= is a valid but deprecated parameter for RHEL5-xx
>- clock= and clocksource= are essentially equivalent
>- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
>- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
>- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
>- clock=pit changes the gtod source but not the WALL source[2]
>- nohpet nopmtimer changes the WALL source to PIT
>- /sys/devices/system/clocksource/clocksource0/...
>  available_clocksource lists the possible clock sources
>  current_clocksource lists the chosen clock source
>  ..but neither of these works in a RHEL5 guest!
>
>For Xen RHEL5 HVM guests:
>- I *think* clock=pit is sufficient for RHEL5-32
>  
>
But still poor accuracy, right?
>- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64
>
>Other info:
>- As of 2.6.24.2, clock= is still valid (though still deprecated)
>
>So, some open questions:
>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>    (I *think* not as it has never come up in any email.)
>  
>
I have not investigated this yet.
>[2] In RHEL5, I *think* it is the WALL source that we care about?
>  
>
I''ll have to check on this too.
>And finally, since invalid command line parameters are ignored.
>I think specifying:
>	clock=pit nohpet nopmtimer
>will force the guest clock sources into the optimal state for
>all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
>question above on tsc).  And we should keep an eye on
>kernel/time/clocksource.c to ensure the __setup("clock="...)
>line doesn''t go away before RHEL6.
>
>Note that if hpet=0 and pmtimer=0 were the default hvm platform
>parameters for all xen hvm guests (on all versions of xen),
>specifying kernel command line parameters would be unnecessary,
>but c''est la vie.
>
>Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
>and timer_mode=2 for 64-bit RHEL guests.
>
>Thanks,
>Dan
>
>  
>
>>-----Original Message-----
>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>Sent: Tuesday, February 19, 2008 8:27 AM
>>To: dan.magenheimer@oracle.com
>>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak
>>Patel
>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>disables pending
>>missed ticks
>>
>>
>>
>>Hi Dan,
>>
>>ltp runs by default loading up only one vcpu.
>>The -x option can be used to run multiple instances, though
>>in this mode you will get test failures.
>>I ran 8 instances on each guest for 16 hours, 25 min
>>and the time error was -11 sec (-.019%) on each guest.
>>
>>Regards,
>>Dave
>>
>>Dave Winchell wrote:
>>
>>    
>>
>>>Hi Dan,
>>>
>>>Mine was oversubscribed.
>>>8 physical cpu, 2 guests, each with 8 vcpu.
>>>I ran one instance of ltp on each guest, continuously. I hope ltp
>>>loaded up all the vcpus. I seem to recall that it did, but I
>>>could be wrong. If it didn''t, that would be a major
difference
>>>between our tests. I''ll verify this afternoon and run 
>>>      
>>>
>>multiple instances,
>>    
>>
>>>if necessary.
>>>
>>>Thanks,
>>>Dave
>>>
>>>
>>>
>>>Dan Magenheimer wrote:
>>>
>>>      
>>>
>>>>Hi Dave --
>>>>
>>>>No new results yet but one other question:
>>>>
>>>>The problems we''ve seen with our testing have been with
a heavily
>>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests
>>>>running LTP simultaneously.
>>>>
>>>>Was your LTP testing oversubscribed or just a single guest?
>>>>
>>>>Thanks,
>>>>Dan
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>-----Original Message-----
>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
>>>>>Sent: Thursday, February 14, 2008 10:56 AM
>>>>>To: dan.magenheimer@oracle.com
>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak
Patel; Dave
>>>>>Winchell
>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
>>>>>          
>>>>>
>>disables pending
>>    
>>
>>>>>missed ticks
>>>>>
>>>>>
>>>>>Dan,
>>>>>
>>>>>Here are some boot snipets for rh4u564 on xen 3.2.
>>>>>
>>>>>
>>>>>#1:
>>>>>
>>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is
ro
>>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
>>>>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
>>>>>          
>>>>>
>>3.4.6 20060404
>>    
>>
>>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>>>>...
>>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro
root=LABEL=/
>>>>>console=ttyS0 clocksource=pit nohpet
>>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
>>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: 
>>>>>          
>>>>>
>>2048 (order: 11,
>>    
>>
>>>>>65536 bytes)
>>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM
timer.
>>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 
>>>>>          
>>>>>
>>MHz processor.
>>    
>>
>>>>>...
>>>>>Feb 14 10:45:00 vs076 kernel: checking TSC 
>>>>>          
>>>>>
>>synchronization across 8
>>    
>>
>>>>>CPUs: passed.
>>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
>>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to 
>>>>>          
>>>>>
>>use of PM timer
>>    
>>
>>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based
timekeeping.
>>>>>
>>>>>
>>>>>
>>>>>#2:
>>>>>
>>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is
ro
>>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer)
>>>>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp
>>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 
>>>>>          
>>>>>
>>3.4.6 20060404
>>    
>>
>>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
>>>>>...
>>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro
root=LABEL=/
>>>>>console=ttyS0 clocksource=pit nohpet nopmtimer
>>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
>>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: 
>>>>>          
>>>>>
>>2048 (order: 11,
>>    
>>
>>>>>65536 bytes)
>>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz 
>>>>>          
>>>>>
>>PIT timer.
>>    
>>
>>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 
>>>>>          
>>>>>
>>MHz processor.
>>    
>>
>>>>>...
>>>>>Feb 14 10:47:59 vs076 kernel: checking TSC 
>>>>>          
>>>>>
>>synchronization across 8
>>    
>>
>>>>>CPUs: passed.
>>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
>>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based 
>>>>>          
>>>>>
>>timekeeping.
>>    
>>
>>>>>As you can see, I only get the pit if I specify nopmtimer.
>>>>>
>>>>>Dan Magenheimer wrote:
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Hi Dave --
>>>>>>
>>>>>>Thanks for continuing to run tests!
>>>>>>
>>>>>>Hmmm... I thought I had noticed that even though Linux
will
>>>>>>            
>>>>>>
>>>>>acknowledge
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>the existence of the pmtimer, it still prints:
>>>>>>
>>>>>>time.c: Using PIT/TSC based timekeeping.
>>>>>>
>>>>>>I will check again, but assuming the clocksource for our
tests is
>>>>>>indeed pit, the huge difference in the results (yours vs
ours) is
>>>>>>baffling. I wonder if the difference may be the 
>>>>>>            
>>>>>>
>>underlying hardware.
>>    
>>
>>>>>>Maybe we will try to ensure we can duplicate the results
on
>>>>>>            
>>>>>>
>>>>>a different
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>box.
>>>>>>
>>>>>>
>>>>>>So your testing was with stock 3.2.0 xen bits (what 
>>>>>>            
>>>>>>
>>cset?) without
>>    
>>
>>>>>>any of your [quote from below] "clock related
tweaks
>>>>>>            
>>>>>>
>>that I haven''t
>>    
>>
>>>>>>submitted, because I''m still characterizing
them"?
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>None of the tweaks I mentioned are in this test.
>>>>>It was stock with some patches. However, none of the 
>>>>>          
>>>>>
>>patches are time
>>    
>>
>>>>>related to
>>>>>my knowledge and I checked vpt.c to make sure that it is 
>>>>>          
>>>>>
>>the same as
>>    
>>
>>>>>what''s in unstable.
>>>>>The only difference is in pt_intr_post, where I set the 
>>>>>          
>>>>>
>>timer mode.
>>    
>>
>>>>>I don''t have timer mode tied into our config
process yet, which
>>>>>is different than official xen method.
>>>>>
>>>>>
>>>>>(In pt_intr_post)
>>>>>     else
>>>>>     {
>>>>>+       if(v->arch.paging.mode->guest_levels == 4)
>>>>>+           
>>>>>          
>>>>>
>>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >>
>>
>>>>>HVMPTM_no_missed_ticks_pending;
>>>>>+       else
>>>>>+           
>>>>>          
>>>>>
>>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >>
>>
>>>>>HVMPTM_delay_for_missed_ticks;
>>>>>         if ( mode_is(v->domain, one_missed_tick_pending)
||
>>>>>              mode_is(v->domain, no_missed_ticks_pending)
)
>>>>>         {
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Could you also send detail on the rhel4u4-64 kernel you
>>>>>>are testing with, just to ensure we are not comparing
apples
>>>>>>and oranges?  (Perhaps there''s some way we can
even share the
>>>>>>identical disk image and vm.cfg file?)
>>>>>>
>>>>>>And if our problem is indeed the pmtimer, I will need to
submit
>>>>>>another patch to Keir to add an hvm pmtimer platform
variable.
>>>>>>(Hmmm... I don''t think he''s even
accepted the hpet variable patch
>>>>>>yet.  I''ll have to check.)
>>>>>>
>>>>>>Thanks,
>>>>>>Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>Sent: Thursday, February 14, 2008 9:00 AM
>>>>>>>To: dan.magenheimer@oracle.com
>>>>>>>Cc: Dave Winchell; Keir Fraser;
>>>>>>>              
>>>>>>>
>>>>>xen-devel@lists.xensource.com; Deepak
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>Patel
>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode
that
>>>>>>>disables pending
>>>>>>>missed ticks
>>>>>>>
>>>>>>>
>>>>>>>Hi Dan,
>>>>>>>
>>>>>>>I ran the ltp tests with 3.2 and found the errors
>>>>>>>for a 16 hour run to be:
>>>>>>>
>>>>>>>rh4u564 -9.9 sec (-.017%)
>>>>>>>rh4u464 -7.3 sec (-.013%)
>>>>>>>
>>>>>>>There were no cliffs and the drift was linear.
>>>>>>>
>>>>>>>I think the problem you had may be due to the use of
the
>>>>>>>pm timer. If you still have the boot log, it would
tell you.
>>>>>>>
>>>>>>>When I first tried a guest on 3.2 with
"clocksource=pit nohpet"
>>>>>>>I noticed that it picked the pm timer. Adding
"nopmtimer", the
>>>>>>>guest will pick the pit.
>>>>>>>
>>>>>>>The reason I didn''t have the problem with
our 3.1 base is that
>>>>>>>I had disabled the hpet and the pmtimer by not
advertising them
>>>>>>>in the acpi tables. I did this so long ago, I forgot
>>>>>>>              
>>>>>>>
>>that I had to
>>    
>>
>>>>>>>disable pmtimer as well as hpet.
>>>>>>>
>>>>>>>So, can you re-run your test with
"clocksource=pit nohpet
>>>>>>>              
>>>>>>>
>>>>>nopmtimer"?
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>You should see this in the boot messages:
>>>>>>>
>>>>>>>time.c: Using PIT/TSC based timekeeping.
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Dave
>>>>>>>
>>>>>>>
>>>>>>>Dave Winchell wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>Hi Dan,
>>>>>>>>
>>>>>>>>Over the weekend the drift was +18 seconds for
each
>>>>>>>>                
>>>>>>>>
>>guest (no ntp).
>>    
>>
>>>>>>>>The duration was 3900 minutes, so the error for
each
>>>>>>>>                
>>>>>>>>
>>was +.0077%.
>>    
>>
>>>>>>>>Looking back through the data, it appears to
drift linearly at
>>>>>>>>this rate. I''ve attached a plot for
rh4u5-64.
>>>>>>>>
>>>>>>>>This accuracy is better than what I''ve
seen before (.03-.05%).
>>>>>>>>This may be due to the different load (ltp vs
usex) or to
>>>>>>>>                
>>>>>>>>
>>>>>one of the
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>changes I''ve made recently.
I''ll do some
>>>>>>>>                
>>>>>>>>
>>experimentation to see if
>>    
>>
>>>>>>>>there is
>>>>>>>>a fix I should propose.
>>>>>>>>
>>>>>>>>This still doesn''t address the radical
drift you saw.
>>>>>>>>The next step for me is to run 3.2 and see if I
can
>>>>>>>>                
>>>>>>>>
>>reproduce it.
>>    
>>
>>>>>>>>Regards,
>>>>>>>>Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Dave Winchell wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>Hi Dan,
>>>>>>>>>
>>>>>>>>>Sorry it took me so long, but I finally ran
an ltp test today.
>>>>>>>>>Its on rh4u4-64. I''m using the
defaults for ltp and
>>>>>>>>>                  
>>>>>>>>>
>>using a script
>>    
>>
>>>>>>>>>called runltp. I had a usex load on
rh4u5-64. No ntpd.
>>>>>>>>>virtual processors / physical processors =
2.
>>>>>>>>>
>>>>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec
(4u4) in
>>>>>>>>>                  
>>>>>>>>>
>>300 minutes
>>    
>>
>>>>>>>>>for -.005% and .008%.
>>>>>>>>>
>>>>>>>>>I''m running a 3.1 based hypervisor
with some clock related
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>tweaks that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>I haven''t submitted, because
I''m still characterizing them.
>>>>>>>>>
>>>>>>>>>I''m stopping the usex load on
4u5-64 now and
>>>>>>>>>                  
>>>>>>>>>
>>replacing it with ltp
>>    
>>
>>>>>>>>>and will leave the two guests running ltp
over the weekend.
>>>>>>>>>
>>>>>>>>>Regards,
>>>>>>>>>Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Dave Winchell wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>Hi Dan, Deepak:
>>>>>>>>>>
>>>>>>>>>>Thanks for the data. Those drifts are
severe - no wonder
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>ntp couldn''t
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>keep then in synch. I''ll try to
reproduce that behaviour
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>here, with
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>my code base.
>>>>>>>>>>If I can''t reproduce it,
I''ll try 3.2.
>>>>>>>>>>
>>>>>>>>>>If you can isolate what ltp is doing
during the cliffs,
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>that would
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>be very
>>>>>>>>>>helpful.
>>>>>>>>>>
>>>>>>>>>>thanks,
>>>>>>>>>>Dave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>>>OK, Deepak repeated the test without
ntpd and using
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>ntpdate -b before
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>the test.
>>>>>>>>>>>
>>>>>>>>>>>The attached graph shows his
results: el5u1-64
>>>>>>>>>>>                      
>>>>>>>>>>>
>>(best=~0.07%),
>>    
>>
>>>>>>>>>>>el4u5-64 (middle=~0.2%), and
el4u5-32 (worst=~0.3%).
>>>>>>>>>>>
>>>>>>>>>>>We will continue to look at LTP to
try to isolate.
>>>>>>>>>>>
>>>>>>>>>>>Thanks,
>>>>>>>>>>>Dan
>>>>>>>>>>>
>>>>>>>>>>>P.S. elXuY is essentially RHEL XuY
with some patches.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>
>>>>>>>>>>>>-----Original Message-----
>>>>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>Sent: Wednesday, January 30,
2008 2:45 PM
>>>>>>>>>>>>To: Deepak Patel
>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com;
Keir Fraser;
>>>>>>>>>>>>xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>Dave Winchell
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH]
Add a timer mode
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>that disables
>>    
>>
>>>>>>>>>>>>pending
>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>Dan, Deeepak,
>>>>>>>>>>>>
>>>>>>>>>>>>It may be that the underlying
clock error is too
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>great for ntp
>>    
>>
>>>>>>>>>>>>to handle. It would be useful if
you did not run ntpd
>>>>>>>>>>>>and, instead did ntpdate -b
<timeserver> at the start
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>of the test
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>for each guest. Then capture the
data as you have
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>been doing.
>>    
>>
>>>>>>>>>>>>If the drift is greater than
.05%, then we need to
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>address that.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>Another option is, when running
ntpd, to enable loop
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>statistics in
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>/etc/ntp.conf
>>>>>>>>>>>>by adding this to the file:
>>>>>>>>>>>>
>>>>>>>>>>>>statistics loopstats
>>>>>>>>>>>>statsdir /var/lib/ntp/
>>>>>>>>>>>>
>>>>>>>>>>>>Then you will see loop data in
that directory.
>>>>>>>>>>>>Correlating the data in the
loopstats files with the
>>>>>>>>>>>>peaks in skew would be
interesting. You will see
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>entries of the form
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>54495 76787.701 -0.045153303
-132.569229 0.020806776
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>239.735511 10
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>Where the second to last column
is the Allan Deviation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>When that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>gets over 1000, ntpd is working
pretty hard. However,
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>I have not
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>seen ntpd
>>>>>>>>>>>>completely loose it like you
have.
>>>>>>>>>>>>
>>>>>>>>>>>>I''m on vacation until
Monday, and won''t be reading
>>>>>>>>>>>>email.
>>>>>>>>>>>>
>>>>>>>>>>>>Thanks for all your work on
this!
>>>>>>>>>>>>
>>>>>>>>>>>>-Dave
>>>>>>>>>>>>
>>>>>>>>>>>>Deepak Patel wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>Is the graph for
RHEL5u1-64? (I''ve never tested
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>this one.)
>>    
>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I do not know which graph
was attached with this. But
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>I saw this
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>behavior in EL4u5 - 32,
EL4U5 - 64 and EL5U1 - 64 hvm
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>guests when I
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>was running ltp tests
continuously.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>What was the behaviour
of the other guests running?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>All pvm guests are fine. But
behavior of most of the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>hvm guests were
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>as described.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>If they had spikes, were
they at the same wall time?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>No. They are not at the same
wall time.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Were the other guests
running ltp as well?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>Yes all 6 guests (4 hvm and
2 pvm) the guests are
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>running ltp
>>    
>>
>>>>>>>>>>>>>continuously.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>How are you measuring
skew?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>I was collecting output of
"ntpdate -q  <timeserver> every
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>300 seconds
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>(5 minutes) and have created
graph based on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Are you running ntpd?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>Yes. ntp was running on all
the guests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>I am investigating what
causes this spikes and
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>let everyone
>>    
>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>know what
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>are my findings.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>Deepak
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                          
>>>>>>>>>>>>>
>>>>>>>>>>>>>>Anything that you can
discover that would be in sync with
>>>>>>>>>>>>>>the spikes would be very
helpful!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>The code that I test
with is our product code,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>which is based
>>    
>>
>>>>>>>>>>>>>>on 3.1. So it is
possible that something in 3.2 other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>than vpt.c
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>is the cause. I can test
with 3.2, if necessary.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>thanks,
>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Dan Magenheimer wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Hi Dave (Keir, see
suggestion below) --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Turning off vhpet
certainly helps a lot (though
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>see below).
>>    
>>
>>>>>>>>>>>>>>>I wonder if
timekeeping with vhpet is so bad that it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>should be
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>turned off by
default (in 3.1, 3.2, and unstable)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>until it is
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>fixed?  (I have a
patch that defaults it off,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>can post it if
>>    
>>
>>>>>>>>>>>>>>>there is agreement
on the above point.)  The whole
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>point of an
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>HPET is to provide
more precise timekeeping and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>if vhpet is
>>    
>>
>>>>>>>>>>>>>>>worse than vpit, it
can only confuse users.  Comments?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>In your testing, are
you just measuring % skew
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>over a long
>>    
>>
>>>>>>>>>>>>>>>period of time?
>>>>>>>>>>>>>>>We are graphing the
skew continuously and
>>>>>>>>>>>>>>>seeing periodic
behavior that is unsettling,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>even with pit.
>>    
>>
>>>>>>>>>>>>>>>See attached. 
Though your algorithm recovers,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>the "cliffs"
>>    
>>
>>>>>>>>>>>>>>>could still cause
real user problems.  I wonder
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>if there is
>>    
>>
>>>>>>>>>>>>>>>anything that can be
done to make the "recovery" more
>>>>>>>>>>>>>>>responsive?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>We are looking into
what part(s) of LTP is causing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>the cliffs.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>>>From: Dave
Winchell [mailto:dwinchell@virtualiron.com]
>>>>>>>>>>>>>>>>Sent: Monday,
January 28, 2008 8:21 AM
>>>>>>>>>>>>>>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>>>>>>>>Cc: Keir Fraser;
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>deepak.patel@oracle.com;
>>>>>>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
>>>>>>>>>>>>>>>>Subject: Re:
[Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>that disables
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>pending
>>>>>>>>>>>>>>>>missed ticks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I guess
I''m a bit out of date calling for clock= usage.
>>>>>>>>>>>>>>>>Looking at linux
2.6.20.4 sources, I think you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>should specify
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>"clocksource=pit
nohpet" on the linux guest bootline.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>You can leave
the xen and dom0 bootlines as they are.
>>>>>>>>>>>>>>>>The xen and
guest clocksources do not need to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>be the same.
>>    
>>
>>>>>>>>>>>>>>>>In my tests, xen
is using the hpet for its
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>timekeeping and
>>    
>>
>>>>>>>>>>>>>>>>that appears to
be the default.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>When you boot
the guests you should see
>>>>>>>>>>>>>>>>time.c: Using
PIT/TSC based timekeeping.
>>>>>>>>>>>>>>>>on the rh4u5-64
guest, and something similar
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>on the others.
>>    
>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>(xm dmesg
shows 8x Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>Platform timer
>>    
>>
>>>>>>>>>>>>>>>>>14.318MHz
HPET.)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>This appears to
be the xen state, which is fine.
>>>>>>>>>>>>>>>>I was wrongly
assuming that this was the guest state.
>>>>>>>>>>>>>>>>You might want
to look in your guest logs and see
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>what they were
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>picking
>>>>>>>>>>>>>>>>for a clock
source.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Dan Magenheimer
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks, I
hadn''t realized that!  No wonder we didn''t
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>see the same
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>improvement
you saw!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Try
specifying clock=pit on the linux boot line...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>I''m
confused... do you mean "clocksource=pit"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>on the Xen
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>command line or
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>"nohpet"
/ "clock=pit" / "clocksource=pit" on the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>guest (or
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>dom0?) command
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>line?  Or
both places?  Since the tests take
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>awhile, it
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>would be nice
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to get this
right the first time.  Do the Xen
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>and guest
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>clocksources
need
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to be the
same?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>>>>*From:* Dave
Winchell
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>[mailto:dwinchell@virtualiron.com]
>>    
>>
>>>>>>>>>>>>>>>>>*Sent:*
Sunday, January 27, 2008 2:22 PM
>>>>>>>>>>>>>>>>>*To:*
dan.magenheimer@oracle.com; Keir Fraser
>>>>>>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>deepak.patel@oracle.com;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
>>>>>>>>>>>>>>>>>*Subject:*
RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>that disables
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>pending
missed ticks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hi Dan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Hpet timer
does have a fairly large error, as I
>>>>>>>>>>>>>>>>>was
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>trying this
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>one
recently.
>>>>>>>>>>>>>>>>>I
don''t remember what I got for error, but 1% sounds
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>about right.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>The problem
is that hpet is not built on top of vpt.c,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>the module
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Keir and I
did
>>>>>>>>>>>>>>>>>all the
recent work in, for its periodic timer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>needs. Try
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>specifying
clock=pit
>>>>>>>>>>>>>>>>>on the linux
boot line. If it still picks the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>hpet, which it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>might, let
me know
>>>>>>>>>>>>>>>>>and
I''ll tell you how to get around this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>--------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>----------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>*From:* Dan
Magenheimer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:dan.magenheimer@oracle.com]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>*Sent:* Fri
1/25/2008 6:50 PM
>>>>>>>>>>>>>>>>>*To:* Dave
Winchell; Keir Fraser
>>>>>>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>deepak.patel@oracle.com;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com
>>>>>>>>>>>>>>>>>*Subject:*
RE: [Xen-devel] [PATCH] Add a timer mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>that disables
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>pending
missed ticks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Sorry for
the very late followup on this but
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>we finally
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>were able
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>to get our
testing set up again on stable 3.1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>bits and have
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>seen some
very bad results on 3.1.3-rc1, on the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>order of 1%.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>Test
enviroment was a 4-socket dual core machine
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>with 24GB of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>memory
running six two-vcpu 2GB domains, four hvm
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>plus two pv.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>All six
guests were running LTP simultaneously.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>The four hvm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>guests were:
RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>RHEL4u5-64.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Timer_mode
was set to 2 for 64-bit guests and 0 for
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>32-bit guests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>All four hvm
guests experienced skew around -1%,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>even the 32-bit
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>guest.  Less
intensive testing didn''t exhibit much
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>skew at all.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>A
representative graph is attached.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Dave, I
wonder if some portion of your patches
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>didn''t end up in
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>the xen
trees?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>(xm dmesg
shows 8x Xeon 3.2GHz stepping 04,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>Platform timer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>14.318MHz
HPET.)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>P.S. Many
thanks to Deepak and Akira for
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>running tests.
>>    
>>
>>>>>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>Dave
Winchell
>>>>>>>>>>>>>>>>>>Sent:
Wednesday, January 09, 2008 9:53 AM
>>>>>>>>>>>>>>>>>>To: Keir
Fraser
>>>>>>>>>>>>>>>>>>Cc:
dan.magenheimer@oracle.com;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Winchell
>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>timer mode that
>>    
>>
>>>>>>>>>>>>>>>>>>disables
pending
>>>>>>>>>>>>>>>>>>missed
ticks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Hi Keir,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>The
latest change, c/s 16690, looks fine.
>>>>>>>>>>>>>>>>>>I agree
that the code in c/s 16690 is equivalent to
>>>>>>>>>>>>>>>>>>the code
I submitted. Also, your version is more
>>>>>>>>>>>>>>>>>>concise.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>The
error tests confirm the equivalence. With
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>overnight cpu
loads,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>the
checked in version was accurate to
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>+.048% for sles
>>    
>>
>>>>>>>>>>>>>>>>>>and
+.038% for red hat. My version was +.046%
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>+.032% in a
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>2 hour
test.
>>>>>>>>>>>>>>>>>>I
don''t think the difference is significant.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>i/o
loads produced errors of +.01%.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Thanks
for all your efforts on this issue.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Keir
Fraser wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>Applied
as c/s 16690, although the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>checked-in patch is
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>smaller.
I think the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>only
important fix is to pt_intr_post() and the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>only bit of
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>the
patch I
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>totally
omitted was the change to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>pt_process_missed_ticks().
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>I
don''t think
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>that
change can be important, but let''s see what
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>happens to the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>error
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>percentage...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>--
Keir
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>On
4/1/08 23:24, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Hi
Dan and Keir,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Attached
is a patch that fixes some
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>issues with the
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>SYNC policy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending).
>>>>>>>>>>>>>>>>>>>>I
have not tried to make the change the
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>minimal one, but,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>rather,
just
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>ported
into
>>>>>>>>>>>>>>>>>>>>the
new code what I know to work well.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>The error for
>>    
>>
>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending
goes from
>>>>>>>>>>>>>>>>>>>>over
3% to .03% with this change according
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>to my testing.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>Dan
Magenheimer wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Hi
Dave --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Did
you get your correction ported?  If so,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>it would be
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>nice to
see this get
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>into
3.1.3.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Note
that I just did some very limited
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>testing with
>>    
>>
>>>>>>>>>>>>>>>>>>timer_mode=2(=SYNC=no
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>missed
ticks pending)
>>>>>>>>>>>>>>>>>>>>>on
tip of xen-3.1-testing (64-bit Linux hv
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>guest) and the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>worst
error I''ve
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>seen
so far
>>>>>>>>>>>>>>>>>>>>>is
0.012%.  But I haven''t tried any exotic
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>loads, just LTP.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>>>>>>>>>From:
Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:dwinchell@virtualiron.com]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Sent:
Wednesday, December 19, 2007 12:33 PM
>>>>>>>>>>>>>>>>>>>>>>To:
dan.magenheimer@oracle.com
>>>>>>>>>>>>>>>>>>>>>>Cc:
Keir Fraser; Shan, Haitao;
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dong,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Eddie;
Jiang, Yunhong; Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>timer mode that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>disables
pending
>>>>>>>>>>>>>>>>>>>>>>missed
ticks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Dan,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I
did some testing with the constant tsc offset
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>SYNC method
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>(now
called
>>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending)
>>>>>>>>>>>>>>>>>>>>>>and
found the error to be very high, much larger
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>than 1 %, as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I
recall.
>>>>>>>>>>>>>>>>>>>>>>I
have not had a chance to submit a
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>correction. I
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>will try to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>do
it later
>>>>>>>>>>>>>>>>>>>>>>this
week or the first week in January. My
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>version of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>constant tsc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>offset
SYNC method
>>>>>>>>>>>>>>>>>>>>>>produces
.02 % error, so I just need to port
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>that into the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>current
code.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>The
error you got for both of those kernels is
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>what I would
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>expect
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>for
the default mode, delay_for_missed_ticks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>I''ll
let Keir answer on how to set the
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>time mode.
>>    
>>
>>>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Dan
Magenheimer wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>Anyone
make measurements on the final patch?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>I
just ran a 64-bit RHEL5.1 pvm kernel and
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>saw a loss of
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>about
0.2% with no load.  This was
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>xen-unstable tip today
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>with
no options specified.  32-bit was
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>about 0.01%.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>I
think I missed something... how do I
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>run the various
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>accounting
choices and which ones are
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>known to be
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>appropriate
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>for
which kernels?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>Thanks,
>>>>>>>>>>>>>>>>>>>>>>>Dan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>-----Original
Message-----
>>>>>>>>>>>>>>>>>>>>>>>>From:
xen-devel-bounces@lists.xensource.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>Keir
Fraser
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Sent:
Thursday, December 06, 2007 4:57 AM
>>>>>>>>>>>>>>>>>>>>>>>>To:
Dave Winchell
>>>>>>>>>>>>>>>>>>>>>>>>Cc:
Shan, Haitao;
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dong,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Eddie;
Jiang,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Yunhong
>>>>>>>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a timer
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>mode that
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>disables
pending
>>>>>>>>>>>>>>>>>>>>>>>>missed
ticks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Please
take a look at xen-unstable
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>changeset 16545.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>--
Keir
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>On
26/11/07 20:57, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
accuracy data I''ve collected for i/o
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>loads for the
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>various
time protocols follows. In
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>addition, the data
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>for
cpu loads is shown.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled cpu and i/o-8 are on an 8
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>processor AMD
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>box.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Two
guests, red hat and sles 64 bit, 8
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>vcpu each.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
cpu load is usex -e36 on each guest.
>>>>>>>>>>>>>>>>>>>>>>>>>(usex
is available at
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>i/o
load is 8 instances of dd if=/dev/hda6
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>of=/dev/null.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled i/o-32 are 32
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>instances of dd.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Also,
these are run on 4 cpu AMD box.
>>>>>>>>>>>>>>>>>>>>>>>>>In
addition, there is an idle rh-32bit guest.
>>>>>>>>>>>>>>>>>>>>>>>>>All
three guests are 8vcpu.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled i/o-4/32 are the same
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>as i/o-32
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>except
that the redhat-64 guest has 4
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>instances of dd.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Date
Duration Protocol sles, rhat error load
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/07
23 hrs 40 min ASYNC -4.96 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>+4.42
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec -.006%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>+.005%
cpu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/09
3 hrs 19 min ASYNC -.13 sec, +1.44
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.001%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>+.012%
cpu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
2 hrs 21 min SYNC -.80 sec, -.34
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.009%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>-.004% cpu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
1 hr 25 min SYNC -.24 sec, -.26 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.005%, -.005%
cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/12
65 hrs 40 min SYNC -18 sec, -8 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.008%, -.003%
cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
28 min MIXED -.75 sec, -.67 sec -.045%,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.040% cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
15 hrs 39 min MIXED -19. sec,-17.4
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.034%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>-.031%
cpu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
17 hrs 17 min ASYNC -6.1
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec,-55.7 sec, -.01%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.09%
i/o-8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/15
2 hrs 44 min ASYNC -1.47
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec,-14.0 sec, -.015%
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.14%
i/o-8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/13
15 hrs 38 min SYNC -9.7 sec,-12.3
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.017%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.022%
i/o-8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
48 min SYNC - .46 sec, - .48 sec,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.017%, -.018%
i/o-8
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
4 hrs 2 min MIXED -2.9 sec, -4.15
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.020%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.029%
i/o-8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/20
16 hrs 2 min MIXED -13.4 sec,-18.1
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>sec, -.023%,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>-.031%
i/o-8
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
28 min MIXED -2.01 sec, -.67
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sec, -.12%,
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.04% i/o-32
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
2 hrs 25 min SYNC -.96 sec, -.43
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.011%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>-.005%
i/o-32
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
40 min ASYNC -2.43 sec, -2.77
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sec -.10%,
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>-.11% i/o-32
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26
113 hrs 46 min MIXED -297. sec,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>13. sec -.07%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>.003%
i/o-4/32
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26
4 hrs 50 min SYNC -3.21 sec, 1.44
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>sec, -.017%,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>.01%
i/o-4/32
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Overhead
measurements:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Progress
in terms of number of passes
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>through a fixed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>system
workload
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>on
an 8 vcpu red hat with an 8 vcpu
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>sles idle.
>>    
>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
workload was usex -b48.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>ASYNC
167 min 145 passes .868 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC
167 min 144 passes .862 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC
1065 min 919 passes .863 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>MIXED
221 min 196 passes .887 passes/min
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Conclusions:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>The
only protocol which meets the
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>.05% accuracy
>>    
>>
>>>>>>>>>>>>>>>>>>requirement
for ntp
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>tracking
under the loads
>>>>>>>>>>>>>>>>>>>>>>>>>above
is the SYNC protocol. The worst case
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>accuracies for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC,
MIXED,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>and
ASYNC
>>>>>>>>>>>>>>>>>>>>>>>>>are
.022%, .12%, and .14%, respectively.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>We
could reduce the cost of the SYNC
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>method by only
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>scheduling
the extra
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>wakeups
if a certain number
>>>>>>>>>>>>>>>>>>>>>>>>>of
ticks are missed.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>Dave
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir
Fraser wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>On
9/11/07 19:22, "Dave Winchell"
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>Since
I had a high error (~.03%) for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>ASYNC method a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>couple
of days ago,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I
ran another ASYNC test. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>there may have
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>been
something
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>wrong
with the code I used a couple of
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>days ago for
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC.
It may have been
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>missing
the immediate delivery of interrupt
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>after context
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>switch
in.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>My
results indicate that either SYNC
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>or ASYNC give
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>acceptable
accuracy,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>each
running consistently around or under
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>.01%. MIXED has
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>a
fairly high
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>error
of
>>>>>>>>>>>>>>>>>>>>>>>>>>>greater
than .03%. Probably too close
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>to .05% ntp
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>threshold
for comfort.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I
don''t have an overnight run with SYNC. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>plan to leave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC
running
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>over
the weekend. If you''d rather I can
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>leave MIXED
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>running
instead.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>It
may be too early to pick the
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>protocol and
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>I can run
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>more
overnight tests
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>next
week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>I''m
a bit worried about any unwanted side
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>effects of the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>approach
-- e.g., whether timer wakeups will
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>cause higher
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>system-wide
CPU
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>contention.
I find it easier to think
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>through the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>implications
of ASYNC. I''m
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>surprised
that MIXED loses time, and is less
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>accurate than
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC.
Perhaps it
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>delivers
more timer interrupts than
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>the other
>>    
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>approaches,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>and
each interrupt
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>event
causes a small accumulated error?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>Overall
I would consider MIXED and ASYNC as
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>favourites and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>if
the latter is
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>actually
more accurate then I can
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>simply revert the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>changeset
that
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>implemented
MIXED.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps
rather than running more of the same
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>workloads you
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>could
try idle
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs
and I/O bound VCPUs (e.g., repeated
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>large disc reads
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>to
/dev/null)? We
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>don''t
have any data on workloads that aren''t
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>CPU bound, so
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>that''s
really an
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>obvious
place to put any further effort imo.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>--
Keir
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>                
>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel
mailing list
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>diff
-r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>>>>>>>>>>>>>>>>>>>>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>2007 +0000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>2008 -0500
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>@@
-58,7 +58,7 @@ static void
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>pt_process_missed_ticks(stru
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
missed_ticks = missed_ticks / (s_time_t)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>pt->period +
1;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>no_missed_ticks_pending)
)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-
pt->do_not_freeze = !pt->pending_intr_nr;
>>>>>>>>>>>>>>>>>>>>+
pt->do_not_freeze = 1;
>>>>>>>>>>>>>>>>>>>>
else
>>>>>>>>>>>>>>>>>>>>
pt->pending_intr_nr += missed_ticks;
>>>>>>>>>>>>>>>>>>>>
pt->scheduled += missed_ticks * pt->period;
>>>>>>>>>>>>>>>>>>>>@@
-127,7 +127,12 @@ static void
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>pt_timer_fn(void *data)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>
pt_lock(pt);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-
pt->pending_intr_nr++;
>>>>>>>>>>>>>>>>>>>>+
if ( mode_is(pt->vcpu->domain,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>no_missed_ticks_pending)
) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr = 1;
>>>>>>>>>>>>>>>>>>>>+
pt->do_not_freeze = 0;
>>>>>>>>>>>>>>>>>>>>+
}
>>>>>>>>>>>>>>>>>>>>+
else
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr++;
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
if ( !pt->one_shot )
>>>>>>>>>>>>>>>>>>>>
{
>>>>>>>>>>>>>>>>>>>>@@
-221,8 +226,6 @@ void pt_intr_post(struct
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>vcpu *v, struct
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
return;
>>>>>>>>>>>>>>>>>>>>
}
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>-
pt->do_not_freeze = 0;
>>>>>>>>>>>>>>>>>>>>-
>>>>>>>>>>>>>>>>>>>>
if ( pt->one_shot )
>>>>>>>>>>>>>>>>>>>>
{
>>>>>>>>>>>>>>>>>>>>
pt->enabled = 0;
>>>>>>>>>>>>>>>>>>>>@@
-235,6 +238,10 @@ void pt_intr_post(struct vcpu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>*v, struct
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
pt->last_plt_gtime
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>hvm_get_guest_time(v);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>>>
pt->pending_intr_nr = 0; /*
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>''collapse'' all
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>>>>>>>>>>>missed
ticks */
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
}
>>>>>>>>>>>>>>>>>>>>+
else if ( mode_is(v->domain,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>no_missed_ticks_pending) ) {
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr--;
>>>>>>>>>>>>>>>>>>>>+
pt->last_plt_gtime = hvm_get_guest_time(v);
>>>>>>>>>>>>>>>>>>>>+
}
>>>>>>>>>>>>>>>>>>>>
else
>>>>>>>>>>>>>>>>>>>>
{
>>>>>>>>>>>>>>>>>>>>
pt->last_plt_gtime
+>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>pt->period_cycles;
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>_______________________________________________
>>>>>>>>>>>>>>>>>>Xen-devel
mailing list
>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>--------------------------------------------------------------
>>>>>>>----------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>          
>>>>>
>>>>
>>>>
>>>>        
>>>>
>>    
>>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >percentage.  However, our 32-bit clock skew seems to
> >show a measureable problem now.
> >
> For the 32 bit guest, which timesource did it pick?
The dmesg output is hard to interpret on a 32-bit guest,
but based on what we''ve seen, I think it was selecting
hpet as timesource (because we specified clocksource=pit
which would have been ignored on RHEL4-32).  We are
running another test with "clock=pit" to see if the
skew goes away.
> >For Xen RHEL5 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL5-32
> >
> But still poor accuracy, right?
Unproven yet but I hope not.  The nohpet and nopmtimer
parameters are ignored on RHEL5-32 so the clock=pit
(or clocksource=pit) is the only way to choose the
clock source, and thus the only way to get good
accuracy on RHEL5-32.

Oops, I see from my long list that I neglected to say
that the two clocks (WALL and GTOD) on RHEL5 are
only reported on RHEL5-64.  RHELx-32 looks to have
only the one, which is overridden with clock=.
> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> I have not investigated this yet.
My *think* is based on: 1) observation of dmesg output for
RHEL5-64 where specifying "nohpet nopmtimer" seems to
select PIT for WALL timer; 2) no mention of tsc in
the generic clocksource.c code nor in the i386-specific
time code.

Still I would sleep better if this were definitive.
> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> I''ll have to check on this too.
My *think* is based on our observations to date that clock=pit
is insufficient to fix the skew problem (and doesn''t change
the dmesg WALL source output on RHELx-64)... nohpet and nopmtimer
is required to change the WALL source output and fix the skew.

Again I would sleep better if this were definitive.

Thanks,
Dan
> -----Original Message-----
> From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> Sent: Tuesday, February 19, 2008 1:50 PM
> To: dan.magenheimer@oracle.com
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave
> Winchell
> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dan,
> 
> Thanks for all the investigation you''ve done!
> 
> -Dave
> 
> 
> Dan Magenheimer wrote:
> 
> >Hi Dave --
> >
> >Thanks for that observation on ltp running on one vcpu!
> >
> >With "clocksource=pit nohpet nopmtimer" our clock skew
> >problems seem to have been reduced to a reasonable
> >percentage.  However, our 32-bit clock skew seems to
> >show a measureable problem now.
> >
> For the 32 bit guest, which timesource did it pick?
> 
> >  As a result,
> >I''ve been doing some digging into kernel sources and
> >have observed the following relative to RHEL4 (2.6.9-based)
> >kernels and RHEL5 (2.6.18-based) kernels and thought I
> >would document them for posterity.  Some of
> >our confusion arises from the fact that invalid command
> >line parameters are silently ignored.
> >
> >RHEL4:
> >- clock= is a valid parameter for RHEL4-32
> >- clocksource= is not a valid parameter for RHEL4-xx
> >- nohpet is a valid parameter for RHEL4-64, not RHEL4-32
> >- nopmtimer is not a valid parameter for RHEL4-xx
> >- notsc is a valid parameter for RHEL4-32, not RHEL4-64
> >- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently
> >
> >For Xen RHEL4 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL4-32 [1]
> >- I *think* nohpet is sufficient for RHEL4-64 [1]
> >
> >RHEL5:
> >- there are two kinds of timekeeping, WALL and gtod
> >- clocksource= is a valid parameter for RHEL5-xx
> >- clock= is a valid but deprecated parameter for RHEL5-xx
> >- clock= and clocksource= are essentially equivalent
> >- nohpet is a valid parameter for RHEL5-64, not RHEL5-32
> >- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32
> >- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1]
> >- clock=pit changes the gtod source but not the WALL source[2]
> >- nohpet nopmtimer changes the WALL source to PIT
> >- /sys/devices/system/clocksource/clocksource0/...
> >  available_clocksource lists the possible clock sources
> >  current_clocksource lists the chosen clock source
> >  ..but neither of these works in a RHEL5 guest!
> >
> >For Xen RHEL5 HVM guests:
> >- I *think* clock=pit is sufficient for RHEL5-32
> >
> >
> But still poor accuracy, right?
> 
> >- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64
> >
> >Other info:
> >- As of 2.6.24.2, clock= is still valid (though still deprecated)
> >
> >So, some open questions:
> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> >
> I have not investigated this yet.
> 
> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> >
> I''ll have to check on this too.
> 
> >And finally, since invalid command line parameters are ignored.
> >I think specifying:
> >	clock=pit nohpet nopmtimer
> >will force the guest clock sources into the optimal state for
> >all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the
> >question above on tsc).  And we should keep an eye on
> >kernel/time/clocksource.c to ensure the __setup("clock="...)
> >line doesn''t go away before RHEL6.
> >
> >Note that if hpet=0 and pmtimer=0 were the default hvm platform
> >parameters for all xen hvm guests (on all versions of xen),
> >specifying kernel command line parameters would be unnecessary,
> >but c''est la vie.
> >
> >Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests
> >and timer_mode=2 for 64-bit RHEL guests.
> >
> >Thanks,
> >Dan
> >
> >
> >
> >>-----Original Message-----
> >>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>Sent: Tuesday, February 19, 2008 8:27 AM
> >>To: dan.magenheimer@oracle.com
> >>Cc: Dave Winchell; Keir Fraser; 
> xen-devel@lists.xensource.com; Deepak
> >>Patel
> >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>disables pending
> >>missed ticks
> >>
> >>
> >>
> >>Hi Dan,
> >>
> >>ltp runs by default loading up only one vcpu.
> >>The -x option can be used to run multiple instances, though
> >>in this mode you will get test failures.
> >>I ran 8 instances on each guest for 16 hours, 25 min
> >>and the time error was -11 sec (-.019%) on each guest.
> >>
> >>Regards,
> >>Dave
> >>
> >>Dave Winchell wrote:
> >>
> >>
> >>
> >>>Hi Dan,
> >>>
> >>>Mine was oversubscribed.
> >>>8 physical cpu, 2 guests, each with 8 vcpu.
> >>>I ran one instance of ltp on each guest, continuously. I hope
ltp
> >>>loaded up all the vcpus. I seem to recall that it did, but I
> >>>could be wrong. If it didn''t, that would be a major
difference
> >>>between our tests. I''ll verify this afternoon and run
> >>>
> >>>
> >>multiple instances,
> >>
> >>
> >>>if necessary.
> >>>
> >>>Thanks,
> >>>Dave
> >>>
> >>>
> >>>
> >>>Dan Magenheimer wrote:
> >>>
> >>>
> >>>
> >>>>Hi Dave --
> >>>>
> >>>>No new results yet but one other question:
> >>>>
> >>>>The problems we''ve seen with our testing have been
with a heavily
> >>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB
guests
> >>>>running LTP simultaneously.
> >>>>
> >>>>Was your LTP testing oversubscribed or just a single guest?
> >>>>
> >>>>Thanks,
> >>>>Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>-----Original Message-----
> >>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>Sent: Thursday, February 14, 2008 10:56 AM
> >>>>>To: dan.magenheimer@oracle.com
> >>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak 
> Patel; Dave
> >>>>>Winchell
> >>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
> >>>>>
> >>>>>
> >>disables pending
> >>
> >>
> >>>>>missed ticks
> >>>>>
> >>>>>
> >>>>>Dan,
> >>>>>
> >>>>>Here are some boot snipets for rh4u564 on xen 3.2.
> >>>>>
> >>>>>
> >>>>>#1:
> >>>>>
> >>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line
is ro
> >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet)
> >>>>>Feb 14 10:44:59 vs076 kernel: Linux version
2.6.9-55.ELsmp
> >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version
> >>>>>
> >>>>>
> >>3.4.6 20060404
> >>
> >>
> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>>>>...
> >>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro 
> root=LABEL=/
> >>>>>console=ttyS0 clocksource=pit nohpet
> >>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0
> >>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries:
> >>>>>
> >>>>>
> >>2048 (order: 11,
> >>
> >>
> >>>>>65536 bytes)
> >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545
MHz
> PM timer.
> >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050
> >>>>>
> >>>>>
> >>MHz processor.
> >>
> >>
> >>>>>...
> >>>>>Feb 14 10:45:00 vs076 kernel: checking TSC
> >>>>>
> >>>>>
> >>synchronization across 8
> >>
> >>
> >>>>>CPUs: passed.
> >>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs
> >>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to
> >>>>>
> >>>>>
> >>use of PM timer
> >>
> >>
> >>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based
timekeeping.
> >>>>>
> >>>>>
> >>>>>
> >>>>>#2:
> >>>>>
> >>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line
is ro
> >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet
nopmtimer)
> >>>>>Feb 14 10:47:57 vs076 kernel: Linux version
2.6.9-55.ELsmp
> >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version
> >>>>>
> >>>>>
> >>3.4.6 20060404
> >>
> >>
> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007
> >>>>>...
> >>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro 
> root=LABEL=/
> >>>>>console=ttyS0 clocksource=pit nohpet nopmtimer
> >>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0
> >>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries:
> >>>>>
> >>>>>
> >>2048 (order: 11,
> >>
> >>
> >>>>>65536 bytes)
> >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182
MHz
> >>>>>
> >>>>>
> >>PIT timer.
> >>
> >>
> >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959
> >>>>>
> >>>>>
> >>MHz processor.
> >>
> >>
> >>>>>...
> >>>>>Feb 14 10:47:59 vs076 kernel: checking TSC
> >>>>>
> >>>>>
> >>synchronization across 8
> >>
> >>
> >>>>>CPUs: passed.
> >>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs
> >>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC
based
> >>>>>
> >>>>>
> >>timekeeping.
> >>
> >>
> >>>>>As you can see, I only get the pit if I specify
nopmtimer.
> >>>>>
> >>>>>Dan Magenheimer wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Hi Dave --
> >>>>>>
> >>>>>>Thanks for continuing to run tests!
> >>>>>>
> >>>>>>Hmmm... I thought I had noticed that even though
Linux will
> >>>>>>
> >>>>>>
> >>>>>acknowledge
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>the existence of the pmtimer, it still prints:
> >>>>>>
> >>>>>>time.c: Using PIT/TSC based timekeeping.
> >>>>>>
> >>>>>>I will check again, but assuming the clocksource
for
> our tests is
> >>>>>>indeed pit, the huge difference in the results
(yours
> vs ours) is
> >>>>>>baffling. I wonder if the difference may be the
> >>>>>>
> >>>>>>
> >>underlying hardware.
> >>
> >>
> >>>>>>Maybe we will try to ensure we can duplicate the
results on
> >>>>>>
> >>>>>>
> >>>>>a different
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>box.
> >>>>>>
> >>>>>>
> >>>>>>So your testing was with stock 3.2.0 xen bits (what
> >>>>>>
> >>>>>>
> >>cset?) without
> >>
> >>
> >>>>>>any of your [quote from below] "clock related
tweaks
> >>>>>>
> >>>>>>
> >>that I haven''t
> >>
> >>
> >>>>>>submitted, because I''m still
characterizing them"?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>None of the tweaks I mentioned are in this test.
> >>>>>It was stock with some patches. However, none of the
> >>>>>
> >>>>>
> >>patches are time
> >>
> >>
> >>>>>related to
> >>>>>my knowledge and I checked vpt.c to make sure that it
is
> >>>>>
> >>>>>
> >>the same as
> >>
> >>
> >>>>>what''s in unstable.
> >>>>>The only difference is in pt_intr_post, where I set the
> >>>>>
> >>>>>
> >>timer mode.
> >>
> >>
> >>>>>I don''t have timer mode tied into our config
process yet, which
> >>>>>is different than official xen method.
> >>>>>
> >>>>>
> >>>>>(In pt_intr_post)
> >>>>>     else
> >>>>>     {
> >>>>>+       if(v->arch.paging.mode->guest_levels ==
4)
> >>>>>+
> >>>>>
> >>>>>
> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >
>>
> >>
> >>>>>HVMPTM_no_missed_ticks_pending;
> >>>>>+       else
> >>>>>+
> >>>>>
> >>>>>
> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] >
>>
> >>
> >>>>>HVMPTM_delay_for_missed_ticks;
> >>>>>         if ( mode_is(v->domain,
one_missed_tick_pending) ||
> >>>>>              mode_is(v->domain,
no_missed_ticks_pending) )
> >>>>>         {
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Could you also send detail on the rhel4u4-64 kernel
you
> >>>>>>are testing with, just to ensure we are not
comparing apples
> >>>>>>and oranges?  (Perhaps there''s some way we
can even share the
> >>>>>>identical disk image and vm.cfg file?)
> >>>>>>
> >>>>>>And if our problem is indeed the pmtimer, I will
need to submit
> >>>>>>another patch to Keir to add an hvm pmtimer
platform variable.
> >>>>>>(Hmmm... I don''t think he''s even
accepted the hpet
> variable patch
> >>>>>>yet.  I''ll have to check.)
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Dan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>-----Original Message-----
> >>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>Sent: Thursday, February 14, 2008 9:00 AM
> >>>>>>>To: dan.magenheimer@oracle.com
> >>>>>>>Cc: Dave Winchell; Keir Fraser;
> >>>>>>>
> >>>>>>>
> >>>>>xen-devel@lists.xensource.com; Deepak
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>Patel
> >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer
mode that
> >>>>>>>disables pending
> >>>>>>>missed ticks
> >>>>>>>
> >>>>>>>
> >>>>>>>Hi Dan,
> >>>>>>>
> >>>>>>>I ran the ltp tests with 3.2 and found the
errors
> >>>>>>>for a 16 hour run to be:
> >>>>>>>
> >>>>>>>rh4u564 -9.9 sec (-.017%)
> >>>>>>>rh4u464 -7.3 sec (-.013%)
> >>>>>>>
> >>>>>>>There were no cliffs and the drift was linear.
> >>>>>>>
> >>>>>>>I think the problem you had may be due to the
use of the
> >>>>>>>pm timer. If you still have the boot log, it
would tell you.
> >>>>>>>
> >>>>>>>When I first tried a guest on 3.2 with
"clocksource=pit nohpet"
> >>>>>>>I noticed that it picked the pm timer. Adding
"nopmtimer", the
> >>>>>>>guest will pick the pit.
> >>>>>>>
> >>>>>>>The reason I didn''t have the problem
with our 3.1 base is that
> >>>>>>>I had disabled the hpet and the pmtimer by not
advertising them
> >>>>>>>in the acpi tables. I did this so long ago, I
forgot
> >>>>>>>
> >>>>>>>
> >>that I had to
> >>
> >>
> >>>>>>>disable pmtimer as well as hpet.
> >>>>>>>
> >>>>>>>So, can you re-run your test with
"clocksource=pit nohpet
> >>>>>>>
> >>>>>>>
> >>>>>nopmtimer"?
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>You should see this in the boot messages:
> >>>>>>>
> >>>>>>>time.c: Using PIT/TSC based timekeeping.
> >>>>>>>
> >>>>>>>Thanks,
> >>>>>>>Dave
> >>>>>>>
> >>>>>>>
> >>>>>>>Dave Winchell wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Hi Dan,
> >>>>>>>>
> >>>>>>>>Over the weekend the drift was +18 seconds
for each
> >>>>>>>>
> >>>>>>>>
> >>guest (no ntp).
> >>
> >>
> >>>>>>>>The duration was 3900 minutes, so the error
for each
> >>>>>>>>
> >>>>>>>>
> >>was +.0077%.
> >>
> >>
> >>>>>>>>Looking back through the data, it appears
to drift linearly at
> >>>>>>>>this rate. I''ve attached a plot
for rh4u5-64.
> >>>>>>>>
> >>>>>>>>This accuracy is better than what
I''ve seen before (.03-.05%).
> >>>>>>>>This may be due to the different load (ltp
vs usex) or to
> >>>>>>>>
> >>>>>>>>
> >>>>>one of the
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>changes I''ve made recently.
I''ll do some
> >>>>>>>>
> >>>>>>>>
> >>experimentation to see if
> >>
> >>
> >>>>>>>>there is
> >>>>>>>>a fix I should propose.
> >>>>>>>>
> >>>>>>>>This still doesn''t address the
radical drift you saw.
> >>>>>>>>The next step for me is to run 3.2 and see
if I can
> >>>>>>>>
> >>>>>>>>
> >>reproduce it.
> >>
> >>
> >>>>>>>>Regards,
> >>>>>>>>Dave
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Dave Winchell wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Hi Dan,
> >>>>>>>>>
> >>>>>>>>>Sorry it took me so long, but I finally
ran an ltp
> test today.
> >>>>>>>>>Its on rh4u4-64. I''m using the
defaults for ltp and
> >>>>>>>>>
> >>>>>>>>>
> >>using a script
> >>
> >>
> >>>>>>>>>called runltp. I had a usex load on
rh4u5-64. No ntpd.
> >>>>>>>>>virtual processors / physical
processors = 2.
> >>>>>>>>>
> >>>>>>>>>The clocks drifted -1 sec (4u5) and
+1.5 sec (4u4) in
> >>>>>>>>>
> >>>>>>>>>
> >>300 minutes
> >>
> >>
> >>>>>>>>>for -.005% and .008%.
> >>>>>>>>>
> >>>>>>>>>I''m running a 3.1 based
hypervisor with some clock related
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>tweaks that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>I haven''t submitted, because
I''m still characterizing them.
> >>>>>>>>>
> >>>>>>>>>I''m stopping the usex load on
4u5-64 now and
> >>>>>>>>>
> >>>>>>>>>
> >>replacing it with ltp
> >>
> >>
> >>>>>>>>>and will leave the two guests running
ltp over the weekend.
> >>>>>>>>>
> >>>>>>>>>Regards,
> >>>>>>>>>Dave
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>Dave Winchell wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>Hi Dan, Deepak:
> >>>>>>>>>>
> >>>>>>>>>>Thanks for the data. Those drifts
are severe - no wonder
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>ntp couldn''t
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>keep then in synch. I''ll
try to reproduce that behaviour
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>here, with
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>my code base.
> >>>>>>>>>>If I can''t reproduce it,
I''ll try 3.2.
> >>>>>>>>>>
> >>>>>>>>>>If you can isolate what ltp is
doing during the cliffs,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>that would
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>be very
> >>>>>>>>>>helpful.
> >>>>>>>>>>
> >>>>>>>>>>thanks,
> >>>>>>>>>>Dave
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>Dan Magenheimer wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>OK, Deepak repeated the test
without ntpd and using
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>ntpdate -b before
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>the test.
> >>>>>>>>>>>
> >>>>>>>>>>>The attached graph shows his
results: el5u1-64
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>(best=~0.07%),
> >>
> >>
> >>>>>>>>>>>el4u5-64 (middle=~0.2%), and
el4u5-32 (worst=~0.3%).
> >>>>>>>>>>>
> >>>>>>>>>>>We will continue to look at LTP
to try to isolate.
> >>>>>>>>>>>
> >>>>>>>>>>>Thanks,
> >>>>>>>>>>>Dan
> >>>>>>>>>>>
> >>>>>>>>>>>P.S. elXuY is essentially RHEL
XuY with some patches.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>-----Original Message-----
> >>>>>>>>>>>>From: Dave Winchell
[mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>Sent: Wednesday, January
30, 2008 2:45 PM
> >>>>>>>>>>>>To: Deepak Patel
> >>>>>>>>>>>>Cc:
dan.magenheimer@oracle.com; Keir Fraser;
>
>>>>>>>>>>>>xen-devel@lists.xensource.com;
akira.ijuin@oracle.com;
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>Dave Winchell
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>Subject: Re: [Xen-devel]
[PATCH] Add a timer mode
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>that disables
> >>
> >>
> >>>>>>>>>>>>pending
> >>>>>>>>>>>>missed ticks
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>Dan, Deeepak,
> >>>>>>>>>>>>
> >>>>>>>>>>>>It may be that the
underlying clock error is too
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>great for ntp
> >>
> >>
> >>>>>>>>>>>>to handle. It would be
useful if you did not run ntpd
> >>>>>>>>>>>>and, instead did ntpdate -b
<timeserver> at the start
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>of the test
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>for each guest. Then
capture the data as you have
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>been doing.
> >>
> >>
> >>>>>>>>>>>>If the drift is greater
than .05%, then we need to
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>address that.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>Another option is, when
running ntpd, to enable loop
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>statistics in
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>/etc/ntp.conf
> >>>>>>>>>>>>by adding this to the file:
> >>>>>>>>>>>>
> >>>>>>>>>>>>statistics loopstats
> >>>>>>>>>>>>statsdir /var/lib/ntp/
> >>>>>>>>>>>>
> >>>>>>>>>>>>Then you will see loop data
in that directory.
> >>>>>>>>>>>>Correlating the data in the
loopstats files with the
> >>>>>>>>>>>>peaks in skew would be
interesting. You will see
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>entries of the form
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>54495 76787.701
-0.045153303 -132.569229 0.020806776
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>239.735511 10
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>Where the second to last
column is the Allan Deviation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>When that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>gets over 1000, ntpd is
working pretty hard. However,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>I have not
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>seen ntpd
> >>>>>>>>>>>>completely loose it like
you have.
> >>>>>>>>>>>>
> >>>>>>>>>>>>I''m on vacation
until Monday, and won''t be reading
> >>>>>>>>>>>>email.
> >>>>>>>>>>>>
> >>>>>>>>>>>>Thanks for all your work on
this!
> >>>>>>>>>>>>
> >>>>>>>>>>>>-Dave
> >>>>>>>>>>>>
> >>>>>>>>>>>>Deepak Patel wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>Is the graph for
RHEL5u1-64? (I''ve never tested
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>this one.)
> >>
> >>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>I do not know which
graph was attached with this. But
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>I saw this
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>behavior in EL4u5 - 32,
EL4U5 - 64 and EL5U1 - 64 hvm
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>guests when I
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>was running ltp tests
continuously.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>What was the
behaviour of the other guests running?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>All pvm guests are
fine. But behavior of most of the
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>hvm guests were
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>as described.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>If they had spikes,
were they at the same wall time?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>No. They are not at the
same wall time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Were the other
guests running ltp as well?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>Yes all 6 guests (4 hvm
and 2 pvm) the guests are
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>running ltp
> >>
> >>
> >>>>>>>>>>>>>continuously.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>How are you
measuring skew?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>I was collecting output
of "ntpdate -q
> <timeserver> every
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>300 seconds
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>(5 minutes) and have
created graph based on that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Are you running
ntpd?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>Yes. ntp was running on
all the guests.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>I am investigating what
causes this spikes and
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>let everyone
> >>
> >>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>know what
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>are my findings.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>Deepak
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>Anything that you
can discover that would be in
> sync with
> >>>>>>>>>>>>>>the spikes would be
very helpful!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>The code that I
test with is our product code,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>which is based
> >>
> >>
> >>>>>>>>>>>>>>on 3.1. So it is
possible that something in 3.2 other
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>than vpt.c
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>is the cause. I can
test with 3.2, if necessary.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>thanks,
> >>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>Dan Magenheimer
wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Hi Dave (Keir,
see suggestion below) --
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Turning off
vhpet certainly helps a lot (though
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>see below).
> >>
> >>
> >>>>>>>>>>>>>>>I wonder if
timekeeping with vhpet is so bad that it
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>should be
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>turned off by
default (in 3.1, 3.2, and unstable)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>until it is
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>fixed?  (I have
a patch that defaults it off,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>can post it if
> >>
> >>
> >>>>>>>>>>>>>>>there is
agreement on the above point.)  The whole
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>point of an
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>HPET is to
provide more precise timekeeping and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>if vhpet is
> >>
> >>
> >>>>>>>>>>>>>>>worse than
vpit, it can only confuse users.  Comments?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>In your
testing, are you just measuring % skew
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>over a long
> >>
> >>
> >>>>>>>>>>>>>>>period of time?
> >>>>>>>>>>>>>>>We are graphing
the skew continuously and
> >>>>>>>>>>>>>>>seeing periodic
behavior that is unsettling,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>even with pit.
> >>
> >>
> >>>>>>>>>>>>>>>See attached. 
Though your algorithm recovers,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>the "cliffs"
> >>
> >>
> >>>>>>>>>>>>>>>could still
cause real user problems.  I wonder
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>if there is
> >>
> >>
> >>>>>>>>>>>>>>>anything that
can be done to make the "recovery" more
> >>>>>>>>>>>>>>>responsive?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>We are looking
into what part(s) of LTP is causing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>the cliffs.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>-----Original
Message-----
> >>>>>>>>>>>>>>>>From: Dave
Winchell [mailto:dwinchell@virtualiron.com]
> >>>>>>>>>>>>>>>>Sent:
Monday, January 28, 2008 8:21 AM
> >>>>>>>>>>>>>>>>To:
dan.magenheimer@oracle.com
> >>>>>>>>>>>>>>>>Cc: Keir
Fraser; xen-devel@lists.xensource.com;
>
>>>>>>>>>>>>>>>>deepak.patel@oracle.com;
>
>>>>>>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
> >>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>that disables
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>pending
> >>>>>>>>>>>>>>>>missed
ticks
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Dan,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>I guess
I''m a bit out of date calling for
> clock= usage.
> >>>>>>>>>>>>>>>>Looking at
linux 2.6.20.4 sources, I think you
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>should specify
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>"clocksource=pit
nohpet" on the linux guest bootline.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>You can
leave the xen and dom0 bootlines as they are.
> >>>>>>>>>>>>>>>>The xen and
guest clocksources do not need to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>be the same.
> >>
> >>
> >>>>>>>>>>>>>>>>In my
tests, xen is using the hpet for its
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>timekeeping and
> >>
> >>
> >>>>>>>>>>>>>>>>that
appears to be the default.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>When you
boot the guests you should see
> >>>>>>>>>>>>>>>>time.c:
Using PIT/TSC based timekeeping.
> >>>>>>>>>>>>>>>>on the
rh4u5-64 guest, and something similar
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>on the others.
> >>
> >>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>(xm
dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>Platform timer
> >>
> >>
>
>>>>>>>>>>>>>>>>>14.318MHz
HPET.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>This
appears to be the xen state, which is fine.
> >>>>>>>>>>>>>>>>I was
wrongly assuming that this was the guest state.
> >>>>>>>>>>>>>>>>You might
want to look in your guest logs and see
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>what they were
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>picking
> >>>>>>>>>>>>>>>>for a clock
source.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Dan
Magenheimer wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks,
I hadn''t realized that!  No wonder we didn''t
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>see the same
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>improvement
you saw!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Try
specifying clock=pit on the linux boot line...
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>I''m
confused... do you mean "clocksource=pit"
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>on the Xen
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>command
line or
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>"nohpet"
/ "clock=pit" / "clocksource=pit" on the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>guest (or
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>dom0?)
command
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>line? 
Or both places?  Since the tests take
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>awhile, it
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>would be
nice
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to get
this right the first time.  Do the Xen
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>and guest
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>clocksources
need
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to be
the same?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>-----Original
Message-----
> >>>>>>>>>>>>>>>>>*From:*
Dave Winchell
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>[mailto:dwinchell@virtualiron.com]
> >>
> >>
> >>>>>>>>>>>>>>>>>*Sent:*
Sunday, January 27, 2008 2:22 PM
> >>>>>>>>>>>>>>>>>*To:*
dan.magenheimer@oracle.com; Keir Fraser
> >>>>>>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>deepak.patel@oracle.com;
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com;
Dave Winchell
>
>>>>>>>>>>>>>>>>>*Subject:*
RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>that disables
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>pending
missed ticks
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Hi Dan,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Hpet
timer does have a fairly large error, as I
> >>>>>>>>>>>>>>>>>was
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>trying this
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>one
recently.
> >>>>>>>>>>>>>>>>>I
don''t remember what I got for error, but 1% sounds
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>about
right.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>The
problem is that hpet is not built on top
> of vpt.c,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>the module
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Keir
and I did
> >>>>>>>>>>>>>>>>>all the
recent work in, for its periodic timer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>needs. Try
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>specifying
clock=pit
> >>>>>>>>>>>>>>>>>on the
linux boot line. If it still picks the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>hpet, which it
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>might,
let me know
> >>>>>>>>>>>>>>>>>and
I''ll tell you how to get around this.
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>Regards,
> >>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>--------------------------------------------------------------
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>----------
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>*From:*
Dan Magenheimer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:dan.magenheimer@oracle.com]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>*Sent:*
Fri 1/25/2008 6:50 PM
> >>>>>>>>>>>>>>>>>*To:*
Dave Winchell; Keir Fraser
> >>>>>>>>>>>>>>>>>*Cc:*
xen-devel@lists.xensource.com;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>deepak.patel@oracle.com;
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>akira.ijuin@oracle.com
>
>>>>>>>>>>>>>>>>>*Subject:*
RE: [Xen-devel] [PATCH] Add a timer mode
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>that
disables
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>pending
missed ticks
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Sorry
for the very late followup on this but
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>we finally
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>were able
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>to get
our testing set up again on stable 3.1
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>bits and have
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>seen
some very bad results on 3.1.3-rc1, on the
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>order of 1%.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>Test
enviroment was a 4-socket dual core machine
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>with 24GB of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>memory
running six two-vcpu 2GB domains, four hvm
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>plus two pv.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>All six
guests were running LTP simultaneously.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>The four hvm
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>guests
were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>RHEL4u5-64.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>Timer_mode
was set to 2 for 64-bit guests and 0 for
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>32-bit
guests.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>All
four hvm guests experienced skew around -1%,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>even the 32-bit
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>guest. 
Less intensive testing didn''t exhibit much
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>skew at all.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>A
representative graph is attached.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Dave, I
wonder if some portion of your patches
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>didn''t end up in
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>the xen
trees?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>(xm
dmesg shows 8x Xeon 3.2GHz stepping 04,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>Platform timer
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>14.318MHz
HPET.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>>>>>>Dan
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>P.S.
Many thanks to Deepak and Akira for
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>running tests.
> >>
> >>
>
>>>>>>>>>>>>>>>>>>-----Original
Message-----
>
>>>>>>>>>>>>>>>>>>From:
xen-devel-bounces@lists.xensource.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>Dave
Winchell
>
>>>>>>>>>>>>>>>>>>Sent:
Wednesday, January 09, 2008 9:53 AM
> >>>>>>>>>>>>>>>>>>To:
Keir Fraser
> >>>>>>>>>>>>>>>>>>Cc:
dan.magenheimer@oracle.com;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>Winchell
>
>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>timer mode that
> >>
> >>
>
>>>>>>>>>>>>>>>>>>disables
pending
>
>>>>>>>>>>>>>>>>>>missed
ticks
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Hi
Keir,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>The
latest change, c/s 16690, looks fine.
> >>>>>>>>>>>>>>>>>>I
agree that the code in c/s 16690 is equivalent to
> >>>>>>>>>>>>>>>>>>the
code I submitted. Also, your version is more
>
>>>>>>>>>>>>>>>>>>concise.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>The
error tests confirm the equivalence. With
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>overnight
cpu loads,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>the
checked in version was accurate to
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>+.048% for sles
> >>
> >>
> >>>>>>>>>>>>>>>>>>and
+.038% for red hat. My version was +.046%
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>and
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>+.032% in a
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>2
hour test.
> >>>>>>>>>>>>>>>>>>I
don''t think the difference is significant.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>i/o
loads produced errors of +.01%.
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>Thanks
for all your efforts on this issue.
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>Regards,
>
>>>>>>>>>>>>>>>>>>Dave
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>Keir
Fraser wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>Applied
as c/s 16690, although the
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>checked-in patch is
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>>smaller.
I think the
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>only
important fix is to pt_intr_post() and the
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>only bit of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>the
patch I
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>totally
omitted was the change to
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>pt_process_missed_ticks().
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>I
don''t think
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>that
change can be important, but let''s see what
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>happens to
the
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>error
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>percentage...
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>--
Keir
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>On
4/1/08 23:24, "Dave Winchell"
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>Hi
Dan and Keir,
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>Attached
is a patch that fixes some
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>issues with the
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>SYNC policy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending).
>
>>>>>>>>>>>>>>>>>>>>I
have not tried to make the change the
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>minimal one, but,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>rather,
just
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>ported
into
>
>>>>>>>>>>>>>>>>>>>>the
new code what I know to work well.
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>The error for
> >>
> >>
>
>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending
goes from
>
>>>>>>>>>>>>>>>>>>>>over
3% to .03% with this change according
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>to my testing.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>Regards,
>
>>>>>>>>>>>>>>>>>>>>Dave
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>Dan
Magenheimer wrote:
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>Hi
Dave --
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>Did
you get your correction ported?  If so,
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>it would be
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>nice to
see this get
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>into
3.1.3.
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>Note
that I just did some very limited
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
> >>testing with
> >>
> >>
>
>>>>>>>>>>>>>>>>>>timer_mode=2(=SYNC=no
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>missed
ticks pending)
>
>>>>>>>>>>>>>>>>>>>>>on
tip of xen-3.1-testing (64-bit Linux hv
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>guest) and the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>worst
error I''ve
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>seen
so far
>
>>>>>>>>>>>>>>>>>>>>>is
0.012%.  But I haven''t tried any exotic
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>loads, just LTP.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>Thanks,
>
>>>>>>>>>>>>>>>>>>>>>Dan
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>-----Original
Message-----
>
>>>>>>>>>>>>>>>>>>>>>>From:
Dave Winchell
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>[mailto:dwinchell@virtualiron.com]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>Sent:
Wednesday, December 19, 2007 12:33 PM
>
>>>>>>>>>>>>>>>>>>>>>>To:
dan.magenheimer@oracle.com
>
>>>>>>>>>>>>>>>>>>>>>>Cc:
Keir Fraser; Shan, Haitao;
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dong,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>Eddie;
Jiang, Yunhong; Dave Winchell
>
>>>>>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>timer mode that
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>disables
pending
>
>>>>>>>>>>>>>>>>>>>>>>missed
ticks
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>Dan,
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>I
did some testing with the constant tsc offset
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>SYNC method
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>(now
called
>
>>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending)
>
>>>>>>>>>>>>>>>>>>>>>>and
found the error to be very high, much larger
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>than 1 %,
as
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>I
recall.
>
>>>>>>>>>>>>>>>>>>>>>>I
have not had a chance to submit a
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>correction. I
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>will try to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>do
it later
>
>>>>>>>>>>>>>>>>>>>>>>this
week or the first week in January. My
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>version of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>constant tsc
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>offset
SYNC method
>
>>>>>>>>>>>>>>>>>>>>>>produces
.02 % error, so I just need to port
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>that into the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>current
code.
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>The
error you got for both of those kernels is
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>what I
would
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>expect
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>for
the default mode, delay_for_missed_ticks.
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>I''ll
let Keir answer on how to set the
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>time mode.
> >>
> >>
>
>>>>>>>>>>>>>>>>>>>>>>Regards,
>
>>>>>>>>>>>>>>>>>>>>>>Dave
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>Dan
Magenheimer wrote:
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>Anyone
make measurements on the final patch?
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>I
just ran a 64-bit RHEL5.1 pvm kernel and
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>saw a loss of
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>about
0.2% with no load.  This was
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>xen-unstable tip today
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>with
no options specified.  32-bit was
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>>>>about 0.01%.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>I
think I missed something... how do I
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>run the various
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>accounting
choices and which ones are
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
> >>known to be
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>appropriate
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>for
which kernels?
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>Thanks,
>
>>>>>>>>>>>>>>>>>>>>>>>Dan
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>-----Original
Message-----
>
>>>>>>>>>>>>>>>>>>>>>>>>From:
xen-devel-bounces@lists.xensource.com
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On
Behalf Of
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>Keir
Fraser
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>Sent:
Thursday, December 06, 2007 4:57 AM
>
>>>>>>>>>>>>>>>>>>>>>>>>To:
Dave Winchell
>
>>>>>>>>>>>>>>>>>>>>>>>>Cc:
Shan, Haitao;
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>xen-devel@lists.xensource.com;
Dong,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>Eddie;
Jiang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>Yunhong
>
>>>>>>>>>>>>>>>>>>>>>>>>Subject:
Re: [Xen-devel] [PATCH] Add a timer
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>mode that
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>disables
pending
>
>>>>>>>>>>>>>>>>>>>>>>>>missed
ticks
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>Please
take a look at xen-unstable
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>changeset 16545.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>--
Keir
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>On
26/11/07 20:57, "Dave Winchell"
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
accuracy data I''ve collected for i/o
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>loads for the
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>various
time protocols follows. In
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>addition, the data
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>for
cpu loads is shown.
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled cpu and i/o-8 are on an 8
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>processor
AMD
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>box.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Two
guests, red hat and sles 64 bit, 8
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>vcpu each.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
cpu load is usex -e36 on each guest.
>
>>>>>>>>>>>>>>>>>>>>>>>>>(usex
is available at
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>i/o
load is 8 instances of dd if=/dev/hda6
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>of=/dev/null.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled i/o-32 are 32
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>instances of dd.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Also,
these are run on 4 cpu AMD box.
>
>>>>>>>>>>>>>>>>>>>>>>>>>In
addition, there is an idle rh-32bit guest.
>
>>>>>>>>>>>>>>>>>>>>>>>>>All
three guests are 8vcpu.
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
loads labeled i/o-4/32 are the same
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>as i/o-32
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>except
that the redhat-64 guest has 4
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>instances of dd.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Date
Duration Protocol sles, rhat error load
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/07
23 hrs 40 min ASYNC -4.96 sec,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>+4.42
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec -.006%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>+.005%
cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/09
3 hrs 19 min ASYNC -.13 sec, +1.44
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.001%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>+.012%
cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
2 hrs 21 min SYNC -.80 sec, -.34
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.009%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>-.004%
cpu
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
1 hr 25 min SYNC -.24 sec, -.26 sec,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.005%,
-.005% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/12
65 hrs 40 min SYNC -18 sec, -8 sec,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.008%,
-.003% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
28 min MIXED -.75 sec, -.67 sec -.045%,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.040% cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/08
15 hrs 39 min MIXED -19. sec,-17.4
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.034%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>-.031%
cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
17 hrs 17 min ASYNC -6.1
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec,-55.7 sec, -.01%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>-.09%
i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/15
2 hrs 44 min ASYNC -1.47
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec,-14.0 sec, -.015%
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>-.14%
i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/13
15 hrs 38 min SYNC -9.7 sec,-12.3
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.017%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>-.022%
i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
48 min SYNC - .46 sec, - .48 sec,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.017%,
-.018% i/o-8
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/14
4 hrs 2 min MIXED -2.9 sec, -4.15
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.020%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>-.029%
i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/20
16 hrs 2 min MIXED -13.4 sec,-18.1
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>sec, -.023%,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>-.031%
i/o-8
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
28 min MIXED -2.01 sec, -.67
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>sec, -.12%,
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.04%
i/o-32
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
2 hrs 25 min SYNC -.96 sec, -.43
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.011%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>-.005%
i/o-32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/21
40 min ASYNC -2.43 sec, -2.77
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>sec -.10%,
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>-.11%
i/o-32
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26
113 hrs 46 min MIXED -297. sec,
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>13. sec -.07%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>.003%
i/o-4/32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>11/26
4 hrs 50 min SYNC -3.21 sec, 1.44
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>sec, -.017%,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>.01%
i/o-4/32
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Overhead
measurements:
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Progress
in terms of number of passes
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>through a fixed
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>system
workload
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>on
an 8 vcpu red hat with an 8 vcpu
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>sles idle.
> >>
> >>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
workload was usex -b48.
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>ASYNC
167 min 145 passes .868 passes/min
>
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC
167 min 144 passes .862 passes/min
>
>>>>>>>>>>>>>>>>>>>>>>>>>SYNC
1065 min 919 passes .863 passes/min
>
>>>>>>>>>>>>>>>>>>>>>>>>>MIXED
221 min 196 passes .887 passes/min
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Conclusions:
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>The
only protocol which meets the
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>.05% accuracy
> >>
> >>
>
>>>>>>>>>>>>>>>>>>requirement
for ntp
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>tracking
under the loads
>
>>>>>>>>>>>>>>>>>>>>>>>>>above
is the SYNC protocol. The worst case
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>accuracies
for
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC,
MIXED,
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>and
ASYNC
>
>>>>>>>>>>>>>>>>>>>>>>>>>are
.022%, .12%, and .14%, respectively.
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>We
could reduce the cost of the SYNC
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>method by only
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>scheduling
the extra
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>wakeups
if a certain number
>
>>>>>>>>>>>>>>>>>>>>>>>>>of
ticks are missed.
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Regards,
>
>>>>>>>>>>>>>>>>>>>>>>>>>Dave
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>Keir
Fraser wrote:
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>On
9/11/07 19:22, "Dave Winchell"
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>><dwinchell@virtualiron.com>
wrote:
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>Since
I had a high error (~.03%) for the
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>ASYNC
method a
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>couple
of days ago,
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I
ran another ASYNC test. I think
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>there may have
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>been
something
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>wrong
with the code I used a couple of
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>days ago for
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC.
It may have been
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>missing
the immediate delivery of interrupt
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>after
context
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>switch
in.
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>My
results indicate that either SYNC
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>or ASYNC give
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>acceptable
accuracy,
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>each
running consistently around or under
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>.01%. MIXED
has
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>a
fairly high
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>error
of
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>greater
than .03%. Probably too close
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>to .05% ntp
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>threshold
for comfort.
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>I
don''t have an overnight run with SYNC. I
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>plan to
leave
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC
running
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>over
the weekend. If you''d rather I can
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>leave MIXED
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>running
instead.
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>It
may be too early to pick the
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>protocol and
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>I can run
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>more
overnight tests
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>next
week.
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>I''m
a bit worried about any unwanted side
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>effects of
the
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>approach
-- e.g., whether timer wakeups will
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>cause
higher
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>system-wide
CPU
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>contention.
I find it easier to think
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>through the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>implications
of ASYNC. I''m
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>surprised
that MIXED loses time, and is less
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>accurate
than
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>ASYNC.
Perhaps it
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>delivers
more timer interrupts than
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>the other
> >>
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>approaches,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>and
each interrupt
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>event
causes a small accumulated error?
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>Overall
I would consider MIXED and ASYNC as
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>favourites
and
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>if
the latter is
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>actually
more accurate then I can
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>simply revert the
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>changeset
that
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>implemented
MIXED.
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps
rather than running more of the same
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>workloads
you
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>could
try idle
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs
and I/O bound VCPUs (e.g., repeated
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>large disc
reads
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>to
/dev/null)? We
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>don''t
have any data on workloads that aren''t
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>CPU bound,
so
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>that''s
really an
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>obvious
place to put any further effort imo.
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>--
Keir
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>_______________________________________________
> >>>>>>>>
> >>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel
mailing list
>
>>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>
>>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>diff
-r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
>
>>>>>>>>>>>>>>>>>>>>---
a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>2007 +0000
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>+++
b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>2008 -0500
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>@@
-58,7 +58,7 @@ static void
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>pt_process_missed_ticks(stru
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
missed_ticks = missed_ticks / (s_time_t)
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>pt->period +
1;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
if ( mode_is(pt->vcpu->domain,
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>no_missed_ticks_pending)
)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>-
pt->do_not_freeze = !pt->pending_intr_nr;
>
>>>>>>>>>>>>>>>>>>>>+
pt->do_not_freeze = 1;
>
>>>>>>>>>>>>>>>>>>>>
else
>
>>>>>>>>>>>>>>>>>>>>
pt->pending_intr_nr += missed_ticks;
>
>>>>>>>>>>>>>>>>>>>>
pt->scheduled += missed_ticks * pt->period;
>
>>>>>>>>>>>>>>>>>>>>@@
-127,7 +127,12 @@ static void
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>pt_timer_fn(void *data)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
pt_lock(pt);
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>-
pt->pending_intr_nr++;
>
>>>>>>>>>>>>>>>>>>>>+
if ( mode_is(pt->vcpu->domain,
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>no_missed_ticks_pending)
) {
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr = 1;
>
>>>>>>>>>>>>>>>>>>>>+
pt->do_not_freeze = 0;
>
>>>>>>>>>>>>>>>>>>>>+
}
>
>>>>>>>>>>>>>>>>>>>>+
else
>
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr++;
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
if ( !pt->one_shot )
>
>>>>>>>>>>>>>>>>>>>>
{
>
>>>>>>>>>>>>>>>>>>>>@@
-221,8 +226,6 @@ void pt_intr_post(struct
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>vcpu *v, struct
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
return;
>
>>>>>>>>>>>>>>>>>>>>
}
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>-
pt->do_not_freeze = 0;
>
>>>>>>>>>>>>>>>>>>>>-
>
>>>>>>>>>>>>>>>>>>>>
if ( pt->one_shot )
>
>>>>>>>>>>>>>>>>>>>>
{
>
>>>>>>>>>>>>>>>>>>>>
pt->enabled = 0;
>
>>>>>>>>>>>>>>>>>>>>@@
-235,6 +238,10 @@ void pt_intr_post(struct vcpu
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>*v, struct
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
pt->last_plt_gtime >
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>hvm_get_guest_time(v);
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
pt->pending_intr_nr = 0; /*
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>''collapse'' all
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
>
>>>>>>>>>>>>>>>>>>missed
ticks */
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
}
>
>>>>>>>>>>>>>>>>>>>>+
else if ( mode_is(v->domain,
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>no_missed_ticks_pending) )
{
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>+
pt->pending_intr_nr--;
>
>>>>>>>>>>>>>>>>>>>>+
pt->last_plt_gtime = hvm_get_guest_time(v);
>
>>>>>>>>>>>>>>>>>>>>+
}
>
>>>>>>>>>>>>>>>>>>>>
else
>
>>>>>>>>>>>>>>>>>>>>
{
>
>>>>>>>>>>>>>>>>>>>>
pt->last_plt_gtime +>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
> >>>>>pt->period_cycles;
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>>>>_______________________________________________
>
>>>>>>>>>>>>>>>>>>Xen-devel
mailing list
>
>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com
>
>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
>>>>>>>--------------------------------------------------------------
> >>>>>>>----------
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >
> >
> >
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > >percentage.  However, our 32-bit clock skew seems to
> > >show a measureable problem now.
> > >
> > For the 32 bit guest, which timesource did it pick?
> 
> The dmesg output is hard to interpret on a 32-bit guest,
> but based on what we''ve seen, I think it was selecting
> hpet as timesource (because we specified clocksource=pit
> which would have been ignored on RHEL4-32).  We are
> running another test with "clock=pit" to see if the
> skew goes away.
Yep, that was the problem.  No clock skew with clock=pit
on RHEL4-32 anymore.
> > >For Xen RHEL5 HVM guests:
> > >- I *think* clock=pit is sufficient for RHEL5-32
> > >
> > But still poor accuracy, right?
> 
> Unproven yet but I hope not.  The nohpet and nopmtimer
> parameters are ignored on RHEL5-32 so the clock=pit
> (or clocksource=pit) is the only way to choose the
> clock source, and thus the only way to get good
> accuracy on RHEL5-32.
We had an interesting firedrill with RHEL5-32.
It seems that the RHEL-specific "divider=10" doesn''t
get along very well with "clock=pit" and causes the
system clock to go whacko, gaining literally HOURS
every second.  Until Deepak discovered this, it was
impossible to boot.  So we haven''t tested this
hypothesis yet, but the dmesg output looks promising.

Dan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dave --

I''ve looked into RHEL5-64 a bit more and would appreciate
any thoughts you might have:
> >So, some open questions:
> >[2] In RHEL5, I *think* it is the WALL source that we care about?
> >
> >
> I''ll have to check on this too.
On second look, I may be wrong.  The GTOD clock seems to be
the one associated with vxtime.mode and vxtime.mode is used in
linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
determine if a tick is lost or not.  Is this the code that we
want timer_mode=2 to influence?
> >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> >    (I *think* not as it has never come up in any email.)
> >
> >
> I have not investigated this yet.
Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
it appears whether notsc is required or not depends in part on the
underlying virtual AND physical system.

notsc definitely is involved in the selection of GTOD time
but notsc can get set not only by kernel command line parameter
but also by the result of unsynchronized_tsc():

if (apic_is_clustered_box) notsc=1
if (box_is_Intel) {
	/* C3 delay is worst case HW latency to enter/exit C3 state */
	if (C3 delay < 1000) notsc=1
}
else { /* e.g. AMD */
	if (num_present_cpus() > 1) notsc=1
}

I''m not sure what constitutes a clustered HVM guest or how that
C3 state latency is determined under Xen, but its clear that the
clocksource can be influenced not only by what clock hardware is
present and command-line parameters but also by the physical CPU
and number of guest vcpus.

Yuck!

Thanks,
Dan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dan,

I''ve added a few comments below ...

Lately I''ve been busy getting the hpet emulation to be accurate.
Right now I have the hpet based clock error down to under .02% with
a limited amount of testing. (The error was 1% or more before my changes.)
I hope to submit a patch soon so others can try this out. I need to do more
testing and reduce my changes to the minimal set required.

One thing I really like about the hpet is that a test designed to see if
the clock will go backwards passes on the hpet where it fails on
the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
backwards, but was unsuccessful. It ended up being easier to fix the hpet
emulation for accuracy.

The clock going backwards test is simply a call to gettimeofday() in a loop,
with no delay between calls except to check for new_time < prev_time.
I run this from a driver calling do_gettimeofday() to make it
more stressful. Then I run an instance of this on all (8) vcpus on
8 physical * 2 guests for a total of 16 vcpus running the test.
The test does a yield now and then so you can actually log into the 
system, etc.

Regards,
Dave

Dan Magenheimer wrote:
>Hi Dave --
>
>I''ve looked into RHEL5-64 a bit more and would appreciate
>any thoughts you might have:
>
>  
>
>>>So, some open questions:
>>>[2] In RHEL5, I *think* it is the WALL source that we care about?
>>>
>>>
>>>      
>>>
>>I''ll have to check on this too.
>>    
>>
>
>On second look, I may be wrong.  The GTOD clock seems to be
>the one associated with vxtime.mode and vxtime.mode is used in
>linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>determine if a tick is lost or not.  Is this the code that we
>want timer_mode=2 to influence?
>  
>
Yes, it is.
>  
>
>>>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>>>   (I *think* not as it has never come up in any email.)
>>>
>>>
>>>      
>>>
>>I have not investigated this yet.
>>    
>>
>
>Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
>it appears whether notsc is required or not depends in part on the
>underlying virtual AND physical system.
>
>notsc definitely is involved in the selection of GTOD time
>but notsc can get set not only by kernel command line parameter
>but also by the result of unsynchronized_tsc():
>  
>
According to the comment in time.c, unsynchronized_tsc() is guessing
whether the tsc is synchronized across processors. On most physical 
platforms they are,
and, on those physical platforms the virtualization layer will present the
same error as the hardware exibits. Right now we don''t have code in xen
to
make a unsynchronized host look synchronized to the guest, though that
wouldn''t be very hard to do as long as the error between physical
processors
remained constant.
>if (apic_is_clustered_box) notsc=1
>if (box_is_Intel) {
>	/* C3 delay is worst case HW latency to enter/exit C3 state */
>	if (C3 delay < 1000) notsc=1
>}
>else { /* e.g. AMD */
>	if (num_present_cpus() > 1) notsc=1
>}
>
>I''m not sure what constitutes a clustered HVM guest or how that
>C3 state latency is determined under Xen, but its clear that the
>clocksource can be influenced not only by what clock hardware is
>present and command-line parameters but also by the physical CPU
>and number of guest vcpus.
>  
>
I haven''t looked into
>Yuck!
>
>Thanks,
>Dan
>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

If this is an integration of hpet into the vpt.c infrastructure then that
would be very welcome.

 -- Keir

On 25/2/08 20:01, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> Hi Dan,
> 
> I''ve added a few comments below ...
> 
> Lately I''ve been busy getting the hpet emulation to be accurate.
> Right now I have the hpet based clock error down to under .02% with
> a limited amount of testing. (The error was 1% or more before my changes.)
> I hope to submit a patch soon so others can try this out. I need to do more
> testing and reduce my changes to the minimal set required.
> 
> One thing I really like about the hpet is that a test designed to see if
> the clock will go backwards passes on the hpet where it fails on
> the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
> backwards, but was unsuccessful. It ended up being easier to fix the hpet
> emulation for accuracy.
> 
> The clock going backwards test is simply a call to gettimeofday() in a
loop,
> with no delay between calls except to check for new_time < prev_time.
> I run this from a driver calling do_gettimeofday() to make it
> more stressful. Then I run an instance of this on all (8) vcpus on
> 8 physical * 2 guests for a total of 16 vcpus running the test.
> The test does a yield now and then so you can actually log into the
> system, etc.
> 
> Regards,
> Dave
> 
> Dan Magenheimer wrote:
> 
>> Hi Dave --
>> 
>> I''ve looked into RHEL5-64 a bit more and would appreciate
>> any thoughts you might have:
>> 
>>  
>> 
>>>> So, some open questions:
>>>> [2] In RHEL5, I *think* it is the WALL source that we care
about?
>>>> 
>>>> 
>>>>      
>>>> 
>>> I''ll have to check on this too.
>>>    
>>> 
>> 
>> On second look, I may be wrong.  The GTOD clock seems to be
>> the one associated with vxtime.mode and vxtime.mode is used in
>> linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>> determine if a tick is lost or not.  Is this the code that we
>> want timer_mode=2 to influence?
>>  
>> 
> Yes, it is.
> 
>>  
>> 
>>>> [1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
>>>>   (I *think* not as it has never come up in any email.)
>>>> 
>>>> 
>>>>      
>>>> 
>>> I have not investigated this yet.
>>>    
>>> 
>> 
>> Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c,
>> it appears whether notsc is required or not depends in part on the
>> underlying virtual AND physical system.
>> 
>> notsc definitely is involved in the selection of GTOD time
>> but notsc can get set not only by kernel command line parameter
>> but also by the result of unsynchronized_tsc():
>>  
>> 
> According to the comment in time.c, unsynchronized_tsc() is guessing
> whether the tsc is synchronized across processors. On most physical
> platforms they are,
> and, on those physical platforms the virtualization layer will present the
> same error as the hardware exibits. Right now we don''t have code
in xen to
> make a unsynchronized host look synchronized to the guest, though that
> wouldn''t be very hard to do as long as the error between physical
processors
> remained constant.
> 
>> if (apic_is_clustered_box) notsc=1
>> if (box_is_Intel) {
>> /* C3 delay is worst case HW latency to enter/exit C3 state */
>> if (C3 delay < 1000) notsc=1
>> }
>> else { /* e.g. AMD */
>> if (num_present_cpus() > 1) notsc=1
>> }
>> 
>> I''m not sure what constitutes a clustered HVM guest or how
that
>> C3 state latency is determined under Xen, but its clear that the
>> clocksource can be influenced not only by what clock hardware is
>> present and command-line parameters but also by the physical CPU
>> and number of guest vcpus.
>>  
>> 
> I haven''t looked into
> 
>> Yuck!
>> 
>> Thanks,
>> Dan
>> 
>> 
>> 
>>  
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir -

Keir Fraser wrote:
>If this is an integration of hpet into the vpt.c infrastructure then that
>would be very welcome.
>  
>
So far it is not, however I may head in that direction.
Last night''s test had an error of .065% on one guest so I still have
some
work to do.

-Dave
> -- Keir
>
>On 25/2/08 20:01, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>Hi Dan,
>>
>>I''ve added a few comments below ...
>>
>>Lately I''ve been busy getting the hpet emulation to be
accurate.
>>Right now I have the hpet based clock error down to under .02% with
>>a limited amount of testing. (The error was 1% or more before my
changes.)
>>I hope to submit a patch soon so others can try this out. I need to do
more
>>testing and reduce my changes to the minimal set required.
>>
>>One thing I really like about the hpet is that a test designed to see if
>>the clock will go backwards passes on the hpet where it fails on
>>the pit/tsc clock. I tried for quite a while to keep pit/tsc from going
>>backwards, but was unsuccessful. It ended up being easier to fix the
hpet
>>emulation for accuracy.
>>
>>The clock going backwards test is simply a call to gettimeofday() in a
loop,
>>with no delay between calls except to check for new_time < prev_time.
>>I run this from a driver calling do_gettimeofday() to make it
>>more stressful. Then I run an instance of this on all (8) vcpus on
>>8 physical * 2 guests for a total of 16 vcpus running the test.
>>The test does a yield now and then so you can actually log into the
>>system, etc.
>>
>>Regards,
>>Dave
>>
>>Dan Magenheimer wrote:
>>
>>    
>>
>>>Hi Dave --
>>>
>>>I''ve looked into RHEL5-64 a bit more and would appreciate
>>>any thoughts you might have:
>>>
>>> 
>>>
>>>      
>>>
>>>>>So, some open questions:
>>>>>[2] In RHEL5, I *think* it is the WALL source that we care
about?
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>I''ll have to check on this too.
>>>>   
>>>>
>>>>        
>>>>
>>>On second look, I may be wrong.  The GTOD clock seems to be
>>>the one associated with vxtime.mode and vxtime.mode is used in
>>>linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
>>>determine if a tick is lost or not.  Is this the code that we
>>>want timer_mode=2 to influence?
>>> 
>>>
>>>      
>>>
>>Yes, it is.
>>
>>    
>>
>>> 
>>>
>>>      
>>>
>>>>>[1] Is notsc necessary for proper ticks for
RHEL4-32/RHEL5-64?
>>>>>  (I *think* not as it has never come up in any email.)
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>I have not investigated this yet.
>>>>   
>>>>
>>>>        
>>>>
>>>Well for RHEL5-64, looking at
linux-2.6.18.8/arch/x86_64/kernel/time.c,
>>>it appears whether notsc is required or not depends in part on the
>>>underlying virtual AND physical system.
>>>
>>>notsc definitely is involved in the selection of GTOD time
>>>but notsc can get set not only by kernel command line parameter
>>>but also by the result of unsynchronized_tsc():
>>> 
>>>
>>>      
>>>
>>According to the comment in time.c, unsynchronized_tsc() is guessing
>>whether the tsc is synchronized across processors. On most physical
>>platforms they are,
>>and, on those physical platforms the virtualization layer will present
the
>>same error as the hardware exibits. Right now we don''t have
code in xen to
>>make a unsynchronized host look synchronized to the guest, though that
>>wouldn''t be very hard to do as long as the error between
physical processors
>>remained constant.
>>
>>    
>>
>>>if (apic_is_clustered_box) notsc=1
>>>if (box_is_Intel) {
>>>/* C3 delay is worst case HW latency to enter/exit C3 state */
>>>if (C3 delay < 1000) notsc=1
>>>}
>>>else { /* e.g. AMD */
>>>if (num_present_cpus() > 1) notsc=1
>>>}
>>>
>>>I''m not sure what constitutes a clustered HVM guest or how
that
>>>C3 state latency is determined under Xen, but its clear that the
>>>clocksource can be influenced not only by what clock hardware is
>>>present and command-line parameters but also by the physical CPU
>>>and number of guest vcpus.
>>> 
>>>
>>>      
>>>
>>I haven''t looked into
>>
>>    
>>
>>>Yuck!
>>>
>>>Thanks,
>>>Dan
>>>
>>>
>>>
>>> 
>>>
>>>      
>>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/2/08 14:45, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
>> If this is an integration of hpet into the vpt.c infrastructure then
that
>> would be very welcome.
>>
> So far it is not, however I may head in that direction.
> Last night''s test had an error of .065% on one guest so I still
have some
> work to do.
Then I hope that the accuracy improvements have come from a set of changes
whose effects are both explicable and generally for the good (rather than
artefacts of how a particular sub-point release of Linux drives the HPET).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
>On 26/2/08 14:45, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>>If this is an integration of hpet into the vpt.c infrastructure then
that
>>>would be very welcome.
>>>
>>>      
>>>
>>So far it is not, however I may head in that direction.
>>Last night''s test had an error of .065% on one guest so I still
have some
>>work to do.
>>    
>>
>
>Then I hope that the accuracy improvements have come from a set of changes
>whose effects are both explicable and generally for the good (rather than
>artefacts of how a particular sub-point release of Linux drives the HPET).
>  
>
I believe that my changes meet these requirements. Your suggestion to layer
on vpt.c is a good one that I will pursue.

- Dave
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir, Dan:

This is an update on the hpet work. With limited testing, the accuracy
of this method appears to be at least 10 times better than the
pit/tsc method for usex loads. Also, as mentioned before, it does
not have the time going backwards problem. Another interesting property
is that the same policy is used for a 32 bit Linux guest and 64 bit Linux.

Two recent 14-16 hour tests were run, one with usex e48 and the other with
usex b48 as loads. The same load was run on three 8 vcpu guests on an 8 cpu
platform. No ntpd running. ntpdate -b used for initial setting of clock and
ntpdate -q used for monitoring drift. The guests were 4u464, 4u564, 4u432
Linux. The results are:

test      duration   drift (secs) 4u464,4u564,4u432      drift %

usex b48    16 hrs       -.68, -.60, -.68                 -.0012
usex e48    15 hrs       -.58, -.55, -.58                 -.0011


The drift with pit/tsc as checked in reported a few months ago:
>> The error tests confirm the equivalence. With overnight cpu loads,
>> the checked in version was accurate to +.048% for sles
>> and +.038% for red hat.
Recall that .05% is the goal for accuracy so that ntpd can synchronize.
With pit/tsc we are rather close to that limit. So far, with hpet,
we are well under that goal.

As for the code structure, I tried layering on vpt.c and had trouble
doing so. Therefore, I focused on a direct approach. I''ll describe the
approach
shortly, after I run tests with other loads and guests.


Regards,
Dave



Keir Fraser wrote:
>On 26/2/08 14:45, "Dave Winchell"
<dwinchell@virtualiron.com> wrote:
>
>  
>
>>>If this is an integration of hpet into the vpt.c infrastructure then
that
>>>would be very welcome.
>>>
>>>      
>>>
>>So far it is not, however I may head in that direction.
>>Last night''s test had an error of .065% on one guest so I still
have some
>>work to do.
>>    
>>
>
>Then I hope that the accuracy improvements have come from a set of changes
>whose effects are both explicable and generally for the good (rather than
>artefacts of how a particular sub-point release of Linux drives the HPET).
>
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/3/08 15:06, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> As for the code structure, I tried layering on vpt.c and had trouble
> doing so. Therefore, I focused on a direct approach. I''ll describe
the
> approach
> shortly, after I run tests with other loads and guests.
Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate
missed ticks, and I want only one missed-tick algorithm in Xen: currently
that is vpt.c.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/3/08 17:25, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
> In 2.6.9, it looks like cur_timer->mark_offset() call from
> timer_interrupt in
> 32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(),
> which computes
> missed ticks based on hpet counter. mark_offset_pit() does nothing.
> mark_offset_tsc() does compute missed ticks.
> 
> In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
> reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.
> 
> So from the code perspective, it looks like missed ticks are computed
> for 32
> and 64 bit Linux using hpet clocksource.
Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the
GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look
like in general Linux 2.6 is robust to missed ticks when using hpet.
That''s
good.

Do you see and simply ignore warning messages from 64-bit Linux when using
hpet (or otherwise not doing missed-tick handling in Xen), by the way? I
know 64-bit Linux is keen to warn about missed ticks, although it does look
like at least the warning is one shot.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
>On 5/3/08 15:06, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
>
>  
>
>>As for the code structure, I tried layering on vpt.c and had trouble
>>doing so. Therefore, I focused on a direct approach. I''ll
describe the
>>approach
>>shortly, after I run tests with other loads and guests.
>>    
>>
>
>Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate
>missed ticks, and I want only one missed-tick algorithm in Xen: currently
>that is vpt.c.
>  
>
No, my results are for 32 bit Linux as well, at least 4u432.
It turns out that for hpet, (rh4u4) 32 bit Linux does handle missed ticks.
I need to try other 32 bit Linux versions, as well as Windows, to see how
broad this behaviour is.

Looking at the code for 2.6.20, it appears to me that if CONFIG_GENERIC_TIME
is set then missed ticks are handled for hpet for 32 bit and 64
bit Linux (see update_wall_time() and read_hpet()).

In 2.6.9, it looks like cur_timer->mark_offset() call from 
timer_interrupt in
32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(), 
which computes
missed ticks based on hpet counter. mark_offset_pit() does nothing.
mark_offset_tsc() does compute missed ticks.

In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.

So from the code perspective, it looks like missed ticks are computed 
for 32
and 64 bit Linux using hpet clocksource.

regards,
Dave
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
>On 5/3/08 17:25, "Dave Winchell" <dwinchell@virtualiron.com>
wrote:
>
>  
>
>>In 2.6.9, it looks like cur_timer->mark_offset() call from
>>timer_interrupt in
>>32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(),
>>which computes
>>missed ticks based on hpet counter. mark_offset_pit() does nothing.
>>mark_offset_tsc() does compute missed ticks.
>>
>>In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet
>>reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks.
>>
>>So from the code perspective, it looks like missed ticks are computed
>>for 32
>>and 64 bit Linux using hpet clocksource.
>>    
>>
>
>Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the
>GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look
>like in general Linux 2.6 is robust to missed ticks when using hpet.
That''s
>good.
>
>Do you see and simply ignore warning messages from 64-bit Linux when using
>hpet (or otherwise not doing missed-tick handling in Xen), by the way? I
>know 64-bit Linux is keen to warn about missed ticks, although it does look
>like at least the warning is one shot.
>  
>
I see the one shot messages on 64 bit and no complaints at all on 4u432 
Linux.
And I have been ignoring them.
> -- Keir
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >Do you see and simply ignore warning messages from 64-bit 
> Linux when using
> >hpet (or otherwise not doing missed-tick handling in Xen), 
> by the way? I
> >know 64-bit Linux is keen to warn about missed ticks, 
> although it does look
> >like at least the warning is one shot.
> >
> >
> I see the one shot messages on 64 bit and no complaints at 
> all on 4u432
> Linux.
> And I have been ignoring them.
Code to print the complaints is x86_64-specific in 2.6.9 and 2.6.18.
On 64-bit, a command line parameter of "report_lost_ticks" will
enable a complaint for every lost tick.

As of 2.6.24, the lost ticks message seems to have gone away
completely.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

(One more time on this old thread...)

Hi Dave --

We have a strange but alarming result:  Running rhel5ga-64 with
timer_mode=0 has no skew but with timer_mode=2 has very bad skew.
With rhel5u1-64 (and all rhel4-64 guests) the opposite is true.
We repeated the tests to verify.  In all cases, dmesg shows:

time.c: Using 1.193182 MHz WALL PIT GTOD PIT/TSC timer.

Could you try your heavy load tests with all four combinations
of rhel5ga-64/rhel5u1-64 and timer_mode=0/2 to see if you
can reproduce our results.

Thanks,
Dan
> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: Monday, February 25, 2008 9:42 AM
> To: ''Dave Winchell''
> Cc: ''Keir Fraser'';
''xen-devel@lists.xensource.com''; ''Deepak
Patel''
> Subject: RE: [Xen-devel] [PATCH] Add a timer mode that 
> disables pending
> missed ticks
> 
> 
> Hi Dave --
> 
> I''ve looked into RHEL5-64 a bit more and would appreciate
> any thoughts you might have:
> 
> > >So, some open questions:
> > >[2] In RHEL5, I *think* it is the WALL source that we care about?
> > >
> > >
> > I''ll have to check on this too.
> 
> On second look, I may be wrong.  The GTOD clock seems to be
> the one associated with vxtime.mode and vxtime.mode is used in
> linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to
> determine if a tick is lost or not.  Is this the code that we
> want timer_mode=2 to influence?
> 
> > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64?
> > >    (I *think* not as it has never come up in any email.)
> > >
> > >
> > I have not investigated this yet.
> 
> Well for RHEL5-64, looking at 
> linux-2.6.18.8/arch/x86_64/kernel/time.c,
> it appears whether notsc is required or not depends in part on the
> underlying virtual AND physical system.
> 
> notsc definitely is involved in the selection of GTOD time
> but notsc can get set not only by kernel command line parameter
> but also by the result of unsynchronized_tsc():
> 
> if (apic_is_clustered_box) notsc=1
> if (box_is_Intel) {
> 	/* C3 delay is worst case HW latency to enter/exit C3 state */
> 	if (C3 delay < 1000) notsc=1
> }
> else { /* e.g. AMD */
> 	if (num_present_cpus() > 1) notsc=1
> }
> 
> I''m not sure what constitutes a clustered HVM guest or how that
> C3 state latency is determined under Xen, but its clear that the
> clocksource can be influenced not only by what clock hardware is
> present and command-line parameters but also by the physical CPU
> and number of guest vcpus.
> 
> Yuck!
> 
> Thanks,
> Dan
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel