Dear developers, I have some issues with time synchronization among dom0 and domUs. In my case helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for domUs and running ntpd also on domUs. I think, running ntpd on domU is not a good idea, but it works acceptably good as a workaround for my configuration: XEN 3.4.1 + kernel 3.6.31.5 with pv_ops (taken from git last week). When I don''t run ntpd on domU clock slowly drifts. I have choosen xen clocksource in both dom0 and domU. I''m not the only one to have time synchronization issues. Please write some document specifying how time synchronization between dom0 and domU should be achieved. I have a number of special questions, but probably there are more questions that should be aswered: 1. What is the mechanics behind the clocksource named xen? How does it depend on phisical time sources available on different platforms? 2. How gets domU''s clock initialized when domU boots? (I observe that just after domU has started its clock differs from dom0. The difference is about 10s.) 3. What are the consequences of not selecting xen clocksource (by selecting one of hpet, acpi_pm, etc.) for dom0 and for domU? 4. What are the best practices to achieve clock synchronization? 5. What has changed recently in timekeeping? What is going to change? Are there any improvements/bugfixes added to redhat or suse kernels? Are these bugfixes planned to be added upstream? Thank you in advance, -- Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland bartoszl @ ics.p.lodz.pl _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Nov-09 16:50 UTC
RE: [Xen-devel] timekeepeing in XEN - request fo comments
What kernels are running in your domUs? Are they also (all) 2.6.31.5? If your domU''s are older kernels (e.g. SLES10 or older, RHEL5 or older) there are many issues that cannot be entirely solved by Xen or dom0.> -----Original Message----- > From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl] > Sent: Monday, November 09, 2009 5:21 AM > To: xen-devel@lists.xensource.com > Subject: [Xen-devel] timekeepeing in XEN - request fo comments > > > Dear developers, > > I have some issues with time synchronization among dom0 and > domUs. In my case > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 > for domUs and > running ntpd also on domUs. I think, running ntpd on domU is > not a good idea, > but it works acceptably good as a workaround for my > configuration: XEN 3.4.1 + > kernel 3.6.31.5 with pv_ops (taken from git last week). When > I don''t run ntpd > on domU clock slowly drifts. I have choosen xen clocksource > in both dom0 and > domU. > > I''m not the only one to have time synchronization issues. > Please write some > document specifying how time synchronization between dom0 and > domU should be > achieved. I have a number of special questions, but probably > there are more > questions that should be aswered: > > 1. What is the mechanics behind the clocksource named xen? > How does it depend > on phisical time sources available on different platforms? > > 2. How gets domU''s clock initialized when domU boots? (I > observe that just > after domU has started its clock differs from dom0. The > difference is about > 10s.) > > 3. What are the consequences of not selecting xen clocksource > (by selecting > one of hpet, acpi_pm, etc.) for dom0 and for domU? > > 4. What are the best practices to achieve clock synchronization? > > 5. What has changed recently in timekeeping? What is going to > change? Are > there any improvements/bugfixes added to redhat or suse > kernels? Are these > bugfixes planned to be added upstream? > > Thank you in advance, > > -- > Bartosz Lis @ Inst. of Information Technology, Technical > Univ. of Lodz, Poland > bartoszl @ ics.p.lodz.pl > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2009-Nov-09 18:49 UTC
Re: [Xen-devel] timekeepeing in XEN - request fo comments
On Mon, Nov 09, 2009 at 08:50:52AM -0800, Dan Magenheimer wrote:> What kernels are running in your domUs? Are they > also (all) 2.6.31.5? If your domU''s are older kernels > (e.g. SLES10 or older, RHEL5 or older) there are > many issues that cannot be entirely solved by > Xen or dom0. >Could you tell more about these timekeeping problems with SLES10/RHEL5 PV kernels? -- Pasi> > -----Original Message----- > > From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl] > > Sent: Monday, November 09, 2009 5:21 AM > > To: xen-devel@lists.xensource.com > > Subject: [Xen-devel] timekeepeing in XEN - request fo comments > > > > > > Dear developers, > > > > I have some issues with time synchronization among dom0 and > > domUs. In my case > > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 > > for domUs and > > running ntpd also on domUs. I think, running ntpd on domU is > > not a good idea, > > but it works acceptably good as a workaround for my > > configuration: XEN 3.4.1 + > > kernel 3.6.31.5 with pv_ops (taken from git last week). When > > I don''t run ntpd > > on domU clock slowly drifts. I have choosen xen clocksource > > in both dom0 and > > domU. > > > > I''m not the only one to have time synchronization issues. > > Please write some > > document specifying how time synchronization between dom0 and > > domU should be > > achieved. I have a number of special questions, but probably > > there are more > > questions that should be aswered: > > > > 1. What is the mechanics behind the clocksource named xen? > > How does it depend > > on phisical time sources available on different platforms? > > > > 2. How gets domU''s clock initialized when domU boots? (I > > observe that just > > after domU has started its clock differs from dom0. The > > difference is about > > 10s.) > > > > 3. What are the consequences of not selecting xen clocksource > > (by selecting > > one of hpet, acpi_pm, etc.) for dom0 and for domU? > > > > 4. What are the best practices to achieve clock synchronization? > > > > 5. What has changed recently in timekeeping? What is going to > > change? Are > > there any improvements/bugfixes added to redhat or suse > > kernels? Are these > > bugfixes planned to be added upstream? > > > > Thank you in advance, > > > > -- > > Bartosz Lis @ Inst. of Information Technology, Technical > > Univ. of Lodz, Poland > > bartoszl @ ics.p.lodz.pl > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Nov-10 00:59 UTC
Re: [Xen-devel] timekeepeing in XEN - request fo comments
On 11/09/09 04:21, Bartosz Lis wrote:> I have some issues with time synchronization among dom0 and domUs. In my case > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for domUs and > running ntpd also on domUs. I think, running ntpd on domU is not a good idea, >Why not?> but it works acceptably good as a workaround for my configuration: XEN 3.4.1 + > kernel 3.6.31.5 with pv_ops (taken from git last week). When I don''t run ntpd > on domU clock slowly drifts. I have choosen xen clocksource in both dom0 and > domU. >Drifts with respect to what? If you run all domains without ntp then they should remain in lockstep, because they are all being run off the same timebase. But the xen clocksource does have a drift correction factor which can be updated via ntpd/adjtimex, so if each kernel updates its factor differently then they will appear to drift with respect to each other.> I''m not the only one to have time synchronization issues. Please write some > document specifying how time synchronization between dom0 and domU should be > achieved. I have a number of special questions, but probably there are more > questions that should be aswered: > > 1. What is the mechanics behind the clocksource named xen? How does it depend > on phisical time sources available on different platforms? >You can think of it as an augmented tsc; the tsc is used to convey fine-grained time information, and a set of additional parameters allow the guest to correct for things like tsc skew between cpus. (And, ideally, differing tsc rates, but it is hard to get things perfectly monotonic in that case.)> 2. How gets domU''s clock initialized when domU boots? (I observe that just > after domU has started its clock differs from dom0. The difference is about > 10s.) >It reads the Xen wallclock at boot to initialize the time of day. Xen''s wallclock time may not be correct/match dom0 if dom0 has failed to update it (which is currently the case in pvops dom0 kernels).> 3. What are the consequences of not selecting xen clocksource (by selecting > one of hpet, acpi_pm, etc.) for dom0 and for domU? >Xen itself uses and claims ownership of hardware resources like hpet and acpi_pm, so they aren''t available to guest kernels. DomU kernels simply don''t have any hardware access; dom0 is specifically not allowed to touch hpet (and I think acpi_pm). You could force a kernel to use tsc a clocksource, but that will only work correctly if the system has inherently synchronized tscs, and there would be discontinuities over suspend/resume/migrate.> 4. What are the best practices to achieve clock synchronization? >ntp is it at present.> 5. What has changed recently in timekeeping? What is going to change? Are > there any improvements/bugfixes added to redhat or suse kernels? Are these > bugfixes planned to be added upstream? >As far as I know the pvops kernel is state of the art. It currently fails to set the Xen wallclock time when asked (not implemented), but aside from that it should be OK. The basic principle is that Xen doesn''t know all that much about time of day, and doesn''t really care. It offers a token get/set time of day interface to access the corresponding hardware, but very little beyond that. Time is really considered to be a local matter for domains to work out. If they want to synchronize, then ntp is a perfectly reasonable mechanism for doing that. Would you like to see any particular changes? What are your requirements? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bartosz Lis
2009-Nov-10 14:43 UTC
Re: [Xen-devel] timekeepeing in XEN - request fo comments
Dnia poniedziałek, 9 listopada 2009 o 17:50:52 Dan Magenheimer napisał(a):> What kernels are running in your domUs? Are they > also (all) 2.6.31.5? If your domU''s are older kernels > (e.g. SLES10 or older, RHEL5 or older) there are > many issues that cannot be entirely solved by > Xen or dom0.Dan, Both mine are 2.6.31.5 from Jeremy''s git tree, but Jordi Espasa Clofent reported on xen-users list problems with 2.6.18-6-xen-amd64. See his post: http://lists.xensource.com/archives/html/xen-users/2009-11/msg00115.html . -- Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland bartoszl @ ics.p.lodz.pl _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bartosz Lis
2009-Nov-10 16:01 UTC
Re: [Xen-devel] timekeepeing in XEN - request fo comments
Jeremy, Thanks very much for your answers. I think, now, I understand what causes my problems. Dnia wtorek, 10 listopada 2009 o 01:59:13 Jeremy Fitzhardinge napisał(a):> On 11/09/09 04:21, Bartosz Lis wrote: > > I have some issues with time synchronization among dom0 and domUs. In my > > case helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for > > domUs and running ntpd also on domUs. I think, running ntpd on domU is > > not a good idea, > > Why not? >I run ntpd in dom0, so it is synchronized to internet time. This synchronization (done by a daemon) costs in terms of processing power, memory usage and network traffic. This cost generated by dom0''s ntpd is small and acceptable. I think it would be much cheaper to synchronize domUs'' clocks to dom0 clock using hypercalls rather than running ntpd daemon on each of them.> > but it works acceptably good as a workaround for my configuration: XEN > > 3.4.1 + kernel 3.6.31.5 with pv_ops (taken from git last week). When I > > don''t run ntpd on domU clock slowly drifts. I have choosen xen > > clocksource in both dom0 and domU. > > Drifts with respect to what?Drifts with respect to dom0''s clock, which is synchronized to internet time by ntpd.> > If you run all domains without ntp then they should remain in lockstep, > because they are all being run off the same timebase. But the xen > clocksource does have a drift correction factor which can be updated via > ntpd/adjtimex, so if each kernel updates its factor differently then > they will appear to drift with respect to each other.I had run ntpd in domUs for several hours, then I switched it off, and observed what happens then. Clocks started drifting from dom0 (which was still synchronized to global time by ntpd) and from each other. Yes, it''s possible that during the period that each domU had ntpd running their drift correction factors were set in a different way. Domains had different loads, so ntpd on more loaded domain could have had worse accuracy.> > > I''m not the only one to have time synchronization issues. Please write > > some document specifying how time synchronization between dom0 and domU > > should be achieved. I have a number of special questions, but probably > > there are more questions that should be aswered: > > > > 1. What is the mechanics behind the clocksource named xen? How does it > > depend on phisical time sources available on different platforms? > > You can think of it as an augmented tsc; the tsc is used to convey > fine-grained time information, and a set of additional parameters allow > the guest to correct for things like tsc skew between cpus. (And, > ideally, differing tsc rates, but it is hard to get things perfectly > monotonic in that case.) > > > 2. How gets domU''s clock initialized when domU boots? (I observe that > > just after domU has started its clock differs from dom0. The difference > > is about 10s.) > > It reads the Xen wallclock at boot to initialize the time of day. Xen''s > wallclock time may not be correct/match dom0 if dom0 has failed to > update it (which is currently the case in pvops dom0 kernels).Please, tell me if I''m right (I''m 99% sure, but, please, assure me): * date - displays and sets system clock (dom0 or domU). * hwclock - displays and sets cmos clock if run in dom0 (in domU doesn''t work from obvious reasons). Is there any shell command (or xm subcommand) to display xen wallclock?> > > 3. What are the consequences of not selecting xen clocksource (by > > selecting one of hpet, acpi_pm, etc.) for dom0 and for domU? > > Xen itself uses and claims ownership of hardware resources like hpet and > acpi_pm, so they aren''t available to guest kernels. DomU kernels simply > don''t have any hardware access; dom0 is specifically not allowed to > touch hpet (and I think acpi_pm). > > You could force a kernel to use tsc a clocksource, but that will only > work correctly if the system has inherently synchronized tscs, and there > would be discontinuities over suspend/resume/migrate. > > > 4. What are the best practices to achieve clock synchronization? > > ntp is it at present. > > > 5. What has changed recently in timekeeping? What is going to change? Are > > there any improvements/bugfixes added to redhat or suse kernels? Are > > these bugfixes planned to be added upstream? > > As far as I know the pvops kernel is state of the art. It currently > fails to set the Xen wallclock time when asked (not implemented), but > aside from that it should be OK. > > The basic principle is that Xen doesn''t know all that much about time of > day, and doesn''t really care. It offers a token get/set time of day > interface to access the corresponding hardware, but very little beyond > that. Time is really considered to be a local matter for domains to > work out. If they want to synchronize, then ntp is a perfectly > reasonable mechanism for doing that. > > Would you like to see any particular changes? What are your requirements?No, I think, that''s enough from the user''s point of view. Kind regards, -- Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland bartoszl @ ics.p.lodz.pl _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Nov-10 17:59 UTC
RE: [Xen-devel] timekeepeing in XEN - request fo comments
Hi Pasi -- The timekeeping problems I am referring to are only(?) for hvm guests. HVM guests with different OS''s and different versions of a Linux kernel (32-bit vs 64-bit, even kernel updates between EL5uX and EL5uX+1) behave differently when ticks are delivered at imperfect intervals (which happens frequently for Xen and other hypervisors). Xen distros and "virtual appliance" distributors may specify certain settings of timer_mode and certain kernel parameters (e.g. "clock=pit") to ensure time is kept as accurately as possible. There is no "one size fits all" solution unfortunately. For PV guests, note that there have been patches to dom0 (2.6.18-xen) that improve timekeeping in dom0 and this may affect pv guests with independent_wallclock==0. I don''t know if the patches have been applied to pv_ops dom0, or if they are even necessary. In fact, I don''t know if there is a concept of independent_wallclock in pv_ops. Dan> -----Original Message----- > From: Pasi Kärkkäinen [mailto:pasik@iki.fi] > Sent: Monday, November 09, 2009 11:50 AM > To: Dan Magenheimer > Cc: Bartosz Lis; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] timekeepeing in XEN - request fo comments > > > On Mon, Nov 09, 2009 at 08:50:52AM -0800, Dan Magenheimer wrote: > > What kernels are running in your domUs? Are they > > also (all) 2.6.31.5? If your domU''s are older kernels > > (e.g. SLES10 or older, RHEL5 or older) there are > > many issues that cannot be entirely solved by > > Xen or dom0. > > > > Could you tell more about these timekeeping problems with > SLES10/RHEL5 PV kernels? > > -- Pasi > > > > -----Original Message----- > > > From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl] > > > Sent: Monday, November 09, 2009 5:21 AM > > > To: xen-devel@lists.xensource.com > > > Subject: [Xen-devel] timekeepeing in XEN - request fo comments > > > > > > > > > Dear developers, > > > > > > I have some issues with time synchronization among dom0 and > > > domUs. In my case > > > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 > > > for domUs and > > > running ntpd also on domUs. I think, running ntpd on domU is > > > not a good idea, > > > but it works acceptably good as a workaround for my > > > configuration: XEN 3.4.1 + > > > kernel 3.6.31.5 with pv_ops (taken from git last week). When > > > I don''t run ntpd > > > on domU clock slowly drifts. I have choosen xen clocksource > > > in both dom0 and > > > domU. > > > > > > I''m not the only one to have time synchronization issues. > > > Please write some > > > document specifying how time synchronization between dom0 and > > > domU should be > > > achieved. I have a number of special questions, but probably > > > there are more > > > questions that should be aswered: > > > > > > 1. What is the mechanics behind the clocksource named xen? > > > How does it depend > > > on phisical time sources available on different platforms? > > > > > > 2. How gets domU''s clock initialized when domU boots? (I > > > observe that just > > > after domU has started its clock differs from dom0. The > > > difference is about > > > 10s.) > > > > > > 3. What are the consequences of not selecting xen clocksource > > > (by selecting > > > one of hpet, acpi_pm, etc.) for dom0 and for domU? > > > > > > 4. What are the best practices to achieve clock synchronization? > > > > > > 5. What has changed recently in timekeeping? What is going to > > > change? Are > > > there any improvements/bugfixes added to redhat or suse > > > kernels? Are these > > > bugfixes planned to be added upstream? > > > > > > Thank you in advance, > > > > > > -- > > > Bartosz Lis @ Inst. of Information Technology, Technical > > > Univ. of Lodz, Poland > > > bartoszl @ ics.p.lodz.pl > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel