Hi In the past I''ve always assumed that if you got the time in dom0 right, then all the domu domains would have the correct time... something is broken in Xen 3.1 because now the domu time can be quite unrelated - and of course it''s not possible to reset it back as you can''t change the time in a domu... This is causing havoc with kerberos.. it''ll work fine for ages then suddenly one or other of the machines will stop authenticating. dom0: goliath:/home/tmh# date -u Wed May 30 11:46:45 UTC 2007 domu: wolf359:/var/log/samba# date -u Wed May 30 09:23:08 UTC 2007 I wonder if this is related to ''time is going backwards'' that seem to scroll constantly up the console? From reading around I''d got the impression that this error was normal. Tony _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Tony Hoyle > Sent: 30 May 2007 12:53 > To: xen-users@lists.xensource.com > Subject: [Xen-users] Xen 3.1 clock problems > > Hi > > In the past I''ve always assumed that if you got the time in > dom0 right, > then all the domu domains would have the correct time... something is > broken in Xen 3.1 because now the domu time can be quite > unrelated - and > of course it''s not possible to reset it back as you can''t change the > time in a domu... > > This is causing havoc with kerberos.. it''ll work fine for ages then > suddenly one or other of the machines will stop authenticating. > > dom0: > goliath:/home/tmh# date -u > Wed May 30 11:46:45 UTC 2007 > > domu: > wolf359:/var/log/samba# date -u > Wed May 30 09:23:08 UTC 2007Is this a HVM or PV domain? I believe in HVM, the clock isn''t synced, and you''ll need to use a NTP server or similar to sync your time. -- Mats> > I wonder if this is related to ''time is going backwards'' that seem to > scroll constantly up the console? From reading around I''d got the > impression that this error was normal. > > Tony > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats wrote:>> dom0: >> goliath:/home/tmh# date -u >> Wed May 30 11:46:45 UTC 2007 >> >> domu: >> wolf359:/var/log/samba# date -u >> Wed May 30 09:23:08 UTC 2007 > > Is this a HVM or PV domain? I believe in HVM, the clock isn''t synced, > and you''ll need to use a NTP server or similar to sync your time.It''s a PV domain.. even runs the same kernel. Tony _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Tony Hoyle > Sent: 30 May 2007 15:29 > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen 3.1 clock problems > > Petersson, Mats wrote: > >> dom0: > >> goliath:/home/tmh# date -u > >> Wed May 30 11:46:45 UTC 2007 > >> > >> domu: > >> wolf359:/var/log/samba# date -u > >> Wed May 30 09:23:08 UTC 2007 > > > > Is this a HVM or PV domain? I believe in HVM, the clock > isn''t synced, > > and you''ll need to use a NTP server or similar to sync your time. > > It''s a PV domain.. even runs the same kernel.And I take you don''t have the "independent wallclock" set? Sorry for not being more help here... -- Mats> > Tony > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have this problem with PV domains as well. I always get a "your clock sucks" answer, but I''ve never had a system this worked right on. However, it is possible to change the time in a DomU. You have to set a sysctl value. From sysctl.conf on my systems: # Allow NTP To Sync Xen Clock xen.independent_wallclock = 1 I combine this with multicast NTP to make my Xen instances all sync to the Dom0s on my systems (which in turn sync to places more reliable). On May 30, 2007, at 5:09 AM, Petersson, Mats wrote:> > >> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com >> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Tony >> Hoyle >> Sent: 30 May 2007 12:53 >> To: xen-users@lists.xensource.com >> Subject: [Xen-users] Xen 3.1 clock problems >> >> Hi >> >> In the past I''ve always assumed that if you got the time in >> dom0 right, >> then all the domu domains would have the correct time... something is >> broken in Xen 3.1 because now the domu time can be quite >> unrelated - and >> of course it''s not possible to reset it back as you can''t change the >> time in a domu... >> >> This is causing havoc with kerberos.. it''ll work fine for ages then >> suddenly one or other of the machines will stop authenticating. >> >> dom0: >> goliath:/home/tmh# date -u >> Wed May 30 11:46:45 UTC 2007 >> >> domu: >> wolf359:/var/log/samba# date -u >> Wed May 30 09:23:08 UTC 2007 > > Is this a HVM or PV domain? I believe in HVM, the clock isn''t synced, > and you''ll need to use a NTP server or similar to sync your time. > > -- > Mats >> >> I wonder if this is related to ''time is going backwards'' that seem to >> scroll constantly up the console? From reading around I''d got the >> impression that this error was normal. >> >> Tony >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> >> > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Jayson Vantuyl Systems Architect Engine Yard jvantuyl@engineyard.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats wrote:> And I take you don''t have the "independent wallclock" set? >Nope.. if only it was that easy! Rebooting the domu seems to fix it.. I''ll see if I can pin down exactly when it happens (it''s not gradual, it''s a sudden jump to being completely wrong). Difficult because it''s random. Tony _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats wrote:> Sorry for not being more help here... >I''ve found on the domu I last rebooted it has a large number of the following: May 30 01:43:42 wolf359 kernel: BUG: soft lockup detected on CPU#1! May 30 01:43:42 wolf359 kernel: May 30 01:43:42 wolf359 kernel: Call Trace: May 30 01:43:42 wolf359 kernel: <IRQ> [<ffffffff8029ea86>] softlockup_tick+0xdb/0xed May 30 01:43:42 wolf359 kernel: [<ffffffff8027d9ae>] run_rebalance_domains+0x14a/0x3a7 May 30 01:43:42 wolf359 kernel: [<ffffffff802663f5>] timer_interrupt+0x39c/0x3fa May 30 01:43:42 wolf359 kernel: [<ffffffff8021129c>] handle_IRQ_event+0x29/0x5b May 30 01:43:42 wolf359 kernel: [<ffffffff802a02ee>] handle_level_irq+0xae/0x100 May 30 01:43:42 wolf359 kernel: [<ffffffff8025b21c>] call_softirq+0x1c/0x28 May 30 01:43:42 wolf359 kernel: [<ffffffff80264c8b>] do_IRQ+0x64/0xcd May 30 01:43:42 wolf359 kernel: [<ffffffff803646f0>] evtchn_do_upcall+0x86/0xe0 May 30 01:43:42 wolf359 kernel: [<ffffffff8025acfe>] do_hypervisor_callback+0x1e/0x30 May 30 01:43:42 wolf359 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 May 30 01:43:42 wolf359 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 May 30 01:43:42 wolf359 kernel: [<ffffffff80265912>] raw_safe_halt+0x7f/0xa3 May 30 01:43:42 wolf359 kernel: [<ffffffff80262c10>] xen_idle+0x34/0x4d May 30 01:43:42 wolf359 kernel: [<ffffffff80247858>] cpu_idle+0x89/0xac May 30 01:43:42 wolf359 kernel: Since rebooting this hasn''t repeated and the time is back to normal. The first one of these is possibly the source of the problem: May 29 19:13:39 wolf359 dhclient: DHCPREQUEST on eth0 to 81.187.26.102 port 67 May 29 19:13:39 wolf359 dhclient: DHCPACK from 81.187.26.102 May 29 19:13:40 wolf359 dhclient: bound to 192.168.44.12 -- renewal in 300 seconds. May 29 19:14:31 wolf359 kernel: Breaking affinity for irq 260 May 29 19:14:31 wolf359 kernel: Breaking affinity for irq 261 May 29 19:14:31 wolf359 kernel: Breaking affinity for irq 262 May 29 16:53:23 wolf359 kernel: netfront: device eth0 has flipping receive path. May 29 16:53:23 wolf359 kernel: netfront: device eth0 has flipping receive path. May 29 16:53:23 wolf359 kernel: Initializing CPU#1 May 29 17:02:34 wolf359 kernel: BUG: soft lockup detected on CPU#1! May 29 17:02:34 wolf359 kernel: May 29 17:02:34 wolf359 kernel: Call Trace: May 29 17:02:34 wolf359 kernel: <IRQ> [<ffffffff8029ea86>] softlockup_tick+0xdb/0xed May 29 17:02:34 wolf359 kernel: [<ffffffff8027d9ae>] run_rebalance_domains+0x14a/0x3a7 May 29 17:02:34 wolf359 kernel: [<ffffffff802663f5>] timer_interrupt+0x39c/0x3fa May 29 17:02:34 wolf359 kernel: [<ffffffff8021129c>] handle_IRQ_event+0x29/0x5b May 29 17:02:34 wolf359 kernel: [<ffffffff802a02ee>] handle_level_irq+0xae/0x100 May 29 17:02:34 wolf359 kernel: [<ffffffff8025b21c>] call_softirq+0x1c/0x28 May 29 17:02:34 wolf359 kernel: [<ffffffff80264c8b>] do_IRQ+0x64/0xcd May 29 17:02:34 wolf359 kernel: [<ffffffff8025af6f>] child_rip+0x11/0x12 May 29 17:02:34 wolf359 kernel: [<ffffffff803646f0>] evtchn_do_upcall+0x86/0xe0 May 29 17:02:34 wolf359 kernel: [<ffffffff8025acfe>] do_hypervisor_callback+0x1e/0x30 May 29 17:02:34 wolf359 kernel: <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 May 29 17:02:34 wolf359 kernel: [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 May 29 17:02:34 wolf359 kernel: [<ffffffff80265912>] raw_safe_halt+0x7f/0xa3 May 29 17:02:34 wolf359 kernel: [<ffffffff80262c10>] xen_idle+0x34/0x4d May 29 17:02:34 wolf359 kernel: [<ffffffff80247858>] cpu_idle+0x89/0xac May 29 17:02:34 wolf359 kernel: May 29 17:17:01 wolf359 /USR/SBIN/CRON[2038]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) May 29 17:17:01 wolf359 kernel: BUG: soft lockup detected on CPU#1! Tony _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users