Setup: RedHat/Centos/Sci. Linux 5 update 3, Dom0: kernel-xen-2.6.18-164.10.1.el5xen 64-bit (on Dell Poweredge 2950 dual quad-core). DomU: kernel-xen-2.6.18-164.10.1.el5xen 32-bit, 1 vcpu 6 domU''s per dom0. We have also seen the same problem with 2.6.18-164.9.1 and 2.6.18-164.6.1 kernels on this branch. Symptom: On 32-bit domU only (we have never seen 64-bit domU be affected), we observe that the clock gets set forward by a random amount of time, between 40-80 minutes. We originally thought this was due to rogue ntp broadcasts because the ntp broadcastclient was disabled, but we have now disabled ntpd on all domU and the problem persists. During the 40-80 minutes it is possible to log into the domU via the Xen console, it appears to be otherwise functional although the clock is stuck at the forward time. The daemons (which are Axis-based web services) continue to function and service requests, the node remains pingable, there is no loss of network. Once the clock on the dom0 catches up to the time on the domU, then the clock advances normally again on the domU as though nothing had happened. It is happening on any given domU on the average of every 2-3 weeks, although they do appear to happen in time clusters. i.e. 3-4 domU''s report the clock skew within the span of a day or two, and then nothing for a couple of weeks, and then we get another round and so forth. Only one of the several domU''s on the machine is affected at any given time. We see this on two different pieces of hardware that are identically configured. There is nothing in dmesg, nothing in /var/log/messages or related files,nothing in the xend logs. On the domU /var/log/messages and related files have a total gap during this time. The logs of the daemons continue to get written at a constant time stamp, although I have seen cases where the system clock is 19:53, a daemon log gets rotated at 19:40 and the time stamp on the rotated log shows 19:40. Any help? Has anyone else seen anything like this? Thanks Steve Timm -- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jan 20, 2010 at 09:46:43AM -0600, Steven Timm wrote:> > Setup: RedHat/Centos/Sci. Linux 5 update 3, > Dom0: kernel-xen-2.6.18-164.10.1.el5xen 64-bit > (on Dell Poweredge 2950 dual quad-core). > DomU: kernel-xen-2.6.18-164.10.1.el5xen 32-bit, 1 vcpu > 6 domU''s per dom0. > > We have also seen the same problem with 2.6.18-164.9.1 and 2.6.18-164.6.1 > kernels on this branch. > > Symptom: On 32-bit domU only (we have never seen 64-bit domU be affected), > we observe that the clock gets set forward by a random amount of time, > between 40-80 minutes. We originally thought this was due to > rogue ntp broadcasts because the ntp broadcastclient was disabled, > but we have now disabled ntpd on all domU and the problem persists. > During the 40-80 minutes it is possible to log into the domU via > the Xen console, it appears to be otherwise functional although > the clock is stuck at the forward time. The daemons > (which are Axis-based web services) continue to function and service > requests, the node remains pingable, there is no loss of network. > Once the clock on the dom0 catches up to the time on the domU, > then the clock advances normally again on the domU as though nothing > had happened. > > It is happening on any given domU on the average of every 2-3 weeks, > although they do appear to happen in time clusters. i.e. 3-4 domU''s > report the clock skew within the span of a day or two, and then > nothing for a couple of weeks, and then we get another round and so forth. > > Only one of the several domU''s on the machine is affected at any > given time. We see this on two different pieces of hardware that > are identically configured. > > There is nothing in dmesg, nothing in /var/log/messages or related > files,nothing in the xend logs. On the domU /var/log/messages > and related files have a total gap during this time. The logs of the > daemons continue to get written at a constant time stamp, although > I have seen cases where the system clock is 19:53, a daemon log > gets rotated at 19:40 and the time stamp on the rotated log shows > 19:40. > > Any help? Has anyone else seen anything like this? >Nope.. I haven''t seen that. Did you search Redhat bugzilla about that? That''s good source for RHEL5 Xen related problems. Also does it still happen if you downgrade domU kernel to, say, 2.6.18-128 ? aka try to figure out if it''s a problem in domU or dom0. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users