CentOS - Nov 2009 - system time automatically fowards in time and then comes back to normal

Hi friends,

I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX
4.0. The issue I am seeing on the server is sometimes nagios is
showing the below messages in /var/log/messages and as the system time
gets changed some false alarms gets generated. I searched it on the
google but I am not able to find the correct solution. I even posted
on the nagios forum and they asked me to see elsewhere why the server
shitfs so much before looking at nagios.


Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change
of 4398 seconds (forwards in time) has been detected.  Compensating...
Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change
of 4398 seconds (backwards in time) has been detected.  Compensating..


Earlier this server was syncing time through ntp daemon and below is
the ntp.conf file. Now I have set a cronjob which sync the time with
the ntp server every 5 minutes but still the problem persist.

ntp.conf file


restrict default ignore
restrict 127.0.0.1
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
#authenticate yes
keys /etc/ntp/keys
restrict 172.16.6.3 nomodify notrap noquery
server 172.16.6.3
restrict 172.16.6.2 nomodify notrap noquery
server 172.16.6.2

Please see the output of hwclock and date at the same time.
hwclock
Sat 21 Nov 2009 08:19:02 PM IST  -0.496922 seconds

date
Sat Nov 21 20:19:55 IST 2009


Please advice what I need to do to fix this error.


Regards

Ankush

On Sun, Nov 22, 2009 at 12:19 AM, ankush grover <ankushcentos at
gmail.com> wrote:
> Hi friends,
>
> I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX
> 4.0. The issue I am seeing on the server is sometimes nagios is
> showing the below messages in /var/log/messages and as the system time
> gets changed some false alarms gets generated. I searched it on the
> google but I am not able to find the correct solution. I even posted
> on the nagios forum and they asked me to see elsewhere why the server
> shitfs so much before looking at nagios.
>
>
> Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change
> of 4398 seconds (forwards in time) has been detected. ?Compensating...
> Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change
> of 4398 seconds (backwards in time) has been detected. ?Compensating..
>
>
> Earlier this server was syncing time through ntp daemon and below is
> the ntp.conf file. Now I have set a cronjob which sync the time with
> the ntp server every 5 minutes but still the problem persist.
>
> ntp.conf file
>
>
> restrict default ignore
> restrict 127.0.0.1
> driftfile /var/lib/ntp/drift
> broadcastdelay 0.008
> #authenticate yes
> keys /etc/ntp/keys
> restrict 172.16.6.3 nomodify notrap noquery
> server 172.16.6.3
> restrict 172.16.6.2 nomodify notrap noquery
> server 172.16.6.2
>
> Please see the output of hwclock and date at the same time.
> hwclock
> Sat 21 Nov 2009 08:19:02 PM IST ?-0.496922 seconds
>
> date
> Sat Nov 21 20:19:55 IST 2009
>
> Please advice what I need to do to fix this error.
>
> Regards
> Ankush

Because you are using VMware you must become very familiar with how
timekeeping works on computer systems.  VMware has some helpful guides
on this:
   
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
    http://www.vmware.com/pdf/vmware_timekeeping.pdf

ankush grover wrote:
> Hi friends,
> 
> I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX
> 4.0. The issue I am seeing on the server is sometimes nagios is
> showing the below messages in /var/log/messages and as the system time
> gets changed some false alarms gets generated. I searched it on the
> google but I am not able to find the correct solution. I even posted
> on the nagios forum and they asked me to see elsewhere why the server
> shitfs so much before looking at nagios.
Well, VMWare ESX sometime causes time jumps ;)

Possibly your service provider is overbooking cpu time of that ESX host ..

--
Eero

ankush grover schrieb:
> Hi friends,
> 
> I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX
> 4.0. The issue I am seeing on the server is sometimes nagios is
> showing the below messages in /var/log/messages and as the system time
> gets changed some false alarms gets generated. I searched it on the
> google but I am not able to find the correct solution. I even posted
> on the nagios forum and they asked me to see elsewhere why the server
> shitfs so much before looking at nagios.
> 
> 
> Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change
> of 4398 seconds (forwards in time) has been detected.  Compensating...
> Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change
> of 4398 seconds (backwards in time) has been detected.  Compensating..
> 
> 
> Earlier this server was syncing time through ntp daemon and below is
> the ntp.conf file. Now I have set a cronjob which sync the time with
> the ntp server every 5 minutes but still the problem persist.
> 
> ntp.conf file
> 
> 
> restrict default ignore
> restrict 127.0.0.1
> driftfile /var/lib/ntp/drift
> broadcastdelay 0.008
> #authenticate yes
> keys /etc/ntp/keys
> restrict 172.16.6.3 nomodify notrap noquery
> server 172.16.6.3
> restrict 172.16.6.2 nomodify notrap noquery
> server 172.16.6.2
> 
> Please see the output of hwclock and date at the same time.
> hwclock
> Sat 21 Nov 2009 08:19:02 PM IST  -0.496922 seconds
> 
> date
> Sat Nov 21 20:19:55 IST 2009
> 
> 
> Please advice what I need to do to fix this error.
> 
> 
> Regards
> 
> Ankush
Be sure that only 1 mechanism for syncing time is active for your VM:
either NTP client configuration through ntpd running (like you have) OR
through the VMware tools. Both being active will result in trouble.

VMware recommends to use time syncronisation through NTP within the VM.

To debug your time sources in NTP make use of "ntpq" and/or
"ntpdc".

Regards

Alexander

ankush grover wrote:
> Earlier this server was syncing time through ntp daemon and below is
> the ntp.conf file. Now I have set a cronjob which sync the time with
Best not to run NTP inside a ESX VM. I've never gotten NTP to sync
inside of VMware outside of a kernel with VMI enabled (no versions
of RHEL support VMI at this time as far as I know).

What I do for my ~40 ESX/ESXi hosts:
- Have your ESX hosts sync to a good NTP server
- Make sure vmware tools is installed and running correctly
  (/etc/init.d/vmware-tools status)
- Enable time sync for your guest, either via the UI or via
  this command in the guest(I have this command run in cron
  every 5 minutes as I have seen for some reason time sync turn
  itself off:
   /usr/sbin/vmware-guestd --cmd "vmx.set_option synctime 0 1"
- On top of all of that I have another cron set to run ntpdate
  every 5 minutes against a local NTP server:
   /usr/sbin/ntpdate `cat /etc/ntp/step-tickers | grep -v \#`

For providing NTP services themselves, currently I run 3 VMs
at each site with Fedora 8 with VMI enabled for the guest VM
(the kernel in FC8 supports VMI, I suspect newer Fedoras work
fine too I just have no reason to change right now). And I have
these FC8 VMs sync to internet hosts(mainly time.nist.gov) so
my internal ESX and other systems can sync against them(they
are load balanced behind a F5 BigIP).

from FC8 kernel log:
VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0
vmi: registering clock event vmi-timer. mult=9483317 shift=22
Booting paravirtualized kernel on vmi
vmi: registering clock source khz=2260999
Time: vmi-timer clocksource has been installed.

I currently run roughly 400 VMs this way and don't have any
noticeable time-related issues.

nate