dovecot - Jan 2011 - Fwd: Re: Dotlock dovecot-uidlist errors / NFS / High Load

Stan,

Thanks for the reply.  In our case we have actually already done most of
the work you suggested to no avail.  We had rebuilt two new ntp servers
that sync against two stratum 1 sources, and all our nfs clients,
regardless of using dovecot, sync to those two machines.  You bring up the
difference between bare metal and hypervisor, and we are running these
machines on vmware 4.0.  All the vmware knowledge base articles tend to
push us towards ntp, and since we are using centos 5.5 there are no kernel
modifications that need to be made regarding timing from what we can find. 
I will give the ntpdate option a try and see what happens.

I was also hoping to understand why the uidlist file is the only file that
uses dotlock, or if there was plans to give it the option to use other
locking mechanisms in the future.

Thanks again!

Michael

-------- Original Message --------
Subject: Re: [Dovecot] Dotlock dovecot-uidlist errors / NFS / High Load
Date: Thu, 20 Jan 2011 10:57:24 -0600
From: Stan Hoeppner <stan at hardwarefreak.com>
To: dovecot at dovecot.org

list at airstreamcomm.net put forth on 1/20/2011 8:32 AM:
> Secondly we thought the issues were due to NTP as the time stamps vary
so
> widely, so we rebuilt our NTP servers and found closer stratum 1 source
> clocks to synchronize to hoping it would alleviate the problem but the
> dotlock errors returned after about 12 hours.  We have fcntl locking set
in
> our configuration file, but it is our understanding from look at the
source
> code that this file is locked with dotlock.  
> 
> Any help troubleshooting is appreciated.
>From your description it sounds as if you're ntpd syncing each of the 4
servers
against an external time source, first stratum 2/3 sources, then stratum 1
sources in an attempt to cure this problem.

In a clustered server environment, _always_ run a local physical
box/router ntpd
server (preferably two) that queries a set of external sources, and
services
your internal machine queries.  With RTTs all on your LAN, and using the
same
internal time sources for every query, this clock drift issue should be
eliminated.  Obviously, when you first set this up, stop ntpd and run
ntpdate to
get an initial time sync for each cluster host.

If after setting this up, and we're dealing with bare metal cluster member
servers, then I'd guess you've got a failed/defective clock chip on one
host.
If this is Linux, you can work around that by changing the local time
source.
There are something like 5 options.  Google for "Linux time" or
similar.
Or,
simply replace the hardware--RTC chip, mobo, etc.

If any of these cluster members are virtual machines, regardless of
hypervisor,
I'd recommend disabling using ntpd, and cron'ing ntpdate to run once
every
5
minutes, or once a a minute, whatever it takes to get the times to remain
synced, against your local ntpd server mentioned above.  I got to the
point with
VMWare ESX that I could make any Linux distro VM of 2.4 or 2.6 stay within
one
minute a month before needing a manual ntdate against our local time
source.
The time required to get to that point is a total waste.  Cron'ing ntpdate
as I
mentioned is the quick, reliable way to solve this issue, if you're using
VMs.

-- 
Stan

list at airstreamcomm.net put forth on 1/20/2011 11:09
AM:
> Stan,
> 
> Thanks for the reply.  In our case we have actually already done most of
> the work you suggested to no avail.  We had rebuilt two new ntp servers
> that sync against two stratum 1 sources, and all our nfs clients,
> regardless of using dovecot, sync to those two machines.  You bring up the
> difference between bare metal and hypervisor, and we are running these
> machines on vmware 4.0.  All the vmware knowledge base articles tend to
> push us towards ntp, and since we are using centos 5.5 there are no kernel
> modifications that need to be made regarding timing from what we can find. 
> I will give the ntpdate option a try and see what happens.
What you're supposed to do, and what VMWare recommends, is to run ntpd _only
in
the ESX host_ and _not_ in each guest.  Each guest kernel needs to be running as
few ticks as possible, and preferably the tickless kernel.  As with many/most
distribution Linux kernels, you're going to need to use boot parameters to
get
accurate guest clock time keeping.  According to:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427

You will need the following _kernel boot parameters_ for CentOS 5.5 guests:

For 32bit kernels:  divider=10, clocksource=acpi_pm
For 64bit kernels:  notsc, divider=10

Also, note at the top of the article that you must run a uniprocessor kernel on
a uniprocessor VM, and an SMP kernel on a "virtual SMP" VM. 
Mismatches here
will cause clock drift.

Once you have all of this setup, your guest kernel timekeeping should be fairly
accurate, and you can cron ntpdate once a week or month as necessary in each
guest, depending on your drift.

I discovered all of this in 2006 when attempting to get accurate clocks on SLES9
and Debian 3 guests for kerberos to work properly, before VMWare had thorough
documentation for ESX2/3 timekeeping with Linux guests.  I spent about two weeks
doing the kernel research, experimenting, and figuring all this out on my own.
I posted my results on the VMWare forums, and my work was used in creating later
VMWare timekeeping documentation.

Monkeying with LILO/Grub boot parameters is often beyond the comfort level of
some SAs.  This is why previously I recommended the "short cut" of
simply
cron'ing ntpdate in each guest.  It used to get one "close enough"
without the
other headaches.  It may not still work today.  I've not tried that method
in a
long time.

I cannot stress enough that you _MUST_ disable the ntpd daemon in each Linux
guests.  ntpd is installed by default with every Linux distro.

So, to recap:

1.  Install, configure and enable ntpd in the ESX 4 shell on each physical host
2.  Disable ntpd in each Linux guest
3.  Modify your LILO/Grub command line in each guest as described above
4.  Document drift in each guest for a month and cron ntpdate to compensate

You need to do _all_ of these things in combination.  Doing some and not all
will leave you with unacceptable clock drift.

-- 
Stan

> I was also hoping to understand why the uidlist file is the only file that
> uses dotlock, or if there was plans to give it the option to use other
> locking mechanisms in the future.
> 
> Thanks again!
> 
> Michael
> 
> -------- Original Message --------
> Subject: Re: [Dovecot] Dotlock dovecot-uidlist errors / NFS / High Load
> Date: Thu, 20 Jan 2011 10:57:24 -0600
> From: Stan Hoeppner <stan at hardwarefreak.com>
> To: dovecot at dovecot.org
> 
> list at airstreamcomm.net put forth on 1/20/2011 8:32 AM:
> 
>> Secondly we thought the issues were due to NTP as the time stamps vary
> so
>> widely, so we rebuilt our NTP servers and found closer stratum 1 source
>> clocks to synchronize to hoping it would alleviate the problem but the
>> dotlock errors returned after about 12 hours.  We have fcntl locking
set
> in
>> our configuration file, but it is our understanding from look at the
> source
>> code that this file is locked with dotlock.  
>>
>> Any help troubleshooting is appreciated.
> 
>>From your description it sounds as if you're ntpd syncing each of
the 4
> servers
> against an external time source, first stratum 2/3 sources, then stratum 1
> sources in an attempt to cure this problem.
> 
> In a clustered server environment, _always_ run a local physical
> box/router ntpd
> server (preferably two) that queries a set of external sources, and
> services
> your internal machine queries.  With RTTs all on your LAN, and using the
> same
> internal time sources for every query, this clock drift issue should be
> eliminated.  Obviously, when you first set this up, stop ntpd and run
> ntpdate to
> get an initial time sync for each cluster host.
> 
> If after setting this up, and we're dealing with bare metal cluster
member
> servers, then I'd guess you've got a failed/defective clock chip on
one
> host.
> If this is Linux, you can work around that by changing the local time
> source.
> There are something like 5 options.  Google for "Linux time" or
similar.
> Or,
> simply replace the hardware--RTC chip, mobo, etc.
> 
> If any of these cluster members are virtual machines, regardless of
> hypervisor,
> I'd recommend disabling using ntpd, and cron'ing ntpdate to run
once every
> 5
> minutes, or once a a minute, whatever it takes to get the times to remain
> synced, against your local ntpd server mentioned above.  I got to the
> point with
> VMWare ESX that I could make any Linux distro VM of 2.4 or 2.6 stay within
> one
> minute a month before needing a manual ntdate against our local time
> source.
> The time required to get to that point is a total waste.  Cron'ing
ntpdate
> as I
> mentioned is the quick, reliable way to solve this issue, if you're
using
> VMs.
>