rsync - Apr 2003 - Solution For Rsync and Cygwin Daylight Savings Timezone Problems

Hi,

we had the same problems last year and we use NTFS.
Luckely we switched to Linux this year.

Kind regards,

Bart Coninckx
Network Administrator
CNE, ASE
*************************************
Sita ICT Services
Lilsedijk 19
B-2340 Beerse
Belgium

e-mail: bart.coninckx@sita.be
Tel: + 32 (0) 14 62 28 22
Fax: + 32 (0) 14 62 41 47
*************************************










                    "Wayne Piekarski" <wayne@cs.unisa.edu.au>
                    Sent by:                                          To:    
"jw schultz" <jw@pegasys.ws>, <rsync@lists.samba.org>
                    rsync-bounces+bart.coninckx=watco.be@lists        cc:
                    .samba.org                                        Subject:  
Re: Solution For Rsync and Cygwin Daylight Savings Timezone
                                                                      Problems

                    02/04/2003 13:25





> Thanks for the reminder.  Unfortunately your email was
> rambling so that it was unclear what can actually be done to
> avoid the problem.  Here in the US Daylight savings time
> will take effect this coming Sunday.
Sorry about the rambling :) I wanted to dump out everything I'd learned
because it was quite confusing initially and there was nothing available on
how to fix it.

The hack I made corrects the problem as such, but I'm not sure if it has
other effects that will break other applications. By disabling the daylight
savings tick box Windows will not automatically adjust its time, but who
knows what else? I wouldn't want to say that this is the correct solution
just yet. I've emailed the Cygwin mailing list with the same thing in case
someone spots something there and maybe a possible fix to make the DLLs
deal
with this.
> With the transition between standard time and daylight
> savings time MS-Windows systems are known to appear to
> change the modification time of existing files.  The effect
> of this is to give the appearance that every file has
> changed.  This will cause affected transfers to take
> inordinate amounts of time.  This affects FAT filesystems
> which store times in localtime and not NTFS which uses UTC.
I have not looked into NTFS but I do know it has wierd issues with UTC as
well, probably not to the extent of FAT but I don't think it is as clean as
Unix-ish implementations.
> The impact of this may be minimized by running rsync with
> the --modify-window=3601 command-line option.  This will
> cause rsync to ignore modification time differences of one
> hour will allow rsync jobs to complete in the usual time
> period with a minimal impact on backup integrity.  To get
> back to normal it will be necessary to run rsync with the
> usual modify-window on all files.  This can be done in
> stages.
If you run with this flag, does it just skip the updates on those files, or
does it update the time stamp mtime as well? I'm thinking it just skips
them
and you will have to run with --modify-window until DST ends a few months
later.

regards,
Wayne

--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Hi,

I have been having a number of problems dealing with time zones and
synchronising files using Rsync to my linux machine on Cygwin with Windows
XP. I have read discussions previously on these mailing lists on how there
are problems that occur when there is a change in daylight savings time.

I saw a page by http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html
that discussed how to set the timezone to UTC and then do an ls to see if
the file dates are really the same. If there is a DST problem the suggestion
was to set --modify-window=3601 to set a 1 hour fuzzy factor to prevent the
problem. I had the same problem but didn't want to set this flag since I was
worried I would miss important updates on some files that had changed
shortly after an rsync run.

I did a lot of reading around about this and in this email I have developed
a solution to the problem that I have not seen anywhere else. My solution is
for WinXP with FAT filesystem but I am not sure what the effect is with NTFS
disks. From what I read up, FAT stores its dates in local time and not in
GMT. So it uses an offset from the clock settings to calculate the GMT value
to report to Rsync. The problem is that these DST calculations have a bug in
that if your clock moves over a DST boundary then files created on the
"other side" of the bounday will mysteriously get a 1 hour offset
applied.
Unix avoids this problem by having proper DST calculations but Windows
doesn't do this right. The references below contain some discussions about
how these are calculated.

In Adelaide where I live, we left Daylight Savings Time at the end of March,
and so now my files are all 3600 seconds off relative to my Linux box. The
files were synchronised while DST is in effect. Now DST is off I need to do
something about this, I don't really want to resync 20 Gb of data over SSH.
If you run a 'stat' program over the files in question, the seconds
since
1970 value will be off by 3600. If you change the time zone nothing seems to
change and the 1970 values on the Windows box remain the same. FAT is
supposed to use the time zone offset to calculate the GMT time but it
doesn't seem to take effect. You have to reboot the box! For some reason
parts of the TZ settings affect Windows immediately and others don't kick in
until many minutes later, so I just reboot to be safe - I spent half a day
to discover this, so it is important!

So all we need to do is get a time zone which is one hour away from where we
are now and it should apply a 1 hour offset to all the GMT values and then
this will fix our problem. I live in Adelaide which is GMT+9:30 normally, so
I need GMT+10:30. There is no time zone like this so we need to hack the
time zone files that Windows uses instead. I downloaded a tool from
Microsoft called TZEDIT.EXE that comes with some versions of Windows
resource kit. Using this tool, you can edit the time zone settings for the
current area you are in. I created a new time zone based on my previous
Adelaide one called "Cygwin Hacked" with GMT+10:30 and daylight
savings
disabled.

Now after doing this you *MUST* activate the time zone change properly. Make
the change using TZEDIT, then you need to go to the control panel time/date
tool and click on the time zone and hit ok to make it take effect. You can't
enable the time zone and then edit it, you have to do it in the right order.
Then you must reboot the machine to make it take effect. (It may have a
delay to take effect but I don't know how long it is and I'm too lazy to
wait all day)

Now after the box reboots you will be in GMT+10:30 and when you run Rsync
everything just works like a dream, just like it did a week ago before DST
finished. You do not need to use --modify-window which could potentially
introduce errors and everything just works very well now.

In hindsight this problem could have been simplified if I had done the first
rsync when DST was not on. So if during July (I am in the southern
hemisphere) I did the rsync, then I could have just unticked the "support
DST" box in the clock control panel and could have avoided these problems.
Unfortunately because my backup was done when DST was enabled I have to now
create a time zone that simulates DST. For now this is my fix for the
problem but I am not sure if there are any other effects that I am not aware
of. Everything seems to be working fine.


Here are some references that I collected while researching this problem.
They contain a lot of information about the ways that Windows deals with
timestamps on NTFS and FAT filesystems.

http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html
http://list-archive.xemacs.org/xemacs-nt/199911/msg00130.html
http://sources.redhat.com/ml/cygwin/2002-12/msg01065.html
http://p2p.wrox.com/archive/c_plus_plus_programming/2001-06/53.asp
http://www.codeproject.com/datetime/dstbugs.asp
http://lists.samba.org/pipermail/rsync/2000-July/002491.html
http://support.microsoft.com/default.aspx?scid=kb;[LN];158588
http://www.cygwin.com/ml/cygwin/2000-06/msg00810.html



Here is the source code for a short program I wrote to find out what the
modified time of a file is, it is important that we print out the seconds
since 1970 GMT because this is the value that Rsync uses to determine if it
needs to copy the file over. I use this tool because it shows the seconds
values and is invariant to the use of the TZ environment variable and other
problems that can crop up.

#include <stdio.h>
#include <sys/stat.h>
#include <unistd.h>

void main (int argc, char *argv[])
{
  struct stat buf;

  if (argc != 2)
    exit (1);
  stat (argv[1], &buf);

  printf ("file %s\n", argv[1]);
  printf ("atime = %d  %s", buf.st_atime, ctime (&buf.st_atime));
  printf ("mtime = %d  %s", buf.st_mtime, ctime (&buf.st_mtime));
  printf ("ctime = %d  %s", buf.st_ctime, ctime (&buf.st_ctime));
}



Sorry for writing such a long email. This has been bugging me for ages and
each time DST kicks in it throws my backup system into chaos. I found pieces
of information all over the web but nothing really proposed a proper
solution to the problem so I wanted to put this all in one place so people
can find it in the future.

regards,
Wayne

----------------------------------------------------------------------------
Wayne Piekarski - PhD Researcher / Lecturer      pho:        +61-8-8302-3669
                                                 fax:        +61-8-8302-3381
Tinmith Project - Wearable Computer Lab          mob:           0407-395-889
Advanced Computing Research Centre               ema:  wayne@cs.unisa.edu.au
University of South Australia                    www: http://www.tinmith.net