rsync - Oct 2002 - important caveat with Rsync on NT and dayligt savings time

Hi,

just a small warning about something that has, like we experienced just
last weekend, great consequences.
We rsync more than 20 Netware servers through a mapping on an NT machine to
central Rsync destination servers (also NT). Last weekend our clock changed
to Daylight Savings Time. It appears that NT changes the file stamps on
NTFS volumes accordingly (Q129574 on their suppor site). This means that
from Sunday 00:00 AM on, all files on the dest. serves had a timestamp of
one hour less than their original counterparts (Netware, which leaves the
timestamps alone). The result is that the syncs have been busy for the last
2 days trying to catch up and hence we don't have a backup for the moment.
Microsoft calls it a "feature", I call it poor programming.


You've been warned.



Bart Coninckx
Network Administrator
CNE, ASE
*************************************
Watco ICT Services
Lilsedijk 19
B-2340 Beerse
Belgium

e-mail: bart.coninckx@sita.be
Tel: + 32 (0) 14 60 99 42
Fax: + 32 (0) 14 62 41 47
*************************************

Howdy...


Fortunately you posted the KB article number, and having read the article, I
see you have a bit of a misunderstanding of what is going on.

First off, NT/2000 uses a offset from GMT, which does not observe Daylight
Savings Time (DST), for storing the time in the Event Log, and NTFS volumes.
This also affects Windows 95, Windows 98, Windows ME, and Windows XP
platforms viewing the same data remotely.

On the other side of file systems, the FAT16, and FAT32 filesystems use the
number of seconds that have elapsed since January 1, 1980 to store the time,
rather than a time offset from GMT.

In case you want to avoid this happening in the future, at least until a
solution can be developed, implemented, and tested, you can turn off DST.
This will mean that for one-half of the year, your clock will be out-of-sync
by 1 hour, but it may be better to be off by 1 hour to avoid this problem in
the future.

I appreciate you warning others about this, as it has identified a cosmetic
bug that I was completely unaware of.  Necessity is the mother of invention,
and this nuance will be addressed in my implementation of rsync as a service
for Win32.


Wolfe

No matter what my understanding's like: the timestamps on the files changed
after last weekend, resulting in a mismatch between source and destination
files. I've changed the clock to a timezone one hour ahead to compensate,
did a "F5" in a particular folder and miraculously the timestamps
followed
along with one hour. Unfortunately, it was already too late for a lot of
files ...


Rgds,

Bart Coninckx
Network Administrator
CNE, ASE
*************************************
Watco ICT Services
Lilsedijk 19
B-2340 Beerse
Belgium

e-mail: bart.coninckx@sita.be
Tel: + 32 (0) 14 60 99 42
Fax: + 32 (0) 14 62 41 47
*************************************







                    <wolfe-mcse@ev1.n
                    et>                      To:     <rsync@samba.org>
                    Sent by:                 cc:    
<bart.coninckx@sita.be>
                    rsync-admin@lists        Subject:     RE: important caveat
with Rsync on NT and dayligt savings time
                    .samba.org


                    10/29/2002 20:25





Howdy...


Fortunately you posted the KB article number, and having read the article,
I
see you have a bit of a misunderstanding of what is going on.

First off, NT/2000 uses a offset from GMT, which does not observe Daylight
Savings Time (DST), for storing the time in the Event Log, and NTFS
volumes.
This also affects Windows 95, Windows 98, Windows ME, and Windows XP
platforms viewing the same data remotely.

On the other side of file systems, the FAT16, and FAT32 filesystems use the
number of seconds that have elapsed since January 1, 1980 to store the
time,
rather than a time offset from GMT.

In case you want to avoid this happening in the future, at least until a
solution can be developed, implemented, and tested, you can turn off DST.
This will mean that for one-half of the year, your clock will be
out-of-sync
by 1 hour, but it may be better to be off by 1 hour to avoid this problem
in
the future.

I appreciate you warning others about this, as it has identified a cosmetic
bug that I was completely unaware of.  Necessity is the mother of
invention,
and this nuance will be addressed in my implementation of rsync as a
service
for Win32.


Wolfe

--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

How do you restrict rsync transfers to only modules in the configuration 
file?
It seems like even though I have a module configured, users can transfer 
files
that they had permission to which is not under the directory of the module.

I.E.
modulename
    path=/web
    ...
    ...
    ...

Users can get /etc/passwd from this machine. How do I restrict that.


Thanks,
AMS :-)

"WorldSecure Server <safeway.com>" made the following
 annotations on 10/29/02 13:13:21
------------------------------------------------------------------------------
Warning: 
All e-mail sent to this address will be received by the Safeway corporate e-mail
system, and is subject to archival and review by someone other than the
recipient.  This e-mail may contain information proprietary to Safeway and is
intended only for the use of the intended recipient(s).  If the reader of this
message is not the intended recipient(s), you are notified that you have
received this message in error and that any review, dissemination, distribution
or copying of this message is strictly prohibited.  If you have received this
message in error, please notify the sender immediately.
  

=============================================================================--------------
next part --------------
Skipped content of type multipart/related

I think this "day light savings" time stamp issue also affects 
Win98 as well - at least on a Win98 PC I use.  
Rsync insists on updating all the files fully.  
Shouldn't it just be resetting the times on the files 
if the files are the same size - or am I missing 
something here?

Lachlan.

>Your description is as confusing as the KB article (at least
>to me).  Part of the problem with the KB article is that it
>focuses on localtime displays rather than internals.
>
>.From what i can tell FAT stores timestamps in localtime.
>NTFS uses a GMT based timestamp.
>
>I can see one of two places where we get bit.  As a matter
>of curiosity i'd like to know which.
>
>1. When the FAT timestamps are converted to GMT it is done
>   based on the current TZ offset rather than the TZ offset
>   in effect when the timestamp was made.
>
>2. The NT equivalent of stat(2) is converting the NTFS
>   timestamps to localtime and then the cygwin libs convert
>   it back to GMT.
>
>It would appear that the problem could be avoided either by
>using NTFS or FAT filesystems.  Yes?   It certainly sounds
>filesystem dependant.
>
>I'd guess that the correct place to fix this is in the
>cygwin libs.  They could be tweaked to recognize the TZ
>calculation error of the NT libs.  There might need to be a
>test for filesystem type (if possible) or a switch to
>determine behavior otherwise what fixes some will break
>others.
-----------------------
Lachlan M. D. Cranswick

Collaborative Computational Project No 14 (CCP14)
    for Single Crystal and Powder Diffraction
  Birkbeck University of London and Daresbury Synchrotron Laboratory 
Postal Address: CCP14 - School of Crystallography,
                Birkbeck College,
                Malet Street, Bloomsbury,
                WC1E 7HX, London,  UK
Tel: (+44) 020 7631 6850   Fax: (+44) 020 7631 6803
E-mail: l.m.d.cranswick@dl.ac.uk   Room: B091
WWW: http://www.ccp14.ac.uk/

Howdy...


The short answer is the DST issue affects volumes using the NTFS filesystem.

The long answer is contained in the Windows KB article that was posted.

I am looking at a solution for this problem, along with a few other issues,
but the solution is a few months off from now.


Wolfe

If I understand the manpages correctly, if you use -t, the criterium for
syncing files is the difference in timestamp, not in size.
There are plenty of situations where the size of a file stays identical,
while it's contents has changed.
If you omit -t, all files are synced.


Rgds,


Bart Coninckx
Network Administrator
CNE, ASE
*************************************
Watco ICT Services
Lilsedijk 19
B-2340 Beerse
Belgium

e-mail: bart.coninckx@sita.be
Tel: + 32 (0) 14 60 99 42
Fax: + 32 (0) 14 62 41 47
*************************************

========================== Disclaimer =================================The
information in this email is confidential, and is intended solely for
the addressee(s). If you are not the intended recipient of this email
please let us know by reply and then delete it from your system;
you should not copy this message or disclose its contents to anyone,
not even by forwarding it.
Due to the integrity risk of sending emails over the Internet,
Sita ICT will accept no liability for any comments and/or attachments
contained within this email.
========================== Disclaimer =================================





                    Lachlan Cranswick
                    <l.m.d.cranswick@        To:     rsync@samba.org
                    dl.ac.uk>                cc:
                    Sent by:                 Subject:     Re: important caveat
with Rsync on NT and dayligt savings time
                    rsync-admin@lists
                    .samba.org


                    10/29/2002 21:35






I think this "day light savings" time stamp issue also affects
Win98 as well - at least on a Win98 PC I use.
Rsync insists on updating all the files fully.
Shouldn't it just be resetting the times on the files
if the files are the same size - or am I missing
something here?

Lachlan.

>Your description is as confusing as the KB article (at least
>to me).  Part of the problem with the KB article is that it
>focuses on localtime displays rather than internals.
>
>.From what i can tell FAT stores timestamps in localtime.
>NTFS uses a GMT based timestamp.
>
>I can see one of two places where we get bit.  As a matter
>of curiosity i'd like to know which.
>
>1. When the FAT timestamps are converted to GMT it is done
>   based on the current TZ offset rather than the TZ offset
>   in effect when the timestamp was made.
>
>2. The NT equivalent of stat(2) is converting the NTFS
>   timestamps to localtime and then the cygwin libs convert
>   it back to GMT.
>
>It would appear that the problem could be avoided either by
>using NTFS or FAT filesystems.  Yes?   It certainly sounds
>filesystem dependant.
>
>I'd guess that the correct place to fix this is in the
>cygwin libs.  They could be tweaked to recognize the TZ
>calculation error of the NT libs.  There might need to be a
>test for filesystem type (if possible) or a switch to
>determine behavior otherwise what fixes some will break
>others.
-----------------------
Lachlan M. D. Cranswick

Collaborative Computational Project No 14 (CCP14)
    for Single Crystal and Powder Diffraction
  Birkbeck University of London and Daresbury Synchrotron Laboratory
Postal Address: CCP14 - School of Crystallography,
                Birkbeck College,
                Malet Street, Bloomsbury,
                WC1E 7HX, London,  UK
Tel: (+44) 020 7631 6850   Fax: (+44) 020 7631 6803
E-mail: l.m.d.cranswick@dl.ac.uk   Room: B091
WWW: http://www.ccp14.ac.uk/

--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Howdy...


I have a printout of the manpage, and the -t states to transfer the
modificaiton time along with the file.  This is to place the same modication
time on the file when it is transferred, rather than the current time.

The easiest way to get around this issue is to use the -c or --checksum
option.  This instructs rsync to EXPLICITLY check the checksum against both,
and if they are the same, teh file is skipped.  Note, this DOES increase the
transfer time, as the 128-bit MD4 checksum is used.


Wolfe

Hi Wolfe,
This problem definitely (ouch) also occurs for FAT32 filesystems.

I discovered the one-hour problem rsyncing to backup files on my WinXP/cygwin
system to a Linux box in a different time zone, using ssh and rsync 2.5.5.  The
files on my WinXP system were on a FAT32 partition.  Both systems have the
correct "date -u" UTC time, checked from a bash shell.  So it seems to
me that cygwin "knows" the correct local time, but somehow
cygwin/Windows are screwing up the file times.  Unfortunately I don't know
have the expertise to pin down where the error is occurring in any reasonable
timeframe.

I'd be happy to help fix the problem, but really just don't know where
to look.  Should we perhaps contact the cygwin maintainers?




?       From: wolfe-mcse 
?       Subject: RE: important caveat with Rsync on NT and dayligt savings time 
?       Date: Tue, 29 Oct 2002 12:50:50 -0800 

Howdy...


The short answer is the DST issue affects volumes using the NTFS filesystem.

The long answer is contained in the Windows KB article that was posted.

I am looking at a solution for this problem, along with a few other issues,
but the solution is a few months off from now.


Wolfe

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html


--
Robert Scholten, University of Melbourne, Australia
On sabbatical at Eindhoven University of Technology
Physics N-laag g2.02
P.O. Box 513, 5600 MB Eindhoven
The Netherlands

Tel:    +31 40 247 4242
Mobile: +31 611 430 467
Fax:    +31 40 245 6050

email: r.scholten@physics.unimelb.edu.au
http://www.ph.unimelb.edu.au/~scholten