Using rsync-2.4.6: Is a times difference supposed to cause a write? Also -t vs -I makes no difference. Below shows the problem, I think: [dmahurin@pc16 /tmp]$ mkdir x y [dmahurin@pc16 /tmp]$ cp /bin/ls x [dmahurin@pc16 /tmp]$ ls -l x/ls -rwxr-xr-x 1 dmahurin users 43024 Nov 13 12:46 x/ls [dmahurin@pc16 /tmp]$ rsync -vrtW x/ y building file list ... done ./ ls ./ wrote 43112 bytes read 36 bytes 86296.00 bytes/sec total size is 43024 speedup is 1.00 [dmahurin@pc16 /tmp]$ touch x/ls [dmahurin@pc16 /tmp]$ rsync -vrtW x/ y building file list ... done ls ./ wrote 43116 bytes read 36 bytes 86304.00 bytes/sec total size is 43024 speedup is 1.00 [dmahurin@pc16 /tmp]$ touch x/ls [dmahurin@pc16 /tmp]$ rsync -vrIW x/ y building file list ... done ls wrote 43116 bytes read 36 bytes 86304.00 bytes/sec total size is 43024 speedup is 1.00
In the example you give, yes, a time difference causes a write. You are using the -W (--whole-file) option, which directs rsync to simply send the file, in its entirety, if there is a discrepency in mtime (ctime too?), or extent. It is used for situations where file access is slow enough that trying to do an incremental update would take more time/resources than simply sending the file... primary example being nfs-mounted filesystems. If you have fast dasd/slow network, you should probably just drop the -W. Tim Conway tim.conway@philips.com 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(nnnnnnnnnnnn, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), ".\n" ' "There are some who call me.... Tim?" Don Mahurin <dmahurin@berkeley.innomedia.com> Sent by: rsync-admin@lists.samba.org 11/13/2001 01:52 PM To: rsync@lists.samba.org cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: times difference causes write Classification: Using rsync-2.4.6: Is a times difference supposed to cause a write? Also -t vs -I makes no difference. Below shows the problem, I think: [dmahurin@pc16 /tmp]$ mkdir x y [dmahurin@pc16 /tmp]$ cp /bin/ls x [dmahurin@pc16 /tmp]$ ls -l x/ls -rwxr-xr-x 1 dmahurin users 43024 Nov 13 12:46 x/ls [dmahurin@pc16 /tmp]$ rsync -vrtW x/ y building file list ... done ./ ls ./ wrote 43112 bytes read 36 bytes 86296.00 bytes/sec total size is 43024 speedup is 1.00 [dmahurin@pc16 /tmp]$ touch x/ls [dmahurin@pc16 /tmp]$ rsync -vrtW x/ y building file list ... done ls ./ wrote 43116 bytes read 36 bytes 86304.00 bytes/sec total size is 43024 speedup is 1.00 [dmahurin@pc16 /tmp]$ touch x/ls [dmahurin@pc16 /tmp]$ rsync -vrIW x/ y building file list ... done ls wrote 43116 bytes read 36 bytes 86304.00 bytes/sec total size is 43024 speedup is 1.00
-W and -C are actually kind of opposite effects... -W means never checksum... if it's different in length or time, send it. -c means ALWAYS checksum, even if time and length are identical. use neither. the default behaviour is to compare date/length, and if they are different, do the rolling checksum. This is done in chunks, then compares the sums, and sends the different blocks, with instructions on where to insert them. If the difference was simply timestamp, and in fact, the data is unchanged, the rolling checksum will sort of say "never mind", and leave the file alone. Unfortunately for your situation, it builds tne new file next to the existing file, no matter what you do, then deletes the existing file and renames the new file to the old name. Every file done by rsync involves a full write. You're going to put a lot of cycles on your flash. I'm thinking rsync might not be the best solution for what you're doing. Flash is not suited to frequently-updated data. Tim Conway tim.conway@philips.com 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(nnnnnnnnnnnn, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), ".\n" ' "There are some who call me.... Tim?" Don Mahurin <dmahurin@berkeley.innomedia.com> Sent by: dmahurin@pc16.berkeley.innomedia.com 11/13/2001 02:36 PM To: Tim Conway/LMT/SC/PHILIPS@AMEC cc: rsync@lists.samba.org Subject: Re: times difference causes write Classification: My first problem is that I am writing to compact flash, so I want the minimal number of writes. My second problem is that the flash is of limited size, so I need some sort of patch rsync that does not keep the old file before writing the new one. My patch now just unlinks the file ahead, and implies -W. So my wish was that a time discrepancy would lead to a checksum, where the files would match. This is not the case, however, as you say. So for now, I must use -c. It's slow, but I know that I get the minimum number of writes. -don tim.conway@philips.com wrote:> In the example you give, yes, a time difference causes a write. You are > using the -W (--whole-file) option, which directs rsync to simply sendthe> file, in its entirety, if there is a discrepency in mtime (ctime too?),or> extent. It is used for situations where file access is slow enough that > trying to do an incremental update would take more time/resources than > simply sending the file... primary example being nfs-mountedfilesystems.> If you have fast dasd/slow network, you should probably just drop the-W.
Don Mahurin [dmahurin@berkeley.innomedia.com] writes:> My second problem is that the flash is of limited size, so I need > some sort of patch rsync that does not keep the old file before > writing the new one. My patch now just unlinks the file ahead, and > implies -W.Sounds reasonable as long as you force the -W.> So my wish was that a time discrepancy would lead to a checksum, where the > files would match. > This is not the case, however, as you say.At least not with -W. In most cases, the time discrepancy would then cause rsync to try to synchronize the file, and during its protocol processing it would determine that it didn't need to send anything, thus the only end result would be adjusting the remote timestamp to match the source. But this requires access to the original source file, so your prior patch (and forcing -W) defeats this as a side effect.> So for now, I must use -c. It's slow, but I know that I get the minimum > number of writes.It definitely sounds like the best match for you. Although -c tends to be used more for cases where files may differ although they appear the same (timestamp/size) than vice versa, it will serve that purpose as well at the expense of some additional I/O and computation. Presumably you could modify your patch so that -c (or some new option) only invoked the checksum if the timestamp differed, since I don't think there's any suitable equivalent currently in rsync. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/