I just confirmed that data corruption can occasionally occur with rsync 2.5.0 when the -z option is used. My command was the following: rsync -vaz --partial --block-size=65536 --checksum remote:/path/ /localdir The files consisted of a year's worth of email (262MB), broken into one file for each day. At least this doesn't seem to occur silently; I get the message "ERROR: file corruption in foo. File changed during transfer?". This happened for 5 files out of 358 total. Repeated rsyncs yield the same message for the same files, and manual md5sums confirm that the files mentioned in the messages differ. When I repeated the rsync without the -z option, the transfers completed normally and the md5sums then matched. Each end is running Debian Potato with zlib1g and zlib1g-dev version 1.1.3-5. Phil
On 12 Dec 2001, rsync@ka9q.net wrote:> I just confirmed that data corruption can occasionally occur with > rsync 2.5.0 when the -z option is used.Please keep the two directories that caused the problems, if they have not already been overwritten. Are you sure you're running 2.5.0 at both ends? Are you using rsh or ssh? David: I suspect this might be because the "bit length overflow" message is being emitted by one end, getting into stderr, and therefore corrupting the stream. If this is true, you should be able to avoid it by upgrading to 2.5.1pre. Thnks, -- Martin
Does rsync support a user-defined compression level option (similar to gzip)? I have read through the mailing list archive and could not find any specific thread which addresses this question. I would like to specify a medium level compression option (use more network speed and less CPU overhead). Using gzip, I would use the -6 or -7 level option instead of the default -9. I am trying to find the balance between CPU overhead and network speed. Any ideas? Thanks, -Ron
rsync@ka9q.net [rsync@ka9q.net] writes:> After I sent my note, I ran some more experiments and found the > problem goes away if I use the default checksum blocksize. So the > problem occurs *only* if I use a large blocksize (65536) *and* enable > compression.Should have read ahead - this is probably the problem I was recalling in my last reply. There was some reason I tended to keep my variable block sizes that my scripts were picking <= 32K or so. If that's it, then I doubt it's the "bit length overflow" issue since I was running into this back with a modified 2.4.3. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/
I just ran into the same corruption problem with 2.5.1pre3. Again, it only happens when I use large checksum block sizes (65536) *and* request compression (-z). Because the corrupted file has a correct size and timestamp, I have to re-run rsync with the --checksum option (and with either a standard block size or no compression) to force the file to be correctly updated. This time I did a diff between the corrupted file and the correct one. It was a mail list archive in mbox format. Interesting how the same error is made in several places, and only in one byte. bash-2.03$ diff cryptography.bad cryptography 1476c1476 < From owner-cryptography-outgoing@wasabisystems.com Tue DeF 11 09:23:27 2001 ---> From owner-cryptography-outgoing@wasabisystems.com Tue Dec 11 09:23:27 20011731c1731 < From owner-cryptography-outgoing@wasabisystems.com Tue DeF 11 10:32:26 2001 ---> From owner-cryptography-outgoing@wasabisystems.com Tue Dec 11 10:32:26 20011807c1807 < From owner-cryptography-outgoing@wasabisystems.com Tue DeF 11 15:07:53 2001 ---> From owner-cryptography-outgoing@wasabisystems.com Tue Dec 11 15:07:53 20011930c1930 < From owner-cryptography-outgoing@wasabisystems.com Wed DeF 12 18:59:52 2001 ---> From owner-cryptography-outgoing@wasabisystems.com Wed Dec 12 18:59:52 2001Phil
On 12 Dec 2001, rsync@ka9q.net wrote:> I just ran into the same corruption problem with 2.5.1pre3. Again, it > only happens when I use large checksum block sizes (65536) *and* > request compression (-z).My apologies, this fix went in after 2.5.1pre3. Would you please try either using CVS HEAD, or ./configure --disable-debug in 2.5.0 or 2.5.1pre3. Thanks -- Martin
On Wed, Dec 12, 2001 at 08:26:31PM -0500, Ron Kelley wrote:> Does rsync support a user-defined compression level option (similar to > gzip)? I have read through the mailing list archive and could not find any > specific thread which addresses this question. > > I would like to specify a medium level compression option (use more network > speed and less CPU overhead). Using gzip, I would use the -6 or -7 level > option instead of the default -9. I am trying to find the balance between > CPU overhead and network speed. > > Any ideas?The compression library that rsync uses has that capability but rsync does not currently have any user interface to change the compress level, sorry. Maybe if somebody submitted a nice clean patch Martin would take it. - Dave Dykstra