Peter Chun
2003-Jul-24 09:52 UTC
bug report: different files are not rsynced, identical files rsynced
Hi, I have only just subscribe to the list. ( only to send this bug report ) Running rsync version 2.5.6 protocol version 26 ( on Solaris 8 sparc ) on both hosts. I have 1 file I wish to sync to a remote machine the md5 checksum is host1: MD5 (030722.mj) = 020397fde83c2e20464b6642c018ce6e host2: MD5 (030722.mj) = 35fcffc896c65c8ec861385b4edb81ac For brevity .. set RSYNC="--archive --verbose --stats --recursive --times --perms --owner --group --links --rsync-path=/usr/local/bin/rsync" rsync $RSYNC --dry-run /path/030722.mj host2:/path/030722.mj building file list ... done Number of files: 1 Number of files transferred: 0 Total file size: 9416161 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 55 Total bytes written: 67 Total bytes read: 20 wrote 67 bytes read 20 bytes 58.00 bytes/sec total size is 9416161 speedup is 108231.74 The file is not listed. There are other files in that path ( 030701.mj to 030722.mj ) All files from 030701 to 030721 have identical md5 checksums rsync $RSYNC --dry-run /path/ host2:/path/ building file list ... done Number of files: 23 Number of files transferred: 0 Total file size: 356814662 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 387 Total bytes written: 399 Total bytes read: 20 wrote 399 bytes read 20 bytes 838.00 bytes/sec total size is 356814662 speedup is 851586.31 I then copy the entire directory to /tmp/path/ rsync $RSYNC --dry-run /path/ host2:/tmp/path/ building file list ... done Number of files: 23 Number of files transferred: 0 Total file size: 356814662 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 387 Total bytes written: 399 Total bytes read: 20 wrote 399 bytes read 20 bytes 279.33 bytes/sec total size is 356814662 speedup is 851586.31 If I make a copy of the file on host1 to /tmp rsync $RSYNC --dry-run /tmp/030722.mj host2:/path/030722.mj building file list ... done 030722.mj Number of files: 1 Number of files transferred: 1 Total file size: 9416161 bytes Total transferred file size: 9416161 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 50 Total bytes written: 66 Total bytes read: 24 wrote 66 bytes read 24 bytes 60.00 bytes/sec total size is 9416161 speedup is 104624.10 MD5 (/path/030722.mj) = 020397fde83c2e20464b6642c018ce6e MD5 (/tmp/030722.mj) = 020397fde83c2e20464b6642c018ce6e Also I have copied the directory to host1 ( remembering that all 030701 to 030721 files have identical md5 checksums ) rsync $RSYNC --dry-run /path/0307/ /path2/0307/ building file list ... done 030701.mj 030702.mj 030703.mj 030704.mj 030705.mj 030706.mj 030707.mj 030708.mj 030709.mj 030710.mj 030711.mj 030712.mj 030713.mj 030714.mj 030715.mj 030716.mj 030717.mj 030718.mj 030719.mj 030720.mj 030721.mj 030722.mj Number of files: 23 Number of files transferred: 22 Total file size: 356814662 bytes Total transferred file size: 356814662 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 387 Total bytes written: 487 Total bytes read: 108 wrote 487 bytes read 108 bytes 1190.00 bytes/sec total size is 356814662 speedup is 599688.51 Note: there are 23 files in the dir, a --delete will remove 030723.mj The --check-sum flag is ( as stated in the docs ) extremely slow. rsyncing without already takes about 3hrs timing tests show that it could take upto 3600 hours to complete with --check-sum These files have cross references, thus you can not just rsync todays file. Thus the need to rsync the entire path. Thanks Peter
jw schultz
2003-Jul-24 10:23 UTC
bug report: different files are not rsynced, identical files rsynced
On Thu, Jul 24, 2003 at 09:52:44AM +1000, Peter Chun wrote:> Hi, > I have only just subscribe to the list. ( only to send this bug report ) > > Running rsync version 2.5.6 protocol version 26 ( on Solaris 8 sparc ) > on both hosts. > > I have 1 file I wish to sync to a remote machine > the md5 checksum is > > host1: MD5 (030722.mj) = 020397fde83c2e20464b6642c018ce6e > host2: MD5 (030722.mj) = 35fcffc896c65c8ec861385b4edb81acWhat is ls -l on them? If that is the same it won't do squat.> For brevity .. > set RSYNC="--archive --verbose --stats --recursive --times --perms > --owner --group --links --rsync-path=/usr/local/bin/rsync"In other words "--archive --verbose --stats --rsync-path=..." the other options are redundant.> rsync $RSYNC --dry-run /path/030722.mj host2:/path/030722.mj > building file list ... done > > Number of files: 1 > Number of files transferred: 0 > Total file size: 9416161 bytes > Total transferred file size: 0 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 55 > Total bytes written: 67 > Total bytes read: 20 > > wrote 67 bytes read 20 bytes 58.00 bytes/sec > total size is 9416161 speedup is 108231.74 > > The file is not listed. > > There are other files in that path ( 030701.mj to 030722.mj ) > All files from 030701 to 030721 have identical md5 checksums > > rsync $RSYNC --dry-run /path/ host2:/path/ > building file list ... done > > Number of files: 23 > Number of files transferred: 0 > Total file size: 356814662 bytes > Total transferred file size: 0 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 387 > Total bytes written: 399 > Total bytes read: 20 > > wrote 399 bytes read 20 bytes 838.00 bytes/sec > total size is 356814662 speedup is 851586.31 > > I then copy the entire directory to /tmp/path/ > rsync $RSYNC --dry-run /path/ host2:/tmp/path/ > building file list ... done > > Number of files: 23 > Number of files transferred: 0 > Total file size: 356814662 bytes > Total transferred file size: 0 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 387 > Total bytes written: 399 > Total bytes read: 20 > > wrote 399 bytes read 20 bytes 279.33 bytes/sec > total size is 356814662 speedup is 851586.31 > > > If I make a copy of the file on host1 to /tmpdid you use "cp -p" or did you allow the copy to have a different modtime?> rsync $RSYNC --dry-run /tmp/030722.mj host2:/path/030722.mj > building file list ... done > 030722.mj > > Number of files: 1 > Number of files transferred: 1 > Total file size: 9416161 bytes > Total transferred file size: 9416161 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 50 > Total bytes written: 66 > Total bytes read: 24 > > wrote 66 bytes read 24 bytes 60.00 bytes/sec > total size is 9416161 speedup is 104624.10 > > MD5 (/path/030722.mj) = 020397fde83c2e20464b6642c018ce6e > MD5 (/tmp/030722.mj) = 020397fde83c2e20464b6642c018ce6e > > > Also I have copied the directory to host1 ( remembering that all 030701 > to 030721 files have identical md5 checksums ) > rsync $RSYNC --dry-run /path/0307/ /path2/0307/ > building file list ... done > 030701.mj[snip]> 030721.mj > 030722.mj > > Number of files: 23 > Number of files transferred: 22 > Total file size: 356814662 bytes > Total transferred file size: 356814662 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 387 > Total bytes written: 487 > Total bytes read: 108 > > wrote 487 bytes read 108 bytes 1190.00 bytes/sec > total size is 356814662 speedup is 599688.51I'll bet the copy had differing timestamps.> Note: there are 23 files in the dir, a --delete will remove 030723.mj > > The --check-sum flag is ( as stated in the docs ) extremely slow. > rsyncing without already takes about 3hrs > timing tests show that it could take upto 3600 hours to complete with > --check-sum > These files have cross references, thus you can not just rsync todays file. > Thus the need to rsync the entire path.-- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt