We're using rsync 2.6.3 to sync two DELL PowerEdge servers with both
Redhat-EL4 and otherwise nearly identical hardware (2.8/3GHz, 1GB RAM
each). The source machine has a SCSI-RAID1, the destination a
SATA-RAID1 disk attached.
There are 5 filesystems which are rsynced via ssh. On the smaller
filesystems with ~200.000 files/7GB, rsync takes 1-3 minutes:
lion:/atg/ ========= Tue Aug 22 12:13:29 CEST 2006 =================
receiving file list ... done
--<snip-snip>--
Number of files: 233780
Number of files transferred: 2
Total file size: 6716087965 bytes
Total transferred file size: 1470 bytes
Literal data: 1470 bytes
Matched data: 0 bytes
File list size: 5015765
Total bytes sent: 68
Total bytes received: 5017339
sent 68 bytes received 5017339 bytes 56060.41 bytes/sec
total size is 6716087965 speedup is 1338.56
lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 =================
233780 files/89 sec = 2626 files/sec
On the large filesystems with ~1.200.000 files/17GB, rsync takes 30
minutes, even when only few files change (and even when invoked with -n):
lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 =================
receiving file list ... done
--<snip-snip>--
Number of files: 1232323
Number of files transferred: 124
Total file size: 17103825898 bytes
Total transferred file size: 1416942923 bytes
Literal data: 27658581 bytes
Matched data: 1389311598 bytes
File list size: 95325482
Total bytes sent: 643209
Total bytes received: 123363867
sent 643209 bytes received 123363867 bytes 62963.73 bytes/sec
total size is 17103825898 speedup is 137.93
lion:/homes/atg/ ========= Tue Aug 22 12:47:48 CEST 2006 =================
1232323 files / 1960 sec = 629 files/sec
In comparison, on the large filesystem, 'find'ing all files on the
source system and 'stat'ing them takes 1.75 minutes:
source# time find /homes -xdev -print0 | xargs -0 stat -c "%s %Y"
| wc -l
1232287
real 1m41.987s
user 0m7.075s
sys 0m33.216s
On the destination 4 minutes:
dest# time find /backup/lion-homes -xdev -print0 | xargs -0 stat -c "%s
%Y" | wc -l
1233202
real 4m3.650s
user 0m9.895s
sys 0m41.047s
Receiving the 123363867 bytes should not take longer than 20sec on a
100Mbit LAN, and transferring a GB to disk should not take longer than
say a minute (we have write transfer rates of ~50MB/sec on those disks
on each machine).
So what is rsync doing in addition that it takes >30 minutes to sync
the large filesystem?
Thanks
R'
Try using major subdirectories one level lower. Example sequence from my automated home machine backup: rsync -ax --stats / /rsync_backup/ rsync -ax --stats /usr/ /rsync_backup/usr/ rsync -ax --stats /home/ /rsync_backup/home/ Compare to "rsync -a --stats / /backup/" Ralf Fassel wrote:> We're using rsync 2.6.3 to sync two DELL PowerEdge servers with both > Redhat-EL4 and otherwise nearly identical hardware (2.8/3GHz, 1GB RAM > each). The source machine has a SCSI-RAID1, the destination a > SATA-RAID1 disk attached. > > There are 5 filesystems which are rsynced via ssh. On the smaller > filesystems with ~200.000 files/7GB, rsync takes 1-3 minutes: > lion:/atg/ ========= Tue Aug 22 12:13:29 CEST 2006 =================> receiving file list ... done > --<snip-snip>-- > Number of files: 233780 > Number of files transferred: 2 > Total file size: 6716087965 bytes > Total transferred file size: 1470 bytes > Literal data: 1470 bytes > Matched data: 0 bytes > File list size: 5015765 > Total bytes sent: 68 > Total bytes received: 5017339 > > sent 68 bytes received 5017339 bytes 56060.41 bytes/sec > total size is 6716087965 speedup is 1338.56 > lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 =================> > 233780 files/89 sec = 2626 files/sec > > On the large filesystems with ~1.200.000 files/17GB, rsync takes 30 > minutes, even when only few files change (and even when invoked with -n): > lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 =================> receiving file list ... done > --<snip-snip>-- > Number of files: 1232323 > Number of files transferred: 124 > Total file size: 17103825898 bytes > Total transferred file size: 1416942923 bytes > Literal data: 27658581 bytes > Matched data: 1389311598 bytes > File list size: 95325482 > Total bytes sent: 643209 > Total bytes received: 123363867 > > sent 643209 bytes received 123363867 bytes 62963.73 bytes/sec > total size is 17103825898 speedup is 137.93 > lion:/homes/atg/ ========= Tue Aug 22 12:47:48 CEST 2006 =================> > 1232323 files / 1960 sec = 629 files/sec > > In comparison, on the large filesystem, 'find'ing all files on the > source system and 'stat'ing them takes 1.75 minutes: > source# time find /homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l > 1232287 > real 1m41.987s > user 0m7.075s > sys 0m33.216s > On the destination 4 minutes: > dest# time find /backup/lion-homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l > 1233202 > real 4m3.650s > user 0m9.895s > sys 0m41.047s > > Receiving the 123363867 bytes should not take longer than 20sec on a > 100Mbit LAN, and transferring a GB to disk should not take longer than > say a minute (we have write transfer rates of ~50MB/sec on those disks > on each machine). > > So what is rsync doing in addition that it takes >30 minutes to sync > the large filesystem? > > Thanks > R'--