Hello together, I have a question about using rsync with many files. We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root at 192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? Btw: The bwlimit should not be the problem, because generating the filelist is the most time. Thank you very much! Cliff Simon
On Mon 06 Jun 2011, Cliff Simon wrote:> > Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not.Are you using a recent version of rsync? One that does incremental recursion? Paul
Hi Paul, Thank you for your reply! Hm...I?m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server has 2.6.9 Do I have to enable incremental recursion and from which version is incremental supported? Cliff Simon> -----Urspr?ngliche Nachricht----- > Von: Paul Slootman [mailto:paul+rsync at wurtel.net] > Gesendet: Montag, 6. Juni 2011 18:00 > An: rsync at lists.samba.org > Betreff: Re: rsync and many files > > On Mon 06 Jun 2011, Cliff Simon wrote: > > > > Now my problem is, that the backup needs about 14 hours - the most time > is to generate the filelist and check whether the files are new/changed or > not. > > Are you using a recent version of rsync? One that does incremental > recursion? > > > Paul > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
In <F992406D6E81B54DBB33210217FE7AFD07F80C84 at exchange1.mtb.netclusive.de>, on 06/06/11 at 12:04 PM, Cliff Simon <cliff.simon at netclusive.com> said: Hi,>We are using rsync via rsnapshot, but this is not elementary. It is used >to backup many (above 100 servers) and works very well. Now there is one >server with many (several millions) files. The files are not very big, so >the complete backup is about 500 GB.>Now my problem is, that the backup needs about 14 hours - the most time >is to generate the filelist and check whether the files are new/changed >or not.>My rsync-command is: >/usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative >--delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh >--link-dest=/dest.path/daily.1/ root at 192.x.x.x:/path.to.backup/>Do you have an idea to reduce the backup time?A bit of math says 2*10^6 / 14 hours is about 40 files/second. How fast do you think rsync should be and how does this compare to backups on your other servers? Are you sure the it is not the hardware that is limiting the rsync's performance? Based on my knowledge of the rsync sources, I believe the file list generation algorithms are pretty efficient. There is quite a bit of code in the code path, but it's hard to avoid this given the number of options available to control the sync process. Steven -- ---------------------------------------------------------------------- "Steven Levine" <steve53 at earthlink.net> eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com ----------------------------------------------------------------------