Hello List, unfortunatelly rsync is beeing REALLY slow and produces a high load when we try to sync lots of files (>250 000 small files). My experience is, that this is fatser if i do it folder by folder. Is there a rsync option which reduces the load and speeds up the rsync somehow? Thanks, Mario
ml@bortal.de wrote:> unfortunatelly rsync is beeing REALLY slow and produces a high load when > we try to sync lots of files (>250 000 small files).What version of rsync are you using?
# rsync --version
rsync version 2.6.9 protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks,
batchfiles, inplace, IPv6, ACLs,
64-bit system inums, 64-bit internal inums
Michael Gardner wrote:> ml@bortal.de wrote:
>
>> unfortunatelly rsync is beeing REALLY slow and produces a high load
when
>> we try to sync lots of files (>250 000 small files).
>>
>
> What version of rsync are you using?
>
>
On 3/3/2009 8:52 AM, ml@bortal.de wrote:>>> unfortunatelly rsync is beeing REALLY slow and produces a high load when >>> we try to sync lots of files (>250 000 small files).>> What version of rsync are you using?> # rsync --version > rsync version 2.6.9 protocol version 29Upgrade to 3.0.5 (on both ends)
On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote:> On 3/3/2009 8:52 AM, ml@bortal.de wrote: > >>> unfortunatelly rsync is beeing REALLY slow and produces a high load when > >>> we try to sync lots of files (>250 000 small files). > > >> What version of rsync are you using? > > > # rsync --version > > rsync version 2.6.9 protocol version 29 > > Upgrade to 3.0.5 (on both ends)OK, besides this, is there any other way to improve the network performance? some thing like change the option or what? -- Daniel
On Tue, Mar 3, 2009 at 9:33 PM, Daniel.Li <daniel_li@usish.com> wrote:>> Upgrade to 3.0.5 (on both ends) > > OK, besides this, is there any other way to improve the network > performance? some thing like change the option or what?This is a well-known performance issue in rsync that was specifically addressed in v3.0, so you need to upgrade to benefit. -- RPM
On 3-Mar-2009, at 20:33, Daniel.Li wrote:> On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: >> On 3/3/2009 8:52 AM, ml@bortal.de wrote: >>>>> unfortunatelly rsync is beeing REALLY slow and produces a high >>>>> load when >>>>> we try to sync lots of files (>250 000 small files). >> >>>> What version of rsync are you using? >> >>> # rsync --version >>> rsync version 2.6.9 protocol version 29 >> >> Upgrade to 3.0.5 (on both ends) > > OK, besides this, is there any other way to improve the network > performance? some thing like change the option or what?Not so much. a large part of 3.0 is specifically to handle large numbers of files. -- And, while it was regarded as pretty good evidence of criminality to be living in a slum, for some reason owning a whole street of them merely got you invited to the very best social occasions.