rsync - Dec 2004 - RSync with /really/ long file lists

Hi everyone!

rsync 2.5.7
linux RHES v3

We need to sync over 4 Million files, and when we run rsync we run out of
memory! :(

We were going to batch things, but the build in Batch mode still will use a
large amount of memory.  Any manualy batching is proving to be harder than
you'd think, even with --include and --exclude flags.  The flags actually
make
it still have to recurse the whole tree.

We would sync things at the top level, one dir at a time, but the tree of files
is not evenly weighted and we still run out of memory.

Questions:

1) What strategies do other people employ?
2) Is there a better tool out there for moving large amounts of files?
3) Is there a way to save the filelist to a file instead of in memory?

Thanks,


-- 
-bk








----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

On Fri, 2004-12-03 19:09:11 -0800, Brandon Knitter
<knitterb@blandsite.org>
wrote in message
<20041203190911.sg5tkem6lgvko4gs@hurl.moil.blandsite.org>:
> 1) What strategies do other people employ?
Some time ago, I had a similar problem; basically, a little number of
application worked on some kind of master repository, that needed to be
synced to other locations (so it's a strict one-way sync).

Since rsync'ing was too slow for me (thus, the remote sites were
possibly too long running with inconsistent data, too long times between
two rsync runs, ...), I took another approach. I wrote a little lib
(loaded through LD_PRELOAD=... in front of the data-generating
applications) that intercepted open(), close(), read(), write(), ... and
replicated it over the network to the other sites.

This way, I only needed to stop the system once, replicate it, and
re-start the "master" side. However, this only works if you've got
*very* reliable hardware: a network outage etc. would block the master
servers.

Unfortunately, this code isn't available freely by any means, but the
idea is there. Maybe I'd rewrite it at some time, since it prooved to be
quite useful.

As an addition, maybe file-checking could be done in advance (size, time
stamps) so that stopping the whole system before accepting a remote site
could be avoided...
> 2) Is there a better tool out there for moving large amounts of files?
Not really. IFF you can actually use rsync, it's a really cool tool. I
use it eg. for keeping Debian repos in sync and it just works (TM).
Doing a more timely remote replication isn't rsync's task, so it
isn't a
good fit for this.
> 3) Is there a way to save the filelist to a file instead of in memory?
Even if you'd press the file list to a file: you'd need to mmap() it, in
parts (4GB limit on 32bit hardware, which you aren't even allowed to
fully exploit...), which would complicate things, and make it possibly a
lot slower.

MfG, JBG
PS: Thanks to all who wrote code for rsync -- I'm really a happy user!

-- 
Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481             _ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg 
_ _ O
 fuer einen Freien Staat voll Freier B?rger" | im Internet! |   im Irak!  
O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url :
http://lists.samba.org/archive/rsync/attachments/20041204/e864404f/attachment.bin

On Fri 03 Dec 2004, Brandon Knitter wrote:
> 
> rsync 2.5.7
> linux RHES v3
> 
> We need to sync over 4 Million files, and when we run rsync we run out of
> memory! :(
Try the latest version, there have been improvements in memory
efficiency since 2.5.7.


Paul Slootman