Hello All, I am having an interesting issue where using the command ?rsync ?avP ?e ssh? seems to want to bail out on filesystems that have more than about 1.2 million files. Using that same command on smaller filesystems works like a charm! The server is rsync 2.6.8 protocol 29 and the destination is rsync 2.6.3 protocol 28. The server is Fedora Core 4 and the client is Mac OS 10.5.2. What am I doing wrong, how can I get around this. I have about four million files on a file system that I need to replicate to a disk in this Mac! Do I just need to upgrade? Can someone point me to binary packages for Fedora Core 4 and Mac OS 10.5 (Intel)??? Help! -------------- next part -------------- HTML attachment scrubbed and removed
It's not the best answer, but it should work to reduce your burden for the
moment....
cd parent-of-directories
for FLDR in dir1 dir2 dir4 dir17
do
cd $FLDR
rsync -avP --e ssh source/$FLDR .
cd ..
done
-dP
-----Original Message-----
From: rsync-bounces+danielp=ap.com@lists.samba.org
[mailto:rsync-bounces+danielp=ap.com@lists.samba.org] On Behalf Of Brian D.
McGrew
Sent: Tuesday, March 25, 2008 5:35 PM
To: rsync@lists.samba.org
Subject: Max files/buffer overflow
Hello All,
I am having an interesting issue where using the command 'rsync -avP -e
ssh'
seems to want to bail out on filesystems that have more than about 1.2
million files.
Using that same command on smaller filesystems works like a charm! The
server is rsync 2.6.8 protocol 29 and the destination is rsync 2.6.3
protocol 28. The server is Fedora Core 4 and the client is Mac OS 10.5.2.
What am I doing wrong, how can I get around this. I have about four million
files on a file system that I need to replicate to a disk in this Mac!
Do I just need to upgrade? Can someone point me to binary packages for
Fedora Core 4 and Mac OS 10.5 (Intel)???
Help!
-------------- next part --------------
HTML attachment scrubbed and removed
On Tue, 2008-03-25 at 17:35 -0700, Brian D. McGrew wrote:> I am having an interesting issue where using the command ?rsync ?avP ? > e ssh? seems to want to bail out on filesystems that have more than > about 1.2 million files. > Using that same command on smaller filesystems works like a charm! > The server is rsync 2.6.8 protocol 29 and the destination is rsync > 2.6.3 protocol 28. The server is Fedora Core 4 and the client is Mac > OS 10.5.2. > > What am I doing wrong, how can I get around this. I have about four > million files on a file system that I need to replicate to a disk in > this Mac! > > Do I just need to upgrade? Can someone point me to binary packages > for Fedora Core 4 and Mac OS 10.5 (Intel)???I assume you mean rsync is running out of memory due to the large number of files? You could upgrade to rsync 3.0.0, which has an incremental recursion mode that avoids holding the entire file list in memory at one time. For the Mac, you could try Rob DuToit's binary: http://rdutoit.home.comcast.net/~rdutoit/dir/lip.zip Fedora Core 4 is pretty old; I don't know of a binary package of rsync 3.0.0 for it. You may be able to rebuild a source RPM from a newer version of Fedora using "rpmbuild --rebuild". My source RPM is at: http://mattmccutchen.net/rpm/rsync-3.0.0-1.matt.src.rpm Or you could just compile and install rsync from the source tarball at: http://rsync.samba.org/ftp/rsync/rsync-3.0.0.tar.gz Matt