What the maximum filesize rsync can transfer? I'm trying to rsync one of my servers to another but the rsync is croaking on a file that's barely 1GB. Tips, hints, suggestions? rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 rsync client is AIX 5.3 ML1 - rsync 2.6.4 Thanks -Jeff -- Jeff Schoby Unix/Network Admin City of Columbia, Missouri 573.874.6320
On Tue 12 Apr 2005, Jeff Schoby wrote:> What the maximum filesize rsync can transfer?2GB (4GB?) should always be possible, when built with the appropriate options for large file support >4GB files are no problem.> I'm trying to rsync one of my servers to another but the rsync is > croaking on a file that's barely 1GB.Well, it shouldn't... Please define "croaking"? Any diagnostic messages? Try running with -vvv Paul Slootman
On Apr 12, Jeff Schoby wrote: | What the maximum filesize rsync can transfer? Depends on whether you have LFS (large file support) enabled (at compile time). | I'm trying to rsync one of my servers to another but the rsync is | croaking on a file that's barely 1GB. That should be no problem. | Tips, hints, suggestions? Define "croaking", give us some error message. | rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 | | rsync client is AIX 5.3 ML1 - rsync 2.6.4 FWIW, I just upgraded to 2.6.4, and that has solved a problem i'd been having for a few weeks where 2.6.3 repeatedly failed to synchronize a 5GB file. Don't remember seeing anything in the NEWS or other making me think upgrading would help, but it did.
How do I build rsync for large file support..I'm going to need > 4GB files thru rsync. It basically stops with an error message of "File too large"...I can re-run rsync later with the -vvv flag and post the results tomorrow.>>> Paul Slootman <paul@debian.org> 04/12/05 11:36 AM >>>On Tue 12 Apr 2005, Jeff Schoby wrote:> What the maximum filesize rsync can transfer?2GB (4GB?) should always be possible, when built with the appropriate options for large file support >4GB files are no problem.> I'm trying to rsync one of my servers to another but the rsync is > croaking on a file that's barely 1GB.Well, it shouldn't... Please define "croaking"? Any diagnostic messages? Try running with -vvv Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
I did what steve suggested here, on both machines. I didn't use any arguments for configure... just did ./configure rsync -Caupvvvz server::myfiles /usr/mydir Here's what happens when it gets to the one file that it says is too big: pathto/myfile recv mapped pathto/myfile of size 1073741312 rsync: write failed on "/usr/mydir/pathto/myfile": File too large (27) rsync error: error in file IO (code 11) at receiver.c (290) _exit_cleanup(code=11, file=receiver.c, line=290): about to call exit(11) rsync: connection unexpectedly closed (1347054 bytes received so far) [generator] rsync: error in rsync protocol data stream (code 12) at io.c(420) _exit_cleanup(code=12, file=io.c, line=420): about to call exit(12)>>> "Steve Bonds" <knnf6cy7w001@sneakemail.com> 04/12/05 12:13 PM >>>On Apr 12, 2005 10:04 AM, Jeff Schoby wrote:> How do I build rsync for large file support..I'm going to need > 4GB > files thru rsync. > > It basically stops with an error message of "File too large"...I can > re-run rsync later with the -vvv flag and post the results tomorrow.It sounds like your version of AIX has similar problems to HP-UX. You will need to re-compile rsync doing something like this: 1) run ./configure with whatever options you normally use 2) edit the config.h file and comment out the line "#define HAVE_SECURE_MKSTEMP 1" 3) continue normally (e.g. "make", "make test", etc.) Here's an older list article that describes the fix: http://lists.samba.org/archive/rsync/2004-July/010199.html -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Jeff Schoby wrote:>What the maximum filesize rsync can transfer? > >I'm trying to rsync one of my servers to another but the rsync is >croaking on a file that's barely 1GB. > >Tips, hints, suggestions? > >rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 > >rsync client is AIX 5.3 ML1 - rsync 2.6.4 > >Thanks >-Jeff > >Please note that, all file size OS limitations aside, rsync has suboptimal performance for too big files. When I get around to it I'll try to create a patch, but in the mean while too big files will have too many non-real hash table collisions, and may become extremely slow. If you run across this problem, please post on list, as we need someone to experience this problem in order to try and fix it. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html
Well, I got -further- by changing the fsize= to -1 in /etc/security/limits on my AIX boxes, but rsync ultimately still did not like my 15GB file I wanted to transfer. Had to resort to good ol' plain vanilla ftp.>>> Shachar Shemesh <rsync@shemesh.biz> 04/13/05 2:52 PM >>>Jeff Schoby wrote:>What the maximum filesize rsync can transfer? > >I'm trying to rsync one of my servers to another but the rsync is >croaking on a file that's barely 1GB. > >Tips, hints, suggestions? > >rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 > >rsync client is AIX 5.3 ML1 - rsync 2.6.4 > >Thanks >-Jeff > >Please note that, all file size OS limitations aside, rsync has suboptimal performance for too big files. When I get around to it I'll try to create a patch, but in the mean while too big files will have too many non-real hash table collisions, and may become extremely slow. If you run across this problem, please post on list, as we need someone to experience this problem in order to try and fix it. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html
Actually, it said that the file was too big, so it wasn't a question of performance, it just wouldn't do it. I set my AIX environment to allow me to have files of unlimited size, rsync still said it was too big.>>> Shachar Shemesh <rsync@shemesh.biz> 04/14/05 1:28 AM >>>Jeff Schoby wrote:>Well, I got -further- by changing the fsize= to -1 in >/etc/security/limits on my AIX boxes, >but rsync ultimately still did not like my 15GB file I wanted to >transfer. > >What does "doesn't like" mean? Does it freeze with too much CPU usage?>Had to resort to good ol' plain vanilla ftp. > >How long does transferring the file via ftp take? A 15GB file puts a load of 194% on the hash table. This means that, *on average*, each lookup will have to scan two areas. I'm not sure whether that should have created a huge slowdown or not. Try rsyncing that file with the "--block-size=524288" (i.e. - a 0.5MB block size), and please report whether rsync's behavior had improved, and in particular, how does it rate against vanilla ftp. Thanks, Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html