rsync - Jun 2003 - rsync limit to file size/file count

Hi,

What are the limits to file size and file count when doing a rsync transfer
using 2.5.6?

I was trying to rsync about 500 GB of data with many files and many
directories, but it has been stuck building the file list for several hours.
First of all, is it possible to transfer 500 GB of data? Secondly, what
would the limit for file count be when doing a rsync transfer? Any comments
or help would be appreciated.

Thanks,
Eric

On Tue, Jun 10, 2003 at 04:46:25PM -0700, Eric Chen
wrote:
> Hi,
> 
> What are the limits to file size and file count when doing a rsync transfer
> using 2.5.6?
File sizes somewhere between 300MB and 1GB 2.5.6 begins
having noticeable issues.  This is not a limit but merely
pushing the capacity of truncated digests. This problem
impacts performance only. It can be mitigated somewhat by
using a larger block size.  These issues have been fixed in
CVS so 2.5.7 and later will be fine with huge files.
> I was trying to rsync about 500 GB of data with many files and many
> directories, but it has been stuck building the file list for several
hours.
> First of all, is it possible to transfer 500 GB of data? Secondly, what
Yes.
> would the limit for file count be when doing a rsync transfer?
There is no hard limit on file count you could hit.
Each file requires approximately 70 + path-length bytes so it
is a question of how much process addressable memory you have.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

I left the rsync running since last night, and it is still running

receiving file list ...
expand file_list to 8000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 8192000 bytes, did move

will this file-list ever be processed? what does the "expand file_list to X
bytes, did move" supposed to mean?

-----Original Message-----
From: rsync-bounces+echen=ateonix.com@lists.samba.org
[mailto:rsync-bounces+echen=ateonix.com@lists.samba.org]On Behalf Of jw
schultz
Sent: Tuesday, June 10, 2003 5:30 PM
To: Rsync List
Subject: Re: rsync limit to file size/file count


On Tue, Jun 10, 2003 at 04:46:25PM -0700, Eric Chen
wrote:
> Hi,
>
> What are the limits to file size and file count when doing a rsync
transfer
> using 2.5.6?
File sizes somewhere between 300MB and 1GB 2.5.6 begins
having noticeable issues.  This is not a limit but merely
pushing the capacity of truncated digests. This problem
impacts performance only. It can be mitigated somewhat by
using a larger block size.  These issues have been fixed in
CVS so 2.5.7 and later will be fine with huge files.
> I was trying to rsync about 500 GB of data with many files and many
> directories, but it has been stuck building the file list for several
hours.
> First of all, is it possible to transfer 500 GB of data? Secondly, what
Yes.
> would the limit for file count be when doing a rsync transfer?
There is no hard limit on file count you could hit.
Each file requires approximately 70 + path-length bytes so it
is a question of how much process addressable memory you have.

--
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html