rsync - Jan 2002 - File system usage

This is a simple question. How much file system overhead is there with this
system? Is it only as large as the largest file transfered or could you
potentially (even if configured correctly) end up with double what you started
out with on the sending or receiving end?

The reason I need to be sure is that I have a file system that's literally
several hundred thousand files. Complete it's around 25-30GB.. these are
sparse files so there's lots of room for compression. ..tar'd and
compressed it's 3-4GB.

I just need to be sure there will be enough room if no single file is larger
than about 200K and I have about 5GB of space left of the partition.

It's just taking way too long to tar and compress the complete directory,
transfer than uncompress and untar.. the last run took around 24 hours front to
back. I'm hoping to cut that to around 4-5 hours. :) (probably optimistic,
but who knows!)

Anybody know of limitations on file count or size I might run into here?

Any help is appreciated for this rsync newbie.

Regards,
Duane Meyer
-------------- next part --------------
HTML attachment scrubbed and removed

Rsync is great. (What is cp? What is tar? Since I started using rsync I
forgot how to use those tools.) I sync over 1,000,000 files (50G) to
multiple destinations every day.  Over medium speed links it takes about
3 hrs. 

The default mode of operation creates a hidden (.file) on the
destination while transferring. This is only done one file at a time so
there is very little overhead unless you have a 10G filesystem with a
single 9G file -- even then I think there are rsync flags to deal with
this situation...

If you run into a file count limitation then split it up do do the rsync
on a dir-by-dir basis. There are scripts in the mailing list archives. I
don't have enough memory/time to easily do the whole tree at once -- I
do each top-level subdir individually.

eric




> Duane Meyer wrote:
> 
> This is a simple question. How much file system overhead is there with
> this system? Is it only as large as the largest file transfered or
> could you potentially (even if configured correctly) end up with
> double what you started out with on the sending or receiving end?
> 
> The reason I need to be sure is that I have a file system that's
> literally several hundred thousand files. Complete it's around
> 25-30GB.. these are sparse files so there's lots of room for
> compression. ..tar'd and compressed it's 3-4GB.
> 
> I just need to be sure there will be enough room if no single file is
> larger than about 200K and I have about 5GB of space left of the
> partition.
> 
> It's just taking way too long to tar and compress the complete
> directory, transfer than uncompress and untar.. the last run took
> around 24 hours front to back. I'm hoping to cut that to around 4-5
> hours. :) (probably optimistic, but who knows!)
> 
> Anybody know of limitations on file count or size I might run into
> here?
> 
> Any help is appreciated for this rsync newbie.
> 
> Regards,
> Duane Meyer