rsync - Jul 2004 - Core dump - Can not sync big data folders of size 800 GB

Hi,

 

I was trying to synchronize data sitting on our Sun Solaris 8 server with
data size of about 800 GB to a remote Linux server in our LAN. Either the
rsync process hangs for ever else I get "core dump" after about 15
minutes
of time on the source host (solaris 8 server). I used rsync 2.6.2 using the
basic command options as shown below

 

# rsync -avz -e rsh <source folder name> 
remote_server_name:<destination
folder name> 

 

I have used this command several number of times to synchronize smaller data
sizes without any problem.

 

Can anyone tell what is the limit of data size that rsync can work.
Moreover, when I tried previously with data sizes of over 50 - 100  GB, I
see that rsync takes too much time to build the file list on source host. Is
this a limitation of rsync or server's resources? Are there any other faster
/ high performance tools available for synchronizing larger data sizes?

 

Thanks in advance.

 

Prasad

On Mon, Jul 12, 2004 at 07:48:27PM +0800, Prasad wrote:
> Can anyone tell what is the limit of data size that rsync can work?
If you run "rsync --version" it will tell you how many bits it is
using
for file sizes (e.g. 64-bit).  However, there is a bug that was just
found and fixed in CVS that affects large-file support in 2.6.2.  You
can try out the fix by applying the attached patch.
> Moreover, when I tried previously with data sizes of over 50 - 100
> GB, I see that rsync takes too much time to build the file list on
> source host. Is this a limitation of rsync or server's resources?
Yes, this is one of the limitations of rsync with large data sets -- the
pre-scan of the source (and an extra scan of the destination when the
--delete option is used) takes a long time and eats up a lot of memory.
> Are there any other faster / high performance tools available for
> synchronizing larger data sizes?
I don't know of any off-hand.  This is an area that future development
is going to be targetted at, so it should improve down the road.

..wayne..
-------------- next part --------------
--- generator.c	13 Jul 2004 01:45:51 -0000	1.95
+++ generator.c	14 Jul 2004 16:40:08 -0000	1.96
@@ -205,7 +205,7 @@ static void sum_sizes_sqroot(struct sum_
  *
  * Generate approximately one checksum every block_len bytes.
  */
-static void generate_and_send_sums(struct map_struct *buf, size_t len, int
f_out)
+static void generate_and_send_sums(struct map_struct *buf, OFF_T len, int
f_out)
 {
 	size_t i;
 	struct sum_struct sum;