rsync - Feb 2013 - rsync memory usage with block-size option

I have been using rsync to sync a SQL database dump(127GB in size) on a
server (cygwin to linux) for many years now w/o a problem, until they put a
maintence plan to reindex the database once a week. This is all done over a
2Mbit link normally took 3-4 hrs now I noticed it was running the next day
and took days so I looked into it more. After I discovered the
--only-write-batch it makes a 52 GB file, even hitting it with 7z it only
gets down to 5G plus it makes it harder to script the process. Then I got
to fooling around with the -B option and found on a smaller test database
that when I get to -B1024 it make the smallest file so I tried that. I get
an Out of memory error in receive_sums so I back it off and when I get to
about -B5120 it is good but only makes the file 36GB. I was wondering is
there a hard limit, the machine has tons of free ram, the program used
1.1GB of RAM at -B5120,the version is 3.0.9-1 in cygwin on the sender via
an ssh connection.
   One other thing I thought about but I don't see that you can is if you
could the --only-write-batch and read batches to/from stdout so I could 7z
them in line I know you can gzip with the -z option and even setting the
level to 9 I still get about %25 better compression with 7z but everything
adds another level of complexity.

Thanks,
Jarrid Graham
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20130228/18fcfc5e/attachment.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

When you say an SQL database dump do you mean you are rsyncing the raw
text file or are you compressing it?  If you are compressing it you
might want to try gzip with the --rsyncable patch as it can
significantly improve rsync's ability to delta-xfer compressed files.

Also, what do you mean by reindex?  That doesn't sound like something
that should affect the dumps unless you are dumping something that
would be easier to regenerate after a restore.

On 02/28/13 14:37, Jarrid Graham wrote:
> I have been using rsync to sync a SQL database dump(127GB in size)
> on a server (cygwin to linux) for many years now w/o a problem,
> until they put a maintence plan to reindex the database once a
> week. This is all done over a 2Mbit link normally took 3-4 hrs now
> I noticed it was running the next day and took days so I looked
> into it more. After I discovered the --only-write-batch it makes a
> 52 GB file, even hitting it with 7z it only gets down to 5G plus it
> makes it harder to script the process. Then I got to fooling around
> with the -B option and found on a smaller test database that when I
> get to -B1024 it make the smallest file so I tried that. I get an
> Out of memory error in receive_sums so I back it off and when I get
> to about -B5120 it is good but only makes the file 36GB. I was
> wondering is there a hard limit, the machine has tons of free ram,
> the program used 1.1GB of RAM at -B5120,the version is 3.0.9-1 in
> cygwin on the sender via an ssh connection. One other thing I
> thought about but I don't see that you can is if you could the
> --only-write-batch and read batches to/from stdout so I could 7z
> them in line I know you can gzip with the -z option and even 
> setting the level to 9 I still get about %25 better compression
> with 7z but everything adds another level of complexity.
> 
> Thanks, Jarrid Graham
> 
> 
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlEvvtMACgkQVKC1jlbQAQeDMgCfXJnfrzijHI+X+kL9b8kbCgcj
IVMAoMMHytQ9tXOeXNX9J7UchEw2AF+F
=1SPZ
-----END PGP SIGNATURE-----