rsync - Nov 2007 - DO NOT REPLY [Bug 5109] New: poor performance on large drives with big bandwidth

https://bugzilla.samba.org/show_bug.cgi?id=5109

           Summary: poor performance on large drives with big bandwidth
           Product: rsync
           Version: 2.6.6
          Platform: x86
        OS/Version: Windows XP
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: james@time4tea.net
         QAContact: rsync-qa@samba.org


I am attempting to sync 2TB systems with about 400GB of differences across a
1Gb link, however I am finding that rsync has very low CPU and network
utilisation for this scenario.

Seeing as how there is such a lot of data to be synced, it would be great if
rsync could use a little more than the 4% of the availble bandwidth it is
managing!

This is using rsync -a -v --progress rsync://a.b.c.d/xx on cygwin.

This is running on 4 CPU boxes - so we would like to be able to sacrifice
performance for a faster sync speed, if that tradeoff is required.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109





------- Comment #1 from matt@mattmccutchen.net  2007-11-26 21:38 CST -------
Figure out what the bottleneck is.  Since your dataset is enormous, I'm
guessing the issue is that rsync is using too much memory and the system is
swapping.  If this is the case, you should try the current development rsync,
which has an incremental recursion mode that dramatically reduces memory usage.
 If you then find that CPU or disk is the bottleneck and the network is
underutilized, pass --whole-file to disable the delta-transfer algorithm.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109





------- Comment #2 from wayned@samba.org  2007-12-15 10:39 CST -------
The latest 3.0.0 pre-release also has improved server-side hashing for really
large files.

Have you had a chance to try using -W (--whole-file)?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109





------- Comment #3 from james@time4tea.net  2007-12-17 06:04 CST -------
I did try it - the files started being transferred more quickly, however the
network utilization was still poor (3-4%).

I don't think that disk write speed was an issue, as i'm writing to a 4
drive
hardware raid 5 volume. Neither box was doing anything else at the time, nor
had any memory issues... in fact "Task Manager" said mem usage for
rsync was
quite low (but who knows if it lies)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109





------- Comment #4 from james@time4tea.net  2007-12-17 06:06 CST -------
i should also say this is using the cygwin version of rsync - but using rsync
protocol not ssh.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109


devzero at web.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |devzero at web.de




------- Comment #5 from devzero at web.de  2010-08-22 05:22 CST -------
you should not blame rsync for that, it`s because of the overhead of cygwin.

cygwin is a posix emulation layer on windows and this introduces a lot of
overhead. 

maybe this can be tuned performance wise, by doing profiling, optimization and
working together with the cygwin folks - but i recommend to try the same thing
from within linux so that you can see the difference.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5109

--- Comment #6 from roland <devzero at web.de> 2013-12-25 23:45:04 UTC ---
besides the fact that this bug is quite old and could perhaps be closed, i
recommend retesting with latest rsync on latest cygwin. performance is much
better with that

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5109

Björn Jacke <bjacke at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

-- 
You are receiving this mail because:
You are the QA Contact for the bug.