rsync - Aug 2017 - [Bug 12964] New: Maybe we can add the '--bind-cpu' option

https://bugzilla.samba.org/show_bug.cgi?id=12964

            Bug ID: 12964
           Summary: Maybe we can add the '--bind-cpu' option
           Product: rsync
           Version: 3.1.2
          Hardware: All
                OS: All
            Status: NEW
          Severity: trivial
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: gongcunjust at gmail.com
        QA Contact: rsync-qa at samba.org

Created attachment 13472
  --> https://bugzilla.samba.org/attachment.cgi?id=13472&action=edit
add '--bind-cpu' option to rsync

We use rsync to take daily backup or log synchronization, but rsync
often trigger high CPU load. I tried to find a solution through
Google, but didn't find a satisfactory answer. Many people suggested
using the '--whole-file' option to reduce the CPU load, or use
'nice'
to lower the rsync execution priority. So I have another idea:

  Maybe we can add the '--bind-cpu' option to tell rsync to run
  on specified processor, like the 'worker_cpu_affinity' in nginx.

Although I'm not sure that is a good idea, because the core issue is
the improvement of the rsync protocol, but I still have made a few
attempts. I made some changes to rsync 3.1.2 to enable it to support
binding CPUs on GNU/Linux, AIX, FreeBSD & Solaris, please see the
attachment. For example, the following option tells rsync to run on
CPUs 0, 2-5 and 7:

  --bind-cpu=0,2-5,7

I also used maketree.py script to test the synchronization of a large
number of files (10000 files), and drew some statistics curves (see
the cpu_time.pdf in attachment). I was just testing on a machine (AIX
7.2 with 8 processors) without testing the remote synchronization, but
I think the conclusion is worth reference. As the size of file
increases, the CPU load will continue to increase by default, but the
CPU load will slowly increase until it is around 55% when binding a
single processor. The disadvantage of binding CPUs is that when file
sizes are growing, they are slower to process than non-binding (it's
about two times slower in my test result). Of course, if I/O or
network bandwidth is the bottleneck, then binding CPUs is not
significant because most of time is waiting.

Is it necessary to add the '--bind-cpu' option ? Your comments are
welcome, happy to answer any question I can !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12964

--- Comment #1 from Stefan Metzmacher <metze at samba.org> ---
On Linux you can use taskset (in combination with nice and ionice)...

-- 
You are receiving this mail because:
You are the QA Contact for the bug.