rsync - Oct 2017 - [Bug 13082] New: [REQ] Hardware / SSE based MD5 operations

https://bugzilla.samba.org/show_bug.cgi?id=13082

            Bug ID: 13082
           Summary: [REQ] Hardware / SSE based MD5 operations
           Product: rsync
           Version: 3.1.3
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: ben.rubson at gmail.com
        QA Contact: rsync-qa at samba.org

Hi,

I made some performance tests and ses that rsync can be a bottleneck while it
is calculating hashes of basis files (not a surprise).

For example, while I can read a big test file directly from the storage at 300
MB/s, rsync checksums it at about 200 MB/s.

Thus impact of calculating hashes for big files can be quite important (some
minutes for files of several hundreds of GB).

Then, what about MD5 implementation based on SSE ?
Do you think we would have a performance improvement ?

Thank you !

Ben

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

--- Comment #1 from Ben RUBSON <ben.rubson at gmx.com> ---
First info / first patch there :
https://lists.samba.org/archive/rsync/2020-May/032175.html

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

--- Comment #2 from Wayne Davison <wayne at opencoder.net> ---
No, that patch has nothing to do with MD5. It is optimizing the rolling
checksum.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

--- Comment #3 from Ben RUBSON <ben.rubson at gmx.com> ---
Yes, and as discussed there also, sounds like xxhash could be a better solution
than SSE / hardware backed MD5.
Feel free to close this then if not so relevant.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #4 from Wayne Davison <wayne at opencoder.net> ---
Jorrit Jongma has supplied a nice patch that provides some assembler
optimizations for MD5 on x86_64 which I have just merged (along with his
optimizations for the rolling checksum algorithm).  Also, my xxhash changes
have finally landed, and it includes a way for rsync to negotiate the best
checksum algorithm shared between the client & server.  This will make it
easier to add new checksum algorithms in the future.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

--- Comment #5 from Ben RUBSON <ben.rubson at gmx.com> ---
Really nice additions, it looks promising, thank you very much Jorrit &
Wayne !

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13082

--- Comment #6 from Andrew Bartlett <abartlet at samba.org> ---
Samba 4.11 moved to GnuTLS for our MD5 and other hash operations, and so uses
any hardware optimisation available there.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.