similar to: [Bug 13082] New: [REQ] Hardware / SSE based MD5 operations

Well, don't get too excited, get_checksum1() (the function optimized here) is not the great performance limiter in this case, it's get_checksum2() and sum_update(), which will be using MD5. You can force using MD4, but on the slower CPU's I've tested in practice that is slower rather than faster, contrary to what would be expected. While this patch will improve things a little, to

I think you're missing a point here. Two different checksum algorithms are used in concert, the Adler-based one and the MD5 one. I SSE-optimized the Adler-based one. The Adler-based hash is used to _find_ blocks that might have shifted, while the MD5 hash is a strong cryptographic hash used to _verify_ blocks and files. You wouldn't want to replace the MD5 hash with the Adler-based hash,

This drop-in patch increases the performance of the get_checksum1() function on x86-64. On the target slow CPU performance of the function increased by nearly 50% in the x86-64 default SSE2 mode, and by nearly 100% if the compiler was told to enable SSSE3 support. The increase was over 200% on the fastest CPU tested in SSSE3 mode. Transfer time improvement with large files existing on both ends

I don't disagree that MD5 could (or even should) be replaced so it is no longer the bottleneck in several real-world cases (including mine). However this patch is not for MD5 performance, rather for the rolling checksum rsync uses to match blocks on existing files on both ends to reduce transfer size. On Mon, May 18, 2020 at 5:44 PM Filipe Maia via rsync <rsync at lists.samba.org>

When a whole-file checksum is performed, hashing was done in 64 byte blocks, causing overhead and limiting performance. Testing showed the performance improvement to go up quickly going from 64 to 512 bytes, with diminishing returns above, 4096 was where it seemed to plateau for me. Re-used CHUNK_SIZE (32 kB) as it already exists and should be fine to use here anyway. Noticed this because

https://bugzilla.samba.org/show_bug.cgi?id=13071 Bug ID: 13071 Summary: [PATCH] Allow --partial-dir with --inplace Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

I've read up some more on the subject, and it seems the proper way to do this with GCC is g++ and target attributes. I've refactored the patch that way, and it indeed uses SSSE3 automatically on supporting CPUs, regardless of the build host, so this should be ideal both for home builders and distros. Getting the code to build right in c++ mode (checksum_sse2.cpp only) was a bit of an

https://bugzilla.samba.org/show_bug.cgi?id=13445 Bug ID: 13445 Summary: Fuzzy searching in link-dest tries to open regular file as directory Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core

https://bugzilla.samba.org/show_bug.cgi?id=10738 Summary: report --stats output when termination signal arrives Product: rsync Version: 3.0.9 Platform: x64 OS/Version: Linux Status: NEW Severity: enhancement Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy:

On 2020-05-22 22:54:18 [-0700], Wayne Davison via rsync wrote: > Thanks for the optimizing patches, Jorrit! I've merged your latest changes > into the git master branch. Wouldn't it be better to add support for a crypto library (like openssl) which would provide optimized algorithms for more than just one platform without the need to maintain it separately? > ..wayne..

What do you base this on? Per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html : "For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default." That reads to me like we're fine for SSE2. As stated in my comments, SSSE3 support must be

https://bugzilla.samba.org/show_bug.cgi?id=12522 Bug ID: 12522 Summary: [PATCH] Send last error messages to sender Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

https://bugzilla.samba.org/show_bug.cgi?id=12817 Bug ID: 12817 Summary: [PATCH] Allow daemon itself to chroot Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

no backup - no mercy ----- On Mar 14, 2018, at 2:19 PM, Ben RUBSON ben.rubson at gmail.com wrote: > On 14 Mar 2018, Lentes, Bernd via rsync wrote: > >> I would now expect a rsync from the snap would transfer just some megay >> bytes to the file from the day before. >> But it doesn't: >> >> ha-idg-1:/cluster/guests/servers_alive # time rsync -av --stats

https://bugzilla.samba.org/show_bug.cgi?id=12498 Bug ID: 12498 Summary: --fuzzy --fuzzy hugely impacts performance even if its' not needed Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core

https://bugzilla.samba.org/show_bug.cgi?id=12525 Bug ID: 12525 Summary: [PATCH] Avoid uploading whole file when transfer fails Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org

https://bugzilla.samba.org/show_bug.cgi?id=13083 Bug ID: 13083 Summary: [PATCH] Use both partial file + basis file Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

https://bugzilla.samba.org/show_bug.cgi?id=12838 Bug ID: 12838 Summary: [PATCH] Log sent/received bytes even in case of error Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org

https://bugzilla.samba.org/show_bug.cgi?id=10338 Summary: Start deletion from the top of the hierarchy Product: rsync Version: 3.1.0 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy: ben.rubson at

--no-whole-file would only make it even worse. It would have to read the remote file over the network in order to do the diff then it would write the whole file over the network anyway (--inplace would help a little). Local copies force --whole-file for a good reason. On 03/14/2018 10:05 AM, Ben RUBSON via rsync wrote: > On 14 Mar 2018, Lentes, Bernd via rsync wrote: > >> ----- On