search for: jongma

...tes rather than remainder bytes like the other hashes do. I don't think it matters, but just in case. GitHub: https://github.com/Chainfire/rsync/commit/aa5ddaae5018180952a09ffaffc1ace88a1fe99d (.patch) -- >From aa5ddaae5018180952a09ffaffc1ace88a1fe99d Mon Sep 17 00:00:00 2001 From: Jorrit Jongma <git at jongma.org> Date: Mon, 25 May 2020 00:30:55 +0200 Subject: [PATCH] Improve performance of file_checksum() Previously files were hashed in blocks of CSUM_CHUNK (64) bytes. This causes significant overhead. The CSUM_CHUNK define cannot be changed as md5.c depends on it, but there is no...

...as 5-10%, with the target machine being CPU limited (still so due to MD5). This same patch on (my) GitHub for easier reading: https://github.com/Chainfire/rsync/commit/f5d0b32df869a23a74b8b8295e4983b0943866df >From f5d0b32df869a23a74b8b8295e4983b0943866df Mon Sep 17 00:00:00 2001 From: Jorrit Jongma <git at jongma.org> Date: Mon, 18 May 2020 00:21:39 +0200 Subject: [PATCH 1/1] SSE2/SSSE3 optimized version of get_checksum1() for x86-64 --- Makefile.in | 2 +- checksum.c | 2 + checksum_sse2.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 246 inse...

...eplace the MD5 hash with a different one, you'd replace it with one of the SHA's or even xxHash. On Mon, May 18, 2020 at 6:21 PM Ben RUBSON via rsync <rsync at lists.samba.org> wrote: > > Thank you Jorrit for your detailed answer. > > > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > > > Well, don't get too excited, get_checksum1() (the function optimized > > here) is not the great performance limiter in this case, it's > > get_checksum2() and sum_update(), which will be using MD5. > > C...

...ven better way to tackle the fundamental problem (the performance limitations) is to use a much faster checksum like xxhash, as has been suggested before: > https://lists.samba.org/archive/rsync/2019-October/031975.html > > Cheers, > Filipe > > On Mon, 18 May 2020 at 17:08, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: >> >> This drop-in patch increases the performance of the get_checksum1() >> function on x86-64. >> >> On the target slow CPU performance of the function increased by nearly >> 50% in the x86-64 default SSE2 mode,...

...Depending on your use-case you may never or rarely even see that performance improvement in action. It applies for my use-case though, so I am looking into this. On Mon, May 18, 2020 at 5:18 PM Ben RUBSON via rsync <rsync at lists.samba.org> wrote: > > On 18 May 2020, at 17:06, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > This drop-in patch increases the performance of the get_checksum1() > function on x86-64. > > > As ref, rather related to this : https://bugzilla.samba.org/show_bug.cgi?id=13082 > > Thank you Jorrit ! > -- > Please...

...patch but, in my view, an even better way to tackle the fundamental problem (the performance limitations) is to use a much faster checksum like xxhash, as has been suggested before: https://lists.samba.org/archive/rsync/2019-October/031975.html Cheers, Filipe On Mon, 18 May 2020 at 17:08, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > This drop-in patch increases the performance of the get_checksum1() > function on x86-64. > > On the target slow CPU performance of the function increased by nearly > 50% in the x86-64 default SSE2 mode, and by nearly 100% if the &...

...quot; ? i dislike having automatic cpu feature switching code in a tool which needs to be reliable for me, this new optimization may have issues - and without such switch it can't be easily workarounded without replacing the binary/package. regards roland Am 19.05.20 um 16:28 schrieb Jorrit Jongma via rsync: > I've read up some more on the subject, and it seems the proper way to > do this with GCC is g++ and target attributes. I've refactored the > patch that way, and it indeed uses SSSE3 automatically on supporting > CPUs, regardless of the build host, so this should be...

...t build on top of it. GitHub: https://github.com/Chainfire/rsync/commit/ef3c13390601752ef652b37c15610e12e2309fea https://github.com/Chainfire/rsync/commit/ef3c13390601752ef652b37c15610e12e2309fea.patch Raw: >From ef3c13390601752ef652b37c15610e12e2309fea Mon Sep 17 00:00:00 2001 From: Jorrit Jongma <git at jongma.org> Date: Tue, 19 May 2020 14:52:40 +0200 Subject: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64 Requires compilation using GCC C++ front end, build scripts have been modified accordingly. C++ is only used when the optimization is enabled (g++ as compiler...

What do you base this on? Per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html : "For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default." That reads to me like we're fine for SSE2. As stated in my comments, SSSE3 support must be

Thank you Jorrit for your detailed answer. > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > Well, don't get too excited, get_checksum1() (the function optimized > here) is not the great performance limiter in this case, it's > get_checksum2() and sum_update(), which will be using MD5. Certainly that all other func...

On 2020-05-18 17:55:58 [+0200], Jorrit Jongma via rsync wrote: > I don't disagree that MD5 could (or even should) be replaced so it is > no longer the bottleneck in several real-world cases (including mine). > > However this patch is not for MD5 performance, rather for the rolling > checksum rsync uses to match blocks on ex...

On 2020-05-18 21:55:13 [+0200], Jorrit Jongma wrote: > What do you base this on? So my memory was wrong. SSE2 is supported by all x86-64bit CPUs. Sorry for that. > would imply that SSSE3 is enabled out of the box on builds on machines > that support it, this is not the case (it certainly isn't on my Ubuntu > box). It would be...

On Tue, May 19, 2020 at 7:29 AM Jorrit Jongma via rsync < rsync at lists.samba.org> wrote: > I've read up some more on the subject, and it seems the proper way to do > this with GCC is g++ and target attributes. I've refactored the patch that > way, and it indeed uses SSSE3 automatically on supporting CPUs, regardless &...

On Fri, May 22, 2020 at 11:08 AM Jorrit Jongma via rsync < rsync at lists.samba.org> wrote: > This patch introduces an optimized assembler version of md5_process(), the > inner loop of MD5 checksumming. It affects the performance of all MD5 > operations in rsync - including block matching and whole-file checksums. > Thanks f...

https://bugzilla.samba.org/show_bug.cgi?id=13082 Bug ID: 13082 Summary: [REQ] Hardware / SSE based MD5 operations Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

...nc/commit/dcab47da4f6853974a952f0412f247126a6f1de8.patch Note: the assembly (.s) file is formatted using tabs, which gmail here may be messing up in the patch below. See the GitHub link above in that case. -- >From dcab47da4f6853974a952f0412f247126a6f1de8 Mon Sep 17 00:00:00 2001 From: Jorrit Jongma <git at jongma.org> Date: Fri, 22 May 2020 19:38:37 +0200 Subject: [PATCH] Optimized assembler version of md5_process() for x86-64 Originally created by Marc Bevand and placed in the public domain --- Makefile.in | 7 +- lib/md5.c | 13 + lib/md5_asm_x86_64.s | 693 ++...

Synology ( https://www.synology.com/en-global ) is one of best selling brands of consumer/prosumer/SMB NASes, with revenue estimated to be in the $100M+ range. Several of their NAS backup options use rsync either explicitly or under the hood (NAS-to-Remote-NAS backup, Shared-Folder-Sync), and they run the rsync daemon (under File Services). rsync is an integral part of their solution. They use a