search for: get_checksum1

Well, don't get too excited, get_checksum1() (the function optimized here) is not the great performance limiter in this case, it's get_checksum2() and sum_update(), which will be using MD5. You can force using MD4, but on the slower CPU's I've tested in practice that is slower rather than faster, contrary to what would be expect...

Thank you Jorrit for your detailed answer. > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > Well, don't get too excited, get_checksum1() (the function optimized > here) is not the great performance limiter in this case, it's > get_checksum2() and sum_update(), which will be using MD5. Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. So that we have a full SSE MD5 support,...

...uire > modifications to configure/Makefile/etc that I'm not comfortable > doing, as my lack of expertise on those would probably lead me to > break the build for somebody else. If someone knowledgable enough in > that area wants to fix it, though... My suggestion would be to have a get_checksum1_sse2() and get_checksum1_sse3() and always build them. The compiler should support it. Then on runtime you would check for sse3 and based on the result get_checksum1() would either invoke the _sse2() or sse3(). Without auto detection it won't be utilized by distros. But yes, this could be impr...

...e a much faster checksum like xxhash, as has been suggested before: https://lists.samba.org/archive/rsync/2019-October/031975.html Cheers, Filipe On Mon, 18 May 2020 at 17:08, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > This drop-in patch increases the performance of the get_checksum1() > function on x86-64. > > On the target slow CPU performance of the function increased by nearly > 50% in the x86-64 default SSE2 mode, and by nearly 100% if the > compiler was told to enable SSSE3 support. The increase was over 200% > on the fastest CPU tested in SSSE3 mode. &g...

This drop-in patch increases the performance of the get_checksum1() function on x86-64. On the target slow CPU performance of the function increased by nearly 50% in the x86-64 default SSE2 mode, and by nearly 100% if the compiler was told to enable SSSE3 support. The increase was over 200% on the fastest CPU tested in SSSE3 mode. Transfer time improvement with...

...on, May 18, 2020 at 6:21 PM Ben RUBSON via rsync <rsync at lists.samba.org> wrote: > > Thank you Jorrit for your detailed answer. > > > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > > > Well, don't get too excited, get_checksum1() (the function optimized > > here) is not the great performance limiter in this case, it's > > get_checksum2() and sum_update(), which will be using MD5. > > Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. > So that we have...

On 2020-05-18 17:55:58 [+0200], Jorrit Jongma via rsync wrote: > I don't disagree that MD5 could (or even should) be replaced so it is > no longer the bottleneck in several real-world cases (including mine). > > However this patch is not for MD5 performance, rather for the rolling > checksum rsync uses to match blocks on existing files on both ends to > reduce transfer size.

On Tue, May 19, 2020 at 7:29 AM Jorrit Jongma via rsync < rsync at lists.samba.org> wrote: > I've read up some more on the subject, and it seems the proper way to do > this with GCC is g++ and target attributes. I've refactored the patch that > way, and it indeed uses SSSE3 automatically on supporting CPUs, regardless > of the build host, so this should be ideal both for

...has been suggested before: > https://lists.samba.org/archive/rsync/2019-October/031975.html > > Cheers, > Filipe > > On Mon, 18 May 2020 at 17:08, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: >> >> This drop-in patch increases the performance of the get_checksum1() >> function on x86-64. >> >> On the target slow CPU performance of the function increased by nearly >> 50% in the x86-64 default SSE2 mode, and by nearly 100% if the >> compiler was told to enable SSSE3 support. The increase was over 200% >> on the fastest CPU...

What do you base this on? Per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html : "For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default." That reads to me like we're fine for SSE2. As stated in my comments, SSSE3 support must be

.../commit/ef3c13390601752ef652b37c15610e12e2309fea.patch > > Raw: > > From ef3c13390601752ef652b37c15610e12e2309fea Mon Sep 17 00:00:00 2001 > From: Jorrit Jongma <git at jongma.org> > Date: Tue, 19 May 2020 14:52:40 +0200 > Subject: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64 > > Requires compilation using GCC C++ front end, build scripts have been > modified accordingly. C++ is only used when the optimization is enabled > (g++ as compiler, x86-64 build target, --disable-sse2 not passed to > configure). > --- > Makefile.in | 12...

...s://github.com/Chainfire/rsync/commit/ef3c13390601752ef652b37c15610e12e2309fea.patch Raw: >From ef3c13390601752ef652b37c15610e12e2309fea Mon Sep 17 00:00:00 2001 From: Jorrit Jongma <git at jongma.org> Date: Tue, 19 May 2020 14:52:40 +0200 Subject: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64 Requires compilation using GCC C++ front end, build scripts have been modified accordingly. C++ is only used when the optimization is enabled (g++ as compiler, x86-64 build target, --disable-sse2 not passed to configure). --- Makefile.in | 12 +- checksum.c | 2 + che...

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...PRJ_SRC:CHECKSUM.TPU on 6-JAN-2002 22:41:06.45 OPENVMS_AXP */ /* Copyright (C) Andrew Tridgell 1996 Copyright (C) Paul Mackerras 1996 @@ -30,7 +31,7 @@ a simple 32 bit checksum that can be upadted from either end (inspired by Mark Adler's Adler-32 checksum) */ -uint32 get_checksum1(char *buf1,int len) +uint32 get_checksum1(const void *buf1,int len) { int i; uint32 s1, s2; @@ -49,7 +50,7 @@ } -void get_checksum2(char *buf,int len,char *sum) +void get_checksum2(const void *buf,int len,unsigned char *sum) { int i; static char *buf1; @@ -82,7...

...;sums[0])*s->count); - if (!s->sums) out_of_memory("generate_sums"); - - for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + write_int(f_out, sum.count); + write_int(f_out, sum.n); + write_int(f_out, sum.remainder); - s->sums[i].sum1 = get_checksum1(map,n1); - get_checksum2(map,n1,s->sums[i].sum2); + for (i=0;i<sum.count;i++) { + int n1 = MIN(len,block_len); + char *map = map_ptr(buf,offset,n1); + uint32 sum1 = get_checksum1(map,n1); + char sum2[SUM_LENGTH]; - s->sums[i].offset = offset; - s->sums[i].len = n1; - s->sum...

...,24 +195,23 @@ s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("generate_sums"); for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + map_ptr(buf,offset,n); - s->sums[i].sum1 = get_checksum1(map,n1); - get_checksum2(map,n1,s->sums[i].sum2); + s->sums[i].sum1 = get_checksum1(buf->m_ptr,buf->m_len); + get_checksum2(buf->m_ptr,buf->m_len,s->sums[i].sum2); s->sums[i].offset = offset; - s->sums[i].len = n1; + s->sums[i].len = buf->m_len; s->s...

Hi all, I am writing a xdelta-like application as a personal experiment and am busy implementing the rsync protocol, so far so good. I am using C++ templates and creating the algorithms so that operate on any stream, array, etc. through iterators. All seems well except that I am getting a lot of false hits with the weak checksum. When generating checksums of blocksize 1024 on the RedHat 7.1

Hi all, today we had a performance issue transfering a big amount of data where one file was over 50GB. Rsync was tunneled over SSH and we expected the data to be synced within hours. However after over 10 hours the data is still not synced ... The sending box has rsync running with 60-80 % CPU load (2GHz Pentium 4) while the receiver is nearly idle. So far I had no acces to the poblematic

...d an adjacent match - the RLL coder will be happy */ i = i2; @@ -232,7 +230,7 @@ matched(f,s,buf,offset,i); offset += s->sums[i].len - 1; - k = MIN((len-offset), s->n); + k = MIN((len-offset), s->blength); map = (schar *)map_ptr(buf,offset,k); sum = get_checksum1((char *)map, k); s1 = sum & 0xFFFF; @@ -262,9 +260,9 @@ running match, the checksum update and the literal send. */ if (offset > last_match && - offset-last_match >= CHUNK_SIZE+s->n && + offset-last_match >= CHUNK_SIZE+s->blength &amp...

...if (i2 == last_i + 1) { if (sum != s->sums[i2].sum1) break; @@ -232,7 +232,7 @@ matched(f,s,buf,offset,i); offset += s->sums[i].len - 1; - k = MIN((len-offset), s->n); + k = MIN((len-offset), (ssize_t)s->n); map = (schar *)map_ptr(buf,offset,k); sum = get_checksum1((char *)map, k); s1 = sum & 0xFFFF; @@ -262,7 +262,7 @@ running match, the checksum update and the literal send. */ if (offset > last_match && - offset-last_match >= CHUNK_SIZE+s->n && + offset-last_match >= (ssize_t)(CHUNK_SIZE+s->n)...