search for: get_checksum2

Well, don't get too excited, get_checksum1() (the function optimized here) is not the great performance limiter in this case, it's get_checksum2() and sum_update(), which will be using MD5. You can force using MD4, but on the slower CPU's I've tested in practice that is slower rather than faster, contrary to what would be expected. While this patch will improve things a little, to improve things a lot we need to tackle or replace MD...

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...@ -30,7 +31,7 @@ a simple 32 bit checksum that can be upadted from either end (inspired by Mark Adler's Adler-32 checksum) */ -uint32 get_checksum1(char *buf1,int len) +uint32 get_checksum1(const void *buf1,int len) { int i; uint32 s1, s2; @@ -49,7 +50,7 @@ } -void get_checksum2(char *buf,int len,char *sum) +void get_checksum2(const void *buf,int len,unsigned char *sum) { int i; static char *buf1; @@ -82,7 +83,7 @@ } -void file_checksum(char *fname,char *sum,OFF_T size) +void file_checksum(const char *fname, void *sum,OFF_T size) { OFF_T...

...if (!s->sums) out_of_memory("generate_sums"); - - for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + write_int(f_out, sum.count); + write_int(f_out, sum.n); + write_int(f_out, sum.remainder); - s->sums[i].sum1 = get_checksum1(map,n1); - get_checksum2(map,n1,s->sums[i].sum2); + for (i=0;i<sum.count;i++) { + int n1 = MIN(len,block_len); + char *map = map_ptr(buf,offset,n1); + uint32 sum1 = get_checksum1(map,n1); + char sum2[SUM_LENGTH]; - s->sums[i].offset = offset; - s->sums[i].len = n1; - s->sums[i].i = i; + get_checksu...

...s = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("generate_sums"); for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + map_ptr(buf,offset,n); - s->sums[i].sum1 = get_checksum1(map,n1); - get_checksum2(map,n1,s->sums[i].sum2); + s->sums[i].sum1 = get_checksum1(buf->m_ptr,buf->m_len); + get_checksum2(buf->m_ptr,buf->m_len,s->sums[i].sum2); s->sums[i].offset = offset; - s->sums[i].len = n1; + s->sums[i].len = buf->m_len; s->sums[i].i = i; if (ver...

Thank you Jorrit for your detailed answer. > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > Well, don't get too excited, get_checksum1() (the function optimized > here) is not the great performance limiter in this case, it's > get_checksum2() and sum_update(), which will be using MD5. Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. So that we have a full SSE MD5 support, wherever rsync is using it (basis file checksum, rolling checksum etc...). I think one nice performance improvemen...

Hi all, today we had a performance issue transfering a big amount of data where one file was over 50GB. Rsync was tunneled over SSH and we expected the data to be synced within hours. However after over 10 hours the data is still not synced ... The sending box has rsync running with 60-80 % CPU load (2GHz Pentium 4) while the receiver is nearly idle. So far I had no acces to the poblematic

...etailed answer. > > > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > > > Well, don't get too excited, get_checksum1() (the function optimized > > here) is not the great performance limiter in this case, it's > > get_checksum2() and sum_update(), which will be using MD5. > > Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. > So that we have a full SSE MD5 support, wherever rsync is using it (basis file checksum, rolling checksum etc...). > > I think one nice...

...// see checksum_sse2.c for SSE2/SSSE3 version /* a simple 32 bit checksum that can be updated from either end (inspired by Mark Adler's Adler-32 checksum) @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) } return (s1 & 0xffff) + (s2 << 16); } +#endif void get_checksum2(char *buf, int32 len, char *sum) { diff --git a/checksum_sse2.c b/checksum_sse2.c new file mode 100644 index 00000000..51662833 --- /dev/null +++ b/checksum_sse2.c @@ -0,0 +1,243 @@ +/* + * SSE2/SSSE3-optimized routines to support checksumming of bytes. + * + * Copyright (C) 1996 Andrew Tridgell +...

Hi, while implementing the rsync protocol in one of our projects I found that the current CVS version still has a MD4 bug. I'm using the FreeBSD libmd implementation and I still had checksum mismatches with protocol version 27 for files whose size was a multiple of 64 - 4 ( - 4 due to checksum_seed). A patch for todays CVS version is attached. Someone should also review the clean_fname()

I'm in the midst of coding a patch set for consideration that will bump the protocol version and have a couple of observations. The current minimum backwards-compatible protocol is 15 but we have code that checks for protocol versions as old as 12. If someone else doesn't beat me to it i'm considering cleaning out the pre-15 compatibility code. A backwards compatibility patch could

...turn MD5_DIGEST_LEN; + case CSUM_SHA1: + return SHA1_DIGEST_LEN; default: /* paranoia to prevent missing case values */ exit_cleanup(RERR_UNSUPPORTED); } @@ -121,30 +128,48 @@ uint32 get_checksum1(char *buf1, int32 len) return (s1 & 0xffff) + (s2 << 16); } +static void get_checksum2_lib(void *buf, int32 len, void *sum) +{ + uchar seedbuf[4]; + MDLIB_MD_CTX *mdctx; + + mdctx = mdlib_new_ctx(); + if (!mdctx) + out_of_memory("get_checksum2"); + + if (xfersum_type == CSUM_MD5) + mdlib_init_md5(mdctx); + else if (xfersum_type == CSUM_SHA1) + mdlib_init_sha1(mdctx); + e...

...f, int bytes_to_write); @@ -23,7 +24,6 @@ void close_batch_delta_file(); int read_batch_delta_file(char *buff, int len); void show_flist(int index, struct file_struct **fptr); void show_argvs(int argc, char *argv[]); -int make_backup(char *fname); uint32 get_checksum1(char *buf1,int len); void get_checksum2(char *buf,int len,char *sum); void file_checksum(char *fname,char *sum,OFF_T size);

...t; /* > a simple 32 bit checksum that can be updated from either end > (inspired by Mark Adler's Adler-32 checksum) > @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) > } > return (s1 & 0xffff) + (s2 << 16); > } > +#endif > > void get_checksum2(char *buf, int32 len, char *sum) > { > diff --git a/checksum_sse2.c b/checksum_sse2.c > new file mode 100644 > index 00000000..51662833 > --- /dev/null > +++ b/checksum_sse2.c > @@ -0,0 +1,243 @@ > +/* > + * SSE2/SSSE3-optimized routines to support checksumming of bytes....

...hecksum that can be updated from either end >> (inspired by Mark Adler's Adler-32 checksum) >> @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) >> } >> return (s1 & 0xffff) + (s2 << 16); >> } >> +#endif >> >> void get_checksum2(char *buf, int32 len, char *sum) >> { >> diff --git a/checksum_sse2.c b/checksum_sse2.c >> new file mode 100644 >> index 00000000..51662833 >> --- /dev/null >> +++ b/checksum_sse2.c >> @@ -0,0 +1,243 @@ >> +/* >> + * SSE2/SSSE3-optimized routin...

...> a simple 32 bit checksum that can be updated from either end > (inspired by Mark Adler's Adler-32 checksum) > @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) > } > return (s1 & 0xffff) + (s2 << 16); > } > +#endif > > void get_checksum2(char *buf, int32 len, char *sum) > { > diff --git a/checksum_sse2.cpp b/checksum_sse2.cpp > new file mode 100644 > index 00000000..9a2ae86b > --- /dev/null > +++ b/checksum_sse2.cpp > @@ -0,0 +1,289 @@ > +/* > + * SSE2/SSSE3-optimized routines to support checksumming of...

What do you base this on? Per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html : "For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default." That reads to me like we're fine for SSE2. As stated in my comments, SSSE3 support must be

> Just curious: what about the rsync+ patch? Thanks for the reminder. I've just committed Jos's rsync+ patch onto the "branch_mbp_rsyncplus_merge" branch. If it works OK and nobody screams I will move it across onto the main tree tomorrow or Wednesday. I see the patch doesn't add documentation about the new options to the man page, so we should fix that in the future.

...+#ifndef ENABLE_SSE2 // see checksum_sse2.cpp /* a simple 32 bit checksum that can be updated from either end (inspired by Mark Adler's Adler-32 checksum) @@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len) } return (s1 & 0xffff) + (s2 << 16); } +#endif void get_checksum2(char *buf, int32 len, char *sum) { diff --git a/checksum_sse2.cpp b/checksum_sse2.cpp new file mode 100644 index 00000000..9a2ae86b --- /dev/null +++ b/checksum_sse2.cpp @@ -0,0 +1,289 @@ +/* + * SSE2/SSSE3-optimized routines to support checksumming of bytes. + * + * Copyright (C) 1996 Andrew Trid...