search for: sum_upd

...checksum(const char *fname, void *sum,OFF_T size) { OFF_T i; struct map_struct *buf; @@ -133,15 +134,17 @@ void sum_init(void) { - char s[4]; + unsigned char s[4]; mdfour_begin(&md); sumresidue=0; SIVAL(s,0,checksum_seed); sum_update(s,4); } -void sum_update(char *p,int len) +void sum_update(void *p1,int len) { +unsigned char * p; + p = (unsigned char *)p1; int i; if (len + sumresidue < CSUM_CHUNK) { memcpy(sumrbuf+sumresidue, p, len); @@ -170,7 +173,7 @@ } } -void...

Well, don't get too excited, get_checksum1() (the function optimized here) is not the great performance limiter in this case, it's get_checksum2() and sum_update(), which will be using MD5. You can force using MD4, but on the slower CPU's I've tested in practice that is slower rather than faster, contrary to what would be expected. While this patch will improve things a little, to improve things a lot we need to tackle or replace MD5. Unfortunat...

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...local_server = 1; shell_path = argv[argc-1]; } diff -E -B -c -r rsync-2.5.6/proto.h rsync-2.5.6-remotebatch/proto.h *** rsync-2.5.6/proto.h Mon Jan 27 03:35:09 2003 --- rsync-2.5.6-remotebatch/proto.h Wed Jul 23 11:27:53 2003 *************** *** 31,37 **** void sum_init(void); void sum_update(char *p, int len); void sum_end(char *sum); ! void close_all(void); void _exit_cleanup(int code, const char *file, int line); void cleanup_disable(void); void cleanup_set(char *fnametmp, char *fname, struct file_struct *file, --- 31,37 ---- void sum_init(void); void sum_update(char *...

...); + if (i <= 0) + return 0; + f_in = -i; + } + if (fd_r >= 0 && size_r > 0) { int32 read_size = MAX(sum.blength * 2, 16*1024); mapbuf = map_file(fd_r, size_r, read_size, sum.blength); @@ -231,11 +245,18 @@ stats.literal_data += i; cleanup_got_literal = 1; - sum_update(data, i); + if (f_in >= 0) + /* no need for md4 during fast copy -goldor */ + sum_update(data, i); if (fd != -1 && write_file(fd,data,i) != i) goto report_write_error; offset += i; + if (f_in < 0 && percent < offset * 100 / total_size) { + /*...

...t;0?0:s->sums[i].len); - data_transfer += n; + data_transfer += send_token(f,i,buf,last_match,n,i<0?0:s->sums[i].len);; if (i >= 0) { stats.matched_data += s->sums[i].len; n += s->sums[i].len; } - for (j=0;j<n;j+=CHUNK_SIZE) { int n1 = MIN(CHUNK_SIZE,n-j); - sum_update(map_ptr(buf,last_match+j,n1),n1); + if (verbose>2) rprintf(FINFO, "#ET#>matched() calling sum_update(buf,%.0f,%.0f)\n", (double)last_match+j,(double)n1); + map_ptr(buf,last_match+j,n1); + if (buf->m_len==0) break; + sum_update(buf->m_ptr,buf->m_len); } -...

...r your detailed answer. > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > Well, don't get too excited, get_checksum1() (the function optimized > here) is not the great performance limiter in this case, it's > get_checksum2() and sum_update(), which will be using MD5. Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. So that we have a full SSE MD5 support, wherever rsync is using it (basis file checksum, rolling checksum etc...). I think one nice performance improvement could be whe...

Hi, I've a directory of large log files (> 100M) that I want to rsync to another server in short intervals (say every minute). With the --append option I can avoid resending whole files at every sync. But rsync still calculates the checksum over the complete files every time, which claims a significant share of CPU and IO resources. Since the log files are always written in append-only

Hi all, today we had a performance issue transfering a big amount of data where one file was over 50GB. Rsync was tunneled over SSH and we expected the data to be synced within hours. However after over 10 hours the data is still not synced ... The sending box has rsync running with 60-80 % CPU load (2GHz Pentium 4) while the receiver is nearly idle. So far I had no acces to the poblematic

...> > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > > > Well, don't get too excited, get_checksum1() (the function optimized > > here) is not the great performance limiter in this case, it's > > get_checksum2() and sum_update(), which will be using MD5. > > Certainly that all other functions using MD5 could be updated to use your SSE-optimized function. > So that we have a full SSE MD5 support, wherever rsync is using it (basis file checksum, rolling checksum etc...). > > I think one nice performance i...

..._struct sum; + struct mdfour mdfour_data; unsigned int len; OFF_T offset = 0; OFF_T offset2; @@ -250,6 +251,8 @@ char *map=NULL; read_sum_head(f_in, &sum); + if (md4) + mdfour_begin(&mdfour_data); sum_init(); @@ -269,6 +272,8 @@ cleanup_got_literal = 1; sum_update(data,i); + if (md4) + mdfour_update(&mdfour_data,data,i); if (fd != -1 && write_file(fd,data,i) != i) { rprintf(FERROR, "write failed on %s: %s\n", @@ -296,6 +301,8 @@ see_token(map, len); sum_update(map,len); + if (md4) + mdfour_update(&amp...

...nsigned int len; OFF_T offset = 0; OFF_T offset2; @@ -199,7 +200,9 @@ char *map=NULL; read_sum_head(f_in, &sum); - + if (md4) + mdfour_begin(&mdfour_data); + sum_init(); while ((i = recv_token(f_in, &data)) != 0) { @@ -216,6 +219,8 @@ cleanup_got_literal = 1; sum_update(data,i); + if (md4) + mdfour_update(&mdfour_data,data,i); if (fd != -1 && write_file(fd,data,i) != i) { rprintf(FERROR, "write failed on %s: %s\n", @@ -243,6 +248,8 @@ see_token(map, len); sum_update(map,len); + if (md4) + mdfour_update(&m...

...nsigned int len; OFF_T offset = 0; OFF_T offset2; @@ -199,7 +200,9 @@ char *map=NULL; read_sum_head(f_in, &sum); - + if (md4) + mdfour_begin(&mdfour_data); + sum_init(); while ((i = recv_token(f_in, &data)) != 0) { @@ -216,6 +219,8 @@ cleanup_got_literal = 1; sum_update(data,i); + if (md4) + mdfour_update(&mdfour_data,data,i); if (fd != -1 && write_file(fd,data,i) != i) { rprintf(FERROR, "write failed on %s: %s\n", @@ -243,6 +248,8 @@ see_token(map, len); sum_update(map,len); + if (md4) + mdfour_update(&m...

...nsigned int len; OFF_T offset = 0; OFF_T offset2; @@ -199,7 +200,9 @@ char *map=NULL; read_sum_head(f_in, &sum); - + if (md4) + mdfour_begin(&mdfour_data); + sum_init(); while ((i = recv_token(f_in, &data)) != 0) { @@ -216,6 +219,8 @@ cleanup_got_literal = 1; sum_update(data,i); + if (md4) + mdfour_update(&mdfour_data,data,i); if (fd != -1 && write_file(fd,data,i) != i) { rprintf(FERROR, "write failed on %s: %s\n", @@ -243,6 +248,8 @@ see_token(map, len); sum_update(map,len); + if (md4) + mdfour_update(&m...

...nsigned int len; OFF_T offset = 0; OFF_T offset2; @@ -199,7 +200,9 @@ char *map=NULL; read_sum_head(f_in, &sum); - + if (md4) + mdfour_begin(&mdfour_data); + sum_init(); while ((i = recv_token(f_in, &data)) != 0) { @@ -216,6 +219,8 @@ cleanup_got_literal = 1; sum_update(data,i); + if (md4) + mdfour_update(&mdfour_data,data,i); if (fd != -1 && write_file(fd,data,i) != i) { rprintf(FERROR, "write failed on %s: %s\n", @@ -243,6 +248,8 @@ see_token(map, len); sum_update(map,len); + if (md4) + mdfour_update(&m...

...2.4.6-fixed/receiver.c Mon Jan 28 19:10:02 2002 @@ -247,7 +247,7 @@ i = -(i+1); offset2 = i*(OFF_T)n; len = n; - if (i == count-1 && remainder != 0) + if (i == (int)(count-1) && remainder != 0) len = remainder; stats.matched_data += len; @@ -263,7 +263,7 @@ sum_update(map,len); } - if (fd != -1 && write_file(fd,map,len) != len) { + if (fd != -1 && write_file(fd,map,len) != (int)len) { rprintf(FERROR,"write failed on %s : %s\n", fname,strerror(errno)); exit_cleanup(RERR_FILEIO); diff -ru rsync-2.4.6/sender.c rsyn...

Hi, while implementing the rsync protocol in one of our projects I found that the current CVS version still has a MD4 bug. I'm using the FreeBSD libmd implementation and I still had checksum mismatches with protocol version 27 for files whose size was a multiple of 64 - 4 ( - 4 due to checksum_seed). A patch for todays CVS version is attached. Someone should also review the clean_fname()

I'm in the midst of coding a patch set for consideration that will bump the protocol version and have a couple of observations. The current minimum backwards-compatible protocol is 15 but we have code that checks for protocol versions as old as 12. If someone else doesn't beat me to it i'm considering cleaning out the pre-15 compatibility code. A backwards compatibility patch could

...char *fn, int is_dir) { @@ -226,6 +227,7 @@ full_fname(fname), strerror(errno)); exit_cleanup(RERR_FILEIO); } + offset += i; continue; } @@ -249,16 +251,28 @@ sum_update(map,len); } - if (fd != -1 && write_file(fd,map,len) != (int) len) { - rprintf(FERROR, "write failed on %s: %s\n", - full_fname(fname), strerror(errno)); - exit_cleanup(RERR_FILEIO); +...

...+ if (compare_auto && am_sender) { + args[ac++] = "--compare-auto"; + } *argc = ac; } diff -aur rsync-2.5.6/proto.h rsync-arh/proto.h --- rsync-2.5.6/proto.h Mon Jan 27 03:35:09 2003 +++ rsync-arh/proto.h Fri Oct 17 08:26:52 2003 @@ -31,7 +31,7 @@ void sum_init(void); void sum_update(char *p, int len); void sum_end(char *sum); -void close_all(void); +void close_all(); void _exit_cleanup(int code, const char *file, int line); void cleanup_disable(void); void cleanup_set(char *fnametmp, char *fname, struct file_struct *file, @@ -93,6 +93,7 @@ char *f_name(struct file_stru...