search for: csum_chunk

...ed CHUNK_SIZE (32 kB) as it already exists and should be fine to use here anyway. Noticed this because I'm playing with a parallel MD5 implementation, and it benchmarked about the same as xxhash on the CPUs I used for testing, which should not be the case. Discovered performance was limited by CSUM_CHUNK reads for multiple checksum types. Have observed near 3x performance gains in rsync --checksum from cached files with xxhash on the i7-7700hq. P.S. Wayne, in the xxhash part of file_checksum(), processing the remainder reads CHUNK bytes rather than remainder bytes like the other hashes do. I don&...

...c, after proper review and testing. Regards, Eran Tromer -------------- next part -------------- diff -r -u4 rsync-2.5.5/checksum.c rsync-patched/checksum.c --- rsync-2.5.5/checksum.c Tue Oct 26 01:04:09 1999 +++ rsync-patched/checksum.c Mon Aug 5 10:05:15 2002 @@ -24,8 +24,9 @@ #define CSUM_CHUNK 64 int checksum_seed = 0; extern int remote_version; +extern int read_devices; /* a simple 32 bit checksum that can be upadted from either end (inspired by Mark Adler's Adler-32 checksum) @@ -73,43 +74,40 @@ for(i = 0; i + CSUM_CHUNK <= len; i += CSUM_CHUNK) { mdfour_upd...

...d5_asm_x86_64.s + $(CC) -c -o $@ $< + tls$(EXEEXT): $(TLS_OBJ) $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(TLS_OBJ) $(LIBS) diff --git a/lib/md5.c b/lib/md5.c index c979d10c..62bb4715 100644 --- a/lib/md5.c +++ b/lib/md5.c @@ -147,6 +147,10 @@ static void md5_process(md_context *ctx, const uchar data[CSUM_CHUNK]) ctx->D += D; } +#if defined(HAVE_SIMD) && (CSUM_CHUNK == 64) +extern void md5_process_asm(md_context *ctx, const void *data, size_t num); +#endif + void md5_update(md_context *ctx, const uchar *input, uint32 length) { uint32 left, fill; @@ -171,11 +175,20 @@ void md5_update(md_...

...emainder; + int ret; + + mdctx = mdlib_new_ctx(); + if (!mdctx) + out_of_memory("HERE"); + + if (checksum_type == CSUM_MD5) + mdlib_init_md5(mdctx); + else if (checksum_type == CSUM_SHA1) + mdlib_init_sha1(mdctx); + else + out_of_memory("wrong checksum"); + + for (i = 0; i + CSUM_CHUNK <= len; i += CSUM_CHUNK) { + ret = mdlib_update(mdctx, map_ptr(buf, i, CSUM_CHUNK), + CSUM_CHUNK); + if (!ret) + openssl_failure(__func__); + } + + remainder = (len - i); + if (remainder > 0) { + ret = mdlib_update(mdctx, map_ptr(buf, i, remainder), + remainder); + if (!re...

Attached is a patch that makes window strides constant when files are walked with a constant block size. In these cases, it completely avoids all memmoves. In my simple local test of rsyncing 57MB of 10 local files, memmoved bytes went from 18MB to zero. I haven't tested this for a big variety of file cases. I think that this will always reduce the memmoves involved with walking a large

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...; mdfour_begin(&md); sumresidue=0; SIVAL(s,0,checksum_seed); sum_update(s,4); } -void sum_update(char *p,int len) +void sum_update(void *p1,int len) { +unsigned char * p; + p = (unsigned char *)p1; int i; if (len + sumresidue < CSUM_CHUNK) { memcpy(sumrbuf+sumresidue, p, len); @@ -170,7 +173,7 @@ } } -void sum_end(char *sum) +void sum_end(void *sum) { if (sumresidue) { mdfour_update(&md, (uchar *)sumrbuf, sumresidue);

...f it had been backed up already. I experimented by adding a new option that causes file_checksum() to not sweep the file linearly but with increasing intervals. As a first approach, I just doubled the index 'i' in each iteration and added another md5_update to be applied at location size-i-CSUM_CHUNK. Thus, the file is checksummed sparsely but with increasing density towards the beginning and the end of the file. This seems to work well enough for me and best of all, it's blazing fast (with enough practical confidence, for me). Further details and changes about implementation and the approa...

I'm in the midst of coding a patch set for consideration that will bump the protocol version and have a couple of observations. The current minimum backwards-compatible protocol is 15 but we have code that checks for protocol versions as old as 12. If someone else doesn't beat me to it i'm considering cleaning out the pre-15 compatibility code. A backwards compatibility patch could