search for: file_checksum

...as xxhash on the CPUs I used for testing, which should not be the case. Discovered performance was limited by CSUM_CHUNK reads for multiple checksum types. Have observed near 3x performance gains in rsync --checksum from cached files with xxhash on the i7-7700hq. P.S. Wayne, in the xxhash part of file_checksum(), processing the remainder reads CHUNK bytes rather than remainder bytes like the other hashes do. I don't think it matters, but just in case. GitHub: https://github.com/Chainfire/rsync/commit/aa5ddaae5018180952a09ffaffc1ace88a1fe99d (.patch) -- >From aa5ddaae5018180952a09ffaffc1ace88a1f...

...uint32 get_checksum1(const void *buf1,int len) { int i; uint32 s1, s2; @@ -49,7 +50,7 @@ } -void get_checksum2(char *buf,int len,char *sum) +void get_checksum2(const void *buf,int len,unsigned char *sum) { int i; static char *buf1; @@ -82,7 +83,7 @@ } -void file_checksum(char *fname,char *sum,OFF_T size) +void file_checksum(const char *fname, void *sum,OFF_T size) { OFF_T i; struct map_struct *buf; @@ -133,15 +134,17 @@ void sum_init(void) { - char s[4]; + unsigned char s[4]; mdfour_begin(&md); sumresidue=0...

...is a further paranoia check, just to make sure that + we really have successfully transferred the file. */ + if (recv_ok && ! am_server && always_checksum) { + char csum[MD4_SUM_LENGTH + 1]; + file_checksum (fnametmp, csum, file->length); + if (memcmp(csum, file->u.sum, MD4_SUM_LENGTH) != 0) { + rprintf (FERROR, "%s checksum does not match remote checksum\n", + full_fname (fnametmp)); +...

...*)(buf1+i), CSUM_CHUNK); } - if (len - i > 0) { + if (len - i > 0) { /* make this ">=" to address http://lists.samba.org/pipermail/rsync/2002-August/008011.html */ mdfour_update(&m, (uchar *)(buf1+i), (len-i)); } mdfour_result(&m, (uchar *)sum); } -void file_checksum(char *fname,char *sum,OFF_T size) +void file_checksum(char *fname,char *sum) { - OFF_T i; + OFF_T i = 0; struct map_struct *buf; int fd; - OFF_T len = size; char tmpchunk[CSUM_CHUNK]; struct mdfour m; memset(sum,0,MD4_SUM_LENGTH); fd = do_open(fname, O_RDONLY, 0); if (fd == -1)...

Attached is a patch that makes window strides constant when files are walked with a constant block size. In these cases, it completely avoids all memmoves. In my simple local test of rsyncing 57MB of 10 local files, memmoved bytes went from 18MB to zero. I haven't tested this for a big variety of file cases. I think that this will always reduce the memmoves involved with walking a large

...s. As per my project requirement, i need to know the checksum of the file being synced. For this purpose i tried debugging the code. I understand that rsync breaks the entire file into blocks and calculates checksum for each block. >From the source code, i have come across a function void file_checksum(char *fname,char *sum,OFF_T size) where i understood that checksum is calculated. Is there a way where i can get the checksum of the entire file at once ? Is what i understood about checksum right ? Can anybody please help me in this issue ? Thanks in Advance. Regards, Vijay Ram.C ------------...

...ng --size-only would have been too coarse: I wanted to peek into the contents of the file a little. So, basically, I needed a quick way to recognize or fingerprint a big blob of data with high probability and check if it had been backed up already. I experimented by adding a new option that causes file_checksum() to not sweep the file linearly but with increasing intervals. As a first approach, I just doubled the index 'i' in each iteration and added another md5_update to be applied at location size-i-CSUM_CHUNK. Thus, the file is checksummed sparsely but with increasing density towards the beginn...

...pathjoin(fnamecmpdest, sizeof fnamecmpdest, - compare_dest, fname); - fname = fnamecmpdest; - } + while ((access(fname, 0) != 0) && compare_dest[i] != NULL) { + pathjoin(fnamecmpdest, sizeof fnamecmpdest, + compare_dest[i], fname); + fname = fnamecmpdest; + i++; } file_checksum(fname,sum,st->st_size); return memcmp(sum, file->u.sum, protocol_version < 21 ? 2 @@ -270,7 +270,7 @@ int statret; char *fnamecmp; char fnamecmpbuf[MAXPATHLEN]; - extern char *compare_dest; + extern char *compare_dest[]; extern int list_only; extern int only_existing; extern...

Hi, while implementing the rsync protocol in one of our projects I found that the current CVS version still has a MD4 bug. I'm using the FreeBSD libmd implementation and I still had checksum mismatches with protocol version 27 for files whose size was a multiple of 64 - 4 ( - 4 due to checksum_seed). A patch for todays CVS version is attached. Someone should also review the clean_fname()

I'm in the midst of coding a patch set for consideration that will bump the protocol version and have a couple of observations. The current minimum backwards-compatible protocol is 15 but we have code that checks for protocol versions as old as 12. If someone else doesn't beat me to it i'm considering cleaning out the pre-15 compatibility code. A backwards compatibility patch could

..._); + } + + remainder = (len - i); + if (remainder > 0) { + ret = mdlib_update(mdctx, map_ptr(buf, i, remainder), + remainder); + if (!ret) + openssl_failure(__func__); + } + + ret = mdlib_finalize(mdctx, sum); + if (!ret) + openssl_failure(__func__); + mdlib_free_ctx(mdctx); +} + void file_checksum(const char *fname, const STRUCT_STAT *st_p, char *sum) { struct map_struct *buf; @@ -207,19 +277,9 @@ void file_checksum(const char *fname, const STRUCT_STAT *st_p, char *sum) buf = map_file(fd, len, MAX_MAP_SIZE, CSUM_CHUNK); switch (checksum_type) { + case CSUM_SHA1: case CSUM_MD5:...

...se_batch_delta_file(); int read_batch_delta_file(char *buff, int len); void show_flist(int index, struct file_struct **fptr); void show_argvs(int argc, char *argv[]); -int make_backup(char *fname); uint32 get_checksum1(char *buf1,int len); void get_checksum2(char *buf,int len,char *sum); void file_checksum(char *fname,char *sum,OFF_T size);

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + checksum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + List.filter ( + fun file -> + if is_supported_format file then + is_new file...

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + checksum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + let files = List.filter ( + fun file -> is_regular_file (cmdline.repo // file) + ) files...

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + checksum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + let files = List.filter ( + fun file -> is_regular_file (cmdline.repo // file) + ) files...

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + checksum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + let files = List.filter ( + fun file -> is_regular_file (cmdline.repo // file) + ) files...

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + checksum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + let files = List.filter ( + fun file -> is_regular_file (cmdline.repo // file) + ) files...

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...Index.checksums = checksums } = + List.find ( + fun (_, { Index.file_uri = file_uri }) -> + Filename.basename file_uri = file + ) index in + let checksum = checksums_get_sha512 checksums in + let path = cmdline.repo // file in + let file_checksum = Checksums.compute_checksum "sha512" path in + match checksum with + | None -> true + | Some sum -> sum <> file_checksum + with Not_found -> true in + let files = Array.to_list (Sys.readdir cmdline.repo) in + let files = List.filter ( +...

Hi all, Here is an updated version of that patch series. Diff to v5: * Apply Pino's comments * Fix indentation issues * Add a default value for arch in builder/index_parser.ml if template is set * Improved new images filtering: don't process image that didn't change. This has been uncovered by introduction of --no-compression Cédric Bosdonnat (10): lib/osinfo.c: