similar to: checksum feature request

I don't disagree that MD5 could (or even should) be replaced so it is no longer the bottleneck in several real-world cases (including mine). However this patch is not for MD5 performance, rather for the rolling checksum rsync uses to match blocks on existing files on both ends to reduce transfer size. On Mon, May 18, 2020 at 5:44 PM Filipe Maia via rsync <rsync at lists.samba.org>

Hi All, Any inputs on this issue? -- Shedi On Wed, Feb 21, 2024 at 5:12?PM Shreenidhi Shedi < shreenidhi.shedi at broadcom.com> wrote: > Hi All, > > Copying the content from the GH issue as is. > Need your inputs on the same. > FWIW, the coredump files generated in linux have xattr values which are > > 32 bytes. > > https://github.com/WayneD/rsync/issues/569

I think you're missing a point here. Two different checksum algorithms are used in concert, the Adler-based one and the MD5 one. I SSE-optimized the Adler-based one. The Adler-based hash is used to _find_ blocks that might have shifted, while the MD5 hash is a strong cryptographic hash used to _verify_ blocks and files. You wouldn't want to replace the MD5 hash with the Adler-based hash,

On Tue, Apr 25, 2017 at 12:55 PM, Vedant Kumar via llvm-dev <llvm-dev at lists.llvm.org> wrote: > >> On Apr 24, 2017, at 5:37 PM, Scott Smith via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >> I've been working on improving the startup performance of lldb, and ran into an issue with llvm::HashString. It works a character at a time, which creates a long

When a whole-file checksum is performed, hashing was done in 64 byte blocks, causing overhead and limiting performance. Testing showed the performance improvement to go up quickly going from 64 to 512 bytes, with diminishing returns above, 4096 was where it seemed to plateau for me. Re-used CHUNK_SIZE (32 kB) as it already exists and should be fine to use here anyway. Noticed this because

Hello all, I'm implementing a watermarking feature for lld that computes a hash of loadable segments and places the result in a note section. Ongoing work can be found here: https://reviews.llvm.org/D70316 https://reviews.llvm.org/D66426 The purpose of this watermark is to enable detection of post-link modifications to the loadable segments of the binary. Such modifications may produce a

On 2020-05-22 22:54:18 [-0700], Wayne Davison via rsync wrote: > Thanks for the optimizing patches, Jorrit! I've merged your latest changes > into the git master branch. Wouldn't it be better to add support for a crypto library (like openssl) which would provide optimized algorithms for more than just one platform without the need to maintain it separately? > ..wayne..

https://bugzilla.samba.org/show_bug.cgi?id=13082 Bug ID: 13082 Summary: [REQ] Hardware / SSE based MD5 operations Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

The ELF file header isn't always covered by a segment but still affects loading. I think everything else that effects loading/dynamic linking is always covered by a PT_LOAD segment. As evidence this is basically what --strip-sections in llvm-strip and eu-strip do and they produce perfectly runnable binaries. Having a hash of the actual memory map is interesting IMO. Build IDs can't really

On Tue, Jan 28, 2003 at 11:22:14PM -0800, Craig Barratt wrote: > And I have several things I would like to work on and submit: > > - Fix the MD4 block and file checksums to comply with the rfc > (currently MD4 is wrong for blocks of size 64*n, or files > longer than 512MB). > > - Adaptive first pass checksum lengths: use 3 or more bytes of the MD4 > block

With rsync-patches/md5p8.diff and --enable-xxhash % gdb --args ./rsync --version Using host libthread_db library "/lib64/libthread_db.so.1". rsync version v3.2.3-29-ge55788bd protocol version 31 Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,

Well, don't get too excited, get_checksum1() (the function optimized here) is not the great performance limiter in this case, it's get_checksum2() and sum_update(), which will be using MD5. You can force using MD4, but on the slower CPU's I've tested in practice that is slower rather than faster, contrary to what would be expected. While this patch will improve things a little, to

Hi, Where can I get good pointers on the rolling checksum algorithm used in rsync? I need an 8-bit or 12-bit rolling checksum too. Any place where "rolling" checksum algorithms are discussed? Seun Osewa ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

Hi, Assume you are in hostile environment, as in you can't trust the DHCP serversss nor the TFTP/HTTP server. So you would want a checksum on kernel and initrd. Which checksum algoritme is available in pxelinux.0? Which checksum algoritme could be integrate into pxelinux.0? In other words: Please advice what could be done to verify what pxelinux.0 did download. Cheers Geert Stappers --

I've been working on improving the startup performance of lldb, and ran into an issue with llvm::HashString. It works a character at a time, which creates a long dependency chain in the processor. On the other hand, the function is very short, which probably works well for short identifiers. I don't know how the mix of identifier length seen by lldb compares with that seen by

Hi, We have seen a huge performance drop in 1.6.3, due to the checksum being enabled by default. I looked at the algorithm being used, and it is actually a CRC32, which is a very strong algorithm for detecting all sorts of problems, such as single bit errors, swapped bytes, and missing bytes. I''ve been experimenting with using a simple XOR algorithm. I''ve been able to recover

Hi All, Copying the content from the GH issue as is. Need your inputs on the same. FWIW, the coredump files generated in linux have xattr values which are > 32 bytes. https://github.com/WayneD/rsync/issues/569 Hi All, System details: root at ph3dev [ ~ ]# rsync --version rsync version 3.2.4 protocol version 31 Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long

2007/10/19, Josh Coalson <xflac@yahoo.com>: > > --- Harry Sack <tranzedude@gmail.com> wrote: > > hi > > > > here some questions about the md5 checksum: > > > > - what happens when the md5 checksum of the decoded audio is > > different > > of the internally stored checksum due to file corruption ? Will > > playing/decoding still be

On 2020-02-20 20:06:39 [+0100], Markus Ueberall wrote: > On 2020-02-09 23:19, Sebastian Andrzej Siewior wrote: > > [...] > > My primar motivation to use SHA1 for checksumming (by default) instead > > of MD5 is not the additional security bits but performance. On a decent > > x86 box the SHA1 performance is almost the same as MD5's but with > > acceleration it

On Fri, 17 Apr 2015 23:45:42 -0400, Kevin Korb wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > So, I am assuming that without --itemize-changes you couldn't tell the > difference between "I am transferring a file" and "I am fixing the > timestamp on a file". > > Even if I am right, you are probably still mis-using --checksum. When