search for: xxhash64

...hecksumming (by default) instead > > of MD5 is not the additional security bits but performance. On a decent > > x86 box the SHA1 performance is almost the same as MD5's but with > > acceleration it outperforms MD5. > > > > The other alternative would be to go for xxHash64 [0] which has the > > superior performance but provides a non-cryptographic hash so I though > > SHA1 would be better here. > > [...] > > With respect to *both* speed and security, wouldn't BLAKE3 be a better, > modern alternative if we're looking at checksumming...

On 2020-03-17 00:03:03 [+0100], Dimitrios Apostolou via rsync wrote: > On Thursday, February 20, 2020 10:34:53 PM CET, Sebastian Andrzej Siewior > via rsync wrote: > > > > I'm still not sure if rsync requires a cryptographic hash _or_ if a > > strong hash like xxHash64 would be just fine for the job. > > I'm fairly sure the hash should *not* be easy to spoof, so I'd say a > cryptographic hash is needed. > > As an example, if a file is replaced by a file of the same size and same > hash, > rsync (if -c is in use) will consider the fi...

Sorry for taking this long to answer here, I would love to push my changes upstream (the ones for xxHash64) but what I did was use the standard xxHash64 c code as that allows a iterative hash computation, the one on LLVM only has the simple interface to pass a stringref to it, so I really don't know how to move this along, would it require me to rewrite the current internal xxHash64 implementation?...

...le, so I need to spend some time examining profiles to see what's going on. On Wed, Jan 31, 2018 at 4:44 AM Leonardo Santagada <santagada at gmail.com> wrote: > So I found all 20 bytes and changed then to GHASH_SIZE (a const I > defined in typehashing.h) and finished the switch to xxHash64, that > saved me around 50 seconds to 56s, then I changed it to uint64_t > instead of a 8 byte uint_8 array and that gave me 48s. With release > config and a pgo pass I'm now linking in 38s... so faster than > link.exe in vs 2017 (which is faster than vs 2015) doing fastlink. > &...

So I found all 20 bytes and changed then to GHASH_SIZE (a const I defined in typehashing.h) and finished the switch to xxHash64, that saved me around 50 seconds to 56s, then I changed it to uint64_t instead of a 8 byte uint_8 array and that gave me 48s. With release config and a pgo pass I'm now linking in 38s... so faster than link.exe in vs 2017 (which is faster than vs 2015) doing fastlink. Now we are in a very good...

...e something like a lib file, but deduplicated debug information (as far as I know actual .lib just put all pdbs or /Z7 debug info inside a file without dedup). Just looking at the code it seems it is much more mature and also the choice of SHA1_8 seems interesting (still don't know why not use xxHash64). ps: My code to add ghashes to msvc compiled .obj files is almost ready to be pushed as an option for llvm-objcopy. -- Leonardo Santagada

...is hardcodes to expect 20-byte sha 1s , anything else and it will recompute them in serial On Tue, Jan 30, 2018 at 12:28 PM Leonardo Santagada <santagada at gmail.com> wrote: > Nice and why are you trying blake2 instead of a faster hash algorithm? And > do you have any guess as to why xxHash64 wasn't faster than SHA1? I still > have to see how many collision I get with it, but it seems so improbable > that collisions on 64 bit hashes would be the problem. > > On 30 Jan 2018 18:39, "Zachary Turner" <zturner at google.com> wrote: > > It turns out there...

...byte sha 1s , anything else and it > will recompute them in serial > On Tue, Jan 30, 2018 at 12:28 PM Leonardo Santagada <santagada at gmail.com> > wrote: > >> Nice and why are you trying blake2 instead of a faster hash algorithm? >> And do you have any guess as to why xxHash64 wasn't faster than SHA1? I >> still have to see how many collision I get with it, but it seems so >> improbable that collisions on 64 bit hashes would be the problem. >> >> On 30 Jan 2018 18:39, "Zachary Turner" <zturner at google.com> wrote: >> &gt...

Nice and why are you trying blake2 instead of a faster hash algorithm? And do you have any guess as to why xxHash64 wasn't faster than SHA1? I still have to see how many collision I get with it, but it seems so improbable that collisions on 64 bit hashes would be the problem. On 30 Jan 2018 18:39, "Zachary Turner" <zturner at google.com> wrote: It turns out there were some problems with the...

...e something like a lib file, but deduplicated debug information (as far as I know actual .lib just put all pdbs or /Z7 debug info inside a file without dedup). Just looking at the code it seems it is much more mature and also the choice of SHA1_8 seems interesting (still don't know why not use xxHash64). ps: My code to add ghashes to msvc compiled .obj files is almost ready to be pushed as an option for llvm-objcopy. -- Leonardo Santagada _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://list...

...Layout: 513 ms ( 1.2%) Commit to Disk: 5175 ms ( 11.9%) Commit Output File: 37 ms ( 0.1%) ------------------------------------------------- Total Link Time: 43366 ms (100.0%) LTO didn't help much :( Now I will try Alexandre patches and switch fo xxHash64 or meow hashing. I need to discover how to enable huge pages on my windows (1809) ps: Need to figure out how to limit the number of link jobs in ninja as that almost used the whole 128GB of ram on my machine. On our distributed build system we can limit linking jobs (which are the only strict loca...

You can make a PDB per lib (consider msvcrtd.pdb which ships with MSVC), but all these per-lib PDBs would have to be merged into a single master PDB at the end, so you still can't avoid that final . In a way, that's similar to the idea behind /DEBUG:FASTLINK (keep the debug info in object files to eliminate the cost of merging types and symbol records) and we know what the problems with

...Still. You claim in your patch that | Benchmarks C SSE2 SSSE3 | - Intel i7-7700hq 1850 MB/s 2550 MB/s 4050 MB/s while xxhash [0] claims on a Core i5-3340M @2.7GHz that: |Version Speed on 64-bit Speed on 32-bit |XXH64 13.8 GB/s 1.9 GB/s so using xxhash64 for that work would also boost !x86 platforms. However your patch has the benefit that no changes are required on the remote side. I like that. [0] https://github.com/Cyan4973/xxHash#benchmarks Sebastian

...Commit to Disk: 5175 ms ( 11.9%) > Commit Output File: 37 ms ( 0.1%) > ------------------------------------------------- > Total Link Time: 43366 ms (100.0%) > > LTO didn't help much :( > > Now I will try Alexandre patches and switch fo xxHash64 or meow hashing. I need to discover how to enable huge pages on my windows > (1809) > > ps: Need to figure out how to limit the number of link jobs in ninja as that almost used the whole 128GB of ram on my machine. On our distributed build system we can limit linking jobs (which are the on...

Today I played around replacing the sha1 with xxHash64 and the results so far are bad. Linking times almost doubled and I can't really explain why, the only thing that comes to mind is hash collisions but on type names they should be very few in 64bit hashes. Any reason why you are trying blake2 and not murmurhash3 or xxHash64? About creating a p...

...deduplicated debug > information (as far as I know actual .lib just put all pdbs or /Z7 > debug info inside a file without dedup). > > Just looking at the code it seems it is much more mature and also the > choice of SHA1_8 seems interesting (still don't know why not use > xxHash64). > > ps: My code to add ghashes to msvc compiled .obj files is almost ready > to be pushed as an option for llvm-objcopy. > > -- > > Leonardo Santagada > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org >...

I don't disagree that MD5 could (or even should) be replaced so it is no longer the bottleneck in several real-world cases (including mine). However this patch is not for MD5 performance, rather for the rolling checksum rsync uses to match blocks on existing files on both ends to reduce transfer size. On Mon, May 18, 2020 at 5:44 PM Filipe Maia via rsync <rsync at lists.samba.org>

...tput File: 37 ms ( 0.1%) > > > ------------------------------------------------- > > > Total Link Time: 43366 ms (100.0%) > > > > > > LTO didn't help much :( > > > > > > Now I will try Alexandre patches and switch fo xxHash64 or meow > hashing. I need to discover how to enable huge pages on my windows > > > (1809) > > > > > > ps: Need to figure out how to limit the number of link jobs in ninja > as that almost used the whole 128GB of ram on my machine. On our > distributed build syste...

...feedback. My primar motivation to use SHA1 for checksumming (by default) instead of MD5 is not the additional security bits but performance. On a decent x86 box the SHA1 performance is almost the same as MD5's but with acceleration it outperforms MD5. The other alternative would be to go for xxHash64 [0] which has the superior performance but provides a non-cryptographic hash so I though SHA1 would be better here. For linking against OpenSSL as of today the rsync license would need an "OpenSSL exception" [1]. The master branch of OpenSSL is licensed under the Apache License 2.0 so we...

...information (as far as I know actual .lib just put all pdbs or > > /Z7 debug info inside a file without dedup). > > > > Just looking at the code it seems it is much more mature and also > > the choice of SHA1_8 seems interesting (still don't know why not use > > xxHash64). > > > > ps: My code to add ghashes to msvc compiled .obj files is almost > > ready to be pushed as an option for llvm-objcopy. > > > > -- > > > > Leonardo Santagada > > _______________________________________________ > > LLVM Developers maili...