search for: xxhash

...r short identifiers. >> >> I don't know how the mix of identifier length seen by lldb compares with that seen by llvm/clang; I imagine they're pretty similar. >> >> I have to different proposals, and wanted to gauge which would be preferred: > > >> 1. Use xxhash instead. > I'd lean towards this option as we already have a copy of xxHash in lib/Support (llvm/Support/xxhash.h). We use it for computing the --build-id in lld as fast/non-crytographic hash variant and has proven to outperform other candidates (e.g. CityHash). Vedant, what do you mean wi...

...it was the checksum components in rsync. Once we found that and tested against the --checksum option, it was glaring that this was slowing us down. We next tested the MD5 vs MD4 checksums and found little difference in speed. So we went out in search of a better checksum algorithm and found xxhash, using the one from the Centos release. Thanks to the way the source is written, it was a fairly easy patch to get this into the src RPM. We have been using this in production now for awhile and see about a 3x speedup over the MD5/4 checksum algorithm which brings it pretty close to the --chec...

With rsync-patches/md5p8.diff and --enable-xxhash % gdb --args ./rsync --version Using host libthread_db library "/lib64/libthread_db.so.1". rsync version v3.2.3-29-ge55788bd protocol version 31 Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-...

...function is very short, which probably works well for short identifiers. I don't know how the mix of identifier length seen by lldb compares with that seen by llvm/clang; I imagine they're pretty similar. I have to different proposals, and wanted to gauge which would be preferred: 1. Use xxhash instead. 2. Use the Intel native crc32q instruction to process 8 bytes at a time, then fall back to byte at a time. Non sse 4.2 capable processors (either early or non Intel/AMD x86) would use the existing algorithm, or possibly #1 above. For my test, both result in approximately the same # of c...

...s to reduce transfer size. On Mon, May 18, 2020 at 5:44 PM Filipe Maia via rsync <rsync at lists.samba.org> wrote: > > I think this is a great patch but, in my view, an even better way to tackle the fundamental problem (the performance limitations) is to use a much faster checksum like xxhash, as has been suggested before: > https://lists.samba.org/archive/rsync/2019-October/031975.html > > Cheers, > Filipe > > On Mon, 18 May 2020 at 17:08, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: >> >> This drop-in patch increases the performance of...

..., hardlink-specials, > hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs, > xattrs, optional protect-args, iconv, prealloc, stop-at, no crtimesOptimizations: > SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5 > Checksum list: > xxh128 xxh3 xxh64 (xxhash) md5 md4 none > Compress list: > zstd lz4 zlibx zlib none > > OS: > Photon OS 3.0 with openssl fips enabled > > I'm using rsync like below: > > rsync -arXHpvxog <src> <dest> > > While doing so, if any file in src or dest location has a xattr valu...

On 2020-05-22 22:54:18 [-0700], Wayne Davison via rsync wrote: > Thanks for the optimizing patches, Jorrit! I've merged your latest changes > into the git master branch. Wouldn't it be better to add support for a crypto library (like openssl) which would provide optimized algorithms for more than just one platform without the need to maintain it separately? > ..wayne..

...particular, IIRC this is the function used for Clang's > identifier maps (via StringMap) and so I'd like to see some measurements > that ensure that these performance improvements translate over to Clang (or > at least don't regress). > > If Clang doesn't regress and xxHash is measurably better for other > HashString workloads, then I don't see a reason not to switch to it. > > -- Sean Silva > > On Mon, Apr 24, 2017 at 5:37 PM, Scott Smith via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> I've been working on improving t...

...ing from 64 to 512 bytes, with diminishing returns above, 4096 was where it seemed to plateau for me. Re-used CHUNK_SIZE (32 kB) as it already exists and should be fine to use here anyway. Noticed this because I'm playing with a parallel MD5 implementation, and it benchmarked about the same as xxhash on the CPUs I used for testing, which should not be the case. Discovered performance was limited by CSUM_CHUNK reads for multiple checksum types. Have observed near 3x performance gains in rsync --checksum from cached files with xxhash on the i7-7700hq. P.S. Wayne, in the xxhash part of file_chec...

...ce, rather for the rolling > checksum rsync uses to match blocks on existing files on both ends to > reduce transfer size. Still. You claim in your patch that | Benchmarks C SSE2 SSSE3 | - Intel i7-7700hq 1850 MB/s 2550 MB/s 4050 MB/s while xxhash [0] claims on a Core i5-3340M @2.7GHz that: |Version Speed on 64-bit Speed on 32-bit |XXH64 13.8 GB/s 1.9 GB/s so using xxhash64 for that work would also boost !x86 platforms. However your patch has the benefit that no changes are required on the remote side. I like that. [0] https://github.co...

...should be a nice improvement for x86 systems. I've tweaked your patch just a bit and changed it to where it is disabled by default unless configured with --enable-sse2, just to be extra cautious. Once it gets some validation testing we can change that to be enabled by default. Thanks! As for xxhash support, the current version of that patch is in the patches git repo (it hasn't been merged yet): https://git.samba.org/?p=rsync-patches.git;a=blob_plain;f=xxhash.diff;hb=6f3c2cd839ccbae8dc74222f9f8a60aef0156d43 ..wayne.. -------------- next part -------------- An HTML attachment was scrubb...

...inks, symtimes, hardlinks, hardlink-specials, hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs, xattrs, optional protect-args, iconv, prealloc, stop-at, no crtimesOptimizations: SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5 Checksum list: xxh128 xxh3 xxh64 (xxhash) md5 md4 none Compress list: zstd lz4 zlibx zlib none OS: Photon OS 3.0 with openssl fips enabled I'm using rsync like below: rsync -arXHpvxog <src> <dest> While doing so, if any file in src or dest location has a xattr value which is more than 32 bytes long, rsync segfaults...

...ell as identifying reliance on unspecified behaviour, on detection of post-link changes we can then look at improving our tooling to support whatever changes had been applied. Its critical for us that the watermark has minimal impact on build time and cryptographic security is not the goal. Hence, xxhash is used as our experiments showed it has minimal overhead. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191121/3a8a52f9/attachment.html>

https://bugzilla.samba.org/show_bug.cgi?id=13082 Bug ID: 13082 Summary: [REQ] Hardware / SSE based MD5 operations Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

...while the MD5 hash is a strong cryptographic hash used to _verify_ blocks and files. You wouldn't want to replace the MD5 hash with the Adler-based hash, they are of a different class. If you'd replace the MD5 hash with a different one, you'd replace it with one of the SHA's or even xxHash. On Mon, May 18, 2020 at 6:21 PM Ben RUBSON via rsync <rsync at lists.samba.org> wrote: > > Thank you Jorrit for your detailed answer. > > > On 18 May 2020, at 17:58, Jorrit Jongma via rsync <rsync at lists.samba.org> wrote: > > > > Well, don't get too ex...

...test this out. On Mon, Feb 25, 2019 at 5:08 PM Alexandre Ganea <alexandre.ganea at ubisoft.com> wrote: > > For enabling large memory pages, see this link: https://support.sisoftware.co.uk/knowledgebase.php?article=52 > > Meow hash isn't in the patch I posted, but you can use xxHash, it is good enough. Just add /hasher:xxhash to the LLD cmd-line. > > > -----Original Message----- > From: Leonardo Santagada <santagada at gmail.com> > Sent: Monday, February 25, 2019 11:05 AM > To: Alexandre Ganea <alexandre.ganea at ubisoft.com> > Cc: Zachary Turn...

...dre Ganea > > <alexandre.ganea at ubisoft.com> wrote: > > > > > > For enabling large memory pages, see this link: > https://support.sisoftware.co.uk/knowledgebase.php?article=52 > > > > > > Meow hash isn't in the patch I posted, but you can use xxHash, it is > good enough. Just add /hasher:xxhash to the LLD cmd-line. > > > > > > > > > -----Original Message----- > > > From: Leonardo Santagada <santagada at gmail.com> > > > Sent: Monday, February 25, 2019 11:05 AM > > > To: Alexandre G...

...go down to about 15sec. The patch doesn’t parallelize (yet) the Type merging itself, but we have an alternate multithread-suitable implementation of DenseHash which already supports lockless, wait-free, insert/fetch/resize. The prototype allows for testing different hashing algorithms, and indeed xxHash seems to be the best general-purpose choice. I’ve also added support for more specialized hardware-based hashes, like Casey Muratori’s Meow Hash (uses hardware AES SSE 4.2 instructions), which brings the figures down a bit. Future changes could write back the computed ghash stream back to OBJs if...

...tion of >> post-link changes we can then look at improving our tooling to support >> whatever >> changes had been applied. >> >> Its critical for us that the watermark has minimal impact on build time >> and >> cryptographic security is not the goal. Hence, xxhash is used as our >> experiments showed it has minimal overhead. >> >> Chris >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> &...

...Layout: 513 ms ( 1.2%) Commit to Disk: 5175 ms ( 11.9%) Commit Output File: 37 ms ( 0.1%) ------------------------------------------------- Total Link Time: 43366 ms (100.0%) LTO didn't help much :( Now I will try Alexandre patches and switch fo xxHash64 or meow hashing. I need to discover how to enable huge pages on my windows (1809) ps: Need to figure out how to limit the number of link jobs in ninja as that almost used the whole 128GB of ram on my machine. On our distributed build system we can limit linking jobs (which are the only strict lo...