similar to: Making LLD PDB generation faster

Leonardo, to answer to your questions, yes to all of them  You can take a look at this prototype/proposal: https://reviews.llvm.org/D55585 Overall, computing ghashes in parallel at link-time and merging Types with them is less costly that the current approach to merging. The 35sec you’re seeing for merging should go down to about 15sec. The patch doesn’t parallelize (yet) the Type merging

How do you compile LLD? There's a big difference between when using MSVC vs Clang. The parallel ghash patch I was mentioning is almost 2x as fast when using Clang 7.0+ vs. MSVC 15.9+, I don't know exactly why. I also suggest you use the Release target. You should also grab this patch: https://reviews.llvm.org/D55056 - I had to revert it because it was causing issues with LLDB. But it

That's good news. For having debug info, you could try adding /Z7 on the cmake cmd-line, such as -DCMAKE_CXX_FLAGS="/Z7". Or use the 'RelWithDebInfo' target instead of 'Release' and add -DCMAKE_CXX_FLAGS="/Ob2" (because that target uses /Ob1 as a default). Can you please send a patch on Phabricator if you fix the LLVM_ENABLE_PDB issue with Clang? The goal

Can you please try using Ninja instead? cmake -G Ninja f:/svn/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_OPTIMIZED_TABLEGEN=true -DLLVM_EXTERNAL_LLD_SOURCE_DIR=f:/svn/lld -DLLVM_TOOL_LLD_BUILD=true -DLLVM_ENABLE_LLD=true -DCMAKE_C_COMPILER="C:/Program Files/LLVM/bin/clang-cl.exe" -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang-cl.exe" -DCMAKE_LINKER="C:/Program

Times for lld compiled with LTO: Input File Reading: 1430 ms ( 3.3%) Code Layout: 486 ms ( 1.1%) PDB Emission (Cumulative): 41042 ms ( 94.6%) Add Objects: 33117 ms ( 76.4%) Type Merging: 25861 ms ( 59.6%) Symbol Merging: 7011 ms ( 16.2%) TPI Stream Layout: 996 ms ( 2.3%) Globals Stream Layout:

I think its a huge bug that it doesn't raise any errors or warnings about it. But I will open a ticket on cmake, they should be using clang-cl.exe and lld-link.exe if T="llvm" probably set host to 64 bit as well. On Mon, Feb 25, 2019 at 3:34 PM Zachary Turner <zturner at google.com> wrote: > > I don’t think changing the compiler or linker is supported with the vs

Yes, -Tllvm works. [cid:image002.jpg at 01D4CCF6.C440CFF0] From: Zachary Turner <zturner at google.com> Sent: Monday, February 25, 2019 10:36 AM To: Leonardo Santagada <santagada at gmail.com> Cc: Alexandre Ganea <alexandre.ganea at ubisoft.com>; Reid Kleckner <rnk at google.com>; llvm-dev <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] Making LLD PDB

Sadly the patch on https://reviews.llvm.org/D55585 didn't apply on my clone of llvm at all :( It will take me quite some time to test this out. On Mon, Feb 25, 2019 at 5:08 PM Alexandre Ganea <alexandre.ganea at ubisoft.com> wrote: > > For enabling large memory pages, see this link: https://support.sisoftware.co.uk/knowledgebase.php?article=52 > > Meow hash isn't in the

This could be ICF. There were lots of issues with ICF on ARM64, but they are not inherently ARM64-specific, they just come up there more often. See https://reviews.llvm.org/D56986 which fixes that. Easiest thing is always to profile or add /time to see what's slow. On Wed, Feb 27, 2019 at 6:30 AM Leonardo Santagada <santagada at gmail.com> wrote: > Anyone would know why lld takes

As for multithreaded ghashes: Even if the hashtable stores 32-bit indices to SeenHashes, you would still need to compare the ghashes for collisions: https://github.com/llvm/llvm-project/blob/master/llvm/include/llvm/ADT/DenseMap.h#L627 Finding the 32-bit index in the hashtable doesn’t necessarily mean it’s the right one. The following table shows the collision distribution when inserting (type)

So I found all 20 bytes and changed then to GHASH_SIZE (a const I defined in typehashing.h) and finished the switch to xxHash64, that saved me around 50 seconds to 56s, then I changed it to uint64_t instead of a 8 byte uint_8 array and that gave me 48s. With release config and a pgo pass I'm now linking in 38s... so faster than link.exe in vs 2017 (which is faster than vs 2015) doing fastlink.

You can make a PDB per lib (consider msvcrtd.pdb which ships with MSVC), but all these per-lib PDBs would have to be merged into a single master PDB at the end, so you still can't avoid that final . In a way, that's similar to the idea behind /DEBUG:FASTLINK (keep the debug info in object files to eliminate the cost of merging types and symbol records) and we know what the problems with

The quickest route would be for you to upload the patches for review and then go through a couple of iterations until it's cleaned up. Do you want to go that route? If not you can upload them anyway and whenever I get some spare cycles I can take it over and get it committed. But that might be slower since I have other things on my plate at the moment so I'm not sure when I'll be

Did you change both the compiler and linker (or make sure that your objcopy was updated to write your 64 bit hashes)? The linker is hardcodes to expect 20-byte sha 1s , anything else and it will recompute them in serial On Tue, Jan 30, 2018 at 12:28 PM Leonardo Santagada <santagada at gmail.com> wrote: > Nice and why are you trying blake2 instead of a faster hash algorithm? And > do

Uhmm I changed only type hashing... Ok back to trying it again. Let's me find where it is looking at 20 bytes instead of using the size of global type hash. On 30 Jan 2018 21:33, "Zachary Turner" <zturner at google.com> wrote: > Did you change both the compiler and linker (or make sure that your > objcopy was updated to write your 64 bit hashes)? > > The linker is

Nice and why are you trying blake2 instead of a faster hash algorithm? And do you have any guess as to why xxHash64 wasn't faster than SHA1? I still have to see how many collision I get with it, but it seems so improbable that collisions on 64 bit hashes would be the problem. On 30 Jan 2018 18:39, "Zachary Turner" <zturner at google.com> wrote: It turns out there were some

Today I played around replacing the sha1 with xxHash64 and the results so far are bad. Linking times almost doubled and I can't really explain why, the only thing that comes to mind is hash collisions but on type names they should be very few in 64bit hashes. Any reason why you are trying blake2 and not murmurhash3 or xxHash64? About creating a pdb per lib, you can say to msvc to put the pdb

Hi, We are starting to test binaries generated by lld on windows and we notices that sometimes the visual studio debugger can't see the content of variables or gets the call stack wrong for deeply nested or complex types. Did anyone else have the same problems, or any way to try to figure out what is missing? I tried llvm-pdbutil dump -symbols but there is a lot of small diferences on

Yes we've discussed many different ideas for incremental linking, but our conclusion is that you can only get one of Fast|Simple. If you want it to be fast it has to be complicated and if you want it to be simple then it's going to be slow. Consider the case where you edit one .cpp file and change this: int x = 0, y = 7; to this: int x = 0; short y = 7; Because different instructions

Does packing obj files in .lib helps linking in any way? My understanding is that there would be no difference. It could help if I could make a pdb per lib, but there is no way to do so... Maybe we could implement this on lld? On 29 Jan 2018 22:14, "Zachary Turner" <zturner at google.com> wrote: > Yes we've discussed many different ideas for incremental linking, but our