llvm dev - Mar 2019 - Making LLD PDB generation faster

On Wed, Feb 27, 2019 at 3:17 PM Leonardo Santagada <santagada at
gmail.com>
wrote:
>
> Also we are investigating why we have 420 million types being linked while
> it appears that 95-99 % of them are not being used. De  anyone know if pch
> can help here? My feeling is not much as template instantiation still
> generates a ton of weak symbols on the pch users, but I might be confused.
>
This probably has to do with the fact that most types are duplicates.  If
there is a class Foo in a header file, and you include that header file in
100 different translation units, all 100 of them of them will get full type
information for that class in its object file.  This is the /Z7 semantics
that clang-cl implements.  The alternative is /Zi which uses a type server
(out of process de-duplicater) to do this merging at compile time, which is
what cl uses by default, and which clang-cl doesn't support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190227/c75d8485/attachment.html>

On Thu, 28 Feb 2019 at 00:28, Zachary Turner <zturner at google.com>
wrote:
>
>
> On Wed, Feb 27, 2019 at 3:17 PM Leonardo Santagada <santagada at
gmail.com>
> wrote:
>
>>
>> Also we are investigating why we have 420 million types being linked
>> while it appears that 95-99 % of them are not being used. De  anyone
know
>> if pch can help here? My feeling is not much as template instantiation
>> still generates a ton of weak symbols on the pch users, but I might be
>> confused.
>>
>
> This probably has to do with the fact that most types are duplicates.  If
> there is a class Foo in a header file, and you include that header file in
> 100 different translation units, all 100 of them of them will get full type
> information for that class in its object file.  This is the /Z7 semantics
> that clang-cl implements.  The alternative is /Zi which uses a type server
> (out of process de-duplicater) to do this merging at compile time, which is
> what cl uses by default, and which clang-cl doesn't support.
>
Sorry that's what I meant, we generate 420 million type definitions that
get de duplicated to less than 2 million.

We use a distributed build system... We use /Zi saving to one pdb per file
for fastlink or 100 pdbs for normal link, but then both compiling and
building is slow.

Maybe if we could make type server serve all distributed machines that
could work, but I don't think mspdbsrv works like that.
> --
Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190228/7dfaaf69/attachment.html>

We could implement /Zi with LLD running on the side, during the compilation, and
have clang pass the debuginfo along to LLD through shared memory pages. However
most of us use distributed builds, I’m not sure how practical that would be. We
would need to maintain a PDB per project, per remote worker, and get it back
once all CPPs are done compiling. Unless you compile all CPPs from a project on
the same remote worker, there wouldn’t be much gain?

An alternative would be to start merging (debug info) locally as soon as the
remote worker(s) sends back OBJs. You would still have the OBJ in the OS cache,
so that could speed up linking a bit; especially if your local host is idle
waiting for remote CPPs to finish compiling. All this would require for LLD to
run in some kind of daemon mode (start with the build process, and end when all
OBJs are available, to emit the final EXE and PDB).

Zachary what do you think?


From: Zachary Turner <zturner at google.com>
Sent: Wednesday, February 27, 2019 6:27 PM
To: Leonardo Santagada <santagada at gmail.com>
Cc: Reid Kleckner <rnk at google.com>; Alexandre Ganea <alexandre.ganea
at ubisoft.com>; llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] Making LLD PDB generation faster


On Wed, Feb 27, 2019 at 3:17 PM Leonardo Santagada <santagada at
gmail.com<mailto:santagada at gmail.com>> wrote:

Also we are investigating why we have 420 million types being linked while it
appears that 95-99 % of them are not being used. De  anyone know if pch can help
here? My feeling is not much as template instantiation still generates a ton of
weak symbols on the pch users, but I might be confused.

This probably has to do with the fact that most types are duplicates.  If there
is a class Foo in a header file, and you include that header file in 100
different translation units, all 100 of them of them will get full type
information for that class in its object file.  This is the /Z7 semantics that
clang-cl implements.  The alternative is /Zi which uses a type server (out of
process de-duplicater) to do this merging at compile time, which is what cl uses
by default, and which clang-cl doesn't support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190301/4fe6acca/attachment.html>

I read a bit and for us seems that using llvm-lib to generate thin
llvm libs with a type record similar to ranlib (with an extra
deduplicated .h).

This nicely form a reduce tree of pre linking. The main problem will
be that it doesn't integrate well with edit and continue solutions
like Live++.

On Fri, Mar 1, 2019 at 3:54 PM Alexandre Ganea
<alexandre.ganea at ubisoft.com> wrote:
>
> We could implement /Zi with LLD running on the side, during the
compilation, and have clang pass the debuginfo along to LLD through shared
memory pages. However most of us use distributed builds, I’m not sure how
practical that would be. We would need to maintain a PDB per project, per remote
worker, and get it back once all CPPs are done compiling. Unless you compile all
CPPs from a project on the same remote worker, there wouldn’t be much gain?
>
>
>
> An alternative would be to start merging (debug info) locally as soon as
the remote worker(s) sends back OBJs. You would still have the OBJ in the OS
cache, so that could speed up linking a bit; especially if your local host is
idle waiting for remote CPPs to finish compiling. All this would require for LLD
to run in some kind of daemon mode (start with the build process, and end when
all OBJs are available, to emit the final EXE and PDB).
>
>
>
> Zachary what do you think?
>
>
>
>
>
> From: Zachary Turner <zturner at google.com>
> Sent: Wednesday, February 27, 2019 6:27 PM
> To: Leonardo Santagada <santagada at gmail.com>
> Cc: Reid Kleckner <rnk at google.com>; Alexandre Ganea
<alexandre.ganea at ubisoft.com>; llvm-dev <llvm-dev at
lists.llvm.org>
> Subject: Re: [llvm-dev] Making LLD PDB generation faster
>
>
>
>
>
> On Wed, Feb 27, 2019 at 3:17 PM Leonardo Santagada <santagada at
gmail.com> wrote:
>
>
>
> Also we are investigating why we have 420 million types being linked while
it appears that 95-99 % of them are not being used. De  anyone know if pch can
help here? My feeling is not much as template instantiation still generates a
ton of weak symbols on the pch users, but I might be confused.
>
>
>
> This probably has to do with the fact that most types are duplicates.  If
there is a class Foo in a header file, and you include that header file in 100
different translation units, all 100 of them of them will get full type
information for that class in its object file.  This is the /Z7 semantics that
clang-cl implements.  The alternative is /Zi which uses a type server (out of
process de-duplicater) to do this merging at compile time, which is what cl uses
by default, and which clang-cl doesn't support.


-- 

Leonardo Santagada