llvm dev - Jan 2018 - [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)

I can totally see something like incremental linking with a simple padding
between obj and a mapping file (which can also help with edit and continue,
something we also would love to have).

We have another developer doing the port to support clang-cl, but although
most of our code also goes trough a version of clang, migrating the rest to
clang-cl has been a fight. From what I heard the main problem is that we
have a copy of parts of windows.h (so not to bring the awful parts of it
like lower case macros) and that totally works on cl, but clang (at least
6.0) complains about two struct/vars with the same name, even though they
are exactly the same. Making clang-cl as broken as cl.exe is not an option
I suppose? I would love to turn on a flag
--accept-that-cl-made-bad-decisions-and-live-with-it and have this at least
until this is completely fixed in our code base.

the biggest win with moving to cl would be a better more standards
compliant compiler, no 1 minute compiles on heavily templated files and
maybe the holy grail of ThinLTO.

On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner <zturner at google.com>
wrote:
> 10-15s will be hard without true incremental linking.
>
> At some point that's going to be the only way to get any faster, but
> incremental linking is hard (putting it lightly), and since our full links
> are already really fast we think we can get reasonably close to link.exe
> incremental speeds with full links.  But it's never enough and I will
> always want it to be faster, so you may see incremental linking in the
> future after we hit a performance wall with full link speed :)
>
> In any case, I'm definitely interested in seeing what kind of numbers
you
> get with /debug:ghash after you get this llvm-objcopy feature implemented.
> So keep me updated :)
>
> As an aside, have you tried building with clang instead of cl?  If you
> build with clang you wouldn't even have to do this llvm-objcopy work,
> because it would "just work".  If you've tried but ran into
issues I'm
> interested in hearing about those too.  On the other hand, it's also
> reasonable to only switch one thing at a time.
>
> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada <santagada at
gmail.com>
> wrote:
>
>> if we get to < 30s I think most users would prefer it to link.exe,
just
>> hopping there is still some more optimizations to get closer to ELF
linking
>> times (around 10-15s here).
>>
>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner <zturner at
google.com>
>> wrote:
>>
>>> Generally speaking a good rule of thumb is that /debug:ghash will
be
>>> close to or faster than /debug:fastlink, but with none of the
penalties
>>> like slow debug time
>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner <zturner at
google.com>
>>> wrote:
>>>
>>>> Chrome is actually one of my exact benchmark cases. When
building
>>>> blink_core.dll and browser_tests.exe, i get anywhere from a
20-40%
>>>> reduction in link time. We have some other optimizations in the
pipeline
>>>> but not upstream yet.
>>>>
>>>> My best time so far (including other optimizations not yet
upstream) is
>>>> 28s on blink_core.dll, compared to 110s with /debug
>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada <
>>>> santagada at gmail.com> wrote:
>>>>
>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner <zturner
at google.com>
>>>>> wrote:
>>>>>
>>>>>> You probably don't want to go down the same route
that clang goes
>>>>>> through to write the object file.  If you think
yaml2coff is convoluted,
>>>>>> the way clang does it will just give you a headache. 
There are multiple
>>>>>> abstractions involved to account for different object
file formats (ELF,
>>>>>> COFF, MachO) and output formats (Assembly, binary
file).  At least with
>>>>>> yaml2coff
>>>>>>
>>>>>
>>>>> I think your phrase got cut there, but yeah I just found
>>>>> AsmPrinter.cpp and it is convoluted.
>>>>>
>>>>>
>>>>>
>>>>>> It's true that yaml2coff is using the COFFParser
structure, but if
>>>>>> you look at the writeCOFF function in yaml2coff
it's pretty
>>>>>> bare-metal.  The logic you need will be almost
identical, except that
>>>>>> instead of checking the COFFParser for the various
fields, you'll check the
>>>>>> existing COFFObjectFile, which should have similar
fields.
>>>>>>
>>>>>> The only thing you need to different is when writing
the section
>>>>>> table and section contents, to insert a new entry. 
Since you're
>>>>>> injecting a section into the middle, you'll also
probably need to push back
>>>>>> the file pointer of all subsequent sections so that
they don't overlap.
>>>>>> (e.g. if the original sections are 1, 2, 3, 4, 5 and
you insert between 2
>>>>>> and 3, then the original sections 3, 4, and 5 would
need to have their
>>>>>> FilePointerToRawData offset by the size of the new
section).
>>>>>>
>>>>>
>>>>> I have the PE/COFF spec open here and I'm happy that I
read a bit of
>>>>> it so I actually know what you are talking about... yeah it
doesn't seem
>>>>> too complicated.
>>>>>
>>>>>
>>>>>
>>>>>> If you need to know what values to put for the other
fields in a
>>>>>> section header, run `dumpbin /headers foo.obj` on a
clang-generated object
>>>>>> file that has a .debug$H section already (e.g. run
clang with
>>>>>> -emit-codeview-ghash-section, and look at the
properties of the .debug$H
>>>>>> section and use the same values).
>>>>>>
>>>>>
>>>>> Thanks I will do that and then also look at how the
CodeView part of
>>>>> the code does it if I can't understand some of it.
>>>>>
>>>>>
>>>>>> The only invariant that needs to be maintained is that
Section[N]->FilePointerOfRawData =>>>>>>
Section[N-1]->FilePointerOfRawData + Section[N-1]->SizeOfRawData
>>>>>>
>>>>>
>>>>> Well, that and all the sections need to be on the final
file... But
>>>>> I'm hopeful.
>>>>>
>>>>>
>>>>> Anyone has times on linking a big project like chrome with
this so
>>>>> that at least I know what kind of performance to expect?
>>>>>
>>>>> My numbers are something like:
>>>>>
>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and 16GB of
ram,
>>>>> lld-link.exe takes 2:30 minutes and ~8GB of ram
>>>>> around 10 pdbs per folder: link.exe takes 1 minute and
2-3GB of ram,
>>>>> lld-link.exe takes 1:30 minutes and ~6GB of ram
>>>>> faslink: link.exe takes 40 seconds, but then 20 seconds of
loading at
>>>>> the first break point in the debugger and we lost DIA
support for listing
>>>>> symbols.
>>>>> incremental: link.exe takes 8 seconds, but it only happens
when very
>>>>> minor changes happen.
>>>>>
>>>>> We have an non negligible number of symbols used on some
runtime
>>>>> systems.
>>>>>
>>>>>
>>>>>>
>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo Santagada
<
>>>>>> santagada at gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for the tips, I now have something that
reads the obj file,
>>>>>>> finds .debug$T sections and global hashes it (proof
of concept kind of
>>>>>>> code). What I can't find is: how does clang
itself writes the coff files
>>>>>>> with global hashes, as that might help me
understand how to create the
>>>>>>> .debug$H section, how to update the file section
count and how to properly
>>>>>>> write this back.
>>>>>>>
>>>>>>> The code on yaml2coff is expecting to be working on
the yaml
>>>>>>> COFFParser struct and I'm having quite a bit of
a headache turning the
>>>>>>> COFFObjectFile into a COFFParser object or
compatible... Tomorrow I might
>>>>>>> try the very non efficient path of coff2yaml and
then yaml2coff with the
>>>>>>> hashes header... but it seems way too inefficient
and convoluted.
>>>>>>>
>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary Turner
<zturner at google.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo
Santagada <
>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM, Zachary
Turner <
>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM
Leonardo Santagada <
>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> No I didn't, I used cl.exe from
the visual studio toolchain.
>>>>>>>>>>> What I'm proposing is a tool
for processing .obj files in COFF format,
>>>>>>>>>>> reading them and generating the
GHASH part.
>>>>>>>>>>>
>>>>>>>>>>> To make our build faster we use
hundreds of unity build files
>>>>>>>>>>> (.cpp's with a lot of other
.cpp's in them aka munch files) but still have
>>>>>>>>>>> a lot of single .cpp's as well
(in total something like 3.4k .obj files).
>>>>>>>>>>>
>>>>>>>>>>> ps: sorry for sending to the wrong
list, I was reading about
>>>>>>>>>>> llvm mailing lists and jumped when
I saw what I thought was a lld exclusive
>>>>>>>>>>> list.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> A tool like this would be useful, yes. 
We've talked about it
>>>>>>>>>> internally as well and agreed it would
be useful, we just haven't
>>>>>>>>>> prioritized it.  If you're
interested in submitting a patch along those
>>>>>>>>>> lines though, I think it would be a
good addition.
>>>>>>>>>>
>>>>>>>>>> I'm not sure what the best place
for it would be.  llvm-readobj
>>>>>>>>>> and llvm-objdump seem like obvious
choices, but they are intended to be
>>>>>>>>>> read-only, so perhaps they wouldn't
be a good fit.
>>>>>>>>>>
>>>>>>>>>> llvm-pdbutil is kind of a hodgepodge of
everything else related
>>>>>>>>>> to PDBs and symbols, so I wouldn't
be opposed to making a new subcommand
>>>>>>>>>> there called "ghash" or
something that could process an object file and
>>>>>>>>>> output a new object file with a
.debug$H section.
>>>>>>>>>>
>>>>>>>>>> A third option would be to make a new
tool for it.
>>>>>>>>>>
>>>>>>>>>> I don't htink it would be that hard
to write.  If you're
>>>>>>>>>> interested in trying to make a patch
for this, I can offer some guidance on
>>>>>>>>>> where to look in the code.  Otherwise
it's something that we'll probably
>>>>>>>>>> get to, I'm just not sure when.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> I would love to write it and contribute it
back, please do tell, I
>>>>>>>>> did find some of the code of ghash in lld,
but in fuzzy on the llvm
>>>>>>>>> codeview part of it and never seen
llvm-readobj/objdump or llvm-pdbutil,
>>>>>>>>> but I'm not afraid to look :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>  Luckily all of the important code is hidden
behind library calls,
>>>>>>>> and it should already just do the right thing,
so I suspect you won't need
>>>>>>>> to know much about CodeView to do this.
>>>>>>>>
>>>>>>>> I think Peter has the right idea about putting
this in llvm-objcopy.
>>>>>>>>
>>>>>>>> You can look at one of the existing CopyBinary
functions there,
>>>>>>>> which currently only work for ELF, but you can
just make a new overload
>>>>>>>> that accepts a COFFObjectFile.
>>>>>>>>
>>>>>>>> I would probably start by iterating over each
of the sections
>>>>>>>> (getNumberOfSections / getSectionName) looking
for .debug$T and .debug$H
>>>>>>>> sections.
>>>>>>>>
>>>>>>>> If you find a .debug$H section then you can
just skip that object
>>>>>>>> file.
>>>>>>>>
>>>>>>>> If you find a .debug$T but not a .debug$H, then
basically do the
>>>>>>>> same thing that LLD does in
PDBLinker::mergeDebugT  (create a CVTypeArray,
>>>>>>>> and pass it to GloballyHashedType::hashTypes. 
That will return an
>>>>>>>> array of hash values.  (the format of .debug$H
is the header, followed by
>>>>>>>> the hash values).  Then when you're writing
the list of sections, just add
>>>>>>>> in the .debug$H section right after the
.debug$T section.
>>>>>>>>
>>>>>>>> Currently llvm-objcopy only writes ELF files,
so it would need to
>>>>>>>> be taught to write COFF files.  We have code to
do this in the yaml2obj
>>>>>>>> utility (specifically, in yaml2coff.cpp in the
function writeCOFF).  There
>>>>>>>> may be a way to move this code to somewhere
else (llvm/Object/COFF.h?) so
>>>>>>>> that it can be re-used by both yaml2coff and
llvm-objcopy, but in the worst
>>>>>>>> case scenario you could copy the code and
re-write it to work with these
>>>>>>>> new structures.
>>>>>>>>
>>>>>>>> Lastly, you'll probably want to put all of
this behind an option in
>>>>>>>> llvm-objcopy such as
-add-codeview-ghash-section
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Leonardo Santagada
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Leonardo Santagada
>>>>>
>>>>
>>
>>
>> --
>>
>> Leonardo Santagada
>>
>

-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180121/5a1edc84/attachment.html>

Clang-cl maintains compatibility with msvc even in cases where it’s non
standards compliant (eg 2 phase name lookup), but we try to keep these
cases few and far between.

To help me understand your case, do you mean you copy windows.h and modify
it? How does this lead to the same struct being defined twice? If i were to
write this:

struct Foo {};
struct Foo {};

Is this a small repro of the issue you’re talking about?
On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada <santagada at
gmail.com>
wrote:
> I can totally see something like incremental linking with a simple padding
> between obj and a mapping file (which can also help with edit and continue,
> something we also would love to have).
>
> We have another developer doing the port to support clang-cl, but although
> most of our code also goes trough a version of clang, migrating the rest to
> clang-cl has been a fight. From what I heard the main problem is that we
> have a copy of parts of windows.h (so not to bring the awful parts of it
> like lower case macros) and that totally works on cl, but clang (at least
> 6.0) complains about two struct/vars with the same name, even though they
> are exactly the same. Making clang-cl as broken as cl.exe is not an option
> I suppose? I would love to turn on a flag
> --accept-that-cl-made-bad-decisions-and-live-with-it and have this at least
> until this is completely fixed in our code base.
>
> the biggest win with moving to cl would be a better more standards
> compliant compiler, no 1 minute compiles on heavily templated files and
> maybe the holy grail of ThinLTO.
>
> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner <zturner at
google.com>
> wrote:
>
>> 10-15s will be hard without true incremental linking.
>>
>> At some point that's going to be the only way to get any faster,
but
>> incremental linking is hard (putting it lightly), and since our full
links
>> are already really fast we think we can get reasonably close to
link.exe
>> incremental speeds with full links.  But it's never enough and I
will
>> always want it to be faster, so you may see incremental linking in the
>> future after we hit a performance wall with full link speed :)
>>
>> In any case, I'm definitely interested in seeing what kind of
numbers you
>> get with /debug:ghash after you get this llvm-objcopy feature
implemented.
>> So keep me updated :)
>>
>> As an aside, have you tried building with clang instead of cl?  If you
>> build with clang you wouldn't even have to do this llvm-objcopy
work,
>> because it would "just work".  If you've tried but ran
into issues I'm
>> interested in hearing about those too.  On the other hand, it's
also
>> reasonable to only switch one thing at a time.
>>
>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada <santagada at
gmail.com>
>> wrote:
>>
>>> if we get to < 30s I think most users would prefer it to
link.exe, just
>>> hopping there is still some more optimizations to get closer to ELF
linking
>>> times (around 10-15s here).
>>>
>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner <zturner at
google.com>
>>> wrote:
>>>
>>>> Generally speaking a good rule of thumb is that /debug:ghash
will be
>>>> close to or faster than /debug:fastlink, but with none of the
penalties
>>>> like slow debug time
>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner <zturner at
google.com>
>>>> wrote:
>>>>
>>>>> Chrome is actually one of my exact benchmark cases. When
building
>>>>> blink_core.dll and browser_tests.exe, i get anywhere from a
20-40%
>>>>> reduction in link time. We have some other optimizations in
the pipeline
>>>>> but not upstream yet.
>>>>>
>>>>> My best time so far (including other optimizations not yet
upstream)
>>>>> is 28s on blink_core.dll, compared to 110s with /debug
>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada <
>>>>> santagada at gmail.com> wrote:
>>>>>
>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner
<zturner at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> You probably don't want to go down the same
route that clang goes
>>>>>>> through to write the object file.  If you think
yaml2coff is convoluted,
>>>>>>> the way clang does it will just give you a
headache.  There are multiple
>>>>>>> abstractions involved to account for different
object file formats (ELF,
>>>>>>> COFF, MachO) and output formats (Assembly, binary
file).  At least with
>>>>>>> yaml2coff
>>>>>>>
>>>>>>
>>>>>> I think your phrase got cut there, but yeah I just
found
>>>>>> AsmPrinter.cpp and it is convoluted.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> It's true that yaml2coff is using the
COFFParser structure, but if
>>>>>>> you look at the writeCOFF function in yaml2coff
it's pretty
>>>>>>> bare-metal.  The logic you need will be almost
identical, except that
>>>>>>> instead of checking the COFFParser for the various
fields, you'll check the
>>>>>>> existing COFFObjectFile, which should have similar
fields.
>>>>>>>
>>>>>>> The only thing you need to different is when
writing the section
>>>>>>> table and section contents, to insert a new entry. 
Since you're
>>>>>>> injecting a section into the middle, you'll
also probably need to push back
>>>>>>> the file pointer of all subsequent sections so that
they don't overlap.
>>>>>>> (e.g. if the original sections are 1, 2, 3, 4, 5
and you insert between 2
>>>>>>> and 3, then the original sections 3, 4, and 5 would
need to have their
>>>>>>> FilePointerToRawData offset by the size of the new
section).
>>>>>>>
>>>>>>
>>>>>> I have the PE/COFF spec open here and I'm happy
that I read a bit of
>>>>>> it so I actually know what you are talking about...
yeah it doesn't seem
>>>>>> too complicated.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> If you need to know what values to put for the
other fields in a
>>>>>>> section header, run `dumpbin /headers foo.obj` on a
clang-generated object
>>>>>>> file that has a .debug$H section already (e.g. run
clang with
>>>>>>> -emit-codeview-ghash-section, and look at the
properties of the .debug$H
>>>>>>> section and use the same values).
>>>>>>>
>>>>>>
>>>>>> Thanks I will do that and then also look at how the
CodeView part of
>>>>>> the code does it if I can't understand some of it.
>>>>>>
>>>>>>
>>>>>>> The only invariant that needs to be maintained is
that
>>>>>>> Section[N]->FilePointerOfRawData ==
Section[N-1]->FilePointerOfRawData +
>>>>>>> Section[N-1]->SizeOfRawData
>>>>>>>
>>>>>>
>>>>>> Well, that and all the sections need to be on the final
file... But
>>>>>> I'm hopeful.
>>>>>>
>>>>>>
>>>>>> Anyone has times on linking a big project like chrome
with this so
>>>>>> that at least I know what kind of performance to
expect?
>>>>>>
>>>>>> My numbers are something like:
>>>>>>
>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and 16GB
of ram,
>>>>>> lld-link.exe takes 2:30 minutes and ~8GB of ram
>>>>>> around 10 pdbs per folder: link.exe takes 1 minute and
2-3GB of ram,
>>>>>> lld-link.exe takes 1:30 minutes and ~6GB of ram
>>>>>> faslink: link.exe takes 40 seconds, but then 20 seconds
of loading at
>>>>>> the first break point in the debugger and we lost DIA
support for listing
>>>>>> symbols.
>>>>>> incremental: link.exe takes 8 seconds, but it only
happens when very
>>>>>> minor changes happen.
>>>>>>
>>>>>> We have an non negligible number of symbols used on
some runtime
>>>>>> systems.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo Santagada
<
>>>>>>> santagada at gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks for the tips, I now have something that
reads the obj file,
>>>>>>>> finds .debug$T sections and global hashes it
(proof of concept kind of
>>>>>>>> code). What I can't find is: how does clang
itself writes the coff files
>>>>>>>> with global hashes, as that might help me
understand how to create the
>>>>>>>> .debug$H section, how to update the file
section count and how to properly
>>>>>>>> write this back.
>>>>>>>>
>>>>>>>> The code on yaml2coff is expecting to be
working on the yaml
>>>>>>>> COFFParser struct and I'm having quite a
bit of a headache turning the
>>>>>>>> COFFObjectFile into a COFFParser object or
compatible... Tomorrow I might
>>>>>>>> try the very non efficient path of coff2yaml
and then yaml2coff with the
>>>>>>>> hashes header... but it seems way too
inefficient and convoluted.
>>>>>>>>
>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary
Turner <
>>>>>>>> zturner at google.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM Leonardo
Santagada <
>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM,
Zachary Turner <
>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29 PM
Leonardo Santagada <
>>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> No I didn't, I used cl.exe
from the visual studio toolchain.
>>>>>>>>>>>> What I'm proposing is a
tool for processing .obj files in COFF format,
>>>>>>>>>>>> reading them and generating the
GHASH part.
>>>>>>>>>>>>
>>>>>>>>>>>> To make our build faster we use
hundreds of unity build files
>>>>>>>>>>>> (.cpp's with a lot of other
.cpp's in them aka munch files) but still have
>>>>>>>>>>>> a lot of single .cpp's as
well (in total something like 3.4k .obj files).
>>>>>>>>>>>>
>>>>>>>>>>>> ps: sorry for sending to the
wrong list, I was reading about
>>>>>>>>>>>> llvm mailing lists and jumped
when I saw what I thought was a lld exclusive
>>>>>>>>>>>> list.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> A tool like this would be useful,
yes.  We've talked about it
>>>>>>>>>>> internally as well and agreed it
would be useful, we just haven't
>>>>>>>>>>> prioritized it.  If you're
interested in submitting a patch along those
>>>>>>>>>>> lines though, I think it would be a
good addition.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure what the best
place for it would be.  llvm-readobj
>>>>>>>>>>> and llvm-objdump seem like obvious
choices, but they are intended to be
>>>>>>>>>>> read-only, so perhaps they
wouldn't be a good fit.
>>>>>>>>>>>
>>>>>>>>>>> llvm-pdbutil is kind of a
hodgepodge of everything else related
>>>>>>>>>>> to PDBs and symbols, so I
wouldn't be opposed to making a new subcommand
>>>>>>>>>>> there called "ghash" or
something that could process an object file and
>>>>>>>>>>> output a new object file with a
.debug$H section.
>>>>>>>>>>>
>>>>>>>>>>> A third option would be to make a
new tool for it.
>>>>>>>>>>>
>>>>>>>>>>> I don't htink it would be that
hard to write.  If you're
>>>>>>>>>>> interested in trying to make a
patch for this, I can offer some guidance on
>>>>>>>>>>> where to look in the code. 
Otherwise it's something that we'll probably
>>>>>>>>>>> get to, I'm just not sure when.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> I would love to write it and contribute
it back, please do tell,
>>>>>>>>>> I did find some of the code of ghash in
lld, but in fuzzy on the llvm
>>>>>>>>>> codeview part of it and never seen
llvm-readobj/objdump or llvm-pdbutil,
>>>>>>>>>> but I'm not afraid to look :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>  Luckily all of the important code is
hidden behind library calls,
>>>>>>>>> and it should already just do the right
thing, so I suspect you won't need
>>>>>>>>> to know much about CodeView to do this.
>>>>>>>>>
>>>>>>>>> I think Peter has the right idea about
putting this in
>>>>>>>>> llvm-objcopy.
>>>>>>>>>
>>>>>>>>> You can look at one of the existing
CopyBinary functions there,
>>>>>>>>> which currently only work for ELF, but you
can just make a new overload
>>>>>>>>> that accepts a COFFObjectFile.
>>>>>>>>>
>>>>>>>>> I would probably start by iterating over
each of the sections
>>>>>>>>> (getNumberOfSections / getSectionName)
looking for .debug$T and .debug$H
>>>>>>>>> sections.
>>>>>>>>>
>>>>>>>>> If you find a .debug$H section then you can
just skip that object
>>>>>>>>> file.
>>>>>>>>>
>>>>>>>>> If you find a .debug$T but not a .debug$H,
then basically do the
>>>>>>>>> same thing that LLD does in
PDBLinker::mergeDebugT  (create a CVTypeArray,
>>>>>>>>> and pass it to
GloballyHashedType::hashTypes.  That will return an array of
>>>>>>>>> hash values.  (the format of .debug$H is
the header, followed by the hash
>>>>>>>>> values).  Then when you're writing the
list of sections, just add in the
>>>>>>>>> .debug$H section right after the .debug$T
section.
>>>>>>>>>
>>>>>>>>> Currently llvm-objcopy only writes ELF
files, so it would need to
>>>>>>>>> be taught to write COFF files.  We have
code to do this in the yaml2obj
>>>>>>>>> utility (specifically, in yaml2coff.cpp in
the function writeCOFF).  There
>>>>>>>>> may be a way to move this code to somewhere
else (llvm/Object/COFF.h?) so
>>>>>>>>> that it can be re-used by both yaml2coff
and llvm-objcopy, but in the worst
>>>>>>>>> case scenario you could copy the code and
re-write it to work with these
>>>>>>>>> new structures.
>>>>>>>>>
>>>>>>>>> Lastly, you'll probably want to put all
of this behind an option
>>>>>>>>> in llvm-objcopy such as
-add-codeview-ghash-section
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Leonardo Santagada
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Leonardo Santagada
>>>>>>
>>>>>
>>>
>>>
>>> --
>>>
>>> Leonardo Santagada
>>>
>>
>
>
> --
>
> Leonardo Santagada
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180121/e5d1f5fc/attachment-0001.html>

Here it is a minimal example, we do this so we don't have to import the
whole windows api everywhere.

https://gist.github.com/santagada/7977e929d31c629c4bf18ebb987f6be3

On Sun, Jan 21, 2018 at 2:31 AM, Zachary Turner <zturner at google.com>
wrote:
> Clang-cl maintains compatibility with msvc even in cases where it’s non
> standards compliant (eg 2 phase name lookup), but we try to keep these
> cases few and far between.
>
> To help me understand your case, do you mean you copy windows.h and modify
> it? How does this lead to the same struct being defined twice? If i were to
> write this:
>
> struct Foo {};
> struct Foo {};
>
> Is this a small repro of the issue you’re talking about?
>
> On Sat, Jan 20, 2018 at 3:44 PM Leonardo Santagada <santagada at
gmail.com>
> wrote:
>
>> I can totally see something like incremental linking with a simple
>> padding between obj and a mapping file (which can also help with edit
and
>> continue, something we also would love to have).
>>
>> We have another developer doing the port to support clang-cl, but
>> although most of our code also goes trough a version of clang,
migrating
>> the rest to clang-cl has been a fight. From what I heard the main
problem
>> is that we have a copy of parts of windows.h (so not to bring the awful
>> parts of it like lower case macros) and that totally works on cl, but
clang
>> (at least 6.0) complains about two struct/vars with the same name, even
>> though they are exactly the same. Making clang-cl as broken as cl.exe
is
>> not an option I suppose? I would love to turn on a flag
>> --accept-that-cl-made-bad-decisions-and-live-with-it and have this at
>> least until this is completely fixed in our code base.
>>
>> the biggest win with moving to cl would be a better more standards
>> compliant compiler, no 1 minute compiles on heavily templated files and
>> maybe the holy grail of ThinLTO.
>>
>> On Sat, Jan 20, 2018 at 10:56 PM, Zachary Turner <zturner at
google.com>
>> wrote:
>>
>>> 10-15s will be hard without true incremental linking.
>>>
>>> At some point that's going to be the only way to get any
faster, but
>>> incremental linking is hard (putting it lightly), and since our
full links
>>> are already really fast we think we can get reasonably close to
link.exe
>>> incremental speeds with full links.  But it's never enough and
I will
>>> always want it to be faster, so you may see incremental linking in
the
>>> future after we hit a performance wall with full link speed :)
>>>
>>> In any case, I'm definitely interested in seeing what kind of
numbers
>>> you get with /debug:ghash after you get this llvm-objcopy feature
>>> implemented.  So keep me updated :)
>>>
>>> As an aside, have you tried building with clang instead of cl?  If
you
>>> build with clang you wouldn't even have to do this llvm-objcopy
work,
>>> because it would "just work".  If you've tried but
ran into issues I'm
>>> interested in hearing about those too.  On the other hand, it's
also
>>> reasonable to only switch one thing at a time.
>>>
>>> On Sat, Jan 20, 2018 at 1:34 PM Leonardo Santagada <santagada at
gmail.com>
>>> wrote:
>>>
>>>> if we get to < 30s I think most users would prefer it to
link.exe, just
>>>> hopping there is still some more optimizations to get closer to
ELF linking
>>>> times (around 10-15s here).
>>>>
>>>> On Sat, Jan 20, 2018 at 9:50 PM, Zachary Turner <zturner at
google.com>
>>>> wrote:
>>>>
>>>>> Generally speaking a good rule of thumb is that
/debug:ghash will be
>>>>> close to or faster than /debug:fastlink, but with none of
the penalties
>>>>> like slow debug time
>>>>> On Sat, Jan 20, 2018 at 12:44 PM Zachary Turner <zturner
at google.com>
>>>>> wrote:
>>>>>
>>>>>> Chrome is actually one of my exact benchmark cases.
When building
>>>>>> blink_core.dll and browser_tests.exe, i get anywhere
from a 20-40%
>>>>>> reduction in link time. We have some other
optimizations in the pipeline
>>>>>> but not upstream yet.
>>>>>>
>>>>>> My best time so far (including other optimizations not
yet upstream)
>>>>>> is 28s on blink_core.dll, compared to 110s with /debug
>>>>>> On Sat, Jan 20, 2018 at 12:28 PM Leonardo Santagada
<
>>>>>> santagada at gmail.com> wrote:
>>>>>>
>>>>>>> On Sat, Jan 20, 2018 at 9:05 PM, Zachary Turner
<zturner at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You probably don't want to go down the same
route that clang goes
>>>>>>>> through to write the object file.  If you think
yaml2coff is convoluted,
>>>>>>>> the way clang does it will just give you a
headache.  There are multiple
>>>>>>>> abstractions involved to account for different
object file formats (ELF,
>>>>>>>> COFF, MachO) and output formats (Assembly,
binary file).  At least with
>>>>>>>> yaml2coff
>>>>>>>>
>>>>>>>
>>>>>>> I think your phrase got cut there, but yeah I just
found
>>>>>>> AsmPrinter.cpp and it is convoluted.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> It's true that yaml2coff is using the
COFFParser structure, but if
>>>>>>>> you look at the writeCOFF function in yaml2coff
it's pretty
>>>>>>>> bare-metal.  The logic you need will be almost
identical, except that
>>>>>>>> instead of checking the COFFParser for the
various fields, you'll check the
>>>>>>>> existing COFFObjectFile, which should have
similar fields.
>>>>>>>>
>>>>>>>> The only thing you need to different is when
writing the section
>>>>>>>> table and section contents, to insert a new
entry.  Since you're
>>>>>>>> injecting a section into the middle, you'll
also probably need to push back
>>>>>>>> the file pointer of all subsequent sections so
that they don't overlap.
>>>>>>>> (e.g. if the original sections are 1, 2, 3, 4,
5 and you insert between 2
>>>>>>>> and 3, then the original sections 3, 4, and 5
would need to have their
>>>>>>>> FilePointerToRawData offset by the size of the
new section).
>>>>>>>>
>>>>>>>
>>>>>>> I have the PE/COFF spec open here and I'm happy
that I read a bit of
>>>>>>> it so I actually know what you are talking about...
yeah it doesn't seem
>>>>>>> too complicated.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> If you need to know what values to put for the
other fields in a
>>>>>>>> section header, run `dumpbin /headers foo.obj`
on a clang-generated object
>>>>>>>> file that has a .debug$H section already (e.g.
run clang with
>>>>>>>> -emit-codeview-ghash-section, and look at the
properties of the .debug$H
>>>>>>>> section and use the same values).
>>>>>>>>
>>>>>>>
>>>>>>> Thanks I will do that and then also look at how the
CodeView part of
>>>>>>> the code does it if I can't understand some of
it.
>>>>>>>
>>>>>>>
>>>>>>>> The only invariant that needs to be maintained
is that Section[N]->FilePointerOfRawData =>>>>>>>>
Section[N-1]->FilePointerOfRawData + Section[N-1]->SizeOfRawData
>>>>>>>>
>>>>>>>
>>>>>>> Well, that and all the sections need to be on the
final file... But
>>>>>>> I'm hopeful.
>>>>>>>
>>>>>>>
>>>>>>> Anyone has times on linking a big project like
chrome with this so
>>>>>>> that at least I know what kind of performance to
expect?
>>>>>>>
>>>>>>> My numbers are something like:
>>>>>>>
>>>>>>> 1 pdb per obj file: link.exe takes ~15 minutes and
16GB of ram,
>>>>>>> lld-link.exe takes 2:30 minutes and ~8GB of ram
>>>>>>> around 10 pdbs per folder: link.exe takes 1 minute
and 2-3GB of ram,
>>>>>>> lld-link.exe takes 1:30 minutes and ~6GB of ram
>>>>>>> faslink: link.exe takes 40 seconds, but then 20
seconds of loading
>>>>>>> at the first break point in the debugger and we
lost DIA support for
>>>>>>> listing symbols.
>>>>>>> incremental: link.exe takes 8 seconds, but it only
happens when very
>>>>>>> minor changes happen.
>>>>>>>
>>>>>>> We have an non negligible number of symbols used on
some runtime
>>>>>>> systems.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Jan 20, 2018 at 11:52 AM Leonardo
Santagada <
>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the tips, I now have something
that reads the obj file,
>>>>>>>>> finds .debug$T sections and global hashes
it (proof of concept kind of
>>>>>>>>> code). What I can't find is: how does
clang itself writes the coff files
>>>>>>>>> with global hashes, as that might help me
understand how to create the
>>>>>>>>> .debug$H section, how to update the file
section count and how to properly
>>>>>>>>> write this back.
>>>>>>>>>
>>>>>>>>> The code on yaml2coff is expecting to be
working on the yaml
>>>>>>>>> COFFParser struct and I'm having quite
a bit of a headache turning the
>>>>>>>>> COFFObjectFile into a COFFParser object or
compatible... Tomorrow I might
>>>>>>>>> try the very non efficient path of
coff2yaml and then yaml2coff with the
>>>>>>>>> hashes header... but it seems way too
inefficient and convoluted.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 19, 2018 at 10:38 PM, Zachary
Turner <
>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 19, 2018 at 1:02 PM
Leonardo Santagada <
>>>>>>>>>> santagada at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 19, 2018 at 9:44 PM,
Zachary Turner <
>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 19, 2018 at 12:29
PM Leonardo Santagada <
>>>>>>>>>>>> santagada at gmail.com>
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> No I didn't, I used
cl.exe from the visual studio toolchain.
>>>>>>>>>>>>> What I'm proposing is a
tool for processing .obj files in COFF format,
>>>>>>>>>>>>> reading them and generating
the GHASH part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To make our build faster we
use hundreds of unity build files
>>>>>>>>>>>>> (.cpp's with a lot of
other .cpp's in them aka munch files) but still have
>>>>>>>>>>>>> a lot of single .cpp's
as well (in total something like 3.4k .obj files).
>>>>>>>>>>>>>
>>>>>>>>>>>>> ps: sorry for sending to
the wrong list, I was reading about
>>>>>>>>>>>>> llvm mailing lists and
jumped when I saw what I thought was a lld exclusive
>>>>>>>>>>>>> list.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> A tool like this would be
useful, yes.  We've talked about it
>>>>>>>>>>>> internally as well and agreed
it would be useful, we just haven't
>>>>>>>>>>>> prioritized it.  If you're
interested in submitting a patch along those
>>>>>>>>>>>> lines though, I think it would
be a good addition.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure what the best
place for it would be.  llvm-readobj
>>>>>>>>>>>> and llvm-objdump seem like
obvious choices, but they are intended to be
>>>>>>>>>>>> read-only, so perhaps they
wouldn't be a good fit.
>>>>>>>>>>>>
>>>>>>>>>>>> llvm-pdbutil is kind of a
hodgepodge of everything else related
>>>>>>>>>>>> to PDBs and symbols, so I
wouldn't be opposed to making a new subcommand
>>>>>>>>>>>> there called "ghash"
or something that could process an object file and
>>>>>>>>>>>> output a new object file with a
.debug$H section.
>>>>>>>>>>>>
>>>>>>>>>>>> A third option would be to make
a new tool for it.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't htink it would be
that hard to write.  If you're
>>>>>>>>>>>> interested in trying to make a
patch for this, I can offer some guidance on
>>>>>>>>>>>> where to look in the code. 
Otherwise it's something that we'll probably
>>>>>>>>>>>> get to, I'm just not sure
when.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> I would love to write it and
contribute it back, please do tell,
>>>>>>>>>>> I did find some of the code of
ghash in lld, but in fuzzy on the llvm
>>>>>>>>>>> codeview part of it and never seen
llvm-readobj/objdump or llvm-pdbutil,
>>>>>>>>>>> but I'm not afraid to look :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>  Luckily all of the important code is
hidden behind library
>>>>>>>>>> calls, and it should already just do
the right thing, so I suspect you
>>>>>>>>>> won't need to know much about
CodeView to do this.
>>>>>>>>>>
>>>>>>>>>> I think Peter has the right idea about
putting this in
>>>>>>>>>> llvm-objcopy.
>>>>>>>>>>
>>>>>>>>>> You can look at one of the existing
CopyBinary functions there,
>>>>>>>>>> which currently only work for ELF, but
you can just make a new overload
>>>>>>>>>> that accepts a COFFObjectFile.
>>>>>>>>>>
>>>>>>>>>> I would probably start by iterating
over each of the sections
>>>>>>>>>> (getNumberOfSections / getSectionName)
looking for .debug$T and .debug$H
>>>>>>>>>> sections.
>>>>>>>>>>
>>>>>>>>>> If you find a .debug$H section then you
can just skip that object
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> If you find a .debug$T but not a
.debug$H, then basically do the
>>>>>>>>>> same thing that LLD does in
PDBLinker::mergeDebugT  (create a CVTypeArray,
>>>>>>>>>> and pass it to
GloballyHashedType::hashTypes.  That will return
>>>>>>>>>> an array of hash values.  (the format
of .debug$H is the header, followed
>>>>>>>>>> by the hash values).  Then when
you're writing the list of sections, just
>>>>>>>>>> add in the .debug$H section right after
the .debug$T section.
>>>>>>>>>>
>>>>>>>>>> Currently llvm-objcopy only writes ELF
files, so it would need to
>>>>>>>>>> be taught to write COFF files.  We have
code to do this in the yaml2obj
>>>>>>>>>> utility (specifically, in yaml2coff.cpp
in the function writeCOFF).  There
>>>>>>>>>> may be a way to move this code to
somewhere else (llvm/Object/COFF.h?) so
>>>>>>>>>> that it can be re-used by both
yaml2coff and llvm-objcopy, but in the worst
>>>>>>>>>> case scenario you could copy the code
and re-write it to work with these
>>>>>>>>>> new structures.
>>>>>>>>>>
>>>>>>>>>> Lastly, you'll probably want to put
all of this behind an option
>>>>>>>>>> in llvm-objcopy such as
-add-codeview-ghash-section
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Leonardo Santagada
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Leonardo Santagada
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Leonardo Santagada
>>>>
>>>
>>
>>
>> --
>>
>> Leonardo Santagada
>>
>

-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180122/9e0ececd/attachment.html>