llvm dev - Jan 2018 - [lldb-dev] Trying out lld to link windows binaries (using msvc as a compiler)

On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada <santagada at
gmail.com>
wrote:
> Hi,
>
> No I didn't, I used cl.exe from the visual studio toolchain. What
I'm
> proposing is a tool for processing .obj files in COFF format, reading them
> and generating the GHASH part.
>
> To make our build faster we use hundreds of unity build files (.cpp's
with
> a lot of other .cpp's in them aka munch files) but still have a lot of
> single .cpp's as well (in total something like 3.4k .obj files).
>
> ps: sorry for sending to the wrong list, I was reading about llvm mailing
> lists and jumped when I saw what I thought was a lld exclusive list.
>
A tool like this would be useful, yes.  We've talked about it internally as
well and agreed it would be useful, we just haven't prioritized it.  If
you're interested in submitting a patch along those lines though, I think
it would be a good addition.

I'm not sure what the best place for it would be.  llvm-readobj and
llvm-objdump seem like obvious choices, but they are intended to be
read-only, so perhaps they wouldn't be a good fit.

llvm-pdbutil is kind of a hodgepodge of everything else related to PDBs and
symbols, so I wouldn't be opposed to making a new subcommand there called
"ghash" or something that could process an object file and output a
new
object file with a .debug$H section.

A third option would be to make a new tool for it.

I don't htink it would be that hard to write.  If you're interested in
trying to make a patch for this, I can offer some guidance on where to look
in the code.  Otherwise it's something that we'll probably get to,
I'm just
not sure when.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180119/146e560a/attachment.html>

On Fri, Jan 19, 2018 at 9:44 PM, Zachary Turner <zturner at google.com>
wrote:
>
>
> On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada <santagada at
gmail.com>
> wrote:
>
>> Hi,
>>
>> No I didn't, I used cl.exe from the visual studio toolchain. What
I'm
>> proposing is a tool for processing .obj files in COFF format, reading
them
>> and generating the GHASH part.
>>
>> To make our build faster we use hundreds of unity build files
(.cpp's
>> with a lot of other .cpp's in them aka munch files) but still have
a lot of
>> single .cpp's as well (in total something like 3.4k .obj files).
>>
>> ps: sorry for sending to the wrong list, I was reading about llvm
mailing
>> lists and jumped when I saw what I thought was a lld exclusive list.
>>
>
> A tool like this would be useful, yes.  We've talked about it
internally
> as well and agreed it would be useful, we just haven't prioritized it. 
If
> you're interested in submitting a patch along those lines though, I
think
> it would be a good addition.
>
> I'm not sure what the best place for it would be.  llvm-readobj and
> llvm-objdump seem like obvious choices, but they are intended to be
> read-only, so perhaps they wouldn't be a good fit.
>
> llvm-pdbutil is kind of a hodgepodge of everything else related to PDBs
> and symbols, so I wouldn't be opposed to making a new subcommand there
> called "ghash" or something that could process an object file and
output a
> new object file with a .debug$H section.
>
> A third option would be to make a new tool for it.
>
> I don't htink it would be that hard to write.  If you're interested
in
> trying to make a patch for this, I can offer some guidance on where to look
> in the code.  Otherwise it's something that we'll probably get to,
I'm just
> not sure when.
>
>>
I would love to write it and contribute it back, please do tell, I did find
some of the code of ghash in lld, but in fuzzy on the llvm codeview part of
it and never seen llvm-readobj/objdump or llvm-pdbutil, but I'm not afraid
to look :)


-- 

Leonardo Santagada
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180119/dd8a3ad6/attachment.html>

On Fri, Jan 19, 2018 at 12:44 PM, Zachary Turner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada <santagada at
gmail.com>
> wrote:
>
>> Hi,
>>
>> No I didn't, I used cl.exe from the visual studio toolchain. What
I'm
>> proposing is a tool for processing .obj files in COFF format, reading
them
>> and generating the GHASH part.
>>
>> To make our build faster we use hundreds of unity build files
(.cpp's
>> with a lot of other .cpp's in them aka munch files) but still have
a lot of
>> single .cpp's as well (in total something like 3.4k .obj files).
>>
>> ps: sorry for sending to the wrong list, I was reading about llvm
mailing
>> lists and jumped when I saw what I thought was a lld exclusive list.
>>
>
> A tool like this would be useful, yes.  We've talked about it
internally
> as well and agreed it would be useful, we just haven't prioritized it. 
If
> you're interested in submitting a patch along those lines though, I
think
> it would be a good addition.
>
> I'm not sure what the best place for it would be.  llvm-readobj and
> llvm-objdump seem like obvious choices, but they are intended to be
> read-only, so perhaps they wouldn't be a good fit.
>
> llvm-pdbutil is kind of a hodgepodge of everything else related to PDBs
> and symbols, so I wouldn't be opposed to making a new subcommand there
> called "ghash" or something that could process an object file and
output a
> new object file with a .debug$H section.
>
> A third option would be to make a new tool for it.
>
Another option would be to make it a feature of llvm-objcopy. That would
probably require adding COFF support to that tool though.

Peter
>
> I don't htink it would be that hard to write.  If you're interested
in
> trying to make a patch for this, I can offer some guidance on where to look
> in the code.  Otherwise it's something that we'll probably get to,
I'm just
> not sure when.
>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180119/1fc6ae6c/attachment.html>

That's a good idea too, I had forgotten about llvm-objcopy.

On Fri, Jan 19, 2018 at 1:02 PM Peter Collingbourne <peter at pcc.me.uk>
wrote:
> On Fri, Jan 19, 2018 at 12:44 PM, Zachary Turner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>>
>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada <santagada at
gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> No I didn't, I used cl.exe from the visual studio toolchain.
What I'm
>>> proposing is a tool for processing .obj files in COFF format,
reading them
>>> and generating the GHASH part.
>>>
>>> To make our build faster we use hundreds of unity build files
(.cpp's
>>> with a lot of other .cpp's in them aka munch files) but still
have a lot of
>>> single .cpp's as well (in total something like 3.4k .obj
files).
>>>
>>> ps: sorry for sending to the wrong list, I was reading about llvm
>>> mailing lists and jumped when I saw what I thought was a lld
exclusive list.
>>>
>>
>> A tool like this would be useful, yes.  We've talked about it
internally
>> as well and agreed it would be useful, we just haven't prioritized
it.  If
>> you're interested in submitting a patch along those lines though, I
think
>> it would be a good addition.
>>
>> I'm not sure what the best place for it would be.  llvm-readobj and
>> llvm-objdump seem like obvious choices, but they are intended to be
>> read-only, so perhaps they wouldn't be a good fit.
>>
>> llvm-pdbutil is kind of a hodgepodge of everything else related to PDBs
>> and symbols, so I wouldn't be opposed to making a new subcommand
there
>> called "ghash" or something that could process an object file
and output a
>> new object file with a .debug$H section.
>>
>> A third option would be to make a new tool for it.
>>
>
> Another option would be to make it a feature of llvm-objcopy. That would
> probably require adding COFF support to that tool though.
>
> Peter
>
>>
>> I don't htink it would be that hard to write.  If you're
interested in
>> trying to make a patch for this, I can offer some guidance on where to
look
>> in the code.  Otherwise it's something that we'll probably get
to, I'm just
>> not sure when.
>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
> --
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180119/1c277d13/attachment.html>

On Fri, Jan 19, 2018 at 1:02 PM Leonardo Santagada <santagada at
gmail.com>
wrote:
> On Fri, Jan 19, 2018 at 9:44 PM, Zachary Turner <zturner at
google.com>
> wrote:
>
>>
>>
>> On Fri, Jan 19, 2018 at 12:29 PM Leonardo Santagada <santagada at
gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> No I didn't, I used cl.exe from the visual studio toolchain.
What I'm
>>> proposing is a tool for processing .obj files in COFF format,
reading them
>>> and generating the GHASH part.
>>>
>>> To make our build faster we use hundreds of unity build files
(.cpp's
>>> with a lot of other .cpp's in them aka munch files) but still
have a lot of
>>> single .cpp's as well (in total something like 3.4k .obj
files).
>>>
>>> ps: sorry for sending to the wrong list, I was reading about llvm
>>> mailing lists and jumped when I saw what I thought was a lld
exclusive list.
>>>
>>
>> A tool like this would be useful, yes.  We've talked about it
internally
>> as well and agreed it would be useful, we just haven't prioritized
it.  If
>> you're interested in submitting a patch along those lines though, I
think
>> it would be a good addition.
>>
>> I'm not sure what the best place for it would be.  llvm-readobj and
>> llvm-objdump seem like obvious choices, but they are intended to be
>> read-only, so perhaps they wouldn't be a good fit.
>>
>> llvm-pdbutil is kind of a hodgepodge of everything else related to PDBs
>> and symbols, so I wouldn't be opposed to making a new subcommand
there
>> called "ghash" or something that could process an object file
and output a
>> new object file with a .debug$H section.
>>
>> A third option would be to make a new tool for it.
>>
>> I don't htink it would be that hard to write.  If you're
interested in
>> trying to make a patch for this, I can offer some guidance on where to
look
>> in the code.  Otherwise it's something that we'll probably get
to, I'm just
>> not sure when.
>>
>>>
> I would love to write it and contribute it back, please do tell, I did
> find some of the code of ghash in lld, but in fuzzy on the llvm codeview
> part of it and never seen llvm-readobj/objdump or llvm-pdbutil, but I'm
not
> afraid to look :)
>
>
 Luckily all of the important code is hidden behind library calls, and it
should already just do the right thing, so I suspect you won't need to know
much about CodeView to do this.

I think Peter has the right idea about putting this in llvm-objcopy.

You can look at one of the existing CopyBinary functions there, which
currently only work for ELF, but you can just make a new overload that
accepts a COFFObjectFile.

I would probably start by iterating over each of the sections
(getNumberOfSections / getSectionName) looking for .debug$T and .debug$H
sections.

If you find a .debug$H section then you can just skip that object file.

If you find a .debug$T but not a .debug$H, then basically do the same thing
that LLD does in PDBLinker::mergeDebugT  (create a CVTypeArray, and pass it
to GloballyHashedType::hashTypes.  That will return an array of hash
values.  (the format of .debug$H is the header, followed by the hash
values).  Then when you're writing the list of sections, just add in the
.debug$H section right after the .debug$T section.

Currently llvm-objcopy only writes ELF files, so it would need to be taught
to write COFF files.  We have code to do this in the yaml2obj utility
(specifically, in yaml2coff.cpp in the function writeCOFF).  There may be a
way to move this code to somewhere else (llvm/Object/COFF.h?) so that it
can be re-used by both yaml2coff and llvm-objcopy, but in the worst case
scenario you could copy the code and re-write it to work with these new
structures.

Lastly, you'll probably want to put all of this behind an option in
llvm-objcopy such as -add-codeview-ghash-section
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180119/455596c9/attachment.html>