llvm dev - Jul 2015 - [LLVMdev] [LLD] support for dlltool generated libs in COFF/PECOFF

Hi Rui,

Thanks for getting back to me on this

Do you think that you can make dlltool to generate short import
libraries?
> If you can, it would be the easiest way to solve the issue.
>
On my current tests ld seems to not like libraries built with short import
libraries so this would require an ABI break in binutils.

dlltool has support for loads of legacy targets and changing this would
also possibly break some of those targets so
Rather than working on dlltool I opted for creating a new tool for mingw-64
to generate libs from def files using the "short import library"
Another reason for me doing this is to have a complete clang based
mingw-w64 toolchain without any gnu gcc or binutils dependencies.

It would be ideal however if we could support what dlltool currently
produces in lld.
mingw.org toolchains will not be using the new tool I am creating and every
distribution of mingw would have to rebuild their crt's just to work with
lld.
This isn't ideal when clang is currenly used like a drop in add on for
mingw-64 toolchains.
So if it is an easy enough fix we should support this in lld
Especially when MSVC already does.

Is there much involved with supporting this on the lld end?

Many Thanks
Martell

On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at google.com> wrote:
> Hi Martell,
>
> Sorry for the belated response. I missed this email.
>
> I took a quick look at the library file you have attached and noticed that
> the file consists of regular object files. LLD expects that all dllimported
> symbols are described using "short import library" (PE/COFF spec
7).
> Because I didn't test LLD with dlltool-style archive files, it's no
> surprise that it didn't work (although it should theoretically work).
>
> Do you think that you can make dlltool to generate short import libraries?
> If you can, it would be the easiest way to solve the issue.
>
> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Hi Ivan, :)
>>
>> For testing purposes, I've managed to make the old COFF linker work
with
>>> a simple test case
>>>
>> Have you got patches for this that I can look at ?
>>
>> Had to make two little changes, and link the exe manually, because the
>>> linker doesn't understand the GNU flags
>>>
>> I've solved the issue of having a GNU driver for the linker so
that's not
>> an issue anymore
>> I sent in patches to the mailing list on this previously :)
>> I keep them updated over in the msys2 project btw
>>
>> By the way, even the MS linker works with the binutils generated import
>>> libraries.
>>>
>> Yes i assume binutils was based from a very old version of a MS linker
>> which did this layout at the time.
>> lld is so much newer and the MS linker has been updated to a more
modern
>> layout which is why we have this issue
>>
>> Kind Regards
>> Martell
>>
>>
>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>> heavenandhell171 at gmail.com> wrote:
>>
>>> Hi Martell.
>>>
>>> I too was trying to make the new COFF linker recognize the
libraries
>>> generated by binutils, but it wasn't resolving the DLLs, just
like you
>>> described. Apparently it is not identifying the libraries as import
>>> libraries but archives.
>>>
>>> For testing purposes, i've managed to make the old COFF linker
work with
>>> a simple test case. Had to make two little changes, and link the
exe
>>> manually, because the linker doesn't understand the GNU flags,
but it
>>> works. By the way, even the MS linker works with the binutils
generated
>>> import libraries.
>>>
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150725/9f90b7e2/attachment.html>

It's doable to support dlltool-style import libraries by LLD. .idata
sections in dlltool-style libraries are designed so that they will
naturally construct the import descriptor table by just sorting them by
section name. Supporting them is not hard.

One thing we need to do is that currently we construct the import
descriptor directly from short import libraries. We don't create
intermediate .idata$X sections from short import libraries. That means we
cannot mix dlltool-style and short import libraries at this moment. We need
to do something for that. Except that, supporting dlltool-style libraries
should be straightforward.

On Sat, Jul 25, 2015 at 3:10 PM, Martell Malone <martellmalone at
gmail.com>
wrote:
> Hi Rui,
>
> Thanks for getting back to me on this
>
> Do you think that you can make dlltool to generate short import libraries?
>> If you can, it would be the easiest way to solve the issue.
>>
> On my current tests ld seems to not like libraries built with short import
> libraries so this would require an ABI break in binutils.
>
> dlltool has support for loads of legacy targets and changing this would
> also possibly break some of those targets so
> Rather than working on dlltool I opted for creating a new tool for
> mingw-64 to generate libs from def files using the "short import
library"
> Another reason for me doing this is to have a complete clang based
> mingw-w64 toolchain without any gnu gcc or binutils dependencies.
>
> It would be ideal however if we could support what dlltool currently
> produces in lld.
> mingw.org toolchains will not be using the new tool I am creating and
> every distribution of mingw would have to rebuild their crt's just to
work
> with lld.
> This isn't ideal when clang is currenly used like a drop in add on for
> mingw-64 toolchains.
> So if it is an easy enough fix we should support this in lld
> Especially when MSVC already does.
>
> Is there much involved with supporting this on the lld end?
>
> Many Thanks
> Martell
>
> On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> Hi Martell,
>>
>> Sorry for the belated response. I missed this email.
>>
>> I took a quick look at the library file you have attached and noticed
>> that the file consists of regular object files. LLD expects that all
>> dllimported symbols are described using "short import
library" (PE/COFF
>> spec 7). Because I didn't test LLD with dlltool-style archive
files, it's
>> no surprise that it didn't work (although it should theoretically
work).
>>
>> Do you think that you can make dlltool to generate short import
>> libraries? If you can, it would be the easiest way to solve the issue.
>>
>> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <martellmalone at
gmail.com>
>> wrote:
>>
>>> Hi Ivan, :)
>>>
>>> For testing purposes, I've managed to make the old COFF linker
work with
>>>> a simple test case
>>>>
>>> Have you got patches for this that I can look at ?
>>>
>>> Had to make two little changes, and link the exe manually, because
the
>>>> linker doesn't understand the GNU flags
>>>>
>>> I've solved the issue of having a GNU driver for the linker so
that's
>>> not an issue anymore
>>> I sent in patches to the mailing list on this previously :)
>>> I keep them updated over in the msys2 project btw
>>>
>>> By the way, even the MS linker works with the binutils generated
import
>>>> libraries.
>>>>
>>> Yes i assume binutils was based from a very old version of a MS
linker
>>> which did this layout at the time.
>>> lld is so much newer and the MS linker has been updated to a more
modern
>>> layout which is why we have this issue
>>>
>>> Kind Regards
>>> Martell
>>>
>>>
>>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>>> heavenandhell171 at gmail.com> wrote:
>>>
>>>> Hi Martell.
>>>>
>>>> I too was trying to make the new COFF linker recognize the
libraries
>>>> generated by binutils, but it wasn't resolving the DLLs,
just like you
>>>> described. Apparently it is not identifying the libraries as
import
>>>> libraries but archives.
>>>>
>>>> For testing purposes, i've managed to make the old COFF
linker work
>>>> with a simple test case. Had to make two little changes, and
link the exe
>>>> manually, because the linker doesn't understand the GNU
flags, but it
>>>> works. By the way, even the MS linker works with the binutils
generated
>>>> import libraries.
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150725/770c7c14/attachment.html>

>
> It's doable to support dlltool-style import libraries by LLD. .idata
> sections in dlltool-style libraries are designed so that they will
> naturally construct the import descriptor table by just sorting them by
> section name. Supporting them is not hard.

I don't really know my way around the code base of COFF well enough todo
this myself.
If you can point me in the right direction as to what functions/sources to
be looking at I think I can come up with a patch.
If you think you can do it fairly easily I can also create a test case for
you if you want ?

One thing we need to do is that currently we construct the
import
> descriptor directly from short import libraries. We don't create
> intermediate .idata$X sections from short import libraries. That means we
> cannot mix dlltool-style and short import libraries at this moment. We need
> to do something for that. Except that, supporting dlltool-style libraries
> should be straightforward.

This is a very unique scenario and I don't know of any tool that generates
both styles of libraries.
I can create one if we need it for a test case but I think this would be
overkill :)


On Sat, Jul 25, 2015 at 11:29 PM, Rui Ueyama <ruiu at google.com> wrote:
> It's doable to support dlltool-style import libraries by LLD. .idata
> sections in dlltool-style libraries are designed so that they will
> naturally construct the import descriptor table by just sorting them by
> section name. Supporting them is not hard.
>
> One thing we need to do is that currently we construct the import
> descriptor directly from short import libraries. We don't create
> intermediate .idata$X sections from short import libraries. That means we
> cannot mix dlltool-style and short import libraries at this moment. We need
> to do something for that. Except that, supporting dlltool-style libraries
> should be straightforward.
>
> On Sat, Jul 25, 2015 at 3:10 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Hi Rui,
>>
>> Thanks for getting back to me on this
>>
>> Do you think that you can make dlltool to generate short import
>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>
>> On my current tests ld seems to not like libraries built with short
>> import libraries so this would require an ABI break in binutils.
>>
>> dlltool has support for loads of legacy targets and changing this would
>> also possibly break some of those targets so
>> Rather than working on dlltool I opted for creating a new tool for
>> mingw-64 to generate libs from def files using the "short import
library"
>> Another reason for me doing this is to have a complete clang based
>> mingw-w64 toolchain without any gnu gcc or binutils dependencies.
>>
>> It would be ideal however if we could support what dlltool currently
>> produces in lld.
>> mingw.org toolchains will not be using the new tool I am creating and
>> every distribution of mingw would have to rebuild their crt's just
to work
>> with lld.
>> This isn't ideal when clang is currenly used like a drop in add on
for
>> mingw-64 toolchains.
>> So if it is an easy enough fix we should support this in lld
>> Especially when MSVC already does.
>>
>> Is there much involved with supporting this on the lld end?
>>
>> Many Thanks
>> Martell
>>
>> On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> Hi Martell,
>>>
>>> Sorry for the belated response. I missed this email.
>>>
>>> I took a quick look at the library file you have attached and
noticed
>>> that the file consists of regular object files. LLD expects that
all
>>> dllimported symbols are described using "short import
library" (PE/COFF
>>> spec 7). Because I didn't test LLD with dlltool-style archive
files, it's
>>> no surprise that it didn't work (although it should
theoretically work).
>>>
>>> Do you think that you can make dlltool to generate short import
>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>
>>> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <martellmalone
at gmail.com
>>> > wrote:
>>>
>>>> Hi Ivan, :)
>>>>
>>>> For testing purposes, I've managed to make the old COFF
linker work
>>>>> with a simple test case
>>>>>
>>>> Have you got patches for this that I can look at ?
>>>>
>>>> Had to make two little changes, and link the exe manually,
because the
>>>>> linker doesn't understand the GNU flags
>>>>>
>>>> I've solved the issue of having a GNU driver for the linker
so that's
>>>> not an issue anymore
>>>> I sent in patches to the mailing list on this previously :)
>>>> I keep them updated over in the msys2 project btw
>>>>
>>>> By the way, even the MS linker works with the binutils
generated import
>>>>> libraries.
>>>>>
>>>> Yes i assume binutils was based from a very old version of a MS
linker
>>>> which did this layout at the time.
>>>> lld is so much newer and the MS linker has been updated to a
more
>>>> modern layout which is why we have this issue
>>>>
>>>> Kind Regards
>>>> Martell
>>>>
>>>>
>>>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>>>> heavenandhell171 at gmail.com> wrote:
>>>>
>>>>> Hi Martell.
>>>>>
>>>>> I too was trying to make the new COFF linker recognize the
libraries
>>>>> generated by binutils, but it wasn't resolving the
DLLs, just like you
>>>>> described. Apparently it is not identifying the libraries
as import
>>>>> libraries but archives.
>>>>>
>>>>> For testing purposes, i've managed to make the old COFF
linker work
>>>>> with a simple test case. Had to make two little changes,
and link the exe
>>>>> manually, because the linker doesn't understand the GNU
flags, but it
>>>>> works. By the way, even the MS linker works with the
binutils generated
>>>>> import libraries.
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150726/67b74c93/attachment.html>