llvm dev - Jul 2015 - [LLVMdev] [LLD] support for dlltool generated libs in COFF/PECOFF

>
> It's doable to support dlltool-style import libraries by LLD. .idata
> sections in dlltool-style libraries are designed so that they will
> naturally construct the import descriptor table by just sorting them by
> section name. Supporting them is not hard.

I don't really know my way around the code base of COFF well enough todo
this myself.
If you can point me in the right direction as to what functions/sources to
be looking at I think I can come up with a patch.
If you think you can do it fairly easily I can also create a test case for
you if you want ?

One thing we need to do is that currently we construct the
import
> descriptor directly from short import libraries. We don't create
> intermediate .idata$X sections from short import libraries. That means we
> cannot mix dlltool-style and short import libraries at this moment. We need
> to do something for that. Except that, supporting dlltool-style libraries
> should be straightforward.

This is a very unique scenario and I don't know of any tool that generates
both styles of libraries.
I can create one if we need it for a test case but I think this would be
overkill :)


On Sat, Jul 25, 2015 at 11:29 PM, Rui Ueyama <ruiu at google.com> wrote:
> It's doable to support dlltool-style import libraries by LLD. .idata
> sections in dlltool-style libraries are designed so that they will
> naturally construct the import descriptor table by just sorting them by
> section name. Supporting them is not hard.
>
> One thing we need to do is that currently we construct the import
> descriptor directly from short import libraries. We don't create
> intermediate .idata$X sections from short import libraries. That means we
> cannot mix dlltool-style and short import libraries at this moment. We need
> to do something for that. Except that, supporting dlltool-style libraries
> should be straightforward.
>
> On Sat, Jul 25, 2015 at 3:10 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Hi Rui,
>>
>> Thanks for getting back to me on this
>>
>> Do you think that you can make dlltool to generate short import
>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>
>> On my current tests ld seems to not like libraries built with short
>> import libraries so this would require an ABI break in binutils.
>>
>> dlltool has support for loads of legacy targets and changing this would
>> also possibly break some of those targets so
>> Rather than working on dlltool I opted for creating a new tool for
>> mingw-64 to generate libs from def files using the "short import
library"
>> Another reason for me doing this is to have a complete clang based
>> mingw-w64 toolchain without any gnu gcc or binutils dependencies.
>>
>> It would be ideal however if we could support what dlltool currently
>> produces in lld.
>> mingw.org toolchains will not be using the new tool I am creating and
>> every distribution of mingw would have to rebuild their crt's just
to work
>> with lld.
>> This isn't ideal when clang is currenly used like a drop in add on
for
>> mingw-64 toolchains.
>> So if it is an easy enough fix we should support this in lld
>> Especially when MSVC already does.
>>
>> Is there much involved with supporting this on the lld end?
>>
>> Many Thanks
>> Martell
>>
>> On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> Hi Martell,
>>>
>>> Sorry for the belated response. I missed this email.
>>>
>>> I took a quick look at the library file you have attached and
noticed
>>> that the file consists of regular object files. LLD expects that
all
>>> dllimported symbols are described using "short import
library" (PE/COFF
>>> spec 7). Because I didn't test LLD with dlltool-style archive
files, it's
>>> no surprise that it didn't work (although it should
theoretically work).
>>>
>>> Do you think that you can make dlltool to generate short import
>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>
>>> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <martellmalone
at gmail.com
>>> > wrote:
>>>
>>>> Hi Ivan, :)
>>>>
>>>> For testing purposes, I've managed to make the old COFF
linker work
>>>>> with a simple test case
>>>>>
>>>> Have you got patches for this that I can look at ?
>>>>
>>>> Had to make two little changes, and link the exe manually,
because the
>>>>> linker doesn't understand the GNU flags
>>>>>
>>>> I've solved the issue of having a GNU driver for the linker
so that's
>>>> not an issue anymore
>>>> I sent in patches to the mailing list on this previously :)
>>>> I keep them updated over in the msys2 project btw
>>>>
>>>> By the way, even the MS linker works with the binutils
generated import
>>>>> libraries.
>>>>>
>>>> Yes i assume binutils was based from a very old version of a MS
linker
>>>> which did this layout at the time.
>>>> lld is so much newer and the MS linker has been updated to a
more
>>>> modern layout which is why we have this issue
>>>>
>>>> Kind Regards
>>>> Martell
>>>>
>>>>
>>>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>>>> heavenandhell171 at gmail.com> wrote:
>>>>
>>>>> Hi Martell.
>>>>>
>>>>> I too was trying to make the new COFF linker recognize the
libraries
>>>>> generated by binutils, but it wasn't resolving the
DLLs, just like you
>>>>> described. Apparently it is not identifying the libraries
as import
>>>>> libraries but archives.
>>>>>
>>>>> For testing purposes, i've managed to make the old COFF
linker work
>>>>> with a simple test case. Had to make two little changes,
and link the exe
>>>>> manually, because the linker doesn't understand the GNU
flags, but it
>>>>> works. By the way, even the MS linker works with the
binutils generated
>>>>> import libraries.
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150726/67b74c93/attachment.html>

On Sat, Jul 25, 2015 at 5:24 PM, Martell Malone <martellmalone at
gmail.com>
wrote:
> It's doable to support dlltool-style import libraries by LLD. .idata
>> sections in dlltool-style libraries are designed so that they will
>> naturally construct the import descriptor table by just sorting them by
>> section name. Supporting them is not hard.
>
>
> I don't really know my way around the code base of COFF well enough
todo
> this myself.
> If you can point me in the right direction as to what functions/sources to
> be looking at I think I can come up with a patch.
> If you think you can do it fairly easily I can also create a test case for
> you if you want ?
>
If it's not urgent, I can take a look after finishing up x86 COFF port and
fixing up remaining x64 bugs.

One thing we need to do is that currently we construct the
import
>> descriptor directly from short import libraries. We don't create
>> intermediate .idata$X sections from short import libraries. That means
we
>> cannot mix dlltool-style and short import libraries at this moment. We
need
>> to do something for that. Except that, supporting dlltool-style
libraries
>> should be straightforward.
>
>
> This is a very unique scenario and I don't know of any tool that
generates
> both styles of libraries.
> I can create one if we need it for a test case but I think this would be
> overkill :)
>
This is not actually very unique but rather common because you usually
always want to link against kernel32.lib which contains short import
libraries.

>
> On Sat, Jul 25, 2015 at 11:29 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> It's doable to support dlltool-style import libraries by LLD.
.idata
>> sections in dlltool-style libraries are designed so that they will
>> naturally construct the import descriptor table by just sorting them by
>> section name. Supporting them is not hard.
>>
>> One thing we need to do is that currently we construct the import
>> descriptor directly from short import libraries. We don't create
>> intermediate .idata$X sections from short import libraries. That means
we
>> cannot mix dlltool-style and short import libraries at this moment. We
need
>> to do something for that. Except that, supporting dlltool-style
libraries
>> should be straightforward.
>>
>> On Sat, Jul 25, 2015 at 3:10 PM, Martell Malone <martellmalone at
gmail.com>
>> wrote:
>>
>>> Hi Rui,
>>>
>>> Thanks for getting back to me on this
>>>
>>> Do you think that you can make dlltool to generate short import
>>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>>
>>> On my current tests ld seems to not like libraries built with short
>>> import libraries so this would require an ABI break in binutils.
>>>
>>> dlltool has support for loads of legacy targets and changing this
would
>>> also possibly break some of those targets so
>>> Rather than working on dlltool I opted for creating a new tool for
>>> mingw-64 to generate libs from def files using the "short
import library"
>>> Another reason for me doing this is to have a complete clang based
>>> mingw-w64 toolchain without any gnu gcc or binutils dependencies.
>>>
>>> It would be ideal however if we could support what dlltool
currently
>>> produces in lld.
>>> mingw.org toolchains will not be using the new tool I am creating
and
>>> every distribution of mingw would have to rebuild their crt's
just to work
>>> with lld.
>>> This isn't ideal when clang is currenly used like a drop in add
on for
>>> mingw-64 toolchains.
>>> So if it is an easy enough fix we should support this in lld
>>> Especially when MSVC already does.
>>>
>>> Is there much involved with supporting this on the lld end?
>>>
>>> Many Thanks
>>> Martell
>>>
>>> On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>
>>>> Hi Martell,
>>>>
>>>> Sorry for the belated response. I missed this email.
>>>>
>>>> I took a quick look at the library file you have attached and
noticed
>>>> that the file consists of regular object files. LLD expects
that all
>>>> dllimported symbols are described using "short import
library" (PE/COFF
>>>> spec 7). Because I didn't test LLD with dlltool-style
archive files, it's
>>>> no surprise that it didn't work (although it should
theoretically work).
>>>>
>>>> Do you think that you can make dlltool to generate short import
>>>> libraries? If you can, it would be the easiest way to solve the
issue.
>>>>
>>>> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <
>>>> martellmalone at gmail.com> wrote:
>>>>
>>>>> Hi Ivan, :)
>>>>>
>>>>> For testing purposes, I've managed to make the old COFF
linker work
>>>>>> with a simple test case
>>>>>>
>>>>> Have you got patches for this that I can look at ?
>>>>>
>>>>> Had to make two little changes, and link the exe manually,
because the
>>>>>> linker doesn't understand the GNU flags
>>>>>>
>>>>> I've solved the issue of having a GNU driver for the
linker so that's
>>>>> not an issue anymore
>>>>> I sent in patches to the mailing list on this previously :)
>>>>> I keep them updated over in the msys2 project btw
>>>>>
>>>>> By the way, even the MS linker works with the binutils
generated
>>>>>> import libraries.
>>>>>>
>>>>> Yes i assume binutils was based from a very old version of
a MS linker
>>>>> which did this layout at the time.
>>>>> lld is so much newer and the MS linker has been updated to
a more
>>>>> modern layout which is why we have this issue
>>>>>
>>>>> Kind Regards
>>>>> Martell
>>>>>
>>>>>
>>>>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>>>>> heavenandhell171 at gmail.com> wrote:
>>>>>
>>>>>> Hi Martell.
>>>>>>
>>>>>> I too was trying to make the new COFF linker recognize
the libraries
>>>>>> generated by binutils, but it wasn't resolving the
DLLs, just like you
>>>>>> described. Apparently it is not identifying the
libraries as import
>>>>>> libraries but archives.
>>>>>>
>>>>>> For testing purposes, i've managed to make the old
COFF linker work
>>>>>> with a simple test case. Had to make two little
changes, and link the exe
>>>>>> manually, because the linker doesn't understand the
GNU flags, but it
>>>>>> works. By the way, even the MS linker works with the
binutils generated
>>>>>> import libraries.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150725/de8759e4/attachment.html>

>
> If it's not urgent, I can take a look after finishing up x86 COFF port
and
> fixing up remaining x64 bugs.
I will create a test case for it then :)

 This is not actually very unique but rather common because you
usually
> always want to link against kernel32.lib which contains short import
> libraries.

Originally I thought you meant having both import types in the same lib.
You actually meant 2 separate libs one with each type. :)

I would note however that mingw doesn't use the kernel32.lib provided by MS
It has def file that was originally generated from the dll with the gendef
tool which is then given to dlltool when building the mingw crt.

If it is a mingw lib being used with visual studio however then yes this
needs to be supported
There are projects that have dll's built with mingw-w64 and that are linked
into visual studio like the vlc media player windows 8 app



On Sun, Jul 26, 2015 at 1:36 AM, Rui Ueyama <ruiu at google.com> wrote:
> On Sat, Jul 25, 2015 at 5:24 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> It's doable to support dlltool-style import libraries by LLD.
.idata
>>> sections in dlltool-style libraries are designed so that they will
>>> naturally construct the import descriptor table by just sorting
them by
>>> section name. Supporting them is not hard.
>>
>>
>> I don't really know my way around the code base of COFF well enough
todo
>> this myself.
>> If you can point me in the right direction as to what functions/sources
>> to be looking at I think I can come up with a patch.
>> If you think you can do it fairly easily I can also create a test case
>> for you if you want ?
>>
>
> If it's not urgent, I can take a look after finishing up x86 COFF port
and
> fixing up remaining x64 bugs.
>
> One thing we need to do is that currently we construct the import
>>> descriptor directly from short import libraries. We don't
create
>>> intermediate .idata$X sections from short import libraries. That
means we
>>> cannot mix dlltool-style and short import libraries at this moment.
We need
>>> to do something for that. Except that, supporting dlltool-style
libraries
>>> should be straightforward.
>>
>>
>> This is a very unique scenario and I don't know of any tool that
>> generates both styles of libraries.
>> I can create one if we need it for a test case but I think this would
be
>> overkill :)
>>
>
> This is not actually very unique but rather common because you usually
> always want to link against kernel32.lib which contains short import
> libraries.
>
>
>>
>> On Sat, Jul 25, 2015 at 11:29 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> It's doable to support dlltool-style import libraries by LLD.
.idata
>>> sections in dlltool-style libraries are designed so that they will
>>> naturally construct the import descriptor table by just sorting
them by
>>> section name. Supporting them is not hard.
>>>
>>> One thing we need to do is that currently we construct the import
>>> descriptor directly from short import libraries. We don't
create
>>> intermediate .idata$X sections from short import libraries. That
means we
>>> cannot mix dlltool-style and short import libraries at this moment.
We need
>>> to do something for that. Except that, supporting dlltool-style
libraries
>>> should be straightforward.
>>>
>>> On Sat, Jul 25, 2015 at 3:10 PM, Martell Malone <martellmalone
at gmail.com
>>> > wrote:
>>>
>>>> Hi Rui,
>>>>
>>>> Thanks for getting back to me on this
>>>>
>>>> Do you think that you can make dlltool to generate short import
>>>>> libraries? If you can, it would be the easiest way to solve
the issue.
>>>>>
>>>> On my current tests ld seems to not like libraries built with
short
>>>> import libraries so this would require an ABI break in
binutils.
>>>>
>>>> dlltool has support for loads of legacy targets and changing
this would
>>>> also possibly break some of those targets so
>>>> Rather than working on dlltool I opted for creating a new tool
for
>>>> mingw-64 to generate libs from def files using the "short
import library"
>>>> Another reason for me doing this is to have a complete clang
based
>>>> mingw-w64 toolchain without any gnu gcc or binutils
dependencies.
>>>>
>>>> It would be ideal however if we could support what dlltool
currently
>>>> produces in lld.
>>>> mingw.org toolchains will not be using the new tool I am
creating and
>>>> every distribution of mingw would have to rebuild their
crt's just to work
>>>> with lld.
>>>> This isn't ideal when clang is currenly used like a drop in
add on for
>>>> mingw-64 toolchains.
>>>> So if it is an easy enough fix we should support this in lld
>>>> Especially when MSVC already does.
>>>>
>>>> Is there much involved with supporting this on the lld end?
>>>>
>>>> Many Thanks
>>>> Martell
>>>>
>>>> On Sat, Jul 25, 2015 at 10:38 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>
>>>>> Hi Martell,
>>>>>
>>>>> Sorry for the belated response. I missed this email.
>>>>>
>>>>> I took a quick look at the library file you have attached
and noticed
>>>>> that the file consists of regular object files. LLD expects
that all
>>>>> dllimported symbols are described using "short import
library" (PE/COFF
>>>>> spec 7). Because I didn't test LLD with dlltool-style
archive files, it's
>>>>> no surprise that it didn't work (although it should
theoretically work).
>>>>>
>>>>> Do you think that you can make dlltool to generate short
import
>>>>> libraries? If you can, it would be the easiest way to solve
the issue.
>>>>>
>>>>> On Sat, Jul 25, 2015 at 6:23 AM, Martell Malone <
>>>>> martellmalone at gmail.com> wrote:
>>>>>
>>>>>> Hi Ivan, :)
>>>>>>
>>>>>> For testing purposes, I've managed to make the old
COFF linker work
>>>>>>> with a simple test case
>>>>>>>
>>>>>> Have you got patches for this that I can look at ?
>>>>>>
>>>>>> Had to make two little changes, and link the exe
manually, because
>>>>>>> the linker doesn't understand the GNU flags
>>>>>>>
>>>>>> I've solved the issue of having a GNU driver for
the linker so that's
>>>>>> not an issue anymore
>>>>>> I sent in patches to the mailing list on this
previously :)
>>>>>> I keep them updated over in the msys2 project btw
>>>>>>
>>>>>> By the way, even the MS linker works with the binutils
generated
>>>>>>> import libraries.
>>>>>>>
>>>>>> Yes i assume binutils was based from a very old version
of a MS
>>>>>> linker which did this layout at the time.
>>>>>> lld is so much newer and the MS linker has been updated
to a more
>>>>>> modern layout which is why we have this issue
>>>>>>
>>>>>> Kind Regards
>>>>>> Martell
>>>>>>
>>>>>>
>>>>>> On Sat, Jul 25, 2015 at 12:43 PM, Ivan Garramona <
>>>>>> heavenandhell171 at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martell.
>>>>>>>
>>>>>>> I too was trying to make the new COFF linker
recognize the libraries
>>>>>>> generated by binutils, but it wasn't resolving
the DLLs, just like you
>>>>>>> described. Apparently it is not identifying the
libraries as import
>>>>>>> libraries but archives.
>>>>>>>
>>>>>>> For testing purposes, i've managed to make the
old COFF linker work
>>>>>>> with a simple test case. Had to make two little
changes, and link the exe
>>>>>>> manually, because the linker doesn't understand
the GNU flags, but it
>>>>>>> works. By the way, even the MS linker works with
the binutils generated
>>>>>>> import libraries.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150726/cc1b220f/attachment.html>