llvm dev - Feb 2017 - RFC: A new llvm-dlltool driver and llvm-lib driver improvements

Hey Rui,
> I wonder how llvm-dlltool would fit in the entire picture of mingw
> support. I don't think dlltool is the last missing piece. What do you
need
> to do other than that to fully support mingw using LLVM toolchain?
Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
within the EmitCommonSymbol function and a single patch for mingw-w64
itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is
infact the only missing part really.
This gives us a fully working clang based mingw-w64 C compiler.

C++ and exception handling is a different story.

libc++ is somewhat working with the following test results

Expected Passes    : 2188
Expected Failures  : 44
Unsupported Tests  : 588
Unexpected Failures: 2816

I was able to build it with exceptions disabled and I actually managed to
bootstrap llvm and clang itself.
It didn't run very well though as you can imagine based on the above tests.

It's not a performance issue but a code maintenance issue. The
initial
> patches to support mingw was trying to add a new linker driver and a
> different linkin semantics to the COFF linker which seemed too complicated
> to me. IIRC, I suggested adding a shim which translates GNU command line
> arguments to MSVC linker arguments. Didn't it work?
There were just so many differences between link and ld I never followed
down that path. I pushed forward with the short import library support
based on your later suggestions. My patch to hack lld into accepting some
very basic gnu front end arguments was enough to get all the above working
which was enough to develop further.

Best,
Martell

On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote:
> Hi Martell,
>
> I wonder how llvm-dlltool would fit in the entire picture of mingw
> support. I don't think dlltool is the last missing piece. What do you
need
> to do other than that to fully support mingw using LLVM toolchain?
>
> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hey llvm'ers,
>>
>> I have been working on a dlltool replacement for llvm.
>> Here is my initial differential https://reviews.llvm.org/D29892
>> It is based on some functionality that already exists in lld.
>> I added functionality to support, PE COFF Weak Externals and of course
a
>> front end to actually use it.
>> I believe the work here can also be used for llvm-lib and lessen the
load
>> on lld.
>> I would like some comments about how this could be be structured to
live
>> in llvm with a shared code base across lib ar and dlltool.
>> I also have a section below called "Difference from lib"
which is
>> somewhat of a rationale for the tool.
>>
>> Many Thanks,
>> Martell
>>
>>
>> Context
>> =========>>
>> Awhile back I talked to various llvm'ers about getting mingw-w64
support
>> for lld.
>> There were a few issues raised but the main issue was that mingw-w64
>> should try best to comply with the PECOFF spec, adding support for
custom
>> sections and various binutils/mingw hacks would impact the performance
of
>> the COFF linker and in general is not something that lld should
support.
>>
>
> It's not a performance issue but a code maintenance issue. The initial
> patches to support mingw was trying to add a new linker driver and a
> different linkin semantics to the COFF linker which seemed too complicated
> to me. IIRC, I suggested adding a shim which translates GNU command line
> arguments to MSVC linker arguments. Didn't it work?
>
>
>> Motivation
>> =========>>
>> The main motivation was because dlltool and ld did not comply with
PECOFF
>> Spec.
>> It has some custom formatting and uses the assembler for some reason to
>> generate import libraries, it did not use the short import library
format
>> at all.
>>
>> There has been many work arounds for the problem this creates such as
the
>> creation of the `reimp` tool. Which imports MSVC built libs creates a
def
>> and uses dlltool so that the binutils linker can use it.
>>
>> We should just be using the short import format and the linker should
>> support that.
>> Thus llvm-dlltool was born.
>>
>>
>> Difference from lib
>> ==================>>
>> Using PE COFF spec (section 8, Import Library Format) should be self
>> explanatory.
>> lib.exe is able to accept def files and create libraries using this
>> format.
>>
>> example
>>
>>   `LIBRARY "user32.dll"
>>    EXPORTS
>>    MessageBoxA`
>>
>> LIB.exe can create a user32.lib with the function MessageBoxA from the
>> above definition.
>>
>> Mingw-w64 is different MSVC in that we need to compile the runtime.
>> MS provide us with their crt prebuilt so lib.exe doesn't have
support for
>> external function aliasing.
>> We often use aliases for posix naming reasons as well as avoid using
the
>> MS version of a function.
>>
>> example
>>
>>   `LIBRARY "user32.dll"
>>    EXPORTS
>>    MessageBoxA
>>    MessageBoxW==MessageBoxA`
>>
>> LIB.exe doesn't not have support for this but dlltool does.
>> The best fit for this within the spec is PE COFF spec (Aux Format 3:
>> Weak Externals)
>>
>> Mingw-w64 also uses lib prefix and .a file extensions for the library
>> name so the driver should be different. The above example would be
>> libuser32.a, for when shared and static versions of the same library
exist
>> there is the .dll.a variant.
>>
>>
>> Issues
>> ======>>
>> Like more of the gnu suite dlltool was not designed in one build multi
>> target manner.
>> We would have to introduce custom arguments to specify the target,
>> "i686", "x86_64", "thumb2pe" etc.
>> This will most likely break some level of backwards compatibility with
>> the binutils version.
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/94383874/attachment.html>

On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at
gmail.com>
wrote:
> Hey Rui,
>
>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>> support. I don't think dlltool is the last missing piece. What do
you need
>> to do other than that to fully support mingw using LLVM toolchain?
>
> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
> within the EmitCommonSymbol function and a single patch for mingw-w64
> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is
> infact the only missing part really.
>
Also you need to make a change to LLD/COFF to accept GNU command arguments,
right? (Looks like you already have that patch locally.)

> This gives us a fully working clang based mingw-w64 C compiler.
>
> C++ and exception handling is a different story.
>
> libc++ is somewhat working with the following test results
>
> Expected Passes    : 2188
> Expected Failures  : 44
> Unsupported Tests  : 588
> Unexpected Failures: 2816
>
> I was able to build it with exceptions disabled and I actually managed to
> bootstrap llvm and clang itself.
> It didn't run very well though as you can imagine based on the above
tests.
>
> It's not a performance issue but a code maintenance issue. The initial
>> patches to support mingw was trying to add a new linker driver and a
>> different linkin semantics to the COFF linker which seemed too
complicated
>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>> arguments to MSVC linker arguments. Didn't it work?
>
> There were just so many differences between link and ld I never followed
> down that path. I pushed forward with the short import library support
> based on your later suggestions. My patch to hack lld into accepting some
> very basic gnu front end arguments was enough to get all the above working
> which was enough to develop further.
>
> Best,
> Martell
>
> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> Hi Martell,
>>
>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>> support. I don't think dlltool is the last missing piece. What do
you need
>> to do other than that to fully support mingw using LLVM toolchain?
>>
>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hey llvm'ers,
>>>
>>> I have been working on a dlltool replacement for llvm.
>>> Here is my initial differential https://reviews.llvm.org/D29892
>>> It is based on some functionality that already exists in lld.
>>> I added functionality to support, PE COFF Weak Externals and of
course
>>> a front end to actually use it.
>>> I believe the work here can also be used for llvm-lib and lessen
the
>>> load on lld.
>>> I would like some comments about how this could be be structured to
live
>>> in llvm with a shared code base across lib ar and dlltool.
>>> I also have a section below called "Difference from lib"
which is
>>> somewhat of a rationale for the tool.
>>>
>>> Many Thanks,
>>> Martell
>>>
>>>
>>> Context
>>> =========>>>
>>> Awhile back I talked to various llvm'ers about getting
mingw-w64 support
>>> for lld.
>>> There were a few issues raised but the main issue was that
mingw-w64
>>> should try best to comply with the PECOFF spec, adding support for
custom
>>> sections and various binutils/mingw hacks would impact the
performance of
>>> the COFF linker and in general is not something that lld should
support.
>>>
>>
>> It's not a performance issue but a code maintenance issue. The
initial
>> patches to support mingw was trying to add a new linker driver and a
>> different linkin semantics to the COFF linker which seemed too
complicated
>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>> arguments to MSVC linker arguments. Didn't it work?
>>
>>
>>> Motivation
>>> =========>>>
>>> The main motivation was because dlltool and ld did not comply with
>>> PECOFF Spec.
>>> It has some custom formatting and uses the assembler for some
reason to
>>> generate import libraries, it did not use the short import library
format
>>> at all.
>>>
>>> There has been many work arounds for the problem this creates such
as
>>> the creation of the `reimp` tool. Which imports MSVC built libs
creates a
>>> def and uses dlltool so that the binutils linker can use it.
>>>
>>> We should just be using the short import format and the linker
should
>>> support that.
>>> Thus llvm-dlltool was born.
>>>
>>>
>>> Difference from lib
>>> ==================>>>
>>> Using PE COFF spec (section 8, Import Library Format) should be
self
>>> explanatory.
>>> lib.exe is able to accept def files and create libraries using this
>>> format.
>>>
>>> example
>>>
>>>   `LIBRARY "user32.dll"
>>>    EXPORTS
>>>    MessageBoxA`
>>>
>>> LIB.exe can create a user32.lib with the function MessageBoxA from
the
>>> above definition.
>>>
>>> Mingw-w64 is different MSVC in that we need to compile the runtime.
>>> MS provide us with their crt prebuilt so lib.exe doesn't have
support
>>> for external function aliasing.
>>> We often use aliases for posix naming reasons as well as avoid
using the
>>> MS version of a function.
>>>
>>> example
>>>
>>>   `LIBRARY "user32.dll"
>>>    EXPORTS
>>>    MessageBoxA
>>>    MessageBoxW==MessageBoxA`
>>>
>>> LIB.exe doesn't not have support for this but dlltool does.
>>> The best fit for this within the spec is PE COFF spec (Aux Format
3:
>>> Weak Externals)
>>>
>>> Mingw-w64 also uses lib prefix and .a file extensions for the
library
>>> name so the driver should be different. The above example would be
>>> libuser32.a, for when shared and static versions of the same
library exist
>>> there is the .dll.a variant.
>>>
>>>
>>> Issues
>>> ======>>>
>>> Like more of the gnu suite dlltool was not designed in one build
multi
>>> target manner.
>>> We would have to introduce custom arguments to specify the target,
>>> "i686", "x86_64", "thumb2pe" etc.
>>> This will most likely break some level of backwards compatibility
with
>>> the binutils version.
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/c9f67d1f/attachment.html>

>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
Yes
> My patch to hack lld into accepting some very basic gnu front end
> arguments was enough to get all the above working which was enough to
> develop further.


On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Hey Rui,
>>
>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>> support. I don't think dlltool is the last missing piece. What
do you need
>>> to do other than that to fully support mingw using LLVM toolchain?
>>
>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
>> within the EmitCommonSymbol function and a single patch for mingw-w64
>> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool
is
>> infact the only missing part really.
>>
>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
>
>
>> This gives us a fully working clang based mingw-w64 C compiler.
>>
>> C++ and exception handling is a different story.
>>
>> libc++ is somewhat working with the following test results
>>
>> Expected Passes    : 2188
>> Expected Failures  : 44
>> Unsupported Tests  : 588
>> Unexpected Failures: 2816
>>
>> I was able to build it with exceptions disabled and I actually managed
to
>> bootstrap llvm and clang itself.
>> It didn't run very well though as you can imagine based on the
above
>> tests.
>>
>> It's not a performance issue but a code maintenance issue. The
initial
>>> patches to support mingw was trying to add a new linker driver and
a
>>> different linkin semantics to the COFF linker which seemed too
complicated
>>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>>> arguments to MSVC linker arguments. Didn't it work?
>>
>> There were just so many differences between link and ld I never
followed
>> down that path. I pushed forward with the short import library support
>> based on your later suggestions. My patch to hack lld into accepting
some
>> very basic gnu front end arguments was enough to get all the above
working
>> which was enough to develop further.
>>
>> Best,
>> Martell
>>
>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> Hi Martell,
>>>
>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>> support. I don't think dlltool is the last missing piece. What
do you need
>>> to do other than that to fully support mingw using LLVM toolchain?
>>>
>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hey llvm'ers,
>>>>
>>>> I have been working on a dlltool replacement for llvm.
>>>> Here is my initial differential https://reviews.llvm.org/D29892
>>>> It is based on some functionality that already exists in lld.
>>>> I added functionality to support, PE COFF Weak Externals and of
course
>>>> a front end to actually use it.
>>>> I believe the work here can also be used for llvm-lib and
lessen the
>>>> load on lld.
>>>> I would like some comments about how this could be be
structured to
>>>> live in llvm with a shared code base across lib ar and dlltool.
>>>> I also have a section below called "Difference from
lib" which is
>>>> somewhat of a rationale for the tool.
>>>>
>>>> Many Thanks,
>>>> Martell
>>>>
>>>>
>>>> Context
>>>> =========>>>>
>>>> Awhile back I talked to various llvm'ers about getting
mingw-w64
>>>> support for lld.
>>>> There were a few issues raised but the main issue was that
mingw-w64
>>>> should try best to comply with the PECOFF spec, adding support
for custom
>>>> sections and various binutils/mingw hacks would impact the
performance of
>>>> the COFF linker and in general is not something that lld should
support.
>>>>
>>>
>>> It's not a performance issue but a code maintenance issue. The
initial
>>> patches to support mingw was trying to add a new linker driver and
a
>>> different linkin semantics to the COFF linker which seemed too
complicated
>>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>>> arguments to MSVC linker arguments. Didn't it work?
>>>
>>>
>>>> Motivation
>>>> =========>>>>
>>>> The main motivation was because dlltool and ld did not comply
with
>>>> PECOFF Spec.
>>>> It has some custom formatting and uses the assembler for some
reason to
>>>> generate import libraries, it did not use the short import
library format
>>>> at all.
>>>>
>>>> There has been many work arounds for the problem this creates
such as
>>>> the creation of the `reimp` tool. Which imports MSVC built libs
creates a
>>>> def and uses dlltool so that the binutils linker can use it.
>>>>
>>>> We should just be using the short import format and the linker
should
>>>> support that.
>>>> Thus llvm-dlltool was born.
>>>>
>>>>
>>>> Difference from lib
>>>> ==================>>>>
>>>> Using PE COFF spec (section 8, Import Library Format) should be
self
>>>> explanatory.
>>>> lib.exe is able to accept def files and create libraries using
this
>>>> format.
>>>>
>>>> example
>>>>
>>>>   `LIBRARY "user32.dll"
>>>>    EXPORTS
>>>>    MessageBoxA`
>>>>
>>>> LIB.exe can create a user32.lib with the function MessageBoxA
from the
>>>> above definition.
>>>>
>>>> Mingw-w64 is different MSVC in that we need to compile the
runtime.
>>>> MS provide us with their crt prebuilt so lib.exe doesn't
have support
>>>> for external function aliasing.
>>>> We often use aliases for posix naming reasons as well as avoid
using
>>>> the MS version of a function.
>>>>
>>>> example
>>>>
>>>>   `LIBRARY "user32.dll"
>>>>    EXPORTS
>>>>    MessageBoxA
>>>>    MessageBoxW==MessageBoxA`
>>>>
>>>> LIB.exe doesn't not have support for this but dlltool does.
>>>> The best fit for this within the spec is PE COFF spec (Aux
Format 3:
>>>> Weak Externals)
>>>>
>>>> Mingw-w64 also uses lib prefix and .a file extensions for the
library
>>>> name so the driver should be different. The above example would
be
>>>> libuser32.a, for when shared and static versions of the same
library exist
>>>> there is the .dll.a variant.
>>>>
>>>>
>>>> Issues
>>>> ======>>>>
>>>> Like more of the gnu suite dlltool was not designed in one
build multi
>>>> target manner.
>>>> We would have to introduce custom arguments to specify the
target,
>>>> "i686", "x86_64", "thumb2pe" etc.
>>>> This will most likely break some level of backwards
compatibility with
>>>> the binutils version.
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/d6beb320/attachment.html>