llvm dev - Feb 2017 - RFC: A new llvm-dlltool driver and llvm-lib driver improvements

>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
Yes
> My patch to hack lld into accepting some very basic gnu front end
> arguments was enough to get all the above working which was enough to
> develop further.


On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Hey Rui,
>>
>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>> support. I don't think dlltool is the last missing piece. What
do you need
>>> to do other than that to fully support mingw using LLVM toolchain?
>>
>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
>> within the EmitCommonSymbol function and a single patch for mingw-w64
>> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool
is
>> infact the only missing part really.
>>
>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
>
>
>> This gives us a fully working clang based mingw-w64 C compiler.
>>
>> C++ and exception handling is a different story.
>>
>> libc++ is somewhat working with the following test results
>>
>> Expected Passes    : 2188
>> Expected Failures  : 44
>> Unsupported Tests  : 588
>> Unexpected Failures: 2816
>>
>> I was able to build it with exceptions disabled and I actually managed
to
>> bootstrap llvm and clang itself.
>> It didn't run very well though as you can imagine based on the
above
>> tests.
>>
>> It's not a performance issue but a code maintenance issue. The
initial
>>> patches to support mingw was trying to add a new linker driver and
a
>>> different linkin semantics to the COFF linker which seemed too
complicated
>>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>>> arguments to MSVC linker arguments. Didn't it work?
>>
>> There were just so many differences between link and ld I never
followed
>> down that path. I pushed forward with the short import library support
>> based on your later suggestions. My patch to hack lld into accepting
some
>> very basic gnu front end arguments was enough to get all the above
working
>> which was enough to develop further.
>>
>> Best,
>> Martell
>>
>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> Hi Martell,
>>>
>>> I wonder how llvm-dlltool would fit in the entire picture of mingw
>>> support. I don't think dlltool is the last missing piece. What
do you need
>>> to do other than that to fully support mingw using LLVM toolchain?
>>>
>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hey llvm'ers,
>>>>
>>>> I have been working on a dlltool replacement for llvm.
>>>> Here is my initial differential https://reviews.llvm.org/D29892
>>>> It is based on some functionality that already exists in lld.
>>>> I added functionality to support, PE COFF Weak Externals and of
course
>>>> a front end to actually use it.
>>>> I believe the work here can also be used for llvm-lib and
lessen the
>>>> load on lld.
>>>> I would like some comments about how this could be be
structured to
>>>> live in llvm with a shared code base across lib ar and dlltool.
>>>> I also have a section below called "Difference from
lib" which is
>>>> somewhat of a rationale for the tool.
>>>>
>>>> Many Thanks,
>>>> Martell
>>>>
>>>>
>>>> Context
>>>> =========>>>>
>>>> Awhile back I talked to various llvm'ers about getting
mingw-w64
>>>> support for lld.
>>>> There were a few issues raised but the main issue was that
mingw-w64
>>>> should try best to comply with the PECOFF spec, adding support
for custom
>>>> sections and various binutils/mingw hacks would impact the
performance of
>>>> the COFF linker and in general is not something that lld should
support.
>>>>
>>>
>>> It's not a performance issue but a code maintenance issue. The
initial
>>> patches to support mingw was trying to add a new linker driver and
a
>>> different linkin semantics to the COFF linker which seemed too
complicated
>>> to me. IIRC, I suggested adding a shim which translates GNU command
line
>>> arguments to MSVC linker arguments. Didn't it work?
>>>
>>>
>>>> Motivation
>>>> =========>>>>
>>>> The main motivation was because dlltool and ld did not comply
with
>>>> PECOFF Spec.
>>>> It has some custom formatting and uses the assembler for some
reason to
>>>> generate import libraries, it did not use the short import
library format
>>>> at all.
>>>>
>>>> There has been many work arounds for the problem this creates
such as
>>>> the creation of the `reimp` tool. Which imports MSVC built libs
creates a
>>>> def and uses dlltool so that the binutils linker can use it.
>>>>
>>>> We should just be using the short import format and the linker
should
>>>> support that.
>>>> Thus llvm-dlltool was born.
>>>>
>>>>
>>>> Difference from lib
>>>> ==================>>>>
>>>> Using PE COFF spec (section 8, Import Library Format) should be
self
>>>> explanatory.
>>>> lib.exe is able to accept def files and create libraries using
this
>>>> format.
>>>>
>>>> example
>>>>
>>>>   `LIBRARY "user32.dll"
>>>>    EXPORTS
>>>>    MessageBoxA`
>>>>
>>>> LIB.exe can create a user32.lib with the function MessageBoxA
from the
>>>> above definition.
>>>>
>>>> Mingw-w64 is different MSVC in that we need to compile the
runtime.
>>>> MS provide us with their crt prebuilt so lib.exe doesn't
have support
>>>> for external function aliasing.
>>>> We often use aliases for posix naming reasons as well as avoid
using
>>>> the MS version of a function.
>>>>
>>>> example
>>>>
>>>>   `LIBRARY "user32.dll"
>>>>    EXPORTS
>>>>    MessageBoxA
>>>>    MessageBoxW==MessageBoxA`
>>>>
>>>> LIB.exe doesn't not have support for this but dlltool does.
>>>> The best fit for this within the spec is PE COFF spec (Aux
Format 3:
>>>> Weak Externals)
>>>>
>>>> Mingw-w64 also uses lib prefix and .a file extensions for the
library
>>>> name so the driver should be different. The above example would
be
>>>> libuser32.a, for when shared and static versions of the same
library exist
>>>> there is the .dll.a variant.
>>>>
>>>>
>>>> Issues
>>>> ======>>>>
>>>> Like more of the gnu suite dlltool was not designed in one
build multi
>>>> target manner.
>>>> We would have to introduce custom arguments to specify the
target,
>>>> "i686", "x86_64", "thumb2pe" etc.
>>>> This will most likely break some level of backwards
compatibility with
>>>> the binutils version.
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/d6beb320/attachment.html>

On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at
gmail.com>
wrote:
> Also you need to make a change to LLD/COFF to accept GNU command
>> arguments, right? (Looks like you already have that patch locally.)
>
> Yes
>
I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
wrapper, meaning that you could add a main function for mingw which
translates all arguments into the MSVC style and then invoke the COFF
linker's main function.

>
> My patch to hack lld into accepting some very basic gnu front end
>> arguments was enough to get all the above working which was enough to
>> develop further.
>
>
>
> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at
gmail.com
>> > wrote:
>>
>>> Hey Rui,
>>>
>>>> I wonder how llvm-dlltool would fit in the entire picture of
mingw
>>>> support. I don't think dlltool is the last missing piece.
What do you need
>>>> to do other than that to fully support mingw using LLVM
toolchain?
>>>
>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use
-aligncomm
>>> within the EmitCommonSymbol function and a single patch for
mingw-w64
>>> itself to pre-populate it's .ctors and .dtors list, so
llvm-dlltool is
>>> infact the only missing part really.
>>>
>>
>> Also you need to make a change to LLD/COFF to accept GNU command
>> arguments, right? (Looks like you already have that patch locally.)
>>
>>
>>> This gives us a fully working clang based mingw-w64 C compiler.
>>>
>>> C++ and exception handling is a different story.
>>>
>>> libc++ is somewhat working with the following test results
>>>
>>> Expected Passes    : 2188
>>> Expected Failures  : 44
>>> Unsupported Tests  : 588
>>> Unexpected Failures: 2816
>>>
>>> I was able to build it with exceptions disabled and I actually
managed
>>> to bootstrap llvm and clang itself.
>>> It didn't run very well though as you can imagine based on the
above
>>> tests.
>>>
>>> It's not a performance issue but a code maintenance issue. The
initial
>>>> patches to support mingw was trying to add a new linker driver
and a
>>>> different linkin semantics to the COFF linker which seemed too
complicated
>>>> to me. IIRC, I suggested adding a shim which translates GNU
command line
>>>> arguments to MSVC linker arguments. Didn't it work?
>>>
>>> There were just so many differences between link and ld I never
followed
>>> down that path. I pushed forward with the short import library
support
>>> based on your later suggestions. My patch to hack lld into
accepting some
>>> very basic gnu front end arguments was enough to get all the above
working
>>> which was enough to develop further.
>>>
>>> Best,
>>> Martell
>>>
>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>
>>>> Hi Martell,
>>>>
>>>> I wonder how llvm-dlltool would fit in the entire picture of
mingw
>>>> support. I don't think dlltool is the last missing piece.
What do you need
>>>> to do other than that to fully support mingw using LLVM
toolchain?
>>>>
>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Hey llvm'ers,
>>>>>
>>>>> I have been working on a dlltool replacement for llvm.
>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>> It is based on some functionality that already exists in
lld.
>>>>> I added functionality to support, PE COFF Weak Externals
and of
>>>>> course a front end to actually use it.
>>>>> I believe the work here can also be used for llvm-lib and
lessen the
>>>>> load on lld.
>>>>> I would like some comments about how this could be be
structured to
>>>>> live in llvm with a shared code base across lib ar and
dlltool.
>>>>> I also have a section below called "Difference from
lib" which is
>>>>> somewhat of a rationale for the tool.
>>>>>
>>>>> Many Thanks,
>>>>> Martell
>>>>>
>>>>>
>>>>> Context
>>>>> =========>>>>>
>>>>> Awhile back I talked to various llvm'ers about getting
mingw-w64
>>>>> support for lld.
>>>>> There were a few issues raised but the main issue was that
mingw-w64
>>>>> should try best to comply with the PECOFF spec, adding
support for custom
>>>>> sections and various binutils/mingw hacks would impact the
performance of
>>>>> the COFF linker and in general is not something that lld
should support.
>>>>>
>>>>
>>>> It's not a performance issue but a code maintenance issue.
The initial
>>>> patches to support mingw was trying to add a new linker driver
and a
>>>> different linkin semantics to the COFF linker which seemed too
complicated
>>>> to me. IIRC, I suggested adding a shim which translates GNU
command line
>>>> arguments to MSVC linker arguments. Didn't it work?
>>>>
>>>>
>>>>> Motivation
>>>>> =========>>>>>
>>>>> The main motivation was because dlltool and ld did not
comply with
>>>>> PECOFF Spec.
>>>>> It has some custom formatting and uses the assembler for
some reason
>>>>> to generate import libraries, it did not use the short
import library
>>>>> format at all.
>>>>>
>>>>> There has been many work arounds for the problem this
creates such as
>>>>> the creation of the `reimp` tool. Which imports MSVC built
libs creates a
>>>>> def and uses dlltool so that the binutils linker can use
it.
>>>>>
>>>>> We should just be using the short import format and the
linker should
>>>>> support that.
>>>>> Thus llvm-dlltool was born.
>>>>>
>>>>>
>>>>> Difference from lib
>>>>> ==================>>>>>
>>>>> Using PE COFF spec (section 8, Import Library Format)
should be self
>>>>> explanatory.
>>>>> lib.exe is able to accept def files and create libraries
using this
>>>>> format.
>>>>>
>>>>> example
>>>>>
>>>>>   `LIBRARY "user32.dll"
>>>>>    EXPORTS
>>>>>    MessageBoxA`
>>>>>
>>>>> LIB.exe can create a user32.lib with the function
MessageBoxA from the
>>>>> above definition.
>>>>>
>>>>> Mingw-w64 is different MSVC in that we need to compile the
runtime.
>>>>> MS provide us with their crt prebuilt so lib.exe
doesn't have support
>>>>> for external function aliasing.
>>>>> We often use aliases for posix naming reasons as well as
avoid using
>>>>> the MS version of a function.
>>>>>
>>>>> example
>>>>>
>>>>>   `LIBRARY "user32.dll"
>>>>>    EXPORTS
>>>>>    MessageBoxA
>>>>>    MessageBoxW==MessageBoxA`
>>>>>
>>>>> LIB.exe doesn't not have support for this but dlltool
does.
>>>>> The best fit for this within the spec is PE COFF spec (Aux
Format 3:
>>>>> Weak Externals)
>>>>>
>>>>> Mingw-w64 also uses lib prefix and .a file extensions for
the library
>>>>> name so the driver should be different. The above example
would be
>>>>> libuser32.a, for when shared and static versions of the
same library exist
>>>>> there is the .dll.a variant.
>>>>>
>>>>>
>>>>> Issues
>>>>> ======>>>>>
>>>>> Like more of the gnu suite dlltool was not designed in one
build multi
>>>>> target manner.
>>>>> We would have to introduce custom arguments to specify the
target,
>>>>> "i686", "x86_64", "thumb2pe"
etc.
>>>>> This will most likely break some level of backwards
compatibility with
>>>>> the binutils version.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/85ffdf36/attachment.html>

>
> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
> wrapper, meaning that you could add a main function for mingw which
> translates all arguments into the MSVC style and then invoke the COFF
> linker's main function.
That should work, if I am understanding this correctly I can create an
argument parser (probably partially based on the ELF one) then I can set
the data in the COFF Config object and call COFF LinkerDriver::run etc
directly from that?

This proposal is for generally moving code from lld into llvm that could be
part of lib.exe / dlltool, what are your thoughts here?

On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> Also you need to make a change to LLD/COFF to accept GNU command
>>> arguments, right? (Looks like you already have that patch locally.)
>>
>> Yes
>>
>
> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
> wrapper, meaning that you could add a main function for mingw which
> translates all arguments into the MSVC style and then invoke the COFF
> linker's main function.
>
>
>>
>> My patch to hack lld into accepting some very basic gnu front end
>>> arguments was enough to get all the above working which was enough
to
>>> develop further.
>>
>>
>>
>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>> martellmalone at gmail.com> wrote:
>>>
>>>> Hey Rui,
>>>>
>>>>> I wonder how llvm-dlltool would fit in the entire picture
of mingw
>>>>> support. I don't think dlltool is the last missing
piece. What do you need
>>>>> to do other than that to fully support mingw using LLVM
toolchain?
>>>>
>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use
-aligncomm
>>>> within the EmitCommonSymbol function and a single patch for
mingw-w64
>>>> itself to pre-populate it's .ctors and .dtors list, so
llvm-dlltool is
>>>> infact the only missing part really.
>>>>
>>>
>>> Also you need to make a change to LLD/COFF to accept GNU command
>>> arguments, right? (Looks like you already have that patch locally.)
>>>
>>>
>>>> This gives us a fully working clang based mingw-w64 C compiler.
>>>>
>>>> C++ and exception handling is a different story.
>>>>
>>>> libc++ is somewhat working with the following test results
>>>>
>>>> Expected Passes    : 2188
>>>> Expected Failures  : 44
>>>> Unsupported Tests  : 588
>>>> Unexpected Failures: 2816
>>>>
>>>> I was able to build it with exceptions disabled and I actually
managed
>>>> to bootstrap llvm and clang itself.
>>>> It didn't run very well though as you can imagine based on
the above
>>>> tests.
>>>>
>>>> It's not a performance issue but a code maintenance issue.
The initial
>>>>> patches to support mingw was trying to add a new linker
driver and a
>>>>> different linkin semantics to the COFF linker which seemed
too complicated
>>>>> to me. IIRC, I suggested adding a shim which translates GNU
command line
>>>>> arguments to MSVC linker arguments. Didn't it work?
>>>>
>>>> There were just so many differences between link and ld I never
>>>> followed down that path. I pushed forward with the short import
library
>>>> support based on your later suggestions. My patch to hack lld
into
>>>> accepting some very basic gnu front end arguments was enough to
get all the
>>>> above working which was enough to develop further.
>>>>
>>>> Best,
>>>> Martell
>>>>
>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>
>>>>> Hi Martell,
>>>>>
>>>>> I wonder how llvm-dlltool would fit in the entire picture
of mingw
>>>>> support. I don't think dlltool is the last missing
piece. What do you need
>>>>> to do other than that to fully support mingw using LLVM
toolchain?
>>>>>
>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via
llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Hey llvm'ers,
>>>>>>
>>>>>> I have been working on a dlltool replacement for llvm.
>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>> It is based on some functionality that already exists
in lld.
>>>>>> I added functionality to support, PE COFF Weak
Externals and of
>>>>>> course a front end to actually use it.
>>>>>> I believe the work here can also be used for llvm-lib
and lessen the
>>>>>> load on lld.
>>>>>> I would like some comments about how this could be be
structured to
>>>>>> live in llvm with a shared code base across lib ar and
dlltool.
>>>>>> I also have a section below called "Difference
from lib" which is
>>>>>> somewhat of a rationale for the tool.
>>>>>>
>>>>>> Many Thanks,
>>>>>> Martell
>>>>>>
>>>>>>
>>>>>> Context
>>>>>> =========>>>>>>
>>>>>> Awhile back I talked to various llvm'ers about
getting mingw-w64
>>>>>> support for lld.
>>>>>> There were a few issues raised but the main issue was
that mingw-w64
>>>>>> should try best to comply with the PECOFF spec, adding
support for custom
>>>>>> sections and various binutils/mingw hacks would impact
the performance of
>>>>>> the COFF linker and in general is not something that
lld should support.
>>>>>>
>>>>>
>>>>> It's not a performance issue but a code maintenance
issue. The initial
>>>>> patches to support mingw was trying to add a new linker
driver and a
>>>>> different linkin semantics to the COFF linker which seemed
too complicated
>>>>> to me. IIRC, I suggested adding a shim which translates GNU
command line
>>>>> arguments to MSVC linker arguments. Didn't it work?
>>>>>
>>>>>
>>>>>> Motivation
>>>>>> =========>>>>>>
>>>>>> The main motivation was because dlltool and ld did not
comply with
>>>>>> PECOFF Spec.
>>>>>> It has some custom formatting and uses the assembler
for some reason
>>>>>> to generate import libraries, it did not use the short
import library
>>>>>> format at all.
>>>>>>
>>>>>> There has been many work arounds for the problem this
creates such as
>>>>>> the creation of the `reimp` tool. Which imports MSVC
built libs creates a
>>>>>> def and uses dlltool so that the binutils linker can
use it.
>>>>>>
>>>>>> We should just be using the short import format and the
linker should
>>>>>> support that.
>>>>>> Thus llvm-dlltool was born.
>>>>>>
>>>>>>
>>>>>> Difference from lib
>>>>>> ==================>>>>>>
>>>>>> Using PE COFF spec (section 8, Import Library Format)
should be self
>>>>>> explanatory.
>>>>>> lib.exe is able to accept def files and create
libraries using this
>>>>>> format.
>>>>>>
>>>>>> example
>>>>>>
>>>>>>   `LIBRARY "user32.dll"
>>>>>>    EXPORTS
>>>>>>    MessageBoxA`
>>>>>>
>>>>>> LIB.exe can create a user32.lib with the function
MessageBoxA from
>>>>>> the above definition.
>>>>>>
>>>>>> Mingw-w64 is different MSVC in that we need to compile
the runtime.
>>>>>> MS provide us with their crt prebuilt so lib.exe
doesn't have support
>>>>>> for external function aliasing.
>>>>>> We often use aliases for posix naming reasons as well
as avoid using
>>>>>> the MS version of a function.
>>>>>>
>>>>>> example
>>>>>>
>>>>>>   `LIBRARY "user32.dll"
>>>>>>    EXPORTS
>>>>>>    MessageBoxA
>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>
>>>>>> LIB.exe doesn't not have support for this but
dlltool does.
>>>>>> The best fit for this within the spec is PE COFF spec
(Aux Format 3:
>>>>>> Weak Externals)
>>>>>>
>>>>>> Mingw-w64 also uses lib prefix and .a file extensions
for the library
>>>>>> name so the driver should be different. The above
example would be
>>>>>> libuser32.a, for when shared and static versions of the
same library exist
>>>>>> there is the .dll.a variant.
>>>>>>
>>>>>>
>>>>>> Issues
>>>>>> ======>>>>>>
>>>>>> Like more of the gnu suite dlltool was not designed in
one build
>>>>>> multi target manner.
>>>>>> We would have to introduce custom arguments to specify
the target,
>>>>>> "i686", "x86_64",
"thumb2pe" etc.
>>>>>> This will most likely break some level of backwards
compatibility
>>>>>> with the binutils version.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/841dea1c/attachment.html>