llvm dev - Feb 2017 - RFC: A new llvm-dlltool driver and llvm-lib driver improvements

>
> No, I meant an even thinner wrapper which textually translates arguments.
> For example, the wrapper would translates "/out:foo.exe foo.obj"
to "-o
> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
anything
> with Config object nor LinkerDriver::run and have absolutely zero knowledge
> on the internals of LLD.
Ohh okay I misunderstood.

(Why I'm asking this is because I want to sure that there's no
difference
> between the regular Windows linker and mingw. If all differences are
> superficial, the command line translator should just work. If not,
there's
> some semantic difference.)
The libraries can be used with MSVC link.exe directly assuming the correct
arguments are passed to the linker.
I tested this after creating a working llvm-dlltool.
The only change I had to make to support this was alter ming-w64 crt to
change all references from __image_base__ to _ImageBase to support that

That should be fine.

Great

On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
>>> wrapper, meaning that you could add a main function for mingw which
>>> translates all arguments into the MSVC style and then invoke the
COFF
>>> linker's main function.
>>
>> That should work, if I am understanding this correctly I can create an
>> argument parser (probably partially based on the ELF one) then I can
set
>> the data in the COFF Config object and call COFF LinkerDriver::run etc
>> directly from that?
>>
>
> No, I meant an even thinner wrapper which textually translates arguments.
> For example, the wrapper would translates "/out:foo.exe foo.obj"
to "-o
> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
anything
> with Config object nor LinkerDriver::run and have absolutely zero knowledge
> on the internals of LLD.
>
> (Why I'm asking this is because I want to sure that there's no
difference
> between the regular Windows linker and mingw. If all differences are
> superficial, the command line translator should just work. If not,
there's
> some semantic difference.)
>
> This proposal is for generally moving code from lld into llvm that could
>> be part of lib.exe / dlltool, what are your thoughts here?
>>
>
> That should be fine.
>
>
>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>> martellmalone at gmail.com> wrote:
>>>
>>>> Also you need to make a change to LLD/COFF to accept GNU
command
>>>>> arguments, right? (Looks like you already have that patch
locally.)
>>>>
>>>> Yes
>>>>
>>>
>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>> wrapper, meaning that you could add a main function for mingw which
>>> translates all arguments into the MSVC style and then invoke the
COFF
>>> linker's main function.
>>>
>>>
>>>>
>>>> My patch to hack lld into accepting some very basic gnu front
end
>>>>> arguments was enough to get all the above working which was
enough to
>>>>> develop further.
>>>>
>>>>
>>>>
>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>
>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>>>> martellmalone at gmail.com> wrote:
>>>>>
>>>>>> Hey Rui,
>>>>>>
>>>>>>> I wonder how llvm-dlltool would fit in the entire
picture of mingw
>>>>>>> support. I don't think dlltool is the last
missing piece. What do you need
>>>>>>> to do other than that to fully support mingw using
LLVM toolchain?
>>>>>>
>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not
use
>>>>>> -aligncomm within the EmitCommonSymbol function and a
single patch for
>>>>>> mingw-w64 itself to pre-populate it's .ctors and
.dtors list, so
>>>>>> llvm-dlltool is infact the only missing part really.
>>>>>>
>>>>>
>>>>> Also you need to make a change to LLD/COFF to accept GNU
command
>>>>> arguments, right? (Looks like you already have that patch
locally.)
>>>>>
>>>>>
>>>>>> This gives us a fully working clang based mingw-w64 C
compiler.
>>>>>>
>>>>>> C++ and exception handling is a different story.
>>>>>>
>>>>>> libc++ is somewhat working with the following test
results
>>>>>>
>>>>>> Expected Passes    : 2188
>>>>>> Expected Failures  : 44
>>>>>> Unsupported Tests  : 588
>>>>>> Unexpected Failures: 2816
>>>>>>
>>>>>> I was able to build it with exceptions disabled and I
actually
>>>>>> managed to bootstrap llvm and clang itself.
>>>>>> It didn't run very well though as you can imagine
based on the above
>>>>>> tests.
>>>>>>
>>>>>> It's not a performance issue but a code maintenance
issue. The
>>>>>>> initial patches to support mingw was trying to add
a new linker driver and
>>>>>>> a different linkin semantics to the COFF linker
which seemed too
>>>>>>> complicated to me. IIRC, I suggested adding a shim
which translates GNU
>>>>>>> command line arguments to MSVC linker arguments.
Didn't it work?
>>>>>>
>>>>>> There were just so many differences between link and ld
I never
>>>>>> followed down that path. I pushed forward with the
short import library
>>>>>> support based on your later suggestions. My patch to
hack lld into
>>>>>> accepting some very basic gnu front end arguments was
enough to get all the
>>>>>> above working which was enough to develop further.
>>>>>>
>>>>>> Best,
>>>>>> Martell
>>>>>>
>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>>>
>>>>>>> Hi Martell,
>>>>>>>
>>>>>>> I wonder how llvm-dlltool would fit in the entire
picture of mingw
>>>>>>> support. I don't think dlltool is the last
missing piece. What do you need
>>>>>>> to do other than that to fully support mingw using
LLVM toolchain?
>>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via
llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> Hey llvm'ers,
>>>>>>>>
>>>>>>>> I have been working on a dlltool replacement
for llvm.
>>>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>>>> It is based on some functionality that already
exists in lld.
>>>>>>>> I added functionality to support, PE COFF Weak
Externals and of
>>>>>>>> course a front end to actually use it.
>>>>>>>> I believe the work here can also be used for
llvm-lib and lessen
>>>>>>>> the load on lld.
>>>>>>>> I would like some comments about how this could
be be structured to
>>>>>>>> live in llvm with a shared code base across lib
ar and dlltool.
>>>>>>>> I also have a section below called
"Difference from lib" which is
>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>
>>>>>>>> Many Thanks,
>>>>>>>> Martell
>>>>>>>>
>>>>>>>>
>>>>>>>> Context
>>>>>>>> =========>>>>>>>>
>>>>>>>> Awhile back I talked to various llvm'ers
about getting mingw-w64
>>>>>>>> support for lld.
>>>>>>>> There were a few issues raised but the main
issue was that
>>>>>>>> mingw-w64 should try best to comply with the
PECOFF spec, adding support
>>>>>>>> for custom sections and various binutils/mingw
hacks would impact the
>>>>>>>> performance of the COFF linker and in general
is not something that lld
>>>>>>>> should support.
>>>>>>>>
>>>>>>>
>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>> initial patches to support mingw was trying to add
a new linker driver and
>>>>>>> a different linkin semantics to the COFF linker
which seemed too
>>>>>>> complicated to me. IIRC, I suggested adding a shim
which translates GNU
>>>>>>> command line arguments to MSVC linker arguments.
Didn't it work?
>>>>>>>
>>>>>>>
>>>>>>>> Motivation
>>>>>>>> =========>>>>>>>>
>>>>>>>> The main motivation was because dlltool and ld
did not comply with
>>>>>>>> PECOFF Spec.
>>>>>>>> It has some custom formatting and uses the
assembler for some
>>>>>>>> reason to generate import libraries, it did not
use the short import
>>>>>>>> library format at all.
>>>>>>>>
>>>>>>>> There has been many work arounds for the
problem this creates such
>>>>>>>> as the creation of the `reimp` tool. Which
imports MSVC built libs creates
>>>>>>>> a def and uses dlltool so that the binutils
linker can use it.
>>>>>>>>
>>>>>>>> We should just be using the short import format
and the linker
>>>>>>>> should support that.
>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>
>>>>>>>>
>>>>>>>> Difference from lib
>>>>>>>>
==================>>>>>>>>
>>>>>>>> Using PE COFF spec (section 8, Import Library
Format) should be
>>>>>>>> self explanatory.
>>>>>>>> lib.exe is able to accept def files and create
libraries using this
>>>>>>>> format.
>>>>>>>>
>>>>>>>> example
>>>>>>>>
>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>    EXPORTS
>>>>>>>>    MessageBoxA`
>>>>>>>>
>>>>>>>> LIB.exe can create a user32.lib with the
function MessageBoxA from
>>>>>>>> the above definition.
>>>>>>>>
>>>>>>>> Mingw-w64 is different MSVC in that we need to
compile the runtime.
>>>>>>>> MS provide us with their crt prebuilt so
lib.exe doesn't have
>>>>>>>> support for external function aliasing.
>>>>>>>> We often use aliases for posix naming reasons
as well as avoid
>>>>>>>> using the MS version of a function.
>>>>>>>>
>>>>>>>> example
>>>>>>>>
>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>    EXPORTS
>>>>>>>>    MessageBoxA
>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>
>>>>>>>> LIB.exe doesn't not have support for this
but dlltool does.
>>>>>>>> The best fit for this within the spec is PE
COFF spec (Aux Format
>>>>>>>> 3: Weak Externals)
>>>>>>>>
>>>>>>>> Mingw-w64 also uses lib prefix and .a file
extensions for the
>>>>>>>> library name so the driver should be different.
The above example would be
>>>>>>>> libuser32.a, for when shared and static
versions of the same library exist
>>>>>>>> there is the .dll.a variant.
>>>>>>>>
>>>>>>>>
>>>>>>>> Issues
>>>>>>>> ======>>>>>>>>
>>>>>>>> Like more of the gnu suite dlltool was not
designed in one build
>>>>>>>> multi target manner.
>>>>>>>> We would have to introduce custom arguments to
specify the target,
>>>>>>>> "i686", "x86_64",
"thumb2pe" etc.
>>>>>>>> This will most likely break some level of
backwards compatibility
>>>>>>>> with the binutils version.
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> LLVM Developers mailing list
>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170214/0aa058ba/attachment.html>

On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at
gmail.com>
wrote:
> No, I meant an even thinner wrapper which textually translates arguments.
>> For example, the wrapper would translates "/out:foo.exe
foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't
do anything
>> with Config object nor LinkerDriver::run and have absolutely zero
knowledge
>> on the internals of LLD.
>
> Ohh okay I misunderstood.
>
> (Why I'm asking this is because I want to sure that there's no
difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not,
there's
>> some semantic difference.)
>
> The libraries can be used with MSVC link.exe directly assuming the correct
> arguments are passed to the linker.
> I tested this after creating a working llvm-dlltool.
> The only change I had to make to support this was alter ming-w64 crt to
> change all references from __image_base__ to _ImageBase to support that
>
You may be able to define __ImageBase as a weak external symbol to
__image_base__. Then, if __ImageBase is not defined, all references against
__ImageBase will be resolved using __image_base__.

> That should be fine.
>
> Great
>
> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at
gmail.com>
>> wrote:
>>
>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>> wrapper, meaning that you could add a main function for mingw
which
>>>> translates all arguments into the MSVC style and then invoke
the COFF
>>>> linker's main function.
>>>
>>> That should work, if I am understanding this correctly I can create
an
>>> argument parser (probably partially based on the ELF one) then I
can set
>>> the data in the COFF Config object and call COFF LinkerDriver::run
etc
>>> directly from that?
>>>
>>
>> No, I meant an even thinner wrapper which textually translates
arguments.
>> For example, the wrapper would translates "/out:foo.exe
foo.obj" to "-o
>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't
do anything
>> with Config object nor LinkerDriver::run and have absolutely zero
knowledge
>> on the internals of LLD.
>>
>> (Why I'm asking this is because I want to sure that there's no
difference
>> between the regular Windows linker and mingw. If all differences are
>> superficial, the command line translator should just work. If not,
there's
>> some semantic difference.)
>>
>> This proposal is for generally moving code from lld into llvm that
could
>>> be part of lib.exe / dlltool, what are your thoughts here?
>>>
>>
>> That should be fine.
>>
>>
>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>
>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>> martellmalone at gmail.com> wrote:
>>>>
>>>>> Also you need to make a change to LLD/COFF to accept GNU
command
>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>
>>>>> Yes
>>>>>
>>>>
>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>> wrapper, meaning that you could add a main function for mingw
which
>>>> translates all arguments into the MSVC style and then invoke
the COFF
>>>> linker's main function.
>>>>
>>>>
>>>>>
>>>>> My patch to hack lld into accepting some very basic gnu
front end
>>>>>> arguments was enough to get all the above working which
was enough to
>>>>>> develop further.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>>
>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <
>>>>>> martellmalone at gmail.com> wrote:
>>>>>>
>>>>>>> Hey Rui,
>>>>>>>
>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>> support. I don't think dlltool is the last
missing piece. What do you need
>>>>>>>> to do other than that to fully support mingw
using LLVM toolchain?
>>>>>>>
>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to
not use
>>>>>>> -aligncomm within the EmitCommonSymbol function and
a single patch for
>>>>>>> mingw-w64 itself to pre-populate it's .ctors
and .dtors list, so
>>>>>>> llvm-dlltool is infact the only missing part
really.
>>>>>>>
>>>>>>
>>>>>> Also you need to make a change to LLD/COFF to accept
GNU command
>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>>
>>>>>>
>>>>>>> This gives us a fully working clang based mingw-w64
C compiler.
>>>>>>>
>>>>>>> C++ and exception handling is a different story.
>>>>>>>
>>>>>>> libc++ is somewhat working with the following test
results
>>>>>>>
>>>>>>> Expected Passes    : 2188
>>>>>>> Expected Failures  : 44
>>>>>>> Unsupported Tests  : 588
>>>>>>> Unexpected Failures: 2816
>>>>>>>
>>>>>>> I was able to build it with exceptions disabled and
I actually
>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>> It didn't run very well though as you can
imagine based on the above
>>>>>>> tests.
>>>>>>>
>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>> initial patches to support mingw was trying to
add a new linker driver and
>>>>>>>> a different linkin semantics to the COFF linker
which seemed too
>>>>>>>> complicated to me. IIRC, I suggested adding a
shim which translates GNU
>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>
>>>>>>> There were just so many differences between link
and ld I never
>>>>>>> followed down that path. I pushed forward with the
short import library
>>>>>>> support based on your later suggestions. My patch
to hack lld into
>>>>>>> accepting some very basic gnu front end arguments
was enough to get all the
>>>>>>> above working which was enough to develop further.
>>>>>>>
>>>>>>> Best,
>>>>>>> Martell
>>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama
<ruiu at google.com> wrote:
>>>>>>>
>>>>>>>> Hi Martell,
>>>>>>>>
>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>> support. I don't think dlltool is the last
missing piece. What do you need
>>>>>>>> to do other than that to fully support mingw
using LLVM toolchain?
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone
via llvm-dev <
>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> Hey llvm'ers,
>>>>>>>>>
>>>>>>>>> I have been working on a dlltool
replacement for llvm.
>>>>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>>>>> It is based on some functionality that
already exists in lld.
>>>>>>>>> I added functionality to support, PE COFF
Weak Externals and of
>>>>>>>>> course a front end to actually use it.
>>>>>>>>> I believe the work here can also be used
for llvm-lib and lessen
>>>>>>>>> the load on lld.
>>>>>>>>> I would like some comments about how this
could be be structured
>>>>>>>>> to live in llvm with a shared code base
across lib ar and dlltool.
>>>>>>>>> I also have a section below called
"Difference from lib" which is
>>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>>
>>>>>>>>> Many Thanks,
>>>>>>>>> Martell
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Context
>>>>>>>>>
=========>>>>>>>>>
>>>>>>>>> Awhile back I talked to various
llvm'ers about getting mingw-w64
>>>>>>>>> support for lld.
>>>>>>>>> There were a few issues raised but the main
issue was that
>>>>>>>>> mingw-w64 should try best to comply with
the PECOFF spec, adding support
>>>>>>>>> for custom sections and various
binutils/mingw hacks would impact the
>>>>>>>>> performance of the COFF linker and in
general is not something that lld
>>>>>>>>> should support.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>> initial patches to support mingw was trying to
add a new linker driver and
>>>>>>>> a different linkin semantics to the COFF linker
which seemed too
>>>>>>>> complicated to me. IIRC, I suggested adding a
shim which translates GNU
>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Motivation
>>>>>>>>>
=========>>>>>>>>>
>>>>>>>>> The main motivation was because dlltool and
ld did not comply with
>>>>>>>>> PECOFF Spec.
>>>>>>>>> It has some custom formatting and uses the
assembler for some
>>>>>>>>> reason to generate import libraries, it did
not use the short import
>>>>>>>>> library format at all.
>>>>>>>>>
>>>>>>>>> There has been many work arounds for the
problem this creates such
>>>>>>>>> as the creation of the `reimp` tool. Which
imports MSVC built libs creates
>>>>>>>>> a def and uses dlltool so that the binutils
linker can use it.
>>>>>>>>>
>>>>>>>>> We should just be using the short import
format and the linker
>>>>>>>>> should support that.
>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Difference from lib
>>>>>>>>>
==================>>>>>>>>>
>>>>>>>>> Using PE COFF spec (section 8, Import
Library Format) should be
>>>>>>>>> self explanatory.
>>>>>>>>> lib.exe is able to accept def files and
create libraries using
>>>>>>>>> this format.
>>>>>>>>>
>>>>>>>>> example
>>>>>>>>>
>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>    EXPORTS
>>>>>>>>>    MessageBoxA`
>>>>>>>>>
>>>>>>>>> LIB.exe can create a user32.lib with the
function MessageBoxA from
>>>>>>>>> the above definition.
>>>>>>>>>
>>>>>>>>> Mingw-w64 is different MSVC in that we need
to compile the runtime.
>>>>>>>>> MS provide us with their crt prebuilt so
lib.exe doesn't have
>>>>>>>>> support for external function aliasing.
>>>>>>>>> We often use aliases for posix naming
reasons as well as avoid
>>>>>>>>> using the MS version of a function.
>>>>>>>>>
>>>>>>>>> example
>>>>>>>>>
>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>    EXPORTS
>>>>>>>>>    MessageBoxA
>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>
>>>>>>>>> LIB.exe doesn't not have support for
this but dlltool does.
>>>>>>>>> The best fit for this within the spec is PE
COFF spec (Aux Format
>>>>>>>>> 3: Weak Externals)
>>>>>>>>>
>>>>>>>>> Mingw-w64 also uses lib prefix and .a file
extensions for the
>>>>>>>>> library name so the driver should be
different. The above example would be
>>>>>>>>> libuser32.a, for when shared and static
versions of the same library exist
>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Issues
>>>>>>>>> ======>>>>>>>>>
>>>>>>>>> Like more of the gnu suite dlltool was not
designed in one build
>>>>>>>>> multi target manner.
>>>>>>>>> We would have to introduce custom arguments
to specify the target,
>>>>>>>>> "i686", "x86_64",
"thumb2pe" etc.
>>>>>>>>> This will most likely break some level of
backwards compatibility
>>>>>>>>> with the binutils version.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
_______________________________________________
>>>>>>>>> LLVM Developers mailing list
>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/c45a1dda/attachment-0001.html>

On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> No, I meant an even thinner wrapper which textually translates
arguments.
>>> For example, the wrapper would translates "/out:foo.exe
foo.obj" to "-o
>>> foo.exe foo.obj" and then call lld::COFFF:link(). It
doesn't do anything
>>> with Config object nor LinkerDriver::run and have absolutely zero
knowledge
>>> on the internals of LLD.
>>
>> Ohh okay I misunderstood.
>>
>> (Why I'm asking this is because I want to sure that there's no
difference
>>> between the regular Windows linker and mingw. If all differences
are
>>> superficial, the command line translator should just work. If not,
there's
>>> some semantic difference.)
>>
>> The libraries can be used with MSVC link.exe directly assuming the
>> correct arguments are passed to the linker.
>> I tested this after creating a working llvm-dlltool.
>> The only change I had to make to support this was alter ming-w64 crt to
>> change all references from __image_base__ to _ImageBase to support that
>>
>
> You may be able to define __ImageBase as a weak external symbol to
> __image_base__. Then, if __ImageBase is not defined, all references against
> __ImageBase will be resolved using __image_base__.
>
Or you could have your wrapper driver pass
"/alternatename:__image_base__=__ImageBase".

Peter

>
>
>> That should be fine.
>>
>> Great
>>
>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone
at gmail.com
>>> > wrote:
>>>
>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>> linker's main function.
>>>>
>>>> That should work, if I am understanding this correctly I can
create an
>>>> argument parser (probably partially based on the ELF one) then
I can set
>>>> the data in the COFF Config object and call COFF
LinkerDriver::run etc
>>>> directly from that?
>>>>
>>>
>>> No, I meant an even thinner wrapper which textually translates
>>> arguments. For example, the wrapper would translates
"/out:foo.exe foo.obj"
>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link().
It doesn't do
>>> anything with Config object nor LinkerDriver::run and have
absolutely zero
>>> knowledge on the internals of LLD.
>>>
>>> (Why I'm asking this is because I want to sure that there's
no
>>> difference between the regular Windows linker and mingw. If all
differences
>>> are superficial, the command line translator should just work. If
not,
>>> there's some semantic difference.)
>>>
>>> This proposal is for generally moving code from lld into llvm that
could
>>>> be part of lib.exe / dlltool, what are your thoughts here?
>>>>
>>>
>>> That should be fine.
>>>
>>>
>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>
>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>>> martellmalone at gmail.com> wrote:
>>>>>
>>>>>> Also you need to make a change to LLD/COFF to accept
GNU command
>>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>>
>>>>>> Yes
>>>>>>
>>>>>
>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>> linker's main function.
>>>>>
>>>>>
>>>>>>
>>>>>> My patch to hack lld into accepting some very basic gnu
front end
>>>>>>> arguments was enough to get all the above working
which was enough to
>>>>>>> develop further.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone
<
>>>>>>> martellmalone at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hey Rui,
>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the
last missing piece. What do you need
>>>>>>>>> to do other than that to fully support
mingw using LLVM toolchain?
>>>>>>>>
>>>>>>>> Other then changing
`lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>>> -aligncomm within the EmitCommonSymbol function
and a single patch for
>>>>>>>> mingw-w64 itself to pre-populate it's
.ctors and .dtors list, so
>>>>>>>> llvm-dlltool is infact the only missing part
really.
>>>>>>>>
>>>>>>>
>>>>>>> Also you need to make a change to LLD/COFF to
accept GNU command
>>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>>>
>>>>>>>
>>>>>>>> This gives us a fully working clang based
mingw-w64 C compiler.
>>>>>>>>
>>>>>>>> C++ and exception handling is a different
story.
>>>>>>>>
>>>>>>>> libc++ is somewhat working with the following
test results
>>>>>>>>
>>>>>>>> Expected Passes    : 2188
>>>>>>>> Expected Failures  : 44
>>>>>>>> Unsupported Tests  : 588
>>>>>>>> Unexpected Failures: 2816
>>>>>>>>
>>>>>>>> I was able to build it with exceptions disabled
and I actually
>>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>>> It didn't run very well though as you can
imagine based on the
>>>>>>>> above tests.
>>>>>>>>
>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying
to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF
linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding
a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>
>>>>>>>> There were just so many differences between
link and ld I never
>>>>>>>> followed down that path. I pushed forward with
the short import library
>>>>>>>> support based on your later suggestions. My
patch to hack lld into
>>>>>>>> accepting some very basic gnu front end
arguments was enough to get all the
>>>>>>>> above working which was enough to develop
further.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Martell
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama
<ruiu at google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Martell,
>>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the
last missing piece. What do you need
>>>>>>>>> to do other than that to fully support
mingw using LLVM toolchain?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell
Malone via llvm-dev <
>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hey llvm'ers,
>>>>>>>>>>
>>>>>>>>>> I have been working on a dlltool
replacement for llvm.
>>>>>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>>>>>> It is based on some functionality that
already exists in lld.
>>>>>>>>>> I added functionality to support, PE
COFF Weak Externals and of
>>>>>>>>>> course a front end to actually use it.
>>>>>>>>>> I believe the work here can also be
used for llvm-lib and lessen
>>>>>>>>>> the load on lld.
>>>>>>>>>> I would like some comments about how
this could be be structured
>>>>>>>>>> to live in llvm with a shared code base
across lib ar and dlltool.
>>>>>>>>>> I also have a section below called
"Difference from lib" which is
>>>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>>>
>>>>>>>>>> Many Thanks,
>>>>>>>>>> Martell
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Context
>>>>>>>>>>
=========>>>>>>>>>>
>>>>>>>>>> Awhile back I talked to various
llvm'ers about getting mingw-w64
>>>>>>>>>> support for lld.
>>>>>>>>>> There were a few issues raised but the
main issue was that
>>>>>>>>>> mingw-w64 should try best to comply
with the PECOFF spec, adding support
>>>>>>>>>> for custom sections and various
binutils/mingw hacks would impact the
>>>>>>>>>> performance of the COFF linker and in
general is not something that lld
>>>>>>>>>> should support.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying
to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF
linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding
a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Motivation
>>>>>>>>>>
=========>>>>>>>>>>
>>>>>>>>>> The main motivation was because dlltool
and ld did not comply
>>>>>>>>>> with PECOFF Spec.
>>>>>>>>>> It has some custom formatting and uses
the assembler for some
>>>>>>>>>> reason to generate import libraries, it
did not use the short import
>>>>>>>>>> library format at all.
>>>>>>>>>>
>>>>>>>>>> There has been many work arounds for
the problem this creates
>>>>>>>>>> such as the creation of the `reimp`
tool. Which imports MSVC built libs
>>>>>>>>>> creates a def and uses dlltool so that
the binutils linker can use it.
>>>>>>>>>>
>>>>>>>>>> We should just be using the short
import format and the linker
>>>>>>>>>> should support that.
>>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Difference from lib
>>>>>>>>>>
==================>>>>>>>>>>
>>>>>>>>>> Using PE COFF spec (section 8, Import
Library Format) should be
>>>>>>>>>> self explanatory.
>>>>>>>>>> lib.exe is able to accept def files and
create libraries using
>>>>>>>>>> this format.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe can create a user32.lib with
the function MessageBoxA
>>>>>>>>>> from the above definition.
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 is different MSVC in that we
need to compile the
>>>>>>>>>> runtime.
>>>>>>>>>> MS provide us with their crt prebuilt
so lib.exe doesn't have
>>>>>>>>>> support for external function aliasing.
>>>>>>>>>> We often use aliases for posix naming
reasons as well as avoid
>>>>>>>>>> using the MS version of a function.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA
>>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe doesn't not have support
for this but dlltool does.
>>>>>>>>>> The best fit for this within the spec
is PE COFF spec (Aux
>>>>>>>>>> Format 3: Weak Externals)
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 also uses lib prefix and .a
file extensions for the
>>>>>>>>>> library name so the driver should be
different. The above example would be
>>>>>>>>>> libuser32.a, for when shared and static
versions of the same library exist
>>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Issues
>>>>>>>>>>
======>>>>>>>>>>
>>>>>>>>>> Like more of the gnu suite dlltool was
not designed in one build
>>>>>>>>>> multi target manner.
>>>>>>>>>> We would have to introduce custom
arguments to specify the
>>>>>>>>>> target, "i686",
"x86_64", "thumb2pe" etc.
>>>>>>>>>> This will most likely break some level
of backwards compatibility
>>>>>>>>>> with the binutils version.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/70e0cfb0/attachment.html>