llvm dev - Feb 2017 - RFC: A new llvm-dlltool driver and llvm-lib driver improvements

On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
>> No, I meant an even thinner wrapper which textually translates
arguments.
>>> For example, the wrapper would translates "/out:foo.exe
foo.obj" to "-o
>>> foo.exe foo.obj" and then call lld::COFFF:link(). It
doesn't do anything
>>> with Config object nor LinkerDriver::run and have absolutely zero
knowledge
>>> on the internals of LLD.
>>
>> Ohh okay I misunderstood.
>>
>> (Why I'm asking this is because I want to sure that there's no
difference
>>> between the regular Windows linker and mingw. If all differences
are
>>> superficial, the command line translator should just work. If not,
there's
>>> some semantic difference.)
>>
>> The libraries can be used with MSVC link.exe directly assuming the
>> correct arguments are passed to the linker.
>> I tested this after creating a working llvm-dlltool.
>> The only change I had to make to support this was alter ming-w64 crt to
>> change all references from __image_base__ to _ImageBase to support that
>>
>
> You may be able to define __ImageBase as a weak external symbol to
> __image_base__. Then, if __ImageBase is not defined, all references against
> __ImageBase will be resolved using __image_base__.
>
Or you could have your wrapper driver pass
"/alternatename:__image_base__=__ImageBase".

Peter

>
>
>> That should be fine.
>>
>> Great
>>
>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone
at gmail.com
>>> > wrote:
>>>
>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>> linker's main function.
>>>>
>>>> That should work, if I am understanding this correctly I can
create an
>>>> argument parser (probably partially based on the ELF one) then
I can set
>>>> the data in the COFF Config object and call COFF
LinkerDriver::run etc
>>>> directly from that?
>>>>
>>>
>>> No, I meant an even thinner wrapper which textually translates
>>> arguments. For example, the wrapper would translates
"/out:foo.exe foo.obj"
>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link().
It doesn't do
>>> anything with Config object nor LinkerDriver::run and have
absolutely zero
>>> knowledge on the internals of LLD.
>>>
>>> (Why I'm asking this is because I want to sure that there's
no
>>> difference between the regular Windows linker and mingw. If all
differences
>>> are superficial, the command line translator should just work. If
not,
>>> there's some semantic difference.)
>>>
>>> This proposal is for generally moving code from lld into llvm that
could
>>>> be part of lib.exe / dlltool, what are your thoughts here?
>>>>
>>>
>>> That should be fine.
>>>
>>>
>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>
>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>>> martellmalone at gmail.com> wrote:
>>>>>
>>>>>> Also you need to make a change to LLD/COFF to accept
GNU command
>>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>>
>>>>>> Yes
>>>>>>
>>>>>
>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>> linker's main function.
>>>>>
>>>>>
>>>>>>
>>>>>> My patch to hack lld into accepting some very basic gnu
front end
>>>>>>> arguments was enough to get all the above working
which was enough to
>>>>>>> develop further.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone
<
>>>>>>> martellmalone at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hey Rui,
>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the
last missing piece. What do you need
>>>>>>>>> to do other than that to fully support
mingw using LLVM toolchain?
>>>>>>>>
>>>>>>>> Other then changing
`lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>>> -aligncomm within the EmitCommonSymbol function
and a single patch for
>>>>>>>> mingw-w64 itself to pre-populate it's
.ctors and .dtors list, so
>>>>>>>> llvm-dlltool is infact the only missing part
really.
>>>>>>>>
>>>>>>>
>>>>>>> Also you need to make a change to LLD/COFF to
accept GNU command
>>>>>>> arguments, right? (Looks like you already have that
patch locally.)
>>>>>>>
>>>>>>>
>>>>>>>> This gives us a fully working clang based
mingw-w64 C compiler.
>>>>>>>>
>>>>>>>> C++ and exception handling is a different
story.
>>>>>>>>
>>>>>>>> libc++ is somewhat working with the following
test results
>>>>>>>>
>>>>>>>> Expected Passes    : 2188
>>>>>>>> Expected Failures  : 44
>>>>>>>> Unsupported Tests  : 588
>>>>>>>> Unexpected Failures: 2816
>>>>>>>>
>>>>>>>> I was able to build it with exceptions disabled
and I actually
>>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>>> It didn't run very well though as you can
imagine based on the
>>>>>>>> above tests.
>>>>>>>>
>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying
to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF
linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding
a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>
>>>>>>>> There were just so many differences between
link and ld I never
>>>>>>>> followed down that path. I pushed forward with
the short import library
>>>>>>>> support based on your later suggestions. My
patch to hack lld into
>>>>>>>> accepting some very basic gnu front end
arguments was enough to get all the
>>>>>>>> above working which was enough to develop
further.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Martell
>>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama
<ruiu at google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Martell,
>>>>>>>>>
>>>>>>>>> I wonder how llvm-dlltool would fit in the
entire picture of mingw
>>>>>>>>> support. I don't think dlltool is the
last missing piece. What do you need
>>>>>>>>> to do other than that to fully support
mingw using LLVM toolchain?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell
Malone via llvm-dev <
>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hey llvm'ers,
>>>>>>>>>>
>>>>>>>>>> I have been working on a dlltool
replacement for llvm.
>>>>>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>>>>>> It is based on some functionality that
already exists in lld.
>>>>>>>>>> I added functionality to support, PE
COFF Weak Externals and of
>>>>>>>>>> course a front end to actually use it.
>>>>>>>>>> I believe the work here can also be
used for llvm-lib and lessen
>>>>>>>>>> the load on lld.
>>>>>>>>>> I would like some comments about how
this could be be structured
>>>>>>>>>> to live in llvm with a shared code base
across lib ar and dlltool.
>>>>>>>>>> I also have a section below called
"Difference from lib" which is
>>>>>>>>>> somewhat of a rationale for the tool.
>>>>>>>>>>
>>>>>>>>>> Many Thanks,
>>>>>>>>>> Martell
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Context
>>>>>>>>>>
=========>>>>>>>>>>
>>>>>>>>>> Awhile back I talked to various
llvm'ers about getting mingw-w64
>>>>>>>>>> support for lld.
>>>>>>>>>> There were a few issues raised but the
main issue was that
>>>>>>>>>> mingw-w64 should try best to comply
with the PECOFF spec, adding support
>>>>>>>>>> for custom sections and various
binutils/mingw hacks would impact the
>>>>>>>>>> performance of the COFF linker and in
general is not something that lld
>>>>>>>>>> should support.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>>> initial patches to support mingw was trying
to add a new linker driver and
>>>>>>>>> a different linkin semantics to the COFF
linker which seemed too
>>>>>>>>> complicated to me. IIRC, I suggested adding
a shim which translates GNU
>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Motivation
>>>>>>>>>>
=========>>>>>>>>>>
>>>>>>>>>> The main motivation was because dlltool
and ld did not comply
>>>>>>>>>> with PECOFF Spec.
>>>>>>>>>> It has some custom formatting and uses
the assembler for some
>>>>>>>>>> reason to generate import libraries, it
did not use the short import
>>>>>>>>>> library format at all.
>>>>>>>>>>
>>>>>>>>>> There has been many work arounds for
the problem this creates
>>>>>>>>>> such as the creation of the `reimp`
tool. Which imports MSVC built libs
>>>>>>>>>> creates a def and uses dlltool so that
the binutils linker can use it.
>>>>>>>>>>
>>>>>>>>>> We should just be using the short
import format and the linker
>>>>>>>>>> should support that.
>>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Difference from lib
>>>>>>>>>>
==================>>>>>>>>>>
>>>>>>>>>> Using PE COFF spec (section 8, Import
Library Format) should be
>>>>>>>>>> self explanatory.
>>>>>>>>>> lib.exe is able to accept def files and
create libraries using
>>>>>>>>>> this format.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe can create a user32.lib with
the function MessageBoxA
>>>>>>>>>> from the above definition.
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 is different MSVC in that we
need to compile the
>>>>>>>>>> runtime.
>>>>>>>>>> MS provide us with their crt prebuilt
so lib.exe doesn't have
>>>>>>>>>> support for external function aliasing.
>>>>>>>>>> We often use aliases for posix naming
reasons as well as avoid
>>>>>>>>>> using the MS version of a function.
>>>>>>>>>>
>>>>>>>>>> example
>>>>>>>>>>
>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>    EXPORTS
>>>>>>>>>>    MessageBoxA
>>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>>
>>>>>>>>>> LIB.exe doesn't not have support
for this but dlltool does.
>>>>>>>>>> The best fit for this within the spec
is PE COFF spec (Aux
>>>>>>>>>> Format 3: Weak Externals)
>>>>>>>>>>
>>>>>>>>>> Mingw-w64 also uses lib prefix and .a
file extensions for the
>>>>>>>>>> library name so the driver should be
different. The above example would be
>>>>>>>>>> libuser32.a, for when shared and static
versions of the same library exist
>>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Issues
>>>>>>>>>>
======>>>>>>>>>>
>>>>>>>>>> Like more of the gnu suite dlltool was
not designed in one build
>>>>>>>>>> multi target manner.
>>>>>>>>>> We would have to introduce custom
arguments to specify the
>>>>>>>>>> target, "i686",
"x86_64", "thumb2pe" etc.
>>>>>>>>>> This will most likely break some level
of backwards compatibility
>>>>>>>>>> with the binutils version.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/70e0cfb0/attachment.html>

On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:
> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at
gmail.com>
>> wrote:
>>
>>> No, I meant an even thinner wrapper which textually translates
>>>> arguments. For example, the wrapper would translates
"/out:foo.exe foo.obj"
>>>> to "-o foo.exe foo.obj" and then call
lld::COFFF:link(). It doesn't do
>>>> anything with Config object nor LinkerDriver::run and have
absolutely zero
>>>> knowledge on the internals of LLD.
>>>
>>> Ohh okay I misunderstood.
>>>
>>> (Why I'm asking this is because I want to sure that there's
no
>>>> difference between the regular Windows linker and mingw. If all
differences
>>>> are superficial, the command line translator should just work.
If not,
>>>> there's some semantic difference.)
>>>
>>> The libraries can be used with MSVC link.exe directly assuming the
>>> correct arguments are passed to the linker.
>>> I tested this after creating a working llvm-dlltool.
>>> The only change I had to make to support this was alter ming-w64
crt to
>>> change all references from __image_base__ to _ImageBase to support
that
>>>
>>
>> You may be able to define __ImageBase as a weak external symbol to
>> __image_base__. Then, if __ImageBase is not defined, all references
against
>> __ImageBase will be resolved using __image_base__.
>>
>
> Or you could have your wrapper driver pass
"/alternatename:__image_base__
> =__ImageBase".
>
Ah, that's much better indeed.

> Peter
>
>
>>
>>
>>> That should be fine.
>>>
>>> Great
>>>
>>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>
>>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <
>>>> martellmalone at gmail.com> wrote:
>>>>
>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an
in-process
>>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>>> linker's main function.
>>>>>
>>>>> That should work, if I am understanding this correctly I
can create an
>>>>> argument parser (probably partially based on the ELF one)
then I can set
>>>>> the data in the COFF Config object and call COFF
LinkerDriver::run etc
>>>>> directly from that?
>>>>>
>>>>
>>>> No, I meant an even thinner wrapper which textually translates
>>>> arguments. For example, the wrapper would translates
"/out:foo.exe foo.obj"
>>>> to "-o foo.exe foo.obj" and then call
lld::COFFF:link(). It doesn't do
>>>> anything with Config object nor LinkerDriver::run and have
absolutely zero
>>>> knowledge on the internals of LLD.
>>>>
>>>> (Why I'm asking this is because I want to sure that
there's no
>>>> difference between the regular Windows linker and mingw. If all
differences
>>>> are superficial, the command line translator should just work.
If not,
>>>> there's some semantic difference.)
>>>>
>>>> This proposal is for generally moving code from lld into llvm
that
>>>>> could be part of lib.exe / dlltool, what are your thoughts
here?
>>>>>
>>>>
>>>> That should be fine.
>>>>
>>>>
>>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at
google.com> wrote:
>>>>>
>>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <
>>>>>> martellmalone at gmail.com> wrote:
>>>>>>
>>>>>>> Also you need to make a change to LLD/COFF to
accept GNU command
>>>>>>>> arguments, right? (Looks like you already have
that patch locally.)
>>>>>>>
>>>>>>> Yes
>>>>>>>
>>>>>>
>>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be
an in-process
>>>>>> wrapper, meaning that you could add a main function for
mingw which
>>>>>> translates all arguments into the MSVC style and then
invoke the COFF
>>>>>> linker's main function.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> My patch to hack lld into accepting some very basic
gnu front end
>>>>>>>> arguments was enough to get all the above
working which was enough to
>>>>>>>> develop further.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama
<ruiu at google.com> wrote:
>>>>>>>
>>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell
Malone <
>>>>>>>> martellmalone at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Rui,
>>>>>>>>>
>>>>>>>>>> I wonder how llvm-dlltool would fit in
the entire picture of
>>>>>>>>>> mingw support. I don't think
dlltool is the last missing piece. What do you
>>>>>>>>>> need to do other than that to fully
support mingw using LLVM toolchain?
>>>>>>>>>
>>>>>>>>> Other then changing
`lib/MC/WinCOFFStreamer.cpp` to not use
>>>>>>>>> -aligncomm within the EmitCommonSymbol
function and a single patch for
>>>>>>>>> mingw-w64 itself to pre-populate it's
.ctors and .dtors list, so
>>>>>>>>> llvm-dlltool is infact the only missing
part really.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Also you need to make a change to LLD/COFF to
accept GNU command
>>>>>>>> arguments, right? (Looks like you already have
that patch locally.)
>>>>>>>>
>>>>>>>>
>>>>>>>>> This gives us a fully working clang based
mingw-w64 C compiler.
>>>>>>>>>
>>>>>>>>> C++ and exception handling is a different
story.
>>>>>>>>>
>>>>>>>>> libc++ is somewhat working with the
following test results
>>>>>>>>>
>>>>>>>>> Expected Passes    : 2188
>>>>>>>>> Expected Failures  : 44
>>>>>>>>> Unsupported Tests  : 588
>>>>>>>>> Unexpected Failures: 2816
>>>>>>>>>
>>>>>>>>> I was able to build it with exceptions
disabled and I actually
>>>>>>>>> managed to bootstrap llvm and clang itself.
>>>>>>>>> It didn't run very well though as you
can imagine based on the
>>>>>>>>> above tests.
>>>>>>>>>
>>>>>>>>> It's not a performance issue but a code
maintenance issue. The
>>>>>>>>>> initial patches to support mingw was
trying to add a new linker driver and
>>>>>>>>>> a different linkin semantics to the
COFF linker which seemed too
>>>>>>>>>> complicated to me. IIRC, I suggested
adding a shim which translates GNU
>>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>>
>>>>>>>>> There were just so many differences between
link and ld I never
>>>>>>>>> followed down that path. I pushed forward
with the short import library
>>>>>>>>> support based on your later suggestions. My
patch to hack lld into
>>>>>>>>> accepting some very basic gnu front end
arguments was enough to get all the
>>>>>>>>> above working which was enough to develop
further.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Martell
>>>>>>>>>
>>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama
<ruiu at google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Martell,
>>>>>>>>>>
>>>>>>>>>> I wonder how llvm-dlltool would fit in
the entire picture of
>>>>>>>>>> mingw support. I don't think
dlltool is the last missing piece. What do you
>>>>>>>>>> need to do other than that to fully
support mingw using LLVM toolchain?
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM,
Martell Malone via llvm-dev <
>>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey llvm'ers,
>>>>>>>>>>>
>>>>>>>>>>> I have been working on a dlltool
replacement for llvm.
>>>>>>>>>>> Here is my initial differential
https://reviews.llvm.org/D29892
>>>>>>>>>>> It is based on some functionality
that already exists in lld.
>>>>>>>>>>> I added functionality to support,
PE COFF Weak Externals and of
>>>>>>>>>>> course a front end to actually use
it.
>>>>>>>>>>> I believe the work here can also be
used for llvm-lib and lessen
>>>>>>>>>>> the load on lld.
>>>>>>>>>>> I would like some comments about
how this could be be structured
>>>>>>>>>>> to live in llvm with a shared code
base across lib ar and dlltool.
>>>>>>>>>>> I also have a section below called
"Difference from lib" which
>>>>>>>>>>> is somewhat of a rationale for the
tool.
>>>>>>>>>>>
>>>>>>>>>>> Many Thanks,
>>>>>>>>>>> Martell
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Context
>>>>>>>>>>>
=========>>>>>>>>>>>
>>>>>>>>>>> Awhile back I talked to various
llvm'ers about getting mingw-w64
>>>>>>>>>>> support for lld.
>>>>>>>>>>> There were a few issues raised but
the main issue was that
>>>>>>>>>>> mingw-w64 should try best to comply
with the PECOFF spec, adding support
>>>>>>>>>>> for custom sections and various
binutils/mingw hacks would impact the
>>>>>>>>>>> performance of the COFF linker and
in general is not something that lld
>>>>>>>>>>> should support.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It's not a performance issue but a
code maintenance issue. The
>>>>>>>>>> initial patches to support mingw was
trying to add a new linker driver and
>>>>>>>>>> a different linkin semantics to the
COFF linker which seemed too
>>>>>>>>>> complicated to me. IIRC, I suggested
adding a shim which translates GNU
>>>>>>>>>> command line arguments to MSVC linker
arguments. Didn't it work?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Motivation
>>>>>>>>>>>
=========>>>>>>>>>>>
>>>>>>>>>>> The main motivation was because
dlltool and ld did not comply
>>>>>>>>>>> with PECOFF Spec.
>>>>>>>>>>> It has some custom formatting and
uses the assembler for some
>>>>>>>>>>> reason to generate import
libraries, it did not use the short import
>>>>>>>>>>> library format at all.
>>>>>>>>>>>
>>>>>>>>>>> There has been many work arounds
for the problem this creates
>>>>>>>>>>> such as the creation of the `reimp`
tool. Which imports MSVC built libs
>>>>>>>>>>> creates a def and uses dlltool so
that the binutils linker can use it.
>>>>>>>>>>>
>>>>>>>>>>> We should just be using the short
import format and the linker
>>>>>>>>>>> should support that.
>>>>>>>>>>> Thus llvm-dlltool was born.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Difference from lib
>>>>>>>>>>>
==================>>>>>>>>>>>
>>>>>>>>>>> Using PE COFF spec (section 8,
Import Library Format) should be
>>>>>>>>>>> self explanatory.
>>>>>>>>>>> lib.exe is able to accept def files
and create libraries using
>>>>>>>>>>> this format.
>>>>>>>>>>>
>>>>>>>>>>> example
>>>>>>>>>>>
>>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>>    EXPORTS
>>>>>>>>>>>    MessageBoxA`
>>>>>>>>>>>
>>>>>>>>>>> LIB.exe can create a user32.lib
with the function MessageBoxA
>>>>>>>>>>> from the above definition.
>>>>>>>>>>>
>>>>>>>>>>> Mingw-w64 is different MSVC in that
we need to compile the
>>>>>>>>>>> runtime.
>>>>>>>>>>> MS provide us with their crt
prebuilt so lib.exe doesn't have
>>>>>>>>>>> support for external function
aliasing.
>>>>>>>>>>> We often use aliases for posix
naming reasons as well as avoid
>>>>>>>>>>> using the MS version of a function.
>>>>>>>>>>>
>>>>>>>>>>> example
>>>>>>>>>>>
>>>>>>>>>>>   `LIBRARY "user32.dll"
>>>>>>>>>>>    EXPORTS
>>>>>>>>>>>    MessageBoxA
>>>>>>>>>>>    MessageBoxW==MessageBoxA`
>>>>>>>>>>>
>>>>>>>>>>> LIB.exe doesn't not have
support for this but dlltool does.
>>>>>>>>>>> The best fit for this within the
spec is PE COFF spec (Aux
>>>>>>>>>>> Format 3: Weak Externals)
>>>>>>>>>>>
>>>>>>>>>>> Mingw-w64 also uses lib prefix and
.a file extensions for the
>>>>>>>>>>> library name so the driver should
be different. The above example would be
>>>>>>>>>>> libuser32.a, for when shared and
static versions of the same library exist
>>>>>>>>>>> there is the .dll.a variant.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Issues
>>>>>>>>>>>
======>>>>>>>>>>>
>>>>>>>>>>> Like more of the gnu suite dlltool
was not designed in one build
>>>>>>>>>>> multi target manner.
>>>>>>>>>>> We would have to introduce custom
arguments to specify the
>>>>>>>>>>> target, "i686",
"x86_64", "thumb2pe" etc.
>>>>>>>>>>> This will most likely break some
level of backwards
>>>>>>>>>>> compatibility with the binutils
version.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
> --
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/e79dc480/attachment.html>

Ohh nice.
With that method I can support it without upsetting ld users by introducing
an api breakage.

On Tue 14 Feb 2017 at 01:32, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at
pcc.me.uk>
> wrote:
>
> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
> No, I meant an even thinner wrapper which textually translates arguments.
> For example, the wrapper would translates "/out:foo.exe foo.obj"
to "-o
> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
anything
> with Config object nor LinkerDriver::run and have absolutely zero knowledge
> on the internals of LLD.
>
> Ohh okay I misunderstood.
>
> (Why I'm asking this is because I want to sure that there's no
difference
> between the regular Windows linker and mingw. If all differences are
> superficial, the command line translator should just work. If not,
there's
> some semantic difference.)
>
> The libraries can be used with MSVC link.exe directly assuming the correct
> arguments are passed to the linker.
> I tested this after creating a working llvm-dlltool.
> The only change I had to make to support this was alter ming-w64 crt to
> change all references from __image_base__ to _ImageBase to support that
>
>
> You may be able to define __ImageBase as a weak external symbol to
> __image_base__. Then, if __ImageBase is not defined, all references against
> __ImageBase will be resolved using __image_base__.
>
>
> Or you could have your wrapper driver pass
> "/alternatename:__image_base__=__ImageBase".
>
>
> Ah, that's much better indeed.
>
>
> Peter
>
>
>
>
> That should be fine.
>
> Great
>
> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
> wrapper, meaning that you could add a main function for mingw which
> translates all arguments into the MSVC style and then invoke the COFF
> linker's main function.
>
> That should work, if I am understanding this correctly I can create an
> argument parser (probably partially based on the ELF one) then I can set
> the data in the COFF Config object and call COFF LinkerDriver::run etc
> directly from that?
>
>
> No, I meant an even thinner wrapper which textually translates arguments.
> For example, the wrapper would translates "/out:foo.exe foo.obj"
to "-o
> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
anything
> with Config object nor LinkerDriver::run and have absolutely zero knowledge
> on the internals of LLD.
>
> (Why I'm asking this is because I want to sure that there's no
difference
> between the regular Windows linker and mingw. If all differences are
> superficial, the command line translator should just work. If not,
there's
> some semantic difference.)
>
> This proposal is for generally moving code from lld into llvm that could
> be part of lib.exe / dlltool, what are your thoughts here?
>
>
> That should be fine.
>
>
> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
>
> Yes
>
>
> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
> wrapper, meaning that you could add a main function for mingw which
> translates all arguments into the MSVC style and then invoke the COFF
> linker's main function.
>
>
>
> My patch to hack lld into accepting some very basic gnu front end
> arguments was enough to get all the above working which was enough to
> develop further.
>
>
>
> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at
gmail.com>
> wrote:
>
> Hey Rui,
>
> I wonder how llvm-dlltool would fit in the entire picture of mingw
> support. I don't think dlltool is the last missing piece. What do you
need
> to do other than that to fully support mingw using LLVM toolchain?
>
> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
> within the EmitCommonSymbol function and a single patch for mingw-w64
> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is
> infact the only missing part really.
>
>
> Also you need to make a change to LLD/COFF to accept GNU command
> arguments, right? (Looks like you already have that patch locally.)
>
>
> This gives us a fully working clang based mingw-w64 C compiler.
>
> C++ and exception handling is a different story.
>
> libc++ is somewhat working with the following test results
>
> Expected Passes    : 2188
> Expected Failures  : 44
> Unsupported Tests  : 588
> Unexpected Failures: 2816
>
> I was able to build it with exceptions disabled and I actually managed to
> bootstrap llvm and clang itself.
> It didn't run very well though as you can imagine based on the above
tests.
>
> It's not a performance issue but a code maintenance issue. The initial
> patches to support mingw was trying to add a new linker driver and a
> different linkin semantics to the COFF linker which seemed too complicated
> to me. IIRC, I suggested adding a shim which translates GNU command line
> arguments to MSVC linker arguments. Didn't it work?
>
> There were just so many differences between link and ld I never followed
> down that path. I pushed forward with the short import library support
> based on your later suggestions. My patch to hack lld into accepting some
> very basic gnu front end arguments was enough to get all the above working
> which was enough to develop further.
>
> Best,
> Martell
>
> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
> Hi Martell,
>
> I wonder how llvm-dlltool would fit in the entire picture of mingw
> support. I don't think dlltool is the last missing piece. What do you
need
> to do other than that to fully support mingw using LLVM toolchain?
>
> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hey llvm'ers,
>
> I have been working on a dlltool replacement for llvm.
> Here is my initial differential https://reviews.llvm.org/D29892
> It is based on some functionality that already exists in lld.
> I added functionality to support, PE COFF Weak Externals and of course a
> front end to actually use it.
> I believe the work here can also be used for llvm-lib and lessen the load
> on lld.
> I would like some comments about how this could be be structured to live
> in llvm with a shared code base across lib ar and dlltool.
> I also have a section below called "Difference from lib" which is
somewhat
> of a rationale for the tool.
>
> Many Thanks,
> Martell
>
>
> Context
> =========>
> Awhile back I talked to various llvm'ers about getting mingw-w64
support
> for lld.
> There were a few issues raised but the main issue was that mingw-w64
> should try best to comply with the PECOFF spec, adding support for custom
> sections and various binutils/mingw hacks would impact the performance of
> the COFF linker and in general is not something that lld should support.
>
>
> It's not a performance issue but a code maintenance issue. The initial
> patches to support mingw was trying to add a new linker driver and a
> different linkin semantics to the COFF linker which seemed too complicated
> to me. IIRC, I suggested adding a shim which translates GNU command line
> arguments to MSVC linker arguments. Didn't it work?
>
>
> Motivation
> =========>
> The main motivation was because dlltool and ld did not comply with PECOFF
> Spec.
> It has some custom formatting and uses the assembler for some reason to
> generate import libraries, it did not use the short import library format
> at all.
>
> There has been many work arounds for the problem this creates such as the
> creation of the `reimp` tool. Which imports MSVC built libs creates a def
> and uses dlltool so that the binutils linker can use it.
>
> We should just be using the short import format and the linker should
> support that.
> Thus llvm-dlltool was born.
>
>
> Difference from lib
> ==================>
> Using PE COFF spec (section 8, Import Library Format) should be self
> explanatory.
> lib.exe is able to accept def files and create libraries using this format.
>
> example
>
>   `LIBRARY "user32.dll"
>    EXPORTS
>    MessageBoxA`
>
> LIB.exe can create a user32.lib with the function MessageBoxA from the
> above definition.
>
> Mingw-w64 is different MSVC in that we need to compile the runtime.
> MS provide us with their crt prebuilt so lib.exe doesn't have support
for
> external function aliasing.
> We often use aliases for posix naming reasons as well as avoid using the
> MS version of a function.
>
> example
>
>   `LIBRARY "user32.dll"
>    EXPORTS
>    MessageBoxA
>    MessageBoxW==MessageBoxA`
>
> LIB.exe doesn't not have support for this but dlltool does.
> The best fit for this within the spec is PE COFF spec (Aux Format 3: Weak
> Externals)
>
> Mingw-w64 also uses lib prefix and .a file extensions for the library name
> so the driver should be different. The above example would be libuser32.a,
> for when shared and static versions of the same library exist there is the
> .dll.a variant.
>
>
> Issues
> ======>
> Like more of the gnu suite dlltool was not designed in one build multi
> target manner.
> We would have to introduce custom arguments to specify the target,
"i686",
> "x86_64", "thumb2pe" etc.
> This will most likely break some level of backwards compatibility with the
> binutils version.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
> --
> --
> Peter
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170214/c3bc4daf/attachment-0001.html>