Peter Collingbourne via llvm-dev
2017-Feb-14 01:26 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com> > wrote: > >> No, I meant an even thinner wrapper which textually translates arguments. >>> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o >>> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything >>> with Config object nor LinkerDriver::run and have absolutely zero knowledge >>> on the internals of LLD. >> >> Ohh okay I misunderstood. >> >> (Why I'm asking this is because I want to sure that there's no difference >>> between the regular Windows linker and mingw. If all differences are >>> superficial, the command line translator should just work. If not, there's >>> some semantic difference.) >> >> The libraries can be used with MSVC link.exe directly assuming the >> correct arguments are passed to the linker. >> I tested this after creating a working llvm-dlltool. >> The only change I had to make to support this was alter ming-w64 crt to >> change all references from __image_base__ to _ImageBase to support that >> > > You may be able to define __ImageBase as a weak external symbol to > __image_base__. Then, if __ImageBase is not defined, all references against > __ImageBase will be resolved using __image_base__. >Or you could have your wrapper driver pass "/alternatename:__image_base__=__ImageBase". Peter> > >> That should be fine. >> >> Great >> >> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote: >> >>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com >>> > wrote: >>> >>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >>>>> wrapper, meaning that you could add a main function for mingw which >>>>> translates all arguments into the MSVC style and then invoke the COFF >>>>> linker's main function. >>>> >>>> That should work, if I am understanding this correctly I can create an >>>> argument parser (probably partially based on the ELF one) then I can set >>>> the data in the COFF Config object and call COFF LinkerDriver::run etc >>>> directly from that? >>>> >>> >>> No, I meant an even thinner wrapper which textually translates >>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj" >>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do >>> anything with Config object nor LinkerDriver::run and have absolutely zero >>> knowledge on the internals of LLD. >>> >>> (Why I'm asking this is because I want to sure that there's no >>> difference between the regular Windows linker and mingw. If all differences >>> are superficial, the command line translator should just work. If not, >>> there's some semantic difference.) >>> >>> This proposal is for generally moving code from lld into llvm that could >>>> be part of lib.exe / dlltool, what are your thoughts here? >>>> >>> >>> That should be fine. >>> >>> >>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote: >>>> >>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone < >>>>> martellmalone at gmail.com> wrote: >>>>> >>>>>> Also you need to make a change to LLD/COFF to accept GNU command >>>>>>> arguments, right? (Looks like you already have that patch locally.) >>>>>> >>>>>> Yes >>>>>> >>>>> >>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >>>>> wrapper, meaning that you could add a main function for mingw which >>>>> translates all arguments into the MSVC style and then invoke the COFF >>>>> linker's main function. >>>>> >>>>> >>>>>> >>>>>> My patch to hack lld into accepting some very basic gnu front end >>>>>>> arguments was enough to get all the above working which was enough to >>>>>>> develop further. >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote: >>>>>> >>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone < >>>>>>> martellmalone at gmail.com> wrote: >>>>>>> >>>>>>>> Hey Rui, >>>>>>>> >>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw >>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need >>>>>>>>> to do other than that to fully support mingw using LLVM toolchain? >>>>>>>> >>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use >>>>>>>> -aligncomm within the EmitCommonSymbol function and a single patch for >>>>>>>> mingw-w64 itself to pre-populate it's .ctors and .dtors list, so >>>>>>>> llvm-dlltool is infact the only missing part really. >>>>>>>> >>>>>>> >>>>>>> Also you need to make a change to LLD/COFF to accept GNU command >>>>>>> arguments, right? (Looks like you already have that patch locally.) >>>>>>> >>>>>>> >>>>>>>> This gives us a fully working clang based mingw-w64 C compiler. >>>>>>>> >>>>>>>> C++ and exception handling is a different story. >>>>>>>> >>>>>>>> libc++ is somewhat working with the following test results >>>>>>>> >>>>>>>> Expected Passes : 2188 >>>>>>>> Expected Failures : 44 >>>>>>>> Unsupported Tests : 588 >>>>>>>> Unexpected Failures: 2816 >>>>>>>> >>>>>>>> I was able to build it with exceptions disabled and I actually >>>>>>>> managed to bootstrap llvm and clang itself. >>>>>>>> It didn't run very well though as you can imagine based on the >>>>>>>> above tests. >>>>>>>> >>>>>>>> It's not a performance issue but a code maintenance issue. The >>>>>>>>> initial patches to support mingw was trying to add a new linker driver and >>>>>>>>> a different linkin semantics to the COFF linker which seemed too >>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU >>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work? >>>>>>>> >>>>>>>> There were just so many differences between link and ld I never >>>>>>>> followed down that path. I pushed forward with the short import library >>>>>>>> support based on your later suggestions. My patch to hack lld into >>>>>>>> accepting some very basic gnu front end arguments was enough to get all the >>>>>>>> above working which was enough to develop further. >>>>>>>> >>>>>>>> Best, >>>>>>>> Martell >>>>>>>> >>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Martell, >>>>>>>>> >>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of mingw >>>>>>>>> support. I don't think dlltool is the last missing piece. What do you need >>>>>>>>> to do other than that to fully support mingw using LLVM toolchain? >>>>>>>>> >>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev < >>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>> >>>>>>>>>> Hey llvm'ers, >>>>>>>>>> >>>>>>>>>> I have been working on a dlltool replacement for llvm. >>>>>>>>>> Here is my initial differential https://reviews.llvm.org/D29892 >>>>>>>>>> It is based on some functionality that already exists in lld. >>>>>>>>>> I added functionality to support, PE COFF Weak Externals and of >>>>>>>>>> course a front end to actually use it. >>>>>>>>>> I believe the work here can also be used for llvm-lib and lessen >>>>>>>>>> the load on lld. >>>>>>>>>> I would like some comments about how this could be be structured >>>>>>>>>> to live in llvm with a shared code base across lib ar and dlltool. >>>>>>>>>> I also have a section below called "Difference from lib" which is >>>>>>>>>> somewhat of a rationale for the tool. >>>>>>>>>> >>>>>>>>>> Many Thanks, >>>>>>>>>> Martell >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Context >>>>>>>>>> =========>>>>>>>>>> >>>>>>>>>> Awhile back I talked to various llvm'ers about getting mingw-w64 >>>>>>>>>> support for lld. >>>>>>>>>> There were a few issues raised but the main issue was that >>>>>>>>>> mingw-w64 should try best to comply with the PECOFF spec, adding support >>>>>>>>>> for custom sections and various binutils/mingw hacks would impact the >>>>>>>>>> performance of the COFF linker and in general is not something that lld >>>>>>>>>> should support. >>>>>>>>>> >>>>>>>>> >>>>>>>>> It's not a performance issue but a code maintenance issue. The >>>>>>>>> initial patches to support mingw was trying to add a new linker driver and >>>>>>>>> a different linkin semantics to the COFF linker which seemed too >>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU >>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work? >>>>>>>>> >>>>>>>>> >>>>>>>>>> Motivation >>>>>>>>>> =========>>>>>>>>>> >>>>>>>>>> The main motivation was because dlltool and ld did not comply >>>>>>>>>> with PECOFF Spec. >>>>>>>>>> It has some custom formatting and uses the assembler for some >>>>>>>>>> reason to generate import libraries, it did not use the short import >>>>>>>>>> library format at all. >>>>>>>>>> >>>>>>>>>> There has been many work arounds for the problem this creates >>>>>>>>>> such as the creation of the `reimp` tool. Which imports MSVC built libs >>>>>>>>>> creates a def and uses dlltool so that the binutils linker can use it. >>>>>>>>>> >>>>>>>>>> We should just be using the short import format and the linker >>>>>>>>>> should support that. >>>>>>>>>> Thus llvm-dlltool was born. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Difference from lib >>>>>>>>>> ==================>>>>>>>>>> >>>>>>>>>> Using PE COFF spec (section 8, Import Library Format) should be >>>>>>>>>> self explanatory. >>>>>>>>>> lib.exe is able to accept def files and create libraries using >>>>>>>>>> this format. >>>>>>>>>> >>>>>>>>>> example >>>>>>>>>> >>>>>>>>>> `LIBRARY "user32.dll" >>>>>>>>>> EXPORTS >>>>>>>>>> MessageBoxA` >>>>>>>>>> >>>>>>>>>> LIB.exe can create a user32.lib with the function MessageBoxA >>>>>>>>>> from the above definition. >>>>>>>>>> >>>>>>>>>> Mingw-w64 is different MSVC in that we need to compile the >>>>>>>>>> runtime. >>>>>>>>>> MS provide us with their crt prebuilt so lib.exe doesn't have >>>>>>>>>> support for external function aliasing. >>>>>>>>>> We often use aliases for posix naming reasons as well as avoid >>>>>>>>>> using the MS version of a function. >>>>>>>>>> >>>>>>>>>> example >>>>>>>>>> >>>>>>>>>> `LIBRARY "user32.dll" >>>>>>>>>> EXPORTS >>>>>>>>>> MessageBoxA >>>>>>>>>> MessageBoxW==MessageBoxA` >>>>>>>>>> >>>>>>>>>> LIB.exe doesn't not have support for this but dlltool does. >>>>>>>>>> The best fit for this within the spec is PE COFF spec (Aux >>>>>>>>>> Format 3: Weak Externals) >>>>>>>>>> >>>>>>>>>> Mingw-w64 also uses lib prefix and .a file extensions for the >>>>>>>>>> library name so the driver should be different. The above example would be >>>>>>>>>> libuser32.a, for when shared and static versions of the same library exist >>>>>>>>>> there is the .dll.a variant. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Issues >>>>>>>>>> ======>>>>>>>>>> >>>>>>>>>> Like more of the gnu suite dlltool was not designed in one build >>>>>>>>>> multi target manner. >>>>>>>>>> We would have to introduce custom arguments to specify the >>>>>>>>>> target, "i686", "x86_64", "thumb2pe" etc. >>>>>>>>>> This will most likely break some level of backwards compatibility >>>>>>>>>> with the binutils version. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> LLVM Developers mailing list >>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-- -- Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/70e0cfb0/attachment.html>
Rui Ueyama via llvm-dev
2017-Feb-14 01:31 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk> wrote:> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com> >> wrote: >> >>> No, I meant an even thinner wrapper which textually translates >>>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj" >>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do >>>> anything with Config object nor LinkerDriver::run and have absolutely zero >>>> knowledge on the internals of LLD. >>> >>> Ohh okay I misunderstood. >>> >>> (Why I'm asking this is because I want to sure that there's no >>>> difference between the regular Windows linker and mingw. If all differences >>>> are superficial, the command line translator should just work. If not, >>>> there's some semantic difference.) >>> >>> The libraries can be used with MSVC link.exe directly assuming the >>> correct arguments are passed to the linker. >>> I tested this after creating a working llvm-dlltool. >>> The only change I had to make to support this was alter ming-w64 crt to >>> change all references from __image_base__ to _ImageBase to support that >>> >> >> You may be able to define __ImageBase as a weak external symbol to >> __image_base__. Then, if __ImageBase is not defined, all references against >> __ImageBase will be resolved using __image_base__. >> > > Or you could have your wrapper driver pass "/alternatename:__image_base__ > =__ImageBase". >Ah, that's much better indeed.> Peter > > >> >> >>> That should be fine. >>> >>> Great >>> >>> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote: >>> >>>> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone < >>>> martellmalone at gmail.com> wrote: >>>> >>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >>>>>> wrapper, meaning that you could add a main function for mingw which >>>>>> translates all arguments into the MSVC style and then invoke the COFF >>>>>> linker's main function. >>>>> >>>>> That should work, if I am understanding this correctly I can create an >>>>> argument parser (probably partially based on the ELF one) then I can set >>>>> the data in the COFF Config object and call COFF LinkerDriver::run etc >>>>> directly from that? >>>>> >>>> >>>> No, I meant an even thinner wrapper which textually translates >>>> arguments. For example, the wrapper would translates "/out:foo.exe foo.obj" >>>> to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do >>>> anything with Config object nor LinkerDriver::run and have absolutely zero >>>> knowledge on the internals of LLD. >>>> >>>> (Why I'm asking this is because I want to sure that there's no >>>> difference between the regular Windows linker and mingw. If all differences >>>> are superficial, the command line translator should just work. If not, >>>> there's some semantic difference.) >>>> >>>> This proposal is for generally moving code from lld into llvm that >>>>> could be part of lib.exe / dlltool, what are your thoughts here? >>>>> >>>> >>>> That should be fine. >>>> >>>> >>>>> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote: >>>>> >>>>>> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone < >>>>>> martellmalone at gmail.com> wrote: >>>>>> >>>>>>> Also you need to make a change to LLD/COFF to accept GNU command >>>>>>>> arguments, right? (Looks like you already have that patch locally.) >>>>>>> >>>>>>> Yes >>>>>>> >>>>>> >>>>>> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >>>>>> wrapper, meaning that you could add a main function for mingw which >>>>>> translates all arguments into the MSVC style and then invoke the COFF >>>>>> linker's main function. >>>>>> >>>>>> >>>>>>> >>>>>>> My patch to hack lld into accepting some very basic gnu front end >>>>>>>> arguments was enough to get all the above working which was enough to >>>>>>>> develop further. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote: >>>>>>> >>>>>>>> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone < >>>>>>>> martellmalone at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hey Rui, >>>>>>>>> >>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of >>>>>>>>>> mingw support. I don't think dlltool is the last missing piece. What do you >>>>>>>>>> need to do other than that to fully support mingw using LLVM toolchain? >>>>>>>>> >>>>>>>>> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use >>>>>>>>> -aligncomm within the EmitCommonSymbol function and a single patch for >>>>>>>>> mingw-w64 itself to pre-populate it's .ctors and .dtors list, so >>>>>>>>> llvm-dlltool is infact the only missing part really. >>>>>>>>> >>>>>>>> >>>>>>>> Also you need to make a change to LLD/COFF to accept GNU command >>>>>>>> arguments, right? (Looks like you already have that patch locally.) >>>>>>>> >>>>>>>> >>>>>>>>> This gives us a fully working clang based mingw-w64 C compiler. >>>>>>>>> >>>>>>>>> C++ and exception handling is a different story. >>>>>>>>> >>>>>>>>> libc++ is somewhat working with the following test results >>>>>>>>> >>>>>>>>> Expected Passes : 2188 >>>>>>>>> Expected Failures : 44 >>>>>>>>> Unsupported Tests : 588 >>>>>>>>> Unexpected Failures: 2816 >>>>>>>>> >>>>>>>>> I was able to build it with exceptions disabled and I actually >>>>>>>>> managed to bootstrap llvm and clang itself. >>>>>>>>> It didn't run very well though as you can imagine based on the >>>>>>>>> above tests. >>>>>>>>> >>>>>>>>> It's not a performance issue but a code maintenance issue. The >>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and >>>>>>>>>> a different linkin semantics to the COFF linker which seemed too >>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU >>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work? >>>>>>>>> >>>>>>>>> There were just so many differences between link and ld I never >>>>>>>>> followed down that path. I pushed forward with the short import library >>>>>>>>> support based on your later suggestions. My patch to hack lld into >>>>>>>>> accepting some very basic gnu front end arguments was enough to get all the >>>>>>>>> above working which was enough to develop further. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Martell >>>>>>>>> >>>>>>>>> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Martell, >>>>>>>>>> >>>>>>>>>> I wonder how llvm-dlltool would fit in the entire picture of >>>>>>>>>> mingw support. I don't think dlltool is the last missing piece. What do you >>>>>>>>>> need to do other than that to fully support mingw using LLVM toolchain? >>>>>>>>>> >>>>>>>>>> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev < >>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>> >>>>>>>>>>> Hey llvm'ers, >>>>>>>>>>> >>>>>>>>>>> I have been working on a dlltool replacement for llvm. >>>>>>>>>>> Here is my initial differential https://reviews.llvm.org/D29892 >>>>>>>>>>> It is based on some functionality that already exists in lld. >>>>>>>>>>> I added functionality to support, PE COFF Weak Externals and of >>>>>>>>>>> course a front end to actually use it. >>>>>>>>>>> I believe the work here can also be used for llvm-lib and lessen >>>>>>>>>>> the load on lld. >>>>>>>>>>> I would like some comments about how this could be be structured >>>>>>>>>>> to live in llvm with a shared code base across lib ar and dlltool. >>>>>>>>>>> I also have a section below called "Difference from lib" which >>>>>>>>>>> is somewhat of a rationale for the tool. >>>>>>>>>>> >>>>>>>>>>> Many Thanks, >>>>>>>>>>> Martell >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Context >>>>>>>>>>> =========>>>>>>>>>>> >>>>>>>>>>> Awhile back I talked to various llvm'ers about getting mingw-w64 >>>>>>>>>>> support for lld. >>>>>>>>>>> There were a few issues raised but the main issue was that >>>>>>>>>>> mingw-w64 should try best to comply with the PECOFF spec, adding support >>>>>>>>>>> for custom sections and various binutils/mingw hacks would impact the >>>>>>>>>>> performance of the COFF linker and in general is not something that lld >>>>>>>>>>> should support. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It's not a performance issue but a code maintenance issue. The >>>>>>>>>> initial patches to support mingw was trying to add a new linker driver and >>>>>>>>>> a different linkin semantics to the COFF linker which seemed too >>>>>>>>>> complicated to me. IIRC, I suggested adding a shim which translates GNU >>>>>>>>>> command line arguments to MSVC linker arguments. Didn't it work? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Motivation >>>>>>>>>>> =========>>>>>>>>>>> >>>>>>>>>>> The main motivation was because dlltool and ld did not comply >>>>>>>>>>> with PECOFF Spec. >>>>>>>>>>> It has some custom formatting and uses the assembler for some >>>>>>>>>>> reason to generate import libraries, it did not use the short import >>>>>>>>>>> library format at all. >>>>>>>>>>> >>>>>>>>>>> There has been many work arounds for the problem this creates >>>>>>>>>>> such as the creation of the `reimp` tool. Which imports MSVC built libs >>>>>>>>>>> creates a def and uses dlltool so that the binutils linker can use it. >>>>>>>>>>> >>>>>>>>>>> We should just be using the short import format and the linker >>>>>>>>>>> should support that. >>>>>>>>>>> Thus llvm-dlltool was born. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Difference from lib >>>>>>>>>>> ==================>>>>>>>>>>> >>>>>>>>>>> Using PE COFF spec (section 8, Import Library Format) should be >>>>>>>>>>> self explanatory. >>>>>>>>>>> lib.exe is able to accept def files and create libraries using >>>>>>>>>>> this format. >>>>>>>>>>> >>>>>>>>>>> example >>>>>>>>>>> >>>>>>>>>>> `LIBRARY "user32.dll" >>>>>>>>>>> EXPORTS >>>>>>>>>>> MessageBoxA` >>>>>>>>>>> >>>>>>>>>>> LIB.exe can create a user32.lib with the function MessageBoxA >>>>>>>>>>> from the above definition. >>>>>>>>>>> >>>>>>>>>>> Mingw-w64 is different MSVC in that we need to compile the >>>>>>>>>>> runtime. >>>>>>>>>>> MS provide us with their crt prebuilt so lib.exe doesn't have >>>>>>>>>>> support for external function aliasing. >>>>>>>>>>> We often use aliases for posix naming reasons as well as avoid >>>>>>>>>>> using the MS version of a function. >>>>>>>>>>> >>>>>>>>>>> example >>>>>>>>>>> >>>>>>>>>>> `LIBRARY "user32.dll" >>>>>>>>>>> EXPORTS >>>>>>>>>>> MessageBoxA >>>>>>>>>>> MessageBoxW==MessageBoxA` >>>>>>>>>>> >>>>>>>>>>> LIB.exe doesn't not have support for this but dlltool does. >>>>>>>>>>> The best fit for this within the spec is PE COFF spec (Aux >>>>>>>>>>> Format 3: Weak Externals) >>>>>>>>>>> >>>>>>>>>>> Mingw-w64 also uses lib prefix and .a file extensions for the >>>>>>>>>>> library name so the driver should be different. The above example would be >>>>>>>>>>> libuser32.a, for when shared and static versions of the same library exist >>>>>>>>>>> there is the .dll.a variant. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Issues >>>>>>>>>>> ======>>>>>>>>>>> >>>>>>>>>>> Like more of the gnu suite dlltool was not designed in one build >>>>>>>>>>> multi target manner. >>>>>>>>>>> We would have to introduce custom arguments to specify the >>>>>>>>>>> target, "i686", "x86_64", "thumb2pe" etc. >>>>>>>>>>> This will most likely break some level of backwards >>>>>>>>>>> compatibility with the binutils version. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> > > > -- > -- > Peter >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/e79dc480/attachment.html>
Martell Malone via llvm-dev
2017-Feb-14 01:37 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
Ohh nice. With that method I can support it without upsetting ld users by introducing an api breakage. On Tue 14 Feb 2017 at 01:32, Rui Ueyama <ruiu at google.com> wrote:> On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk> > wrote: > > On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > No, I meant an even thinner wrapper which textually translates arguments. > For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o > foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything > with Config object nor LinkerDriver::run and have absolutely zero knowledge > on the internals of LLD. > > Ohh okay I misunderstood. > > (Why I'm asking this is because I want to sure that there's no difference > between the regular Windows linker and mingw. If all differences are > superficial, the command line translator should just work. If not, there's > some semantic difference.) > > The libraries can be used with MSVC link.exe directly assuming the correct > arguments are passed to the linker. > I tested this after creating a working llvm-dlltool. > The only change I had to make to support this was alter ming-w64 crt to > change all references from __image_base__ to _ImageBase to support that > > > You may be able to define __ImageBase as a weak external symbol to > __image_base__. Then, if __ImageBase is not defined, all references against > __ImageBase will be resolved using __image_base__. > > > Or you could have your wrapper driver pass > "/alternatename:__image_base__=__ImageBase". > > > Ah, that's much better indeed. > > > Peter > > > > > That should be fine. > > Great > > On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > I wonder if it can be a wrapper for LLD/COFF. It can be an in-process > wrapper, meaning that you could add a main function for mingw which > translates all arguments into the MSVC style and then invoke the COFF > linker's main function. > > That should work, if I am understanding this correctly I can create an > argument parser (probably partially based on the ELF one) then I can set > the data in the COFF Config object and call COFF LinkerDriver::run etc > directly from that? > > > No, I meant an even thinner wrapper which textually translates arguments. > For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o > foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything > with Config object nor LinkerDriver::run and have absolutely zero knowledge > on the internals of LLD. > > (Why I'm asking this is because I want to sure that there's no difference > between the regular Windows linker and mingw. If all differences are > superficial, the command line translator should just work. If not, there's > some semantic difference.) > > This proposal is for generally moving code from lld into llvm that could > be part of lib.exe / dlltool, what are your thoughts here? > > > That should be fine. > > > On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Also you need to make a change to LLD/COFF to accept GNU command > arguments, right? (Looks like you already have that patch locally.) > > Yes > > > I wonder if it can be a wrapper for LLD/COFF. It can be an in-process > wrapper, meaning that you could add a main function for mingw which > translates all arguments into the MSVC style and then invoke the COFF > linker's main function. > > > > My patch to hack lld into accepting some very basic gnu front end > arguments was enough to get all the above working which was enough to > develop further. > > > > On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Hey Rui, > > I wonder how llvm-dlltool would fit in the entire picture of mingw > support. I don't think dlltool is the last missing piece. What do you need > to do other than that to fully support mingw using LLVM toolchain? > > Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm > within the EmitCommonSymbol function and a single patch for mingw-w64 > itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is > infact the only missing part really. > > > Also you need to make a change to LLD/COFF to accept GNU command > arguments, right? (Looks like you already have that patch locally.) > > > This gives us a fully working clang based mingw-w64 C compiler. > > C++ and exception handling is a different story. > > libc++ is somewhat working with the following test results > > Expected Passes : 2188 > Expected Failures : 44 > Unsupported Tests : 588 > Unexpected Failures: 2816 > > I was able to build it with exceptions disabled and I actually managed to > bootstrap llvm and clang itself. > It didn't run very well though as you can imagine based on the above tests. > > It's not a performance issue but a code maintenance issue. The initial > patches to support mingw was trying to add a new linker driver and a > different linkin semantics to the COFF linker which seemed too complicated > to me. IIRC, I suggested adding a shim which translates GNU command line > arguments to MSVC linker arguments. Didn't it work? > > There were just so many differences between link and ld I never followed > down that path. I pushed forward with the short import library support > based on your later suggestions. My patch to hack lld into accepting some > very basic gnu front end arguments was enough to get all the above working > which was enough to develop further. > > Best, > Martell > > On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote: > > Hi Martell, > > I wonder how llvm-dlltool would fit in the entire picture of mingw > support. I don't think dlltool is the last missing piece. What do you need > to do other than that to fully support mingw using LLVM toolchain? > > On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Hey llvm'ers, > > I have been working on a dlltool replacement for llvm. > Here is my initial differential https://reviews.llvm.org/D29892 > It is based on some functionality that already exists in lld. > I added functionality to support, PE COFF Weak Externals and of course a > front end to actually use it. > I believe the work here can also be used for llvm-lib and lessen the load > on lld. > I would like some comments about how this could be be structured to live > in llvm with a shared code base across lib ar and dlltool. > I also have a section below called "Difference from lib" which is somewhat > of a rationale for the tool. > > Many Thanks, > Martell > > > Context > =========> > Awhile back I talked to various llvm'ers about getting mingw-w64 support > for lld. > There were a few issues raised but the main issue was that mingw-w64 > should try best to comply with the PECOFF spec, adding support for custom > sections and various binutils/mingw hacks would impact the performance of > the COFF linker and in general is not something that lld should support. > > > It's not a performance issue but a code maintenance issue. The initial > patches to support mingw was trying to add a new linker driver and a > different linkin semantics to the COFF linker which seemed too complicated > to me. IIRC, I suggested adding a shim which translates GNU command line > arguments to MSVC linker arguments. Didn't it work? > > > Motivation > =========> > The main motivation was because dlltool and ld did not comply with PECOFF > Spec. > It has some custom formatting and uses the assembler for some reason to > generate import libraries, it did not use the short import library format > at all. > > There has been many work arounds for the problem this creates such as the > creation of the `reimp` tool. Which imports MSVC built libs creates a def > and uses dlltool so that the binutils linker can use it. > > We should just be using the short import format and the linker should > support that. > Thus llvm-dlltool was born. > > > Difference from lib > ==================> > Using PE COFF spec (section 8, Import Library Format) should be self > explanatory. > lib.exe is able to accept def files and create libraries using this format. > > example > > `LIBRARY "user32.dll" > EXPORTS > MessageBoxA` > > LIB.exe can create a user32.lib with the function MessageBoxA from the > above definition. > > Mingw-w64 is different MSVC in that we need to compile the runtime. > MS provide us with their crt prebuilt so lib.exe doesn't have support for > external function aliasing. > We often use aliases for posix naming reasons as well as avoid using the > MS version of a function. > > example > > `LIBRARY "user32.dll" > EXPORTS > MessageBoxA > MessageBoxW==MessageBoxA` > > LIB.exe doesn't not have support for this but dlltool does. > The best fit for this within the spec is PE COFF spec (Aux Format 3: Weak > Externals) > > Mingw-w64 also uses lib prefix and .a file extensions for the library name > so the driver should be different. The above example would be libuser32.a, > for when shared and static versions of the same library exist there is the > .dll.a variant. > > > Issues > ======> > Like more of the gnu suite dlltool was not designed in one build multi > target manner. > We would have to introduce custom arguments to specify the target, "i686", > "x86_64", "thumb2pe" etc. > This will most likely break some level of backwards compatibility with the > binutils version. > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > -- > -- > Peter > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170214/c3bc4daf/attachment-0001.html>