Martell Malone via llvm-dev
2017-Feb-14 02:06 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
Well that means it would just be the only plan then. I assume the first step would be to move the code from lld into this new library and lld can use that? I can then re add the extra functionality for weak externals? We can then have lib and dlltool use this then. On Tue 14 Feb 2017 at 01:58, Rui Ueyama <ruiu at google.com> wrote:> On Mon, Feb 13, 2017 at 5:56 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Yes, that would be the long term plan. > llvm-link ideally would share the same library. > I'm not sure where it should live though. > > > What is the short-term plan? I don't think you can submit this > llvm-dlltool as-is because it has to much duplicate code with LLD. > > > > On Tue 14 Feb 2017 at 01:49, Rui Ueyama <ruiu at google.com> wrote: > > I took a look at the code. Looks like you need a library to create import > library files in LLVM and use that from llvm-dlltool and LLD. Is that what > you are planning? > > On Mon, Feb 13, 2017 at 5:37 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Ohh nice. > With that method I can support it without upsetting ld users by > introducing an api breakage. > > On Tue 14 Feb 2017 at 01:32, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk> > wrote: > > On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > No, I meant an even thinner wrapper which textually translates arguments. > For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o > foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything > with Config object nor LinkerDriver::run and have absolutely zero knowledge > on the internals of LLD. > > Ohh okay I misunderstood. > > (Why I'm asking this is because I want to sure that there's no difference > between the regular Windows linker and mingw. If all differences are > superficial, the command line translator should just work. If not, there's > some semantic difference.) > > The libraries can be used with MSVC link.exe directly assuming the correct > arguments are passed to the linker. > I tested this after creating a working llvm-dlltool. > The only change I had to make to support this was alter ming-w64 crt to > change all references from __image_base__ to _ImageBase to support that > > > You may be able to define __ImageBase as a weak external symbol to > __image_base__. Then, if __ImageBase is not defined, all references against > __ImageBase will be resolved using __image_base__. > > > Or you could have your wrapper driver pass > "/alternatename:__image_base__=__ImageBase". > > > Ah, that's much better indeed. > > > Peter > > > > > That should be fine. > > Great > > On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > I wonder if it can be a wrapper for LLD/COFF. It can be an in-process > wrapper, meaning that you could add a main function for mingw which > translates all arguments into the MSVC style and then invoke the COFF > linker's main function. > > That should work, if I am understanding this correctly I can create an > argument parser (probably partially based on the ELF one) then I can set > the data in the COFF Config object and call COFF LinkerDriver::run etc > directly from that? > > > No, I meant an even thinner wrapper which textually translates arguments. > For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o > foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything > with Config object nor LinkerDriver::run and have absolutely zero knowledge > on the internals of LLD. > > (Why I'm asking this is because I want to sure that there's no difference > between the regular Windows linker and mingw. If all differences are > superficial, the command line translator should just work. If not, there's > some semantic difference.) > > This proposal is for generally moving code from lld into llvm that could > be part of lib.exe / dlltool, what are your thoughts here? > > > That should be fine. > > > On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Also you need to make a change to LLD/COFF to accept GNU command > arguments, right? (Looks like you already have that patch locally.) > > Yes > > > I wonder if it can be a wrapper for LLD/COFF. It can be an in-process > wrapper, meaning that you could add a main function for mingw which > translates all arguments into the MSVC style and then invoke the COFF > linker's main function. > > > > My patch to hack lld into accepting some very basic gnu front end > arguments was enough to get all the above working which was enough to > develop further. > > > > On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote: > > On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at gmail.com> > wrote: > > Hey Rui, > > I wonder how llvm-dlltool would fit in the entire picture of mingw > support. I don't think dlltool is the last missing piece. What do you need > to do other than that to fully support mingw using LLVM toolchain? > > Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm > within the EmitCommonSymbol function and a single patch for mingw-w64 > itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is > infact the only missing part really. > > > Also you need to make a change to LLD/COFF to accept GNU command > arguments, right? (Looks like you already have that patch locally.) > > > This gives us a fully working clang based mingw-w64 C compiler. > > C++ and exception handling is a different story. > > libc++ is somewhat working with the following test results > > Expected Passes : 2188 > Expected Failures : 44 > Unsupported Tests : 588 > Unexpected Failures: 2816 > > I was able to build it with exceptions disabled and I actually managed to > bootstrap llvm and clang itself. > It didn't run very well though as you can imagine based on the above tests. > > It's not a performance issue but a code maintenance issue. The initial > patches to support mingw was trying to add a new linker driver and a > different linkin semantics to the COFF linker which seemed too complicated > to me. IIRC, I suggested adding a shim which translates GNU command line > arguments to MSVC linker arguments. Didn't it work? > > There were just so many differences between link and ld I never followed > down that path. I pushed forward with the short import library support > based on your later suggestions. My patch to hack lld into accepting some > very basic gnu front end arguments was enough to get all the above working > which was enough to develop further. > > Best, > Martell > > On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote: > > Hi Martell, > > I wonder how llvm-dlltool would fit in the entire picture of mingw > support. I don't think dlltool is the last missing piece. What do you need > to do other than that to fully support mingw using LLVM toolchain? > > On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Hey llvm'ers, > > I have been working on a dlltool replacement for llvm. > Here is my initial differential https://reviews.llvm.org/D29892 > It is based on some functionality that already exists in lld. > I added functionality to support, PE COFF Weak Externals and of course a > front end to actually use it. > I believe the work here can also be used for llvm-lib and lessen the load > on lld. > I would like some comments about how this could be be structured to live > in llvm with a shared code base across lib ar and dlltool. > I also have a section below called "Difference from lib" which is somewhat > of a rationale for the tool. > > Many Thanks, > Martell > > > Context > =========> > Awhile back I talked to various llvm'ers about getting mingw-w64 support > for lld. > There were a few issues raised but the main issue was that mingw-w64 > should try best to comply with the PECOFF spec, adding support for custom > sections and various binutils/mingw hacks would impact the performance of > the COFF linker and in general is not something that lld should support. > > > It's not a performance issue but a code maintenance issue. The initial > patches to support mingw was trying to add a new linker driver and a > different linkin semantics to the COFF linker which seemed too complicated > to me. IIRC, I suggested adding a shim which translates GNU command line > arguments to MSVC linker arguments. Didn't it work? > > > Motivation > =========> > The main motivation was because dlltool and ld did not comply with PECOFF > Spec. > It has some custom formatting and uses the assembler for some reason to > generate import libraries, it did not use the short import library format > at all. > > There has been many work arounds for the problem this creates such as the > creation of the `reimp` tool. Which imports MSVC built libs creates a def > and uses dlltool so that the binutils linker can use it. > > We should just be using the short import format and the linker should > support that. > Thus llvm-dlltool was born. > > > Difference from lib > ==================> > Using PE COFF spec (section 8, Import Library Format) should be self > explanatory. > lib.exe is able to accept def files and create libraries using this format. > > example > > `LIBRARY "user32.dll" > EXPORTS > MessageBoxA` > > LIB.exe can create a user32.lib with the function MessageBoxA from the > above definition. > > Mingw-w64 is different MSVC in that we need to compile the runtime. > MS provide us with their crt prebuilt so lib.exe doesn't have support for > external function aliasing. > We often use aliases for posix naming reasons as well as avoid using the > MS version of a function. > > example > > `LIBRARY "user32.dll" > EXPORTS > MessageBoxA > MessageBoxW==MessageBoxA` > > LIB.exe doesn't not have support for this but dlltool does. > The best fit for this within the spec is PE COFF spec (Aux Format 3: Weak > Externals) > > Mingw-w64 also uses lib prefix and .a file extensions for the library name > so the driver should be different. The above example would be libuser32.a, > for when shared and static versions of the same library exist there is the > .dll.a variant. > > > Issues > ======> > Like more of the gnu suite dlltool was not designed in one build multi > target manner. > We would have to introduce custom arguments to specify the target, "i686", > "x86_64", "thumb2pe" etc. > This will most likely break some level of backwards compatibility with the > binutils version. > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > -- > -- > Peter > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170214/ee105b64/attachment-0001.html>
Rui Ueyama via llvm-dev
2017-Feb-14 02:19 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
The code in LLD is somewhat tightly coupled with the rest of the linker. Probably it is better to create a new library rather than trying to split LLD into two. I believe this would result in a better API. Of course you can reuse code, but what we need is a clean interface. The library shouldn't be complicated. In fact, it would export only one function. std::vector<uint8_t> createImportLibrary(std::vector<ExportSymbol>) I.e. it takes a list of DLLExported symbols and returns a .lib file. ExportSymbol type contains information about DLLExported symbols. On Mon, Feb 13, 2017 at 6:06 PM, Martell Malone <martellmalone at gmail.com> wrote:> Well that means it would just be the only plan then. > I assume the first step would be to move the code from lld into this new > library and lld can use that? I can then re add the extra functionality for > weak externals? > We can then have lib and dlltool use this then. > > On Tue 14 Feb 2017 at 01:58, Rui Ueyama <ruiu at google.com> wrote: > >> On Mon, Feb 13, 2017 at 5:56 PM, Martell Malone <martellmalone at gmail.com> >> wrote: >> >> Yes, that would be the long term plan. >> llvm-link ideally would share the same library. >> I'm not sure where it should live though. >> >> >> What is the short-term plan? I don't think you can submit this >> llvm-dlltool as-is because it has to much duplicate code with LLD. >> >> >> >> On Tue 14 Feb 2017 at 01:49, Rui Ueyama <ruiu at google.com> wrote: >> >> I took a look at the code. Looks like you need a library to create import >> library files in LLVM and use that from llvm-dlltool and LLD. Is that what >> you are planning? >> >> On Mon, Feb 13, 2017 at 5:37 PM, Martell Malone <martellmalone at gmail.com> >> wrote: >> >> Ohh nice. >> With that method I can support it without upsetting ld users by >> introducing an api breakage. >> >> On Tue 14 Feb 2017 at 01:32, Rui Ueyama <ruiu at google.com> wrote: >> >> On Mon, Feb 13, 2017 at 5:26 PM, Peter Collingbourne <peter at pcc.me.uk> >> wrote: >> >> On Mon, Feb 13, 2017 at 5:20 PM, Rui Ueyama via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> On Mon, Feb 13, 2017 at 4:46 PM, Martell Malone <martellmalone at gmail.com> >> wrote: >> >> No, I meant an even thinner wrapper which textually translates arguments. >> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o >> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything >> with Config object nor LinkerDriver::run and have absolutely zero knowledge >> on the internals of LLD. >> >> Ohh okay I misunderstood. >> >> (Why I'm asking this is because I want to sure that there's no difference >> between the regular Windows linker and mingw. If all differences are >> superficial, the command line translator should just work. If not, there's >> some semantic difference.) >> >> The libraries can be used with MSVC link.exe directly assuming the >> correct arguments are passed to the linker. >> I tested this after creating a working llvm-dlltool. >> The only change I had to make to support this was alter ming-w64 crt to >> change all references from __image_base__ to _ImageBase to support that >> >> >> You may be able to define __ImageBase as a weak external symbol to >> __image_base__. Then, if __ImageBase is not defined, all references against >> __ImageBase will be resolved using __image_base__. >> >> >> Or you could have your wrapper driver pass "/alternatename:__image_base__ >> =__ImageBase". >> >> >> Ah, that's much better indeed. >> >> >> Peter >> >> >> >> >> That should be fine. >> >> Great >> >> On Mon, Feb 13, 2017 at 9:34 PM, Rui Ueyama <ruiu at google.com> wrote: >> >> On Mon, Feb 13, 2017 at 1:16 PM, Martell Malone <martellmalone at gmail.com> >> wrote: >> >> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >> wrapper, meaning that you could add a main function for mingw which >> translates all arguments into the MSVC style and then invoke the COFF >> linker's main function. >> >> That should work, if I am understanding this correctly I can create an >> argument parser (probably partially based on the ELF one) then I can set >> the data in the COFF Config object and call COFF LinkerDriver::run etc >> directly from that? >> >> >> No, I meant an even thinner wrapper which textually translates arguments. >> For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o >> foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything >> with Config object nor LinkerDriver::run and have absolutely zero knowledge >> on the internals of LLD. >> >> (Why I'm asking this is because I want to sure that there's no difference >> between the regular Windows linker and mingw. If all differences are >> superficial, the command line translator should just work. If not, there's >> some semantic difference.) >> >> This proposal is for generally moving code from lld into llvm that could >> be part of lib.exe / dlltool, what are your thoughts here? >> >> >> That should be fine. >> >> >> On Mon, Feb 13, 2017 at 8:57 PM, Rui Ueyama <ruiu at google.com> wrote: >> >> On Mon, Feb 13, 2017 at 12:52 PM, Martell Malone <martellmalone at gmail.com >> > wrote: >> >> Also you need to make a change to LLD/COFF to accept GNU command >> arguments, right? (Looks like you already have that patch locally.) >> >> Yes >> >> >> I wonder if it can be a wrapper for LLD/COFF. It can be an in-process >> wrapper, meaning that you could add a main function for mingw which >> translates all arguments into the MSVC style and then invoke the COFF >> linker's main function. >> >> >> >> My patch to hack lld into accepting some very basic gnu front end >> arguments was enough to get all the above working which was enough to >> develop further. >> >> >> >> On Mon, Feb 13, 2017 at 8:41 PM, Rui Ueyama <ruiu at google.com> wrote: >> >> On Mon, Feb 13, 2017 at 12:33 PM, Martell Malone <martellmalone at gmail.com >> > wrote: >> >> Hey Rui, >> >> I wonder how llvm-dlltool would fit in the entire picture of mingw >> support. I don't think dlltool is the last missing piece. What do you need >> to do other than that to fully support mingw using LLVM toolchain? >> >> Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm >> within the EmitCommonSymbol function and a single patch for mingw-w64 >> itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is >> infact the only missing part really. >> >> >> Also you need to make a change to LLD/COFF to accept GNU command >> arguments, right? (Looks like you already have that patch locally.) >> >> >> This gives us a fully working clang based mingw-w64 C compiler. >> >> C++ and exception handling is a different story. >> >> libc++ is somewhat working with the following test results >> >> Expected Passes : 2188 >> Expected Failures : 44 >> Unsupported Tests : 588 >> Unexpected Failures: 2816 >> >> I was able to build it with exceptions disabled and I actually managed to >> bootstrap llvm and clang itself. >> It didn't run very well though as you can imagine based on the above >> tests. >> >> It's not a performance issue but a code maintenance issue. The initial >> patches to support mingw was trying to add a new linker driver and a >> different linkin semantics to the COFF linker which seemed too complicated >> to me. IIRC, I suggested adding a shim which translates GNU command line >> arguments to MSVC linker arguments. Didn't it work? >> >> There were just so many differences between link and ld I never followed >> down that path. I pushed forward with the short import library support >> based on your later suggestions. My patch to hack lld into accepting some >> very basic gnu front end arguments was enough to get all the above working >> which was enough to develop further. >> >> Best, >> Martell >> >> On Mon, Feb 13, 2017 at 8:08 PM, Rui Ueyama <ruiu at google.com> wrote: >> >> Hi Martell, >> >> I wonder how llvm-dlltool would fit in the entire picture of mingw >> support. I don't think dlltool is the last missing piece. What do you need >> to do other than that to fully support mingw using LLVM toolchain? >> >> On Mon, Feb 13, 2017 at 8:56 AM, Martell Malone via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> Hey llvm'ers, >> >> I have been working on a dlltool replacement for llvm. >> Here is my initial differential https://reviews.llvm.org/D29892 >> It is based on some functionality that already exists in lld. >> I added functionality to support, PE COFF Weak Externals and of course a >> front end to actually use it. >> I believe the work here can also be used for llvm-lib and lessen the load >> on lld. >> I would like some comments about how this could be be structured to live >> in llvm with a shared code base across lib ar and dlltool. >> I also have a section below called "Difference from lib" which is >> somewhat of a rationale for the tool. >> >> Many Thanks, >> Martell >> >> >> Context >> =========>> >> Awhile back I talked to various llvm'ers about getting mingw-w64 support >> for lld. >> There were a few issues raised but the main issue was that mingw-w64 >> should try best to comply with the PECOFF spec, adding support for custom >> sections and various binutils/mingw hacks would impact the performance of >> the COFF linker and in general is not something that lld should support. >> >> >> It's not a performance issue but a code maintenance issue. The initial >> patches to support mingw was trying to add a new linker driver and a >> different linkin semantics to the COFF linker which seemed too complicated >> to me. IIRC, I suggested adding a shim which translates GNU command line >> arguments to MSVC linker arguments. Didn't it work? >> >> >> Motivation >> =========>> >> The main motivation was because dlltool and ld did not comply with PECOFF >> Spec. >> It has some custom formatting and uses the assembler for some reason to >> generate import libraries, it did not use the short import library format >> at all. >> >> There has been many work arounds for the problem this creates such as the >> creation of the `reimp` tool. Which imports MSVC built libs creates a def >> and uses dlltool so that the binutils linker can use it. >> >> We should just be using the short import format and the linker should >> support that. >> Thus llvm-dlltool was born. >> >> >> Difference from lib >> ==================>> >> Using PE COFF spec (section 8, Import Library Format) should be self >> explanatory. >> lib.exe is able to accept def files and create libraries using this >> format. >> >> example >> >> `LIBRARY "user32.dll" >> EXPORTS >> MessageBoxA` >> >> LIB.exe can create a user32.lib with the function MessageBoxA from the >> above definition. >> >> Mingw-w64 is different MSVC in that we need to compile the runtime. >> MS provide us with their crt prebuilt so lib.exe doesn't have support for >> external function aliasing. >> We often use aliases for posix naming reasons as well as avoid using the >> MS version of a function. >> >> example >> >> `LIBRARY "user32.dll" >> EXPORTS >> MessageBoxA >> MessageBoxW==MessageBoxA` >> >> LIB.exe doesn't not have support for this but dlltool does. >> The best fit for this within the spec is PE COFF spec (Aux Format 3: >> Weak Externals) >> >> Mingw-w64 also uses lib prefix and .a file extensions for the library >> name so the driver should be different. The above example would be >> libuser32.a, for when shared and static versions of the same library exist >> there is the .dll.a variant. >> >> >> Issues >> ======>> >> Like more of the gnu suite dlltool was not designed in one build multi >> target manner. >> We would have to introduce custom arguments to specify the target, >> "i686", "x86_64", "thumb2pe" etc. >> This will most likely break some level of backwards compatibility with >> the binutils version. >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> >> >> -- >> -- >> Peter >> >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/5d877e21/attachment-0001.html>
Rui Ueyama via llvm-dev
2017-Feb-14 02:21 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
The code in LLD is somewhat tightly coupled with the rest of the linker. Probably it is better to create a new library rather than trying to split LLD into two. I believe this would result in a better API. Of course you can reuse code, but what we need is a clean interface. The library shouldn't be complicated. In fact, it would export only one function. std::vector<uint8_t> createImportLibrary(std::vector<ExportSymbol>) I.e. it takes a list of DLLExported symbols and returns a .lib file. ExportSymbol type contains information about DLLExported symbols. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170213/5aa5ea78/attachment.html>
Martell Malone via llvm-dev
2017-Mar-27 18:00 UTC
[llvm-dev] RFC: A new llvm-dlltool driver and llvm-lib driver improvements
Just to follow up here I found some time to progress. Anyone who wants to follow along can see here. https://reviews.llvm.org/D29892 Kind Regards Martell On Tue, Feb 14, 2017 at 2:21 AM, Rui Ueyama <ruiu at google.com> wrote:> The code in LLD is somewhat tightly coupled with the rest of the linker. > Probably it is better to create a new library rather than trying to split > LLD into two. I believe this would result in a better API. Of course you > can reuse code, but what we need is a clean interface. > > The library shouldn't be complicated. In fact, it would export only one > function. > > std::vector<uint8_t> createImportLibrary(std::vector<ExportSymbol>) > > I.e. it takes a list of DLLExported symbols and returns a .lib file. > ExportSymbol type contains information about DLLExported symbols. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170327/bb04769d/attachment.html>