llvm dev - Oct 2017 - Make LLD output COFF relocatable object file (like ELF's -r does). How much work is required to implement this?

Hi,

How far are we from having '-r' in the LLD COFF linker?
I'd try to implement this if not too much effort is required.
Any suggestions and/or pointers?

Cheers,
Kyra

As far as I know, no one has ever tried to add the -r option to the lld
COFF linker. It shouldn't be super hard to add it to the COFF linker, but
from our experience of implementing it to lld ELF linker, I can say that it
was tricky and somewhat fragile. We had to add a number of small pieces of
code here and there.

We wanted to support it in the ELF linker because that's an existing
feature and people are actually using it. Otherwise, we wouldn't have added
it. So, what is the motivation of adding the feature to the COFF linker? I
don't think MSVC linker supports it.

(For those who are not familiar with -r,  the option is to make the linker
emit a .o file instead of an executable or a shared library. With the
option, you can combine multiple object files into one object file.)

On Mon, Oct 9, 2017 at 8:02 AM, kyra via llvm-dev <llvm-dev at
lists.llvm.org>
wrote:
> Hi,
>
> How far are we from having '-r' in the LLD COFF linker?
> I'd try to implement this if not too much effort is required.
> Any suggestions and/or pointers?
>
> Cheers,
> Kyra
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171009/0b07fdb6/attachment.html>

TL;DR:
I'm trying to evaluate if LLD can be used with GHC (Glasgow Haskell 
Compiler) on Windows.

Haskell binary code is usually deployed in "packages". A package 
typically provides static library(ies) and optionally – shared 
library(ies) and/or prelinked ('ld -r') object file. The latter is the 
best way to satisfy GHC runtime linker, since it requires no separate 
compile/link pass (as shared library requires), and is much faster to 
consume by GHC runtime linker than a static library.

Long story:

To prevent linking unused code GHC have always been supported splitting 
intermediate assembly which is horribly slow when compiling. Now GHC 
supports a direct analogue of '-ffunction-sections'
('-split-sections'
in GHC parlance), which dramatically improves compile times, but now BFD 
linker is horribly slow on the files with a *lot* of sections. In the 
*nix world they have gold linker, in the windows world we have nothing 
other than GNU BFD ld ATM.

GHC on Windows uses Mingw tools and LLD doesn't fit into Mingw ecosystem 
yet (I know that some support have creeped into LLD recently, but it is 
still far from being complete), moreover, when assemling GHC native 
codegen output, GNU assembler produces peculiar non-standard COFF files 
(with 0x11 relocations), and finally binutils doesn't (and probably 
never would) support bigobj extension in the 32-bit case.

Windows GHC relies heavily on GCC, especially it's runtime system's code
is full of gnu-isms, but Clang has a unique ability to combine gnu-ish 
frontend with ms-ish backend, I've experimented a bit and have concluded 
that replacing GCC as a C compiler/system assembler with Clang in GHC on 
Windows is very much doable.

GHC uses object file combining ('ld -r') when C stubs/wrappers 
generation is triggered, these stubs/wrappers are compiled with gcc and 
are linked back into the 'main' object file. In the MS world this use 
case can easily be satisfied by packing the object files into a library 
since MS linker looks into libraries both when linking final exe/dll 
*and/or* creating another library (i.e. when creating another library it 
unpacks all object files from all libraries it is fed with, and repacks 
them into the output library, llvm-lib doesn't support this ATM, and 
AFAIR LLVM developers are aware of this).

But my question is motivated by another important use-case: when 
packaging compiled Haskell code it is very desirable to provide not only 
a static library, but also to partially link this library's object 
modules into the one big object file, which can further be consumed by 
GHC runtime linker. GHC runtime linker can link binary code in any form, 
but linking static library is much slower than linking the single object 
file.

On 10/10/2017 2:39 AM, Rui Ueyama wrote:
> As far as I know, no one has ever tried to add the -r option to the 
> lld COFF linker. It shouldn't be super hard to add it to the COFF 
> linker, but from our experience of implementing it to lld ELF linker, 
> I can say that it was tricky and somewhat fragile. We had to add a 
> number of small pieces of code here and there.
>
> We wanted to support it in the ELF linker because that's an existing 
> feature and people are actually using it. Otherwise, we wouldn't have 
> added it. So, what is the motivation of adding the feature to the COFF 
> linker? I don't think MSVC linker supports it.
>
> (For those who are not familiar with -r,  the option is to make the 
> linker emit a .o file instead of an executable or a shared library. 
> With the option, you can combine multiple object files into one object 
> file.)
>
> On Mon, Oct 9, 2017 at 8:02 AM, kyra via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     Hi,
>
>     How far are we from having '-r' in the LLD COFF linker?
>     I'd try to implement this if not too much effort is required.
>     Any suggestions and/or pointers?
>
>     Cheers,
>     Kyra
>
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>