llvm dev - Jan 2016 - lld: ELF/COFF main() interface

In the process of migrating from old lld ELF linker to new (previously
ELF2) I noticed the interface lost several important features (ordered by
importance for my use case):

1. Detecting errors in the first place. New linker seems to call exit(1)
for any error.

2. Reporting messages to non-stderr outputs. Previously all link functions
had a raw_ostream argument so it was possible to delay the error output,
aggregate it for multiple linked files, output via a different format, etc.

3. Linking multiple outputs in parallel (useful for test drivers) in a
single process. Not really an interface issue but there are at least two
global pointers (Config & Driver) that refer to stack variables and are
used in various places in the code.

All of this seems to indicate a departure from the linker being useable as
a library. To maintain the previous behavior you'd have to use a linker
binary & popen.

Is this a conscious design decision or a temporary limitation?

If it's a limitation, how would one go about fixing this? I'm not too
familiar with the idiomatical error handling in LLVM. Normally in this
situation I'd just throw from error() but lld is probably compiled without
exceptions. Is ErrorOr the established practice? How does LLVM generally
deal with error handling for parsers (like linker script parser) where it's
a lot of mutually recursive functions where every single one can fail?

Arseny
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/09a9ef49/attachment.html>

On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> In the process of migrating from old lld ELF linker to new (previously
> ELF2) I noticed the interface lost several important features (ordered by
> importance for my use case):
>
> 1. Detecting errors in the first place. New linker seems to call exit(1)
> for any error.
>
> 2. Reporting messages to non-stderr outputs. Previously all link functions
> had a raw_ostream argument so it was possible to delay the error output,
> aggregate it for multiple linked files, output via a different format, etc.
>
> 3. Linking multiple outputs in parallel (useful for test drivers) in a
> single process. Not really an interface issue but there are at least two
> global pointers (Config & Driver) that refer to stack variables and are
> used in various places in the code.
>
> All of this seems to indicate a departure from the linker being useable as
> a library. To maintain the previous behavior you'd have to use a linker
> binary & popen.
>
> Is this a conscious design decision or a temporary limitation?
>
That the new ELF and COFF linkers are designed as commands instead of
libraries is very much an intended design change.

> If it's a limitation, how would one go about fixing this? I'm not
too
> familiar with the idiomatical error handling in LLVM. Normally in this
> situation I'd just throw from error() but lld is probably compiled
without
> exceptions. Is ErrorOr the established practice? How does LLVM generally
> deal with error handling for parsers (like linker script parser) where
it's
> a lot of mutually recursive functions where every single one can fail?
>
Since it's as designed, please run the linker as an external command using
fork/exec (or wrappers of them) instead of trying to use that inside the
same process.

> Arseny
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/aaefe17a/attachment.html>

This is really unfortunate.

I've read the discussion threads for the atom/chunk controversy and I feel
like I understand the reasons for rewriting the linker, but this does not
seem to have anything to do with whether the linker is usable as a library
or not.

As it stands, not only does lld have two completely different linkers (I'm
treating COFF/ELF2 as one since they are really two different
implementations of the same concept, AFAIU), but one is usable as a library
(and even does not require round-tripping generated code through an object
file! I was really happy to use that) and the other one isn't. Not sure
what the future plans are for Mach-O linker (at this point it seems logical
to rewrite that using the new designs but I'm not sure if it ever happens),
so maybe at some point we'll just have one linker application instead of a
library and an application.

Anyway, since linker is the only missing piece for full compilation stack
(source language to runnable executable), it's sad to see this specific
part of LLVM not working as a library when everything else does.

Are there specific concerns in terms of implementation that prevent new lld
from being a library? I understand that using global variables and error()
function is simpler, but the rest of LLVM does not do that and the codebase
there is significantly larger. Am I missing any other issues except for the
ones I mentioned in my original e-mail that will come up in a library-like
usage scenario?

Arseny

On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote:
> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> In the process of migrating from old lld ELF linker to new (previously
>> ELF2) I noticed the interface lost several important features (ordered
by
>> importance for my use case):
>>
>> 1. Detecting errors in the first place. New linker seems to call
exit(1)
>> for any error.
>>
>> 2. Reporting messages to non-stderr outputs. Previously all link
>> functions had a raw_ostream argument so it was possible to delay the
error
>> output, aggregate it for multiple linked files, output via a different
>> format, etc.
>>
>> 3. Linking multiple outputs in parallel (useful for test drivers) in a
>> single process. Not really an interface issue but there are at least
two
>> global pointers (Config & Driver) that refer to stack variables and
are
>> used in various places in the code.
>>
>> All of this seems to indicate a departure from the linker being useable
>> as a library. To maintain the previous behavior you'd have to use a
linker
>> binary & popen.
>>
>> Is this a conscious design decision or a temporary limitation?
>>
>
> That the new ELF and COFF linkers are designed as commands instead of
> libraries is very much an intended design change.
>
>
>> If it's a limitation, how would one go about fixing this? I'm
not too
>> familiar with the idiomatical error handling in LLVM. Normally in this
>> situation I'd just throw from error() but lld is probably compiled
without
>> exceptions. Is ErrorOr the established practice? How does LLVM
generally
>> deal with error handling for parsers (like linker script parser) where
it's
>> a lot of mutually recursive functions where every single one can fail?
>>
>
> Since it's as designed, please run the linker as an external command
using
> fork/exec (or wrappers of them) instead of trying to use that inside the
> same process.
>
>
>> Arseny
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/c28d3f18/attachment.html>

On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> In the process of migrating from old lld ELF linker to new (previously
>> ELF2) I noticed the interface lost several important features (ordered
by
>> importance for my use case):
>>
>> 1. Detecting errors in the first place. New linker seems to call
exit(1)
>> for any error.
>>
>> 2. Reporting messages to non-stderr outputs. Previously all link
>> functions had a raw_ostream argument so it was possible to delay the
error
>> output, aggregate it for multiple linked files, output via a different
>> format, etc.
>>
>> 3. Linking multiple outputs in parallel (useful for test drivers) in a
>> single process. Not really an interface issue but there are at least
two
>> global pointers (Config & Driver) that refer to stack variables and
are
>> used in various places in the code.
>>
>> All of this seems to indicate a departure from the linker being useable
>> as a library. To maintain the previous behavior you'd have to use a
linker
>> binary & popen.
>>
>> Is this a conscious design decision or a temporary limitation?
>>
>
> That the new ELF and COFF linkers are designed as commands instead of
> libraries is very much an intended design change.
>
I disagree.

During the discussion, there was a *specific* discussion of both the new
COFF port and ELF port continuing to be libraries with a common command
line driver.

If you want to consider changing that, we should have a fresh (and broad)
discussion, but it goes pretty firmly against the design of the entire LLVM
project. I also don't really understand why it would be beneficial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/65168e5c/attachment.html>