llvm dev - Jan 2016 - lld: ELF/COFF main() interface

On Thu, Jan 7, 2016 at 3:18 PM Rui Ueyama <ruiu at google.com> wrote:
> On Thu, Jan 7, 2016 at 2:56 PM, Chandler Carruth <chandlerc at
gmail.com>
> wrote:
>
>> On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> In the process of migrating from old lld ELF linker to new
(previously
>>>> ELF2) I noticed the interface lost several important features
(ordered by
>>>> importance for my use case):
>>>>
>>>> 1. Detecting errors in the first place. New linker seems to
call
>>>> exit(1) for any error.
>>>>
>>>> 2. Reporting messages to non-stderr outputs. Previously all
link
>>>> functions had a raw_ostream argument so it was possible to
delay the error
>>>> output, aggregate it for multiple linked files, output via a
different
>>>> format, etc.
>>>>
>>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
>>>> single process. Not really an interface issue but there are at
least two
>>>> global pointers (Config & Driver) that refer to stack
variables and are
>>>> used in various places in the code.
>>>>
>>>> All of this seems to indicate a departure from the linker being
useable
>>>> as a library. To maintain the previous behavior you'd have
to use a linker
>>>> binary & popen.
>>>>
>>>> Is this a conscious design decision or a temporary limitation?
>>>>
>>>
>>> That the new ELF and COFF linkers are designed as commands instead
of
>>> libraries is very much an intended design change.
>>>
>>
>> I disagree.
>>
>> During the discussion, there was a *specific* discussion of both the
new
>> COFF port and ELF port continuing to be libraries with a common command
>> line driver.
>>
>
> There was a discussion that we would keep the same entry point for the old
> and the new, but I don't remember if I promised that we were going to
> organize the new linker as a library.
>
Ok, myself and essentially everyone else thought this was clear. If it
isn't lets clarify:

I think it is absolutely critical and important that LLD's architecture
remain one where all functionality is available as a library. This is *the*
design goal of LLVM and all of LLVM's infrastructure. This applies just as
much to LLD as it does to Clang.

You say that it isn't compelling to match Clang's design, but in fact it
is. You would need an overwhelming argument to *diverge* from Clang's
design.

The fact that it makes the design more challenging is not compelling at
all. Yes, building libraries that can be re-used and making the binary
calling it equally efficient is more challenging, but that is the express
mission of LLVM and every project within it.

> The new one is designed as a command from day one. (Precisely speaking,
> the original code propagates errors all the way up to the entry point, so
> you can call it and expect it to always return. Rafael introduced error()
> function later and we now depends on that function does not return.)
>
I think this last was a mistake.

The fact that the code propagates errors all the way up is fine, and even
good. We don't necessarily need to be able to *recover* from link errors
and try some other path.

But we absolutely need the design to be a *library* that can be embedded
into other programs and tools. I can't even begin to count the use cases
for this.

So please, let's go back to where we *do not* rely on never-returning error
handling. That is an absolute mistake.

>
> If you want to consider changing that, we should have a fresh (and broad)
>> discussion, but it goes pretty firmly against the design of the entire
LLVM
>> project. I also don't really understand why it would be beneficial.
>>
>
> I'm not against organizing it as a library as long as it does not make
> things too complicated
>
I am certain that it will make things more complicated, but that is the
technical challenge that we must overcome. It will be hard, but I am
absolutely confident it is possible to have an elegant library design here.
It may not be as simple as a pure command line tool, but it will be
*dramatically* more powerful, general, and broadly applicable.

The design of LLVM is not the simplest way to build a compiler. But it is
valuable to all of those working on it precisely because of this
flexibility imparted by its library oriented design. This is absolutely not
something that we should lose from the linker.

> , and I guess reorganizing the existing code as a library is relatively
> easy because it's still pretty small, but I don't really want to
focus on
> that until it becomes usable as an alternative to GNU ld or gold. I want to
> focus on the linker features themselves at this moment. Once it's
complete,
> it becomes more clear how to organize it.
>
Ok, now we're talking about something totally reasonable.

If it is easier for you all to develop this first as a command line tool,
and then make it work as a library, sure, go for it. You're doing the work,
I can hardly tell you how to go about it. ;]

However, I think it is super important to be clear that getting the library
architecture is a *hard requirement* for the LLD project. Without that, it
doesn't even make sense as part of LLVM.

And as a consequence, I think it is unacceptable to replace the old ELF
port with the new one until this is true. That is removing functionality
that users of LLD realistically were depending on, which you're seeing in
this thread. That's not cool. We don't really tolerate dramatic
regressions
in functionality like this, and even if we've already done it, we should
revert back to a state where both are available until the new port is
actually ready. And ready in LLVM land means, functional as a library.

-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/a1ad8f50/attachment.html>

Hi All,
> Not sure what the future plans are for Mach-O linker (at this point it
seems logical to rewrite that using the new designs but I'm not sure if it
ever happens), so maybe at some point we'll just have one linker
application instead of a library and an application.

We plan to continue with the existing atom-based linker for MachO. This is
an area of ongoing active development (notwithstanding me being on holidays
for the last two months).

Seconding Chandler's (and others) comments: I'd really like to see LLD
retain a library architecture, and I can guarantee we'll be aiming for this
on the MachO side of the codebase.
> Rewriting every function as ErrorOr sounds terrible and we should avoid
that as much as possible...
>From my recent experience the problem is that if you handle errors lazily
they can come up very deep in the stack, and you're stuck propagating them
out one way or another. ErrorOr feels really heavy, but I'm coming around
to the idea that it's the least-worst solution.

Cheers,
Lang.

On Thu, Jan 7, 2016 at 3:44 PM, Chandler Carruth via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Thu, Jan 7, 2016 at 3:18 PM Rui Ueyama <ruiu at google.com> wrote:
>
>> On Thu, Jan 7, 2016 at 2:56 PM, Chandler Carruth <chandlerc at
gmail.com>
>> wrote:
>>
>>> On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> In the process of migrating from old lld ELF linker to new
(previously
>>>>> ELF2) I noticed the interface lost several important
features (ordered by
>>>>> importance for my use case):
>>>>>
>>>>> 1. Detecting errors in the first place. New linker seems to
call
>>>>> exit(1) for any error.
>>>>>
>>>>> 2. Reporting messages to non-stderr outputs. Previously all
link
>>>>> functions had a raw_ostream argument so it was possible to
delay the error
>>>>> output, aggregate it for multiple linked files, output via
a different
>>>>> format, etc.
>>>>>
>>>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
>>>>> single process. Not really an interface issue but there are
at least two
>>>>> global pointers (Config & Driver) that refer to stack
variables and are
>>>>> used in various places in the code.
>>>>>
>>>>> All of this seems to indicate a departure from the linker
being
>>>>> useable as a library. To maintain the previous behavior
you'd have to use a
>>>>> linker binary & popen.
>>>>>
>>>>> Is this a conscious design decision or a temporary
limitation?
>>>>>
>>>>
>>>> That the new ELF and COFF linkers are designed as commands
instead of
>>>> libraries is very much an intended design change.
>>>>
>>>
>>> I disagree.
>>>
>>> During the discussion, there was a *specific* discussion of both
the new
>>> COFF port and ELF port continuing to be libraries with a common
command
>>> line driver.
>>>
>>
>> There was a discussion that we would keep the same entry point for the
>> old and the new, but I don't remember if I promised that we were
going to
>> organize the new linker as a library.
>>
>
> Ok, myself and essentially everyone else thought this was clear. If it
> isn't lets clarify:
>
> I think it is absolutely critical and important that LLD's architecture
> remain one where all functionality is available as a library. This is *the*
> design goal of LLVM and all of LLVM's infrastructure. This applies just
as
> much to LLD as it does to Clang.
>
> You say that it isn't compelling to match Clang's design, but in
fact it
> is. You would need an overwhelming argument to *diverge* from Clang's
> design.
>
> The fact that it makes the design more challenging is not compelling at
> all. Yes, building libraries that can be re-used and making the binary
> calling it equally efficient is more challenging, but that is the express
> mission of LLVM and every project within it.
>
>
>> The new one is designed as a command from day one. (Precisely speaking,
>> the original code propagates errors all the way up to the entry point,
so
>> you can call it and expect it to always return. Rafael introduced
error()
>> function later and we now depends on that function does not return.)
>>
>
> I think this last was a mistake.
>
> The fact that the code propagates errors all the way up is fine, and even
> good. We don't necessarily need to be able to *recover* from link
errors
> and try some other path.
>
> But we absolutely need the design to be a *library* that can be embedded
> into other programs and tools. I can't even begin to count the use
cases
> for this.
>
> So please, let's go back to where we *do not* rely on never-returning
> error handling. That is an absolute mistake.
>
>
>>
>> If you want to consider changing that, we should have a fresh (and
broad)
>>> discussion, but it goes pretty firmly against the design of the
entire LLVM
>>> project. I also don't really understand why it would be
beneficial.
>>>
>>
>> I'm not against organizing it as a library as long as it does not
make
>> things too complicated
>>
>
> I am certain that it will make things more complicated, but that is the
> technical challenge that we must overcome. It will be hard, but I am
> absolutely confident it is possible to have an elegant library design here.
> It may not be as simple as a pure command line tool, but it will be
> *dramatically* more powerful, general, and broadly applicable.
>
> The design of LLVM is not the simplest way to build a compiler. But it is
> valuable to all of those working on it precisely because of this
> flexibility imparted by its library oriented design. This is absolutely not
> something that we should lose from the linker.
>
>
>> , and I guess reorganizing the existing code as a library is relatively
>> easy because it's still pretty small, but I don't really want
to focus on
>> that until it becomes usable as an alternative to GNU ld or gold. I
want to
>> focus on the linker features themselves at this moment. Once it's
complete,
>> it becomes more clear how to organize it.
>>
>
> Ok, now we're talking about something totally reasonable.
>
> If it is easier for you all to develop this first as a command line tool,
> and then make it work as a library, sure, go for it. You're doing the
work,
> I can hardly tell you how to go about it. ;]
>
> However, I think it is super important to be clear that getting the
> library architecture is a *hard requirement* for the LLD project. Without
> that, it doesn't even make sense as part of LLVM.
>
> And as a consequence, I think it is unacceptable to replace the old ELF
> port with the new one until this is true. That is removing functionality
> that users of LLD realistically were depending on, which you're seeing
in
> this thread. That's not cool. We don't really tolerate dramatic
regressions
> in functionality like this, and even if we've already done it, we
should
> revert back to a state where both are available until the new port is
> actually ready. And ready in LLVM land means, functional as a library.
>
> -Chandler
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/7a205226/attachment-0001.html>

> And as a consequence, I think it is unacceptable to replace the old ELF
port with the new one until this is true. That is removing functionality
that users of LLD realistically were depending on, which you're seeing in
this thread. That's not cool. We don't really tolerate dramatic
regressions
in functionality like this, and even if we've already done it, we should
revert back to a state where both are available until the new port is
actually ready. And ready in LLVM land means, functional as a library.

Just in case I want to clarify that right now I can still use the original
ELF linker as part of lld. I was looking at migrating to get faster linking
and since I got the impression old ELF linker is basically frozen, but I
can still use the old one in the mean time.

Thank you for the rest; happy to see other people share my opinion on
library vs application issue.

Arseny

On Thu, Jan 7, 2016 at 3:44 PM, Chandler Carruth <chandlerc at gmail.com>
wrote:
> On Thu, Jan 7, 2016 at 3:18 PM Rui Ueyama <ruiu at google.com> wrote:
>
>> On Thu, Jan 7, 2016 at 2:56 PM, Chandler Carruth <chandlerc at
gmail.com>
>> wrote:
>>
>>> On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> In the process of migrating from old lld ELF linker to new
(previously
>>>>> ELF2) I noticed the interface lost several important
features (ordered by
>>>>> importance for my use case):
>>>>>
>>>>> 1. Detecting errors in the first place. New linker seems to
call
>>>>> exit(1) for any error.
>>>>>
>>>>> 2. Reporting messages to non-stderr outputs. Previously all
link
>>>>> functions had a raw_ostream argument so it was possible to
delay the error
>>>>> output, aggregate it for multiple linked files, output via
a different
>>>>> format, etc.
>>>>>
>>>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
>>>>> single process. Not really an interface issue but there are
at least two
>>>>> global pointers (Config & Driver) that refer to stack
variables and are
>>>>> used in various places in the code.
>>>>>
>>>>> All of this seems to indicate a departure from the linker
being
>>>>> useable as a library. To maintain the previous behavior
you'd have to use a
>>>>> linker binary & popen.
>>>>>
>>>>> Is this a conscious design decision or a temporary
limitation?
>>>>>
>>>>
>>>> That the new ELF and COFF linkers are designed as commands
instead of
>>>> libraries is very much an intended design change.
>>>>
>>>
>>> I disagree.
>>>
>>> During the discussion, there was a *specific* discussion of both
the new
>>> COFF port and ELF port continuing to be libraries with a common
command
>>> line driver.
>>>
>>
>> There was a discussion that we would keep the same entry point for the
>> old and the new, but I don't remember if I promised that we were
going to
>> organize the new linker as a library.
>>
>
> Ok, myself and essentially everyone else thought this was clear. If it
> isn't lets clarify:
>
> I think it is absolutely critical and important that LLD's architecture
> remain one where all functionality is available as a library. This is *the*
> design goal of LLVM and all of LLVM's infrastructure. This applies just
as
> much to LLD as it does to Clang.
>
> You say that it isn't compelling to match Clang's design, but in
fact it
> is. You would need an overwhelming argument to *diverge* from Clang's
> design.
>
> The fact that it makes the design more challenging is not compelling at
> all. Yes, building libraries that can be re-used and making the binary
> calling it equally efficient is more challenging, but that is the express
> mission of LLVM and every project within it.
>
>
>> The new one is designed as a command from day one. (Precisely speaking,
>> the original code propagates errors all the way up to the entry point,
so
>> you can call it and expect it to always return. Rafael introduced
error()
>> function later and we now depends on that function does not return.)
>>
>
> I think this last was a mistake.
>
> The fact that the code propagates errors all the way up is fine, and even
> good. We don't necessarily need to be able to *recover* from link
errors
> and try some other path.
>
> But we absolutely need the design to be a *library* that can be embedded
> into other programs and tools. I can't even begin to count the use
cases
> for this.
>
> So please, let's go back to where we *do not* rely on never-returning
> error handling. That is an absolute mistake.
>
>
>>
>> If you want to consider changing that, we should have a fresh (and
broad)
>>> discussion, but it goes pretty firmly against the design of the
entire LLVM
>>> project. I also don't really understand why it would be
beneficial.
>>>
>>
>> I'm not against organizing it as a library as long as it does not
make
>> things too complicated
>>
>
> I am certain that it will make things more complicated, but that is the
> technical challenge that we must overcome. It will be hard, but I am
> absolutely confident it is possible to have an elegant library design here.
> It may not be as simple as a pure command line tool, but it will be
> *dramatically* more powerful, general, and broadly applicable.
>
> The design of LLVM is not the simplest way to build a compiler. But it is
> valuable to all of those working on it precisely because of this
> flexibility imparted by its library oriented design. This is absolutely not
> something that we should lose from the linker.
>
>
>> , and I guess reorganizing the existing code as a library is relatively
>> easy because it's still pretty small, but I don't really want
to focus on
>> that until it becomes usable as an alternative to GNU ld or gold. I
want to
>> focus on the linker features themselves at this moment. Once it's
complete,
>> it becomes more clear how to organize it.
>>
>
> Ok, now we're talking about something totally reasonable.
>
> If it is easier for you all to develop this first as a command line tool,
> and then make it work as a library, sure, go for it. You're doing the
work,
> I can hardly tell you how to go about it. ;]
>
> However, I think it is super important to be clear that getting the
> library architecture is a *hard requirement* for the LLD project. Without
> that, it doesn't even make sense as part of LLVM.
>
> And as a consequence, I think it is unacceptable to replace the old ELF
> port with the new one until this is true. That is removing functionality
> that users of LLD realistically were depending on, which you're seeing
in
> this thread. That's not cool. We don't really tolerate dramatic
regressions
> in functionality like this, and even if we've already done it, we
should
> revert back to a state where both are available until the new port is
> actually ready. And ready in LLVM land means, functional as a library.
>
> -Chandler
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/01bf81d8/attachment.html>

By organizing it as a library, I'm expecting something coarse. I don't
expect to reorganize the linker itself as a collection of small libraries,
but make the entire linker available as a library, so that you can link
stuff in-process. More specifically, I expect that the library would
basically export one function, link(std::vector<StringRef>), which takes
command line arguments, and returns a memory buffer for a newly created
executable. We may want to allow a mix of StringRef and MemoryBuffer as
input, so that you can directly pass in-memory objects to the linker, but
the basic idea remains the same.

Are we on the same page?

On Thu, Jan 7, 2016 at 3:44 PM, Chandler Carruth <chandlerc at gmail.com>
wrote:
> On Thu, Jan 7, 2016 at 3:18 PM Rui Ueyama <ruiu at google.com> wrote:
>
>> On Thu, Jan 7, 2016 at 2:56 PM, Chandler Carruth <chandlerc at
gmail.com>
>> wrote:
>>
>>> On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> In the process of migrating from old lld ELF linker to new
(previously
>>>>> ELF2) I noticed the interface lost several important
features (ordered by
>>>>> importance for my use case):
>>>>>
>>>>> 1. Detecting errors in the first place. New linker seems to
call
>>>>> exit(1) for any error.
>>>>>
>>>>> 2. Reporting messages to non-stderr outputs. Previously all
link
>>>>> functions had a raw_ostream argument so it was possible to
delay the error
>>>>> output, aggregate it for multiple linked files, output via
a different
>>>>> format, etc.
>>>>>
>>>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
>>>>> single process. Not really an interface issue but there are
at least two
>>>>> global pointers (Config & Driver) that refer to stack
variables and are
>>>>> used in various places in the code.
>>>>>
>>>>> All of this seems to indicate a departure from the linker
being
>>>>> useable as a library. To maintain the previous behavior
you'd have to use a
>>>>> linker binary & popen.
>>>>>
>>>>> Is this a conscious design decision or a temporary
limitation?
>>>>>
>>>>
>>>> That the new ELF and COFF linkers are designed as commands
instead of
>>>> libraries is very much an intended design change.
>>>>
>>>
>>> I disagree.
>>>
>>> During the discussion, there was a *specific* discussion of both
the new
>>> COFF port and ELF port continuing to be libraries with a common
command
>>> line driver.
>>>
>>
>> There was a discussion that we would keep the same entry point for the
>> old and the new, but I don't remember if I promised that we were
going to
>> organize the new linker as a library.
>>
>
> Ok, myself and essentially everyone else thought this was clear. If it
> isn't lets clarify:
>
> I think it is absolutely critical and important that LLD's architecture
> remain one where all functionality is available as a library. This is *the*
> design goal of LLVM and all of LLVM's infrastructure. This applies just
as
> much to LLD as it does to Clang.
>
> You say that it isn't compelling to match Clang's design, but in
fact it
> is. You would need an overwhelming argument to *diverge* from Clang's
> design.
>
> The fact that it makes the design more challenging is not compelling at
> all. Yes, building libraries that can be re-used and making the binary
> calling it equally efficient is more challenging, but that is the express
> mission of LLVM and every project within it.
>
>
>> The new one is designed as a command from day one. (Precisely speaking,
>> the original code propagates errors all the way up to the entry point,
so
>> you can call it and expect it to always return. Rafael introduced
error()
>> function later and we now depends on that function does not return.)
>>
>
> I think this last was a mistake.
>
> The fact that the code propagates errors all the way up is fine, and even
> good. We don't necessarily need to be able to *recover* from link
errors
> and try some other path.
>
> But we absolutely need the design to be a *library* that can be embedded
> into other programs and tools. I can't even begin to count the use
cases
> for this.
>
> So please, let's go back to where we *do not* rely on never-returning
> error handling. That is an absolute mistake.
>
>
>>
>> If you want to consider changing that, we should have a fresh (and
broad)
>>> discussion, but it goes pretty firmly against the design of the
entire LLVM
>>> project. I also don't really understand why it would be
beneficial.
>>>
>>
>> I'm not against organizing it as a library as long as it does not
make
>> things too complicated
>>
>
> I am certain that it will make things more complicated, but that is the
> technical challenge that we must overcome. It will be hard, but I am
> absolutely confident it is possible to have an elegant library design here.
> It may not be as simple as a pure command line tool, but it will be
> *dramatically* more powerful, general, and broadly applicable.
>
> The design of LLVM is not the simplest way to build a compiler. But it is
> valuable to all of those working on it precisely because of this
> flexibility imparted by its library oriented design. This is absolutely not
> something that we should lose from the linker.
>
>
>> , and I guess reorganizing the existing code as a library is relatively
>> easy because it's still pretty small, but I don't really want
to focus on
>> that until it becomes usable as an alternative to GNU ld or gold. I
want to
>> focus on the linker features themselves at this moment. Once it's
complete,
>> it becomes more clear how to organize it.
>>
>
> Ok, now we're talking about something totally reasonable.
>
> If it is easier for you all to develop this first as a command line tool,
> and then make it work as a library, sure, go for it. You're doing the
work,
> I can hardly tell you how to go about it. ;]
>
It is not only easier for me to develop but is also super important for
avoiding over-designing the API of the library. Until we know what we need
to do and what can be done, it is too easy to make mistake to design API
that is supposed to cover everything -- including hypothetical unrealistic
ones. Such API would slow down the development speed significantly, and
it's a pain when we abandon that when we realize that that was not needed.

However, I think it is super important to be clear that getting the
library
> architecture is a *hard requirement* for the LLD project. Without that, it
> doesn't even make sense as part of LLVM.
>
> And as a consequence, I think it is unacceptable to replace the old ELF
> port with the new one until this is true. That is removing functionality
> that users of LLD realistically were depending on, which you're seeing
in
> this thread. That's not cool. We don't really tolerate dramatic
regressions
> in functionality like this, and even if we've already done it, we
should
> revert back to a state where both are available until the new port is
> actually ready. And ready in LLVM land means, functional as a library.
>
> -Chandler
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/54966ca6/attachment.html>

On Thu, Jan 7, 2016 at 4:05 PM Rui Ueyama <ruiu at google.com> wrote:
> By organizing it as a library, I'm expecting something coarse. I
don't
> expect to reorganize the linker itself as a collection of small libraries,
> but make the entire linker available as a library, so that you can link
> stuff in-process. More specifically, I expect that the library would
> basically export one function, link(std::vector<StringRef>), which
takes
> command line arguments, and returns a memory buffer for a newly created
> executable. We may want to allow a mix of StringRef and MemoryBuffer as
> input, so that you can directly pass in-memory objects to the linker, but
> the basic idea remains the same.
>
> Are we on the same page?
>
Let me answer this below, where I think you get to the core of the problem.

>
> On Thu, Jan 7, 2016 at 3:44 PM, Chandler Carruth <chandlerc at
gmail.com>
> wrote:
>
>> On Thu, Jan 7, 2016 at 3:18 PM Rui Ueyama <ruiu at google.com>
wrote:
>>
>>> On Thu, Jan 7, 2016 at 2:56 PM, Chandler Carruth <chandlerc at
gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jan 7, 2016 at 7:18 AM Rui Ueyama via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via
llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> In the process of migrating from old lld ELF linker to
new
>>>>>> (previously ELF2) I noticed the interface lost several
important features
>>>>>> (ordered by importance for my use case):
>>>>>>
>>>>>> 1. Detecting errors in the first place. New linker
seems to call
>>>>>> exit(1) for any error.
>>>>>>
>>>>>> 2. Reporting messages to non-stderr outputs. Previously
all link
>>>>>> functions had a raw_ostream argument so it was possible
to delay the error
>>>>>> output, aggregate it for multiple linked files, output
via a different
>>>>>> format, etc.
>>>>>>
>>>>>> 3. Linking multiple outputs in parallel (useful for
test drivers) in
>>>>>> a single process. Not really an interface issue but
there are at least two
>>>>>> global pointers (Config & Driver) that refer to
stack variables and are
>>>>>> used in various places in the code.
>>>>>>
>>>>>> All of this seems to indicate a departure from the
linker being
>>>>>> useable as a library. To maintain the previous behavior
you'd have to use a
>>>>>> linker binary & popen.
>>>>>>
>>>>>> Is this a conscious design decision or a temporary
limitation?
>>>>>>
>>>>>
>>>>> That the new ELF and COFF linkers are designed as commands
instead of
>>>>> libraries is very much an intended design change.
>>>>>
>>>>
>>>> I disagree.
>>>>
>>>> During the discussion, there was a *specific* discussion of
both the
>>>> new COFF port and ELF port continuing to be libraries with a
common command
>>>> line driver.
>>>>
>>>
>>> There was a discussion that we would keep the same entry point for
the
>>> old and the new, but I don't remember if I promised that we
were going to
>>> organize the new linker as a library.
>>>
>>
>> Ok, myself and essentially everyone else thought this was clear. If it
>> isn't lets clarify:
>>
>> I think it is absolutely critical and important that LLD's
architecture
>> remain one where all functionality is available as a library. This is
*the*
>> design goal of LLVM and all of LLVM's infrastructure. This applies
just as
>> much to LLD as it does to Clang.
>>
>> You say that it isn't compelling to match Clang's design, but
in fact it
>> is. You would need an overwhelming argument to *diverge* from
Clang's
>> design.
>>
>> The fact that it makes the design more challenging is not compelling at
>> all. Yes, building libraries that can be re-used and making the binary
>> calling it equally efficient is more challenging, but that is the
express
>> mission of LLVM and every project within it.
>>
>>
>>> The new one is designed as a command from day one. (Precisely
speaking,
>>> the original code propagates errors all the way up to the entry
point, so
>>> you can call it and expect it to always return. Rafael introduced
error()
>>> function later and we now depends on that function does not
return.)
>>>
>>
>> I think this last was a mistake.
>>
>> The fact that the code propagates errors all the way up is fine, and
even
>> good. We don't necessarily need to be able to *recover* from link
errors
>> and try some other path.
>>
>> But we absolutely need the design to be a *library* that can be
embedded
>> into other programs and tools. I can't even begin to count the use
cases
>> for this.
>>
>> So please, let's go back to where we *do not* rely on
never-returning
>> error handling. That is an absolute mistake.
>>
>>
>>>
>>> If you want to consider changing that, we should have a fresh (and
>>>> broad) discussion, but it goes pretty firmly against the design
of the
>>>> entire LLVM project. I also don't really understand why it
would be
>>>> beneficial.
>>>>
>>>
>>> I'm not against organizing it as a library as long as it does
not make
>>> things too complicated
>>>
>>
>> I am certain that it will make things more complicated, but that is the
>> technical challenge that we must overcome. It will be hard, but I am
>> absolutely confident it is possible to have an elegant library design
here.
>> It may not be as simple as a pure command line tool, but it will be
>> *dramatically* more powerful, general, and broadly applicable.
>>
>> The design of LLVM is not the simplest way to build a compiler. But it
is
>> valuable to all of those working on it precisely because of this
>> flexibility imparted by its library oriented design. This is absolutely
not
>> something that we should lose from the linker.
>>
>>
>>> , and I guess reorganizing the existing code as a library is
relatively
>>> easy because it's still pretty small, but I don't really
want to focus on
>>> that until it becomes usable as an alternative to GNU ld or gold. I
want to
>>> focus on the linker features themselves at this moment. Once
it's complete,
>>> it becomes more clear how to organize it.
>>>
>>
>> Ok, now we're talking about something totally reasonable.
>>
>> If it is easier for you all to develop this first as a command line
tool,
>> and then make it work as a library, sure, go for it. You're doing
the work,
>> I can hardly tell you how to go about it. ;]
>>
>
> It is not only easier for me to develop but is also super important for
> avoiding over-designing the API of the library. Until we know what we need
> to do and what can be done, it is too easy to make mistake to design API
> that is supposed to cover everything -- including hypothetical unrealistic
> ones. Such API would slow down the development speed significantly, and
> it's a pain when we abandon that when we realize that that was not
needed.
>
I'm very sympathetic to the problem of not wanting to design an API until
the concrete use cases for it appear. That makes perfect sense.

We just need to be *ready* to extend the library API (and potentially find
a more fine grained layering if one is actually called for) when a
reasonable and real use case arises for some users of LLD. Once we have
people that actually have a use case and want to introduce a certain
interface to the library that supports it, we need to work with them to
figure out how to effectively support their use case.

At the least, we clearly need the super simple interface[1] that the
command line tool would use, but an in-process linker could also probably
use.

We might need minor extensions to effectively support Arseny's use case (I
think an in-process linker is a *very* reasonable thing to support, I'd
even like to teach the Clang driver to optionally work that way to be more
efficient on platforms like Windows). But I have to imagine that the
interface for an in-process static linker and the command line linker are
extremely similar if not precisely the same.

At some point, it might also make sense to support more interesting linking
scenarios such as linking a PIC "shared object" that can be mapped
into the
running process for JIT users. But I think it is reasonable to build the
interface that those users need when those users are ready to leverage LLD.
That way we can work with them to make sure we don't build the wrong
interface or an overly complicated one (as you say).

Make sense?
-Chandler
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160108/f5068799/attachment-0001.html>

On Thu, Jan 07, 2016 at 11:44:28PM +0000, Chandler Carruth via llvm-dev
wrote:
> The fact that the code propagates errors all the way up is fine, and even
> good. We don't necessarily need to be able to *recover* from link
errors
> and try some other path.
Just to clarify this: errors can be fatal as far as the linking process
is considered, but they should allow enough recovery to get rid of the
LLD instance and associated ressources.

Joerg