llvm dev - Dec 2016 - LLD status update and performance chart

>I am on PTO, so slow to respond.
>
>Some items that are left:
>
>* Debug fission
>* Single file debug fission
>* Range extension thunks
>* All of freebsd links and works
>* Very good performance when all that is in
Looks we have initial version of debug fusion implemented. 
r289790, r289810 commits from yesterday did the rest of main job I believe.
I do not know what is "Single file debug fission" ? (quick googling
gives nothing and I never heard about that before I think)

George.

I talked several people and found that this is more like a communication
issue rather than a technical/philosophical issue. I believe communication
problems won't solve themselves. As a person who is on the owners file of
LLD, I think I need to say something about that issue. Also, I guess people
who were just watching this thread wondered why my happy pre-holiday status
report suddenly turned into a heated discussion, and they are probably
still wondering what's wrong with LLD. I want to address that, too.

So, as a project, there is no anti-library policy in LLD. I think this is
the misunderstanding one side had. We already provide main-as-a-library
feature so that you can embed the linker to your program. We as a project
welcome other ideas to export linker features at a well-defined boundary.
For example, I think abstracting the file system access so that you can
hook file operations could be a well-defined, useful API for those who want
to do in-memory linking (I expressed that opinion already in this thread).
Just like LLVM, we won't guarantee API compatibility between releases, and
we are unlikely to be able to expose deep internals of the linker, but as
long as you think you found a reasonable coarse API boundary, there should
be nothing preventing you from bringing that to the table.

On the other hand, as far as I talked, no one who is on the "library"
side
requested LLD expose deep internals. This is the misunderstanding the other
side had. If we as a project said that LLD should not support any library
interface at all, they would be upset and speak out loudly, but again,
that's not a project policy.

So, correct me if I'm wrong, but I don't see no serious conflicts here.
The
conflict I saw in the thread is I believe superficial, and I strongly
believe that it could have been addressed calmly and nicely if we have used
more words to explain thoughts instead of small number of strong words.

Hope this helps.

Rui

On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> >I am on PTO, so slow to respond.
> >
> >Some items that are left:
> >
> >* Debug fission
> >* Single file debug fission
> >* Range extension thunks
> >* All of freebsd links and works
> >* Very good performance when all that is in
>
> Looks we have initial version of debug fusion implemented.
> r289790, r289810 commits from yesterday did the rest of main job I believe.
> I do not know what is "Single file debug fission" ? (quick
googling gives
> nothing and I never heard about that before I think)
>
> George.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161216/60304c26/attachment.html>

On 16 December 2016 at 18:15, Rui Ueyama via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> So, correct me if I'm wrong, but I don't see no serious conflicts
here. The
> conflict I saw in the thread is I believe superficial, and I strongly
> believe that it could have been addressed calmly and nicely if we have used
> more words to explain thoughts instead of small number of strong words.
Hi Rui,

Thank you for your comments. I agree with your views that the issue
was much more about communication than technical. It's how things are
said, rather that what is said, and we should put that aside.

On the technical side, I agree that the project needs a solid base
before fancy new features get incorporated, but I also believe that
parallel development, even on trunk, can happen as it does in LLVM
(see GlobalISel, pass managers, register allocators, back-ends).

I'm very happy that LLD works well on AArch64, bootstrapping Clang and
passing the test-suite. We aim to reach the same objective next year
on ARM, and go beyond. I'm also happy that the FreeBSD community is
looking at it with serious eyes and trying to make it the default
linker on x86_64.

In the interest of collaboration, I think we should set some goals for
the project in general, getting feedback from the community that works
in it and the stakeholders, at least until we reach production quality
in one architecture. From my point of view, having LLD as the default
linker for FreeBSD/ELF/x86_64 is a strong indication that it "works
well in a large range of situations". I'd happily call that stage
"Production Beta". But that's not all. We need other architectures
(AArch64 and ARM will probably come next), as well as different object
formats (COFF, MachO) to be able to call it "Production Stable".

However, my humble opinion is that we don't need to be in "Production
Stable" to start adding new features, especially when that means a
larger portion of the LLVM ecosystem will begin to contribute more to
the project. Those new features can stay disabled by default and be
isolated from the main code, like we do in LLVM. Yes, that will mean
more work for all of us. Yes, that will mean longer test cycles, more
test configurations. But that will also mean more people working on
it, validating in ways you didn't even know it was possible. That
value is worth the extra trouble, IMVHO, and LLVM's success is living
proof of that.

LLD may be a separate project, and a young one full of energy, but it
is also an "LLVM Project". As such, it's bound to the same level
of
design goals that all LLVM projects share. Not all share all values,
but two that we share amongst all projects is collaboration and
modularisation. Those values reflect our multiple ranges of users and
developers as well as the need to re-use code above raw performance.

As was said in this list, Clang is not faster than GCC for a long
time, even though it was one of the shiny distinctions. The faster the
code we produce, the slower the compiler runs. It's an obvious
relationship, and one that will (hopefully) happen with the linker if
it has any expectation of being used by the whole community, not just
the limited number of developers today.

cheers,
--renato

Thank you for writing this!

I totally agree with this characterization of the discussion as a huge
misunderstanding -- I was planning to write an email saying effectively the
same thing, except, posed more as a question. :)



On Fri, Dec 16, 2016 at 1:15 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I talked several people and found that this is more like a communication
> issue rather than a technical/philosophical issue. I believe communication
> problems won't solve themselves. As a person who is on the owners file
of
> LLD, I think I need to say something about that issue. Also, I guess people
> who were just watching this thread wondered why my happy pre-holiday status
> report suddenly turned into a heated discussion, and they are probably
> still wondering what's wrong with LLD. I want to address that, too.
>
> So, as a project, there is no anti-library policy in LLD. I think this is
> the misunderstanding one side had. We already provide main-as-a-library
> feature so that you can embed the linker to your program. We as a project
> welcome other ideas to export linker features at a well-defined boundary.
> For example, I think abstracting the file system access so that you can
> hook file operations could be a well-defined, useful API for those who want
> to do in-memory linking (I expressed that opinion already in this thread).
> Just like LLVM, we won't guarantee API compatibility between releases,
and
> we are unlikely to be able to expose deep internals of the linker, but as
> long as you think you found a reasonable coarse API boundary, there should
> be nothing preventing you from bringing that to the table.
>
> On the other hand, as far as I talked, no one who is on the
"library" side
> requested LLD expose deep internals. This is the misunderstanding the other
> side had. If we as a project said that LLD should not support any library
> interface at all, they would be upset and speak out loudly, but again,
> that's not a project policy.
>
> So, correct me if I'm wrong, but I don't see no serious conflicts
here.
> The conflict I saw in the thread is I believe superficial, and I strongly
> believe that it could have been addressed calmly and nicely if we have used
> more words to explain thoughts instead of small number of strong words.
>
> Hope this helps.
>
> Rui
>
> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> >I am on PTO, so slow to respond.
>> >
>> >Some items that are left:
>> >
>> >* Debug fission
>> >* Single file debug fission
>> >* Range extension thunks
>> >* All of freebsd links and works
>> >* Very good performance when all that is in
>>
>> Looks we have initial version of debug fusion implemented.
>> r289790, r289810 commits from yesterday did the rest of main job I
>> believe.
>> I do not know what is "Single file debug fission" ? (quick
googling gives
>> nothing and I never heard about that before I think)
>>
>> George.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161216/f96ee88d/attachment.html>

Hi Rui

I agree separating the components out in to libraries only makes sense when
there is a clear reason to do so.  However, just this year there was a very
involved discussion about what it means to be a library.  Specifically, I
don't think your current 'main-as-library' argument is valid while
you call exit or (if you) rely on mutable global state.  Having a single entry
point via a main function is fine, but that function cannot then kill the
process which its linked in to.

If you want  context then the relevant piece of the thread is
http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html
<http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html>.

Arseny summarized things very well there, so i'll just quote him at the end
here.  I understand that you and others want to first write a fast linker tool
and i don't think anyone has any problem with that, but there is also a
clear desire from folks to have it be usable as a library and I would hope any
patches to do so are accepted, even if they make the code more complex, or
slower.
>>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
>>> llvm-dev at lists.llvm.org
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>>>
>>>> In the process of migrating from old lld ELF linker to new
(previously
>>>> ELF2) I noticed the interface lost several important features
(ordered by
>>>> importance for my use case):
>>>>
>>>> 1. Detecting errors in the first place. New linker seems to
call
>>>> exit(1) for any error.
>>>>
>>>> 2. Reporting messages to non-stderr outputs. Previously all
link
>>>> functions had a raw_ostream argument so it was possible to
delay the error
>>>> output, aggregate it for multiple linked files, output via a
different
>>>> format, etc.
>>>>
>>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
>>>> single process. Not really an interface issue but there are at
least two
>>>> global pointers (Config & Driver) that refer to stack
variables and are
>>>> used in various places in the code.
>>>>
>>>> All of this seems to indicate a departure from the linker being
useable
>>>> as a library. To maintain the previous behavior you'd have
to use a linker
>>>> binary & popen.
Pete
> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I talked several people and found that this is more like a communication
issue rather than a technical/philosophical issue. I believe communication
problems won't solve themselves. As a person who is on the owners file of
LLD, I think I need to say something about that issue. Also, I guess people who
were just watching this thread wondered why my happy pre-holiday status report
suddenly turned into a heated discussion, and they are probably still wondering
what's wrong with LLD. I want to address that, too.
> 
> So, as a project, there is no anti-library policy in LLD. I think this is
the misunderstanding one side had. We already provide main-as-a-library feature
so that you can embed the linker to your program. We as a project welcome other
ideas to export linker features at a well-defined boundary. For example, I think
abstracting the file system access so that you can hook file operations could be
a well-defined, useful API for those who want to do in-memory linking (I
expressed that opinion already in this thread). Just like LLVM, we won't
guarantee API compatibility between releases, and we are unlikely to be able to
expose deep internals of the linker, but as long as you think you found a
reasonable coarse API boundary, there should be nothing preventing you from
bringing that to the table.
> 
> On the other hand, as far as I talked, no one who is on the
"library" side requested LLD expose deep internals. This is the
misunderstanding the other side had. If we as a project said that LLD should not
support any library interface at all, they would be upset and speak out loudly,
but again, that's not a project policy.
> 
> So, correct me if I'm wrong, but I don't see no serious conflicts
here. The conflict I saw in the thread is I believe superficial, and I strongly
believe that it could have been addressed calmly and nicely if we have used more
words to explain thoughts instead of small number of strong words.
> 
> Hope this helps.
> 
> Rui
> 
> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >I am on PTO, so slow to respond.
> >
> >Some items that are left:
> >
> >* Debug fission
> >* Single file debug fission
> >* Range extension thunks
> >* All of freebsd links and works
> >* Very good performance when all that is in
> 
> Looks we have initial version of debug fusion implemented.
> r289790, r289810 commits from yesterday did the rest of main job I believe.
> I do not know what is "Single file debug fission" ? (quick
googling gives nothing and I never heard about that before I think)
> 
> George.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161216/076c3388/attachment.html>

George Rimar <grimar at accesssoftek.com> writes:
>>I am on PTO, so slow to respond.
>>
>>Some items that are left:
>>
>>* Debug fission
>>* Single file debug fission
>>* Range extension thunks
>>* All of freebsd links and works
>>* Very good performance when all that is in
>
> Looks we have initial version of debug fusion implemented. 
> r289790, r289810 commits from yesterday did the rest of main job I believe.
> I do not know what is "Single file debug fission" ? (quick
googling gives nothing and I never heard about that before I think)
The idea is to design and implement a way to keep the .dwo data in the
same file as the .o. Having two files is nice for a cluster build, but
not for a local build with ccache.

Cheers,
Rafael

Rui Ueyama <ruiu at google.com> writes:

> So, as a project, there is no anti-library policy in LLD. I think this is
> the misunderstanding one side had. We already provide main-as-a-library
> feature so that you can embed the linker to your program. We as a project
> welcome other ideas to export linker features at a well-defined boundary.
> For example, I think abstracting the file system access so that you can
> hook file operations could be a well-defined, useful API for those who want
> to do in-memory linking (I expressed that opinion already in this thread).
> Just like LLVM, we won't guarantee API compatibility between releases,
and
> we are unlikely to be able to expose deep internals of the linker, but as
> long as you think you found a reasonable coarse API boundary, there should
> be nothing preventing you from bringing that to the table.
It really depends on what is meant by a library.

If we have to remove all calls to exit, all memory pools and all global
variables I am very opposed to it.

lld is an actual project with actual value for its users. I really don't
want to compromise its maintainability by turning every

for (auto &X : check(...))

into

auto XsOrErr = ...;
if (std::error_code EC = XsOrErr.getError())
  return EC; // also update callers. Add more context to the return if
             // you don't want to degrade the error message.
auto &Xs = *XsOrErr;
for (auto &X : Xs)

In the name of a theoretical user. Really, can we put this on hold
until the MachO linker has shown how to get its parts actually reused?

Cheers,
Rafael