llvm dev - Dec 2016 - LLD status update and performance chart

> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com> wrote:
> 
> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at apple.com
<mailto:peter_cooper at apple.com>> wrote:
> Hi Rui
> 
> I agree separating the components out in to libraries only makes sense when
there is a clear reason to do so.  However, just this year there was a very
involved discussion about what it means to be a library.  Specifically, I
don't think your current 'main-as-library' argument is valid while
you call exit or (if you) rely on mutable global state.  Having a single entry
point via a main function is fine, but that function cannot then kill the
process which its linked in to.
> 
> Our main function returns as long as input object files are not corrupted.
If you are doing in-memory linking, I think it is unlikely that the object files
in memory are corrupted (especially when you just created them using LLVM), so I
think this satisfies most users needs in practice. Do you have a concern about
that?
Ultimately my concern is that there is *any* code path calling exit.  I would
say that this prevents the lld library from being used in-process.  But others
opinions may differ, and I honestly don't have a use case in mind, just that
I don't think library code should ever call exit.
> 
> For the situation that you need to handle foreign object files in the same
process (I'd recommend you to sandbox a process in that case though), we can
write a verifier to check for file correctness rigorously so that we can
guarantee that object files are as trustworthy as freshly-created object files.
I think this feature is a reasonable addition to the linker.
That sounds great.  Having written some parts of the MachO lld linker and seen
Kevin's work on llvm-objdump, I can appreciate that is not easy.  For
example, I wrote the logic to process EH FDE's which may need to error out
if invalid.  You don't necessarily want to validate them all up front as it
may be too slow, so I can understand that this isn't necessarily trivial to
handle in a performant way.
> 
> As to the mutable shared state, my current (unproved) idea is to make them
thread local variables. Since no one yet has come up to say "hey, we are
actually trying to run multiple instances of the linker in the same process
simultaneously but LLD doesn't allow that", that's not implemented
yet, but technically I think it's doable, and that's needless to say a
reasonable feature request.
LLVM uses the LLVMContext for this (and begs users to look the other way with
regards to cl::opt's).  I don't know if there's been a discussion in
LLVM about whether TLV's would be better there too, but seems like a
reasonable discussion to have.  Certainly I don't think anyone should say
you can't use them without good reason.
> 
> As I repeatedly said in the thread that speed is not the only goal for us.
Honestly, it's going to be the best selling point of LLD, because most
people do not use that many linker features but just use it to create
executables (and sometimes wait for a long period of time). I reported about the
performance in this thread because I thought people would be happy to hear the
speed improvement we've made this year. Also, because I was happy about
that, I probably emphasized that too much. But that's not our single goal.
I meant to commend you for both sending out a summary email, and the results. 
Having this fast a linker on ELF/COFF is going to be a huge win for developers. 
And I personally really like status updates for major projects/features as it
can be hard to follow along with all the email traffic.  So thank you for doing
that.

My only concern with performance is that I felt like you would be against
changes to the code which make it slower but add functionality.  Error handling
is such a use case.  LLVM and clang continue to get bigger each year and
sometimes that means a little slower too.  The linker may be faster next year
than it is now, or it may be slower but have a feature which makes that a
worthwhile tradeoff.  I don't want to slow down any of the code for any
reason, but its natural that sometimes it'll happen with good reason.

Thanks,
Pete
> 
> If you want  context then the relevant piece of the thread is
http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html
<http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html>.
> 
> Arseny summarized things very well there, so i'll just quote him at the
end here.  I understand that you and others want to first write a fast linker
tool and i don't think anyone has any problem with that, but there is also a
clear desire from folks to have it be usable as a library and I would hope any
patches to do so are accepted, even if they make the code more complex, or
slower.
> 
> >>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev
<
> >>> llvm-dev at lists.llvm.org
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
> >>>
> >>>> In the process of migrating from old lld ELF linker to new
(previously
> >>>> ELF2) I noticed the interface lost several important
features (ordered by
> >>>> importance for my use case):
> >>>>
> >>>> 1. Detecting errors in the first place. New linker seems
to call
> >>>> exit(1) for any error.
> >>>>
> >>>> 2. Reporting messages to non-stderr outputs. Previously
all link
> >>>> functions had a raw_ostream argument so it was possible to
delay the error
> >>>> output, aggregate it for multiple linked files, output via
a different
> >>>> format, etc.
> >>>>
> >>>> 3. Linking multiple outputs in parallel (useful for test
drivers) in a
> >>>> single process. Not really an interface issue but there
are at least two
> >>>> global pointers (Config & Driver) that refer to stack
variables and are
> >>>> used in various places in the code.
> >>>>
> >>>> All of this seems to indicate a departure from the linker
being useable
> >>>> as a library. To maintain the previous behavior you'd
have to use a linker
> >>>> binary & popen.
> Pete
>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> I talked several people and found that this is more like a
communication issue rather than a technical/philosophical issue. I believe
communication problems won't solve themselves. As a person who is on the
owners file of LLD, I think I need to say something about that issue. Also, I
guess people who were just watching this thread wondered why my happy
pre-holiday status report suddenly turned into a heated discussion, and they are
probably still wondering what's wrong with LLD. I want to address that, too.
>> 
>> So, as a project, there is no anti-library policy in LLD. I think this
is the misunderstanding one side had. We already provide main-as-a-library
feature so that you can embed the linker to your program. We as a project
welcome other ideas to export linker features at a well-defined boundary. For
example, I think abstracting the file system access so that you can hook file
operations could be a well-defined, useful API for those who want to do
in-memory linking (I expressed that opinion already in this thread). Just like
LLVM, we won't guarantee API compatibility between releases, and we are
unlikely to be able to expose deep internals of the linker, but as long as you
think you found a reasonable coarse API boundary, there should be nothing
preventing you from bringing that to the table.
>> 
>> On the other hand, as far as I talked, no one who is on the
"library" side requested LLD expose deep internals. This is the
misunderstanding the other side had. If we as a project said that LLD should not
support any library interface at all, they would be upset and speak out loudly,
but again, that's not a project policy.
>> 
>> So, correct me if I'm wrong, but I don't see no serious
conflicts here. The conflict I saw in the thread is I believe superficial, and I
strongly believe that it could have been addressed calmly and nicely if we have
used more words to explain thoughts instead of small number of strong words.
>> 
>> Hope this helps.
>> 
>> Rui
>> 
>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >I am on PTO, so slow to respond.
>> >
>> >Some items that are left:
>> >
>> >* Debug fission
>> >* Single file debug fission
>> >* Range extension thunks
>> >* All of freebsd links and works
>> >* Very good performance when all that is in
>> 
>> Looks we have initial version of debug fusion implemented.
>> r289790, r289810 commits from yesterday did the rest of main job I
believe.
>> I do not know what is "Single file debug fission" ? (quick
googling gives nothing and I never heard about that before I think)
>> 
>> George.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161216/757da8e3/attachment.html>

On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper <peter_cooper at apple.com>
wrote:
>
> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at
apple.com>
> wrote:
>
>> Hi Rui
>>
>> I agree separating the components out in to libraries only makes sense
>> when there is a clear reason to do so.  However, just this year there
was a
>> very involved discussion about what it means to be a library.
>> Specifically, I don't think your current 'main-as-library'
argument is
>> valid while you call exit or (if you) rely on mutable global state. 
Having
>> a single entry point via a main function is fine, but that function
cannot
>> then kill the process which its linked in to.
>>
>
> Our main function returns as long as input object files are not corrupted.
> If you are doing in-memory linking, I think it is unlikely that the object
> files in memory are corrupted (especially when you just created them using
> LLVM), so I think this satisfies most users needs in practice. Do you have
> a concern about that?
>
> Ultimately my concern is that there is *any* code path calling exit.  I
> would say that this prevents the lld library from being used in-process.
> But others opinions may differ, and I honestly don't have a use case in
> mind, just that I don't think library code should ever call exit.
>
There is a duality of LLD: lld-as-a-command and lld-as-a-library. This
duality is not necessarily a bad thing. Given that we have a verifier, any
path that leads to check for an impossible error condition and call exit()
should be thought as an assert() when it is used as a library since they
should never happen or there is a bug in code (and that's what assert
actually does). We already have lots of asserts in our libraries, and
that's I think essentially the same.

> For the situation that you need to handle foreign object files in the same
> process (I'd recommend you to sandbox a process in that case though),
we
> can write a verifier to check for file correctness rigorously so that we
> can guarantee that object files are as trustworthy as freshly-created
> object files. I think this feature is a reasonable addition to the linker.
>
> That sounds great.  Having written some parts of the MachO lld linker and
> seen Kevin's work on llvm-objdump, I can appreciate that is not easy. 
For
> example, I wrote the logic to process EH FDE's which may need to error
out
> if invalid.  You don't necessarily want to validate them all up front
as it
> may be too slow, so I can understand that this isn't necessarily
trivial to
> handle in a performant way.
>
That's I don't know yet. My gut is that doing error checking beforehand
makes code easy to read and maintain, just like semantic analysis doesn't
have to handle syntactic errors. But I don't know the answer, so I cannot
exclude neither possibilities. We have to experiment that and compare.

>
> As to the mutable shared state, my current (unproved) idea is to make them
> thread local variables. Since no one yet has come up to say "hey, we
are
> actually trying to run multiple instances of the linker in the same process
> simultaneously but LLD doesn't allow that", that's not
implemented yet, but
> technically I think it's doable, and that's needless to say a
reasonable
> feature request.
>
> LLVM uses the LLVMContext for this (and begs users to look the other way
> with regards to cl::opt's).  I don't know if there's been a
discussion in
> LLVM about whether TLV's would be better there too, but seems like a
> reasonable discussion to have.  Certainly I don't think anyone should
say
> you can't use them without good reason.
>
That's also another thing no one knows the answer. As far as I can say,
global states in the LLD/ELF makes things easy to maintain, and looks like
a majority of people working on it are in favor of it. Of course people who
have different taste may not like it that much, I understand that, and I
don't say that that's the best way, but it's there and it works
fairly
satisfactory. The most important thing for external users is the API, no?
We can discuss what is the best way to have a linker-global state
internally, but as long as we provide a sane API, everything else should
fall in the internal design stuff category.

>
> As I repeatedly said in the thread that speed is not the only goal for us.
> Honestly, it's going to be the best selling point of LLD, because most
> people do not use that many linker features but just use it to create
> executables (and sometimes wait for a long period of time). I reported
> about the performance in this thread because I thought people would be
> happy to hear the speed improvement we've made this year. Also, because
I
> was happy about that, I probably emphasized that too much. But that's
not
> our single goal.
>
> I meant to commend you for both sending out a summary email, and the
> results.  Having this fast a linker on ELF/COFF is going to be a huge win
> for developers.  And I personally really like status updates for major
> projects/features as it can be hard to follow along with all the email
> traffic.  So thank you for doing that.
>
> My only concern with performance is that I felt like you would be against
> changes to the code which make it slower but add functionality.  Error
> handling is such a use case.  LLVM and clang continue to get bigger each
> year and sometimes that means a little slower too.  The linker may be
> faster next year than it is now, or it may be slower but have a feature
> which makes that a worthwhile tradeoff.  I don't want to slow down any
of
> the code for any reason, but its natural that sometimes it'll happen
with
> good reason.
>
I don't know if you believe me by repeating the thing I said many times in
this thread, but I did not sacrificing functionality for speed.

If you take a look at the performance chart that I sent in this thread,
you'll notice the pattern that the linker gradually became slower and then
suddenly became faster. As we add more safety measures, error checks and
features, the linker get slower and slower. Each one is small but they
accumulates. And then we sometimes ran a profiler to nail down a
bottleneck, came up with a good optimization, and implement it. That's what
you see as steep speedups in the chart. We do not optimize by removing
features. We just did better.

So, I don't know what I can do for you to believe me, but I have never said
that the performance is the only goal, and you can find it by my actual
behavior. I believe I've been trying to always be helpful. Please ask
LLD/ELF developers about that. If you find me doing the opposite in the
code review or discussion, please point it out, so that I can correct that.

> Thanks,
> Pete
>
>
> If you want  context then the relevant piece of the thread is
>> http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html.
>>
>> Arseny summarized things very well there, so i'll just quote him at
the
>> end here.  I understand that you and others want to first write a fast
>> linker tool and i don't think anyone has any problem with that, but
there
>> is also a clear desire from folks to have it be usable as a library and
I
>> would hope any patches to do so are accepted, even if they make the
code
>> more complex, or slower.
>>
>> >>>* On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via
llvm-dev <
>> *>>>* llvm-dev at lists.llvm.org
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>> *>>>>>>>* In the process of migrating from old lld
ELF linker to new (previously
>> *>>>>* ELF2) I noticed the interface lost several important
features (ordered by
>> *>>>>* importance for my use case):
>> *>>>>>>>>* 1. Detecting errors in the first
place. New linker seems to call
>> *>>>>* exit(1) for any error.
>> *>>>>>>>>* 2. Reporting messages to non-stderr
outputs. Previously all link
>> *>>>>* functions had a raw_ostream argument so it was
possible to delay the error
>> *>>>>* output, aggregate it for multiple linked files,
output via a different
>> *>>>>* format, etc.
>> *>>>>>>>>* 3. Linking multiple outputs in
parallel (useful for test drivers) in a
>> *>>>>* single process. Not really an interface issue but
there are at least two
>> *>>>>* global pointers (Config & Driver) that refer to
stack variables and are
>> *>>>>* used in various places in the code.
>> *>>>>>>>>* All of this seems to indicate a
departure from the linker being useable
>> *>>>>* as a library. To maintain the previous behavior
you'd have to use a linker
>> *>>>>* binary & popen.*
>>
>> Pete
>>
>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> I talked several people and found that this is more like a
communication
>> issue rather than a technical/philosophical issue. I believe
communication
>> problems won't solve themselves. As a person who is on the owners
file of
>> LLD, I think I need to say something about that issue. Also, I guess
people
>> who were just watching this thread wondered why my happy pre-holiday
status
>> report suddenly turned into a heated discussion, and they are probably
>> still wondering what's wrong with LLD. I want to address that, too.
>>
>> So, as a project, there is no anti-library policy in LLD. I think this
is
>> the misunderstanding one side had. We already provide main-as-a-library
>> feature so that you can embed the linker to your program. We as a
project
>> welcome other ideas to export linker features at a well-defined
boundary.
>> For example, I think abstracting the file system access so that you can
>> hook file operations could be a well-defined, useful API for those who
want
>> to do in-memory linking (I expressed that opinion already in this
thread).
>> Just like LLVM, we won't guarantee API compatibility between
releases, and
>> we are unlikely to be able to expose deep internals of the linker, but
as
>> long as you think you found a reasonable coarse API boundary, there
should
>> be nothing preventing you from bringing that to the table.
>>
>> On the other hand, as far as I talked, no one who is on the
"library"
>> side requested LLD expose deep internals. This is the misunderstanding
the
>> other side had. If we as a project said that LLD should not support any
>> library interface at all, they would be upset and speak out loudly, but
>> again, that's not a project policy.
>>
>> So, correct me if I'm wrong, but I don't see no serious
conflicts here.
>> The conflict I saw in the thread is I believe superficial, and I
strongly
>> believe that it could have been addressed calmly and nicely if we have
used
>> more words to explain thoughts instead of small number of strong words.
>>
>> Hope this helps.
>>
>> Rui
>>
>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> >I am on PTO, so slow to respond.
>>> >
>>> >Some items that are left:
>>> >
>>> >* Debug fission
>>> >* Single file debug fission
>>> >* Range extension thunks
>>> >* All of freebsd links and works
>>> >* Very good performance when all that is in
>>>
>>> Looks we have initial version of debug fusion implemented.
>>> r289790, r289810 commits from yesterday did the rest of main job I
>>> believe.
>>> I do not know what is "Single file debug fission" ?
(quick googling
>>> gives nothing and I never heard about that before I think)
>>>
>>> George.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161216/3b7a12da/attachment-0001.html>

On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at
apple.com>
> wrote:
>
>> Hi Rui
>>
>> I agree separating the components out in to libraries only makes sense
>> when there is a clear reason to do so.  However, just this year there
was a
>> very involved discussion about what it means to be a library.
>> Specifically, I don't think your current 'main-as-library'
argument is
>> valid while you call exit or (if you) rely on mutable global state. 
Having
>> a single entry point via a main function is fine, but that function
cannot
>> then kill the process which its linked in to.
>>
>
> Our main function returns as long as input object files are not corrupted.
> If you are doing in-memory linking, I think it is unlikely that the object
> files in memory are corrupted (especially when you just created them using
> LLVM), so I think this satisfies most users needs in practice. Do you have
> a concern about that?
>
> Ultimately my concern is that there is *any* code path calling exit.  I
> would say that this prevents the lld library from being used in-process.
> But others opinions may differ, and I honestly don't have a use case in
> mind, just that I don't think library code should ever call exit.
>
I agreed with the sentiment at first, but after thinking about it for a
while, I actually have convinced myself that it doesn't hold water under
closer inspection.

The fundamental thing is that the LLVM libraries actually do have tons of
fatal errors; they're just in the form of assert's (or we'll
dereference a
null pointer, or run off the end of a data structure, or go into an
infinite loop, etc.).

If you pass a corrupted Module to LLVM through the library API, you can
certainly trip tons of "fatal errors" (in the form of failed
assertions or
UB). The way that LLVM gets around this is by having a policy of "if you
pass it corrupted Module that doesn't pass the verifier, it's your
fault,
you're using our API wrong". Why can't an LLD library API have that
same
requirement?

If it is safe for clang to uses the LLVM library API without running the
verifier as its default configuration for non-development builds, why would
it be unsafe for (say) clang to pass an object file directly to LLD as a
library without verification? Like Rui said, it's absolutely possible to
create a verifier pass for LLD; it just hasn't been written because most
object files we've seen so far seem to come from a small number of
well-tested codepaths that always (as in the `assert` meaning of
"always")
create valid ELF files. In fact, we've added graceful recovery as
appropriate (e.g. r259831), which is a step above error handling!

Also, I'd like to point out that Clang, even when it does run the LLVM
verifier (which is not the default except in development builds), runs it
with fatal error handling. Is anybody aware of a program that uses LLVM as
a library, produces IR in memory, runs the verifier, and does not simply
abort if the verifier fails in non-development builds?

-- Sean Silva

>
> For the situation that you need to handle foreign object files in the same
> process (I'd recommend you to sandbox a process in that case though),
we
> can write a verifier to check for file correctness rigorously so that we
> can guarantee that object files are as trustworthy as freshly-created
> object files. I think this feature is a reasonable addition to the linker.
>
> That sounds great.  Having written some parts of the MachO lld linker and
> seen Kevin's work on llvm-objdump, I can appreciate that is not easy. 
For
> example, I wrote the logic to process EH FDE's which may need to error
out
> if invalid.  You don't necessarily want to validate them all up front
as it
> may be too slow, so I can understand that this isn't necessarily
trivial to
> handle in a performant way.
>
>
> As to the mutable shared state, my current (unproved) idea is to make them
> thread local variables. Since no one yet has come up to say "hey, we
are
> actually trying to run multiple instances of the linker in the same process
> simultaneously but LLD doesn't allow that", that's not
implemented yet, but
> technically I think it's doable, and that's needless to say a
reasonable
> feature request.
>
> LLVM uses the LLVMContext for this (and begs users to look the other way
> with regards to cl::opt's).  I don't know if there's been a
discussion in
> LLVM about whether TLV's would be better there too, but seems like a
> reasonable discussion to have.  Certainly I don't think anyone should
say
> you can't use them without good reason.
>
>
> As I repeatedly said in the thread that speed is not the only goal for us.
> Honestly, it's going to be the best selling point of LLD, because most
> people do not use that many linker features but just use it to create
> executables (and sometimes wait for a long period of time). I reported
> about the performance in this thread because I thought people would be
> happy to hear the speed improvement we've made this year. Also, because
I
> was happy about that, I probably emphasized that too much. But that's
not
> our single goal.
>
> I meant to commend you for both sending out a summary email, and the
> results.  Having this fast a linker on ELF/COFF is going to be a huge win
> for developers.  And I personally really like status updates for major
> projects/features as it can be hard to follow along with all the email
> traffic.  So thank you for doing that.
>
> My only concern with performance is that I felt like you would be against
> changes to the code which make it slower but add functionality.  Error
> handling is such a use case.  LLVM and clang continue to get bigger each
> year and sometimes that means a little slower too.  The linker may be
> faster next year than it is now, or it may be slower but have a feature
> which makes that a worthwhile tradeoff.  I don't want to slow down any
of
> the code for any reason, but its natural that sometimes it'll happen
with
> good reason.
>
> Thanks,
> Pete
>
>
> If you want  context then the relevant piece of the thread is
>> http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html.
>>
>> Arseny summarized things very well there, so i'll just quote him at
the
>> end here.  I understand that you and others want to first write a fast
>> linker tool and i don't think anyone has any problem with that, but
there
>> is also a clear desire from folks to have it be usable as a library and
I
>> would hope any patches to do so are accepted, even if they make the
code
>> more complex, or slower.
>>
>> >>>* On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via
llvm-dev <
>> *>>>* llvm-dev at lists.llvm.org
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>> *>>>>>>>* In the process of migrating from old lld
ELF linker to new (previously
>> *>>>>* ELF2) I noticed the interface lost several important
features (ordered by
>> *>>>>* importance for my use case):
>> *>>>>>>>>* 1. Detecting errors in the first
place. New linker seems to call
>> *>>>>* exit(1) for any error.
>> *>>>>>>>>* 2. Reporting messages to non-stderr
outputs. Previously all link
>> *>>>>* functions had a raw_ostream argument so it was
possible to delay the error
>> *>>>>* output, aggregate it for multiple linked files,
output via a different
>> *>>>>* format, etc.
>> *>>>>>>>>* 3. Linking multiple outputs in
parallel (useful for test drivers) in a
>> *>>>>* single process. Not really an interface issue but
there are at least two
>> *>>>>* global pointers (Config & Driver) that refer to
stack variables and are
>> *>>>>* used in various places in the code.
>> *>>>>>>>>* All of this seems to indicate a
departure from the linker being useable
>> *>>>>* as a library. To maintain the previous behavior
you'd have to use a linker
>> *>>>>* binary & popen.*
>>
>> Pete
>>
>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> I talked several people and found that this is more like a
communication
>> issue rather than a technical/philosophical issue. I believe
communication
>> problems won't solve themselves. As a person who is on the owners
file of
>> LLD, I think I need to say something about that issue. Also, I guess
people
>> who were just watching this thread wondered why my happy pre-holiday
status
>> report suddenly turned into a heated discussion, and they are probably
>> still wondering what's wrong with LLD. I want to address that, too.
>>
>> So, as a project, there is no anti-library policy in LLD. I think this
is
>> the misunderstanding one side had. We already provide main-as-a-library
>> feature so that you can embed the linker to your program. We as a
project
>> welcome other ideas to export linker features at a well-defined
boundary.
>> For example, I think abstracting the file system access so that you can
>> hook file operations could be a well-defined, useful API for those who
want
>> to do in-memory linking (I expressed that opinion already in this
thread).
>> Just like LLVM, we won't guarantee API compatibility between
releases, and
>> we are unlikely to be able to expose deep internals of the linker, but
as
>> long as you think you found a reasonable coarse API boundary, there
should
>> be nothing preventing you from bringing that to the table.
>>
>> On the other hand, as far as I talked, no one who is on the
"library"
>> side requested LLD expose deep internals. This is the misunderstanding
the
>> other side had. If we as a project said that LLD should not support any
>> library interface at all, they would be upset and speak out loudly, but
>> again, that's not a project policy.
>>
>> So, correct me if I'm wrong, but I don't see no serious
conflicts here.
>> The conflict I saw in the thread is I believe superficial, and I
strongly
>> believe that it could have been addressed calmly and nicely if we have
used
>> more words to explain thoughts instead of small number of strong words.
>>
>> Hope this helps.
>>
>> Rui
>>
>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> >I am on PTO, so slow to respond.
>>> >
>>> >Some items that are left:
>>> >
>>> >* Debug fission
>>> >* Single file debug fission
>>> >* Range extension thunks
>>> >* All of freebsd links and works
>>> >* Very good performance when all that is in
>>>
>>> Looks we have initial version of debug fusion implemented.
>>> r289790, r289810 commits from yesterday did the rest of main job I
>>> believe.
>>> I do not know what is "Single file debug fission" ?
(quick googling
>>> gives nothing and I never heard about that before I think)
>>>
>>> George.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161217/221409c2/attachment.html>

On Sat, Dec 17, 2016 at 9:32 PM, Sean Silva via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at
apple.com>
>> wrote:
>>
>>> Hi Rui
>>>
>>> I agree separating the components out in to libraries only makes
sense
>>> when there is a clear reason to do so.  However, just this year
there was a
>>> very involved discussion about what it means to be a library.
>>> Specifically, I don't think your current
'main-as-library' argument is
>>> valid while you call exit or (if you) rely on mutable global state.
Having
>>> a single entry point via a main function is fine, but that function
cannot
>>> then kill the process which its linked in to.
>>>
>>
>> Our main function returns as long as input object files are not
>> corrupted. If you are doing in-memory linking, I think it is unlikely
that
>> the object files in memory are corrupted (especially when you just
created
>> them using LLVM), so I think this satisfies most users needs in
practice.
>> Do you have a concern about that?
>>
>> Ultimately my concern is that there is *any* code path calling exit.  I
>> would say that this prevents the lld library from being used
in-process.
>> But others opinions may differ, and I honestly don't have a use
case in
>> mind, just that I don't think library code should ever call exit.
>>
>
> I agreed with the sentiment at first, but after thinking about it for a
> while, I actually have convinced myself that it doesn't hold water
under
> closer inspection.
>
> The fundamental thing is that the LLVM libraries actually do have tons of
> fatal errors; they're just in the form of assert's (or we'll
dereference a
> null pointer, or run off the end of a data structure, or go into an
> infinite loop, etc.).
>
> If you pass a corrupted Module to LLVM through the library API, you can
> certainly trip tons of "fatal errors" (in the form of failed
assertions or
> UB). The way that LLVM gets around this is by having a policy of "if
you
> pass it corrupted Module that doesn't pass the verifier, it's your
fault,
> you're using our API wrong". Why can't an LLD library API have
that same
> requirement?
>
I agree that if an API user violates the API of a library, it is
appropriate for the library to abort with a fatal error.

However if the API is used correctly, but some error occurs, this error
should be reported back to the API consumer.

> If it is safe for clang to uses the LLVM library API without running the
> verifier as its default configuration for non-development builds, why would
> it be unsafe for (say) clang to pass an object file directly to LLD as a
> library without verification? Like Rui said, it's absolutely possible
to
> create a verifier pass for LLD; it just hasn't been written because
most
> object files we've seen so far seem to come from a small number of
> well-tested codepaths that always (as in the `assert` meaning of
"always")
> create valid ELF files. In fact, we've added graceful recovery as
> appropriate (e.g. r259831), which is a step above error handling!
>
> Also, I'd like to point out that Clang, even when it does run the LLVM
> verifier (which is not the default except in development builds), runs it
> with fatal error handling. Is anybody aware of a program that uses LLVM as
> a library, produces IR in memory, runs the verifier, and does not simply
> abort if the verifier fails in non-development builds?
>
I'm doing the same as clang:

#ifndef NDEBUG
    char *error = nullptr;
    LLVMVerifyModule(g->module, LLVMAbortProcessAction, &error);
#endif

However the LLVM API is defined such that trying to call codegen functions
on an invalid module is undefined, and aborting in this case makes sense.
This is really just a more sophisticated assert().

I'm a fan of assert(); I like having assertions on during development. It
makes sense for a library to assert if API is violated. But errors not due
to API violations should be reported back to the caller instead of aborting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161217/97e62834/attachment.html>

> On Dec 17, 2016, at 6:32 PM, Sean Silva via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> 
> On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> 
>> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com
<mailto:ruiu at google.com>> wrote:
>> 
>> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at
apple.com <mailto:peter_cooper at apple.com>> wrote:
>> Hi Rui
>> 
>> I agree separating the components out in to libraries only makes sense
when there is a clear reason to do so.  However, just this year there was a very
involved discussion about what it means to be a library.  Specifically, I
don't think your current 'main-as-library' argument is valid while
you call exit or (if you) rely on mutable global state.  Having a single entry
point via a main function is fine, but that function cannot then kill the
process which its linked in to.
>> 
>> Our main function returns as long as input object files are not
corrupted. If you are doing in-memory linking, I think it is unlikely that the
object files in memory are corrupted (especially when you just created them
using LLVM), so I think this satisfies most users needs in practice. Do you have
a concern about that?
> Ultimately my concern is that there is *any* code path calling exit.  I
would say that this prevents the lld library from being used in-process.  But
others opinions may differ, and I honestly don't have a use case in mind,
just that I don't think library code should ever call exit.
> 
> I agreed with the sentiment at first, but after thinking about it for a
while, I actually have convinced myself that it doesn't hold water under
closer inspection.
> 
> The fundamental thing is that the LLVM libraries actually do have tons of
fatal errors; they're just in the form of assert's (or we'll
dereference a null pointer, or run off the end of a data structure, or go into
an infinite loop, etc.).
> 
> If you pass a corrupted Module to LLVM through the library API, you can
certainly trip tons of "fatal errors" (in the form of failed
assertions or UB). The way that LLVM gets around this is by having a policy of
"if you pass it corrupted Module that doesn't pass the verifier,
it's your fault, you're using our API wrong". Why can't an LLD
library API have that same requirement?
> 
> If it is safe for clang to uses the LLVM library API without running the
verifier as its default configuration for non-development builds, why would it
be unsafe for (say) clang to pass an object file directly to LLD as a library
without verification?
It would be OK, but in practice this is not what happens, does it?
We run the IR verifier on every LTO input for instance.

— 
Mehdi

> Like Rui said, it's absolutely possible to create a verifier pass for
LLD; it just hasn't been written because most object files we've seen so
far seem to come from a small number of well-tested codepaths that always (as in
the `assert` meaning of "always") create valid ELF files. In fact,
we've added graceful recovery as appropriate (e.g. r259831), which is a step
above error handling!
> 
> Also, I'd like to point out that Clang, even when it does run the LLVM
verifier (which is not the default except in development builds), runs it with
fatal error handling. Is anybody aware of a program that uses LLVM as a library,
produces IR in memory, runs the verifier, and does not simply abort if the
verifier fails in non-development builds?
> 
> -- Sean Silva
> 
> 
>> 
>> For the situation that you need to handle foreign object files in the
same process (I'd recommend you to sandbox a process in that case though),
we can write a verifier to check for file correctness rigorously so that we can
guarantee that object files are as trustworthy as freshly-created object files.
I think this feature is a reasonable addition to the linker.
> That sounds great.  Having written some parts of the MachO lld linker and
seen Kevin's work on llvm-objdump, I can appreciate that is not easy.  For
example, I wrote the logic to process EH FDE's which may need to error out
if invalid.  You don't necessarily want to validate them all up front as it
may be too slow, so I can understand that this isn't necessarily trivial to
handle in a performant way.
>> 
>> As to the mutable shared state, my current (unproved) idea is to make
them thread local variables. Since no one yet has come up to say "hey, we
are actually trying to run multiple instances of the linker in the same process
simultaneously but LLD doesn't allow that", that's not implemented
yet, but technically I think it's doable, and that's needless to say a
reasonable feature request.
> LLVM uses the LLVMContext for this (and begs users to look the other way
with regards to cl::opt's).  I don't know if there's been a
discussion in LLVM about whether TLV's would be better there too, but seems
like a reasonable discussion to have.  Certainly I don't think anyone should
say you can't use them without good reason.
>> 
>> As I repeatedly said in the thread that speed is not the only goal for
us. Honestly, it's going to be the best selling point of LLD, because most
people do not use that many linker features but just use it to create
executables (and sometimes wait for a long period of time). I reported about the
performance in this thread because I thought people would be happy to hear the
speed improvement we've made this year. Also, because I was happy about
that, I probably emphasized that too much. But that's not our single goal.
> I meant to commend you for both sending out a summary email, and the
results.  Having this fast a linker on ELF/COFF is going to be a huge win for
developers.  And I personally really like status updates for major
projects/features as it can be hard to follow along with all the email traffic. 
So thank you for doing that.
> 
> My only concern with performance is that I felt like you would be against
changes to the code which make it slower but add functionality.  Error handling
is such a use case.  LLVM and clang continue to get bigger each year and
sometimes that means a little slower too.  The linker may be faster next year
than it is now, or it may be slower but have a feature which makes that a
worthwhile tradeoff.  I don't want to slow down any of the code for any
reason, but its natural that sometimes it'll happen with good reason.
> 
> Thanks,
> Pete
> 
>> 
>> If you want  context then the relevant piece of the thread is
http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html
<http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html>.
>> 
>> Arseny summarized things very well there, so i'll just quote him at
the end here.  I understand that you and others want to first write a fast
linker tool and i don't think anyone has any problem with that, but there is
also a clear desire from folks to have it be usable as a library and I would
hope any patches to do so are accepted, even if they make the code more complex,
or slower.
>> 
>> >>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via
llvm-dev <
>> >>> llvm-dev at lists.llvm.org
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>> >>>
>> >>>> In the process of migrating from old lld ELF linker to
new (previously
>> >>>> ELF2) I noticed the interface lost several important
features (ordered by
>> >>>> importance for my use case):
>> >>>>
>> >>>> 1. Detecting errors in the first place. New linker
seems to call
>> >>>> exit(1) for any error.
>> >>>>
>> >>>> 2. Reporting messages to non-stderr outputs.
Previously all link
>> >>>> functions had a raw_ostream argument so it was
possible to delay the error
>> >>>> output, aggregate it for multiple linked files, output
via a different
>> >>>> format, etc.
>> >>>>
>> >>>> 3. Linking multiple outputs in parallel (useful for
test drivers) in a
>> >>>> single process. Not really an interface issue but
there are at least two
>> >>>> global pointers (Config & Driver) that refer to
stack variables and are
>> >>>> used in various places in the code.
>> >>>>
>> >>>> All of this seems to indicate a departure from the
linker being useable
>> >>>> as a library. To maintain the previous behavior
you'd have to use a linker
>> >>>> binary & popen.
>> Pete
>>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> 
>>> I talked several people and found that this is more like a
communication issue rather than a technical/philosophical issue. I believe
communication problems won't solve themselves. As a person who is on the
owners file of LLD, I think I need to say something about that issue. Also, I
guess people who were just watching this thread wondered why my happy
pre-holiday status report suddenly turned into a heated discussion, and they are
probably still wondering what's wrong with LLD. I want to address that, too.
>>> 
>>> So, as a project, there is no anti-library policy in LLD. I think
this is the misunderstanding one side had. We already provide main-as-a-library
feature so that you can embed the linker to your program. We as a project
welcome other ideas to export linker features at a well-defined boundary. For
example, I think abstracting the file system access so that you can hook file
operations could be a well-defined, useful API for those who want to do
in-memory linking (I expressed that opinion already in this thread). Just like
LLVM, we won't guarantee API compatibility between releases, and we are
unlikely to be able to expose deep internals of the linker, but as long as you
think you found a reasonable coarse API boundary, there should be nothing
preventing you from bringing that to the table.
>>> 
>>> On the other hand, as far as I talked, no one who is on the
"library" side requested LLD expose deep internals. This is the
misunderstanding the other side had. If we as a project said that LLD should not
support any library interface at all, they would be upset and speak out loudly,
but again, that's not a project policy.
>>> 
>>> So, correct me if I'm wrong, but I don't see no serious
conflicts here. The conflict I saw in the thread is I believe superficial, and I
strongly believe that it could have been addressed calmly and nicely if we have
used more words to explain thoughts instead of small number of strong words.
>>> 
>>> Hope this helps.
>>> 
>>> Rui
>>> 
>>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>> >I am on PTO, so slow to respond.
>>> >
>>> >Some items that are left:
>>> >
>>> >* Debug fission
>>> >* Single file debug fission
>>> >* Range extension thunks
>>> >* All of freebsd links and works
>>> >* Very good performance when all that is in
>>> 
>>> Looks we have initial version of debug fusion implemented.
>>> r289790, r289810 commits from yesterday did the rest of main job I
believe.
>>> I do not know what is "Single file debug fission" ?
(quick googling gives nothing and I never heard about that before I think)
>>> 
>>> George.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161217/14e4733c/attachment.html>