llvm dev - Dec 2016 - LLD status update and performance chart

On Tue, Dec 13, 2016 at 10:37 AM, Hal Finkel via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> ----- Original Message -----
> > From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev
at lists.llvm.org>
> > To: "Mehdi Amini" <mehdi.amini at apple.com>
> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> > Sent: Tuesday, December 13, 2016 12:10:08 PM
> > Subject: Re: [llvm-dev] LLD status update and performance chart
> >
> > Mehdi Amini <mehdi.amini at apple.com> writes:
> >
> > >> On Dec 13, 2016, at 9:40 AM, Rafael Avila de Espindola
> > >> <rafael.espindola at gmail.com> wrote:
> > >>
> > >> Mehdi Amini <mehdi.amini at apple.com> writes:
> > >>
> > >>>> On Dec 13, 2016, at 5:55 AM, Rafael Avila de
Espindola via
> > >>>> llvm-dev <llvm-dev at lists.llvm.org> wrote:
> > >>>>
> > >>>> Sean Silva via llvm-dev <llvm-dev at
lists.llvm.org> writes:
> > >>>>> This will also greatly facilitate certain
measurements I'd like
> > >>>>> to do
> > >>>>> w.r.t. different strategies for avoiding memory
costs for input
> > >>>>> files (esp.
> > >>>>> minor faults and dTLB costs). I've almost
gotten to the point
> > >>>>> of
> > >>>>> implementing this just to do those measurements.
> > >>>>
> > >>>> If you do please keep it local. The bare minimum we
have of
> > >>>> library
> > >>>> support is already disproportionately painful and
prevents
> > >>>> easier sharing
> > >>>> with COFF. We should really not add more until the
linker is
> > >>>> done.
> > >>>
> > >>> This is so much in contrast with the LLVM development, I
find it
> > >>> quite hard to see this as an acceptable position on
llvm-dev.
> > >>
> > >> Why? What is wrong with setting priorities and observing that
what
> > >> library support we already have has had a disproportional
cost?
> > >
> > > The library-hostile lld development goes against one the core
> > > principles that, I believe, drives the LLVM development:
providing
> > > libraries and reusable components.
> >
> > Because it is trying to do something fundamentally different. We are
> > trying to write a *program*.
>
> But this is not a technical argument. As a project, we rarely write
> programs, as such. We generally create reusable components that happen to
> have driver executables. At least long term, I think there's consensus
that
> this is the best path. If we're going to make a different choice in
this
> case, we need concrete reasons. We should discuss this in the context of
> the reasons you've provided (error handling, etc.).
>
Please tell me what you think about how reusable components would be like.
Which parts of the linker can be reusable components? Is that really
feasible? You are saying that the linker should be written in different way
by comparing it with an ideal linker (modular, reusable and fast! and by
the way the current LLD is much more reusable and extensible than before in
my opinion), but you can say anything if you compare with an ideal one. You
need to prove that it's doable so we should do that way instead of this. We
(or I) did a large experiment with the old LLD for years but couldn't find
a way to make it possible in a reasonable manner. I'm still trying to find
one, by distilling ELF and COFF linkers common parts, but still couldn't
make it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/6ed4a48a/attachment.html>

----- Original Message -----
> From: "Rui Ueyama" <ruiu at google.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Rafael Avila de Espindola" <rafael.espindola at
gmail.com>,
> "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Tuesday, December 13, 2016 12:56:41 PM
> Subject: Re: [llvm-dev] LLD status update and performance chart
> On Tue, Dec 13, 2016 at 10:37 AM, Hal Finkel via llvm-dev <
> llvm-dev at lists.llvm.org > wrote:
> > ----- Original Message -----
> 
> > > From: "Rafael Avila de Espindola via llvm-dev" <
> > > llvm-dev at lists.llvm.org >
> 
> > > To: "Mehdi Amini" < mehdi.amini at apple.com >
> 
> > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >
> 
> > > Sent: Tuesday, December 13, 2016 12:10:08 PM
> 
> > > Subject: Re: [llvm-dev] LLD status update and performance chart
> 
> > >
> 
> > > Mehdi Amini < mehdi.amini at apple.com > writes:
> 
> > >
> 
> > > >> On Dec 13, 2016, at 9:40 AM, Rafael Avila de Espindola
> 
> > > >> < rafael.espindola at gmail.com > wrote:
> 
> > > >>
> 
> > > >> Mehdi Amini < mehdi.amini at apple.com > writes:
> 
> > > >>
> 
> > > >>>> On Dec 13, 2016, at 5:55 AM, Rafael Avila de
Espindola via
> 
> > > >>>> llvm-dev < llvm-dev at lists.llvm.org >
wrote:
> 
> > > >>>>
> 
> > > >>>> Sean Silva via llvm-dev < llvm-dev at
lists.llvm.org > writes:
> 
> > > >>>>> This will also greatly facilitate certain
measurements I'd
> > > >>>>> like
> 
> > > >>>>> to do
> 
> > > >>>>> w.r.t. different strategies for avoiding
memory costs for
> > > >>>>> input
> 
> > > >>>>> files (esp.
> 
> > > >>>>> minor faults and dTLB costs). I've
almost gotten to the
> > > >>>>> point
> 
> > > >>>>> of
> 
> > > >>>>> implementing this just to do those
measurements.
> 
> > > >>>>
> 
> > > >>>> If you do please keep it local. The bare minimum
we have of
> 
> > > >>>> library
> 
> > > >>>> support is already disproportionately painful
and prevents
> 
> > > >>>> easier sharing
> 
> > > >>>> with COFF. We should really not add more until
the linker is
> 
> > > >>>> done.
> 
> > > >>>
> 
> > > >>> This is so much in contrast with the LLVM
development, I find
> > > >>> it
> 
> > > >>> quite hard to see this as an acceptable position on
llvm-dev.
> 
> > > >>
> 
> > > >> Why? What is wrong with setting priorities and observing
that
> > > >> what
> 
> > > >> library support we already have has had a
disproportional
> > > >> cost?
> 
> > > >
> 
> > > > The library-hostile lld development goes against one the
core
> 
> > > > principles that, I believe, drives the LLVM development:
> > > > providing
> 
> > > > libraries and reusable components.
> 
> > >
> 
> > > Because it is trying to do something fundamentally different. We
> > > are
> 
> > > trying to write a *program*.
> 
> > But this is not a technical argument. As a project, we rarely write
> > programs, as such. We generally create reusable components that
> > happen to have driver executables. At least long term, I think
> > there's consensus that this is the best path. If we're going
to
> > make
> > a different choice in this case, we need concrete reasons. We
> > should
> > discuss this in the context of the reasons you've provided (error
> > handling, etc.).
> 
> Please tell me what you think about how reusable components would be
> like. Which parts of the linker can be reusable components? Is that
> really feasible?
As far as I'm concerned, your response, "That said, I think the current
our 'API' to allow users call our linker's main function hit the
sweet spot. I know at least a few LLVM-based language developers who want to
eliminate external dependencies and embed a linker to their compilers.
That's a reasonable usage, and I think allowing them to pass a map from
filename to MemoryBuffer objects makes sense, too. That would be done without
affecting the overall linker architecture. I don't oppose to that idea, and
if someone wrote a patch, I'm fine with that" is perfectly appropriate.
We need to guide these discussions with use cases, and that's the use case
that's been provided so far.

Longer term, we also should take a serious look at how to unify the
functionality in LLD with that in our JIT runtime linker. Having two linkers in
the LLVM project, one for use with the JIT and one for other things, seems
suboptimal.

Thanks again, 
Hal 
> You are saying that the linker should be written in different way by
> comparing it with an ideal linker (modular, reusable and fast! and
> by the way the current LLD is much more reusable and extensible than
> before in my opinion), but you can say anything if you compare with
> an ideal one. You need to prove that it's doable so we should do
> that way instead of this. We (or I) did a large experiment with the
> old LLD for years but couldn't find a way to make it possible in a
> reasonable manner. I'm still trying to find one, by distilling ELF
> and COFF linkers common parts, but still couldn't make it.
-- 

Hal Finkel 
Lead, Compiler Technology and Programming Languages 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/77215b67/attachment.html>

Hal Finkel <hfinkel at anl.gov> writes:
>
>> Please tell me what you think about how reusable components would be
>> like. Which parts of the linker can be reusable components? Is that
>> really feasible?
> As far as I'm concerned, your response, "That said, I think the
current our 'API' to allow users call our linker's main function hit
the sweet spot. I know at least a few LLVM-based language developers who want to
eliminate external dependencies and embed a linker to their compilers.
That's a reasonable usage, and I think allowing them to pass a map from
filename to MemoryBuffer objects makes sense, too. That would be done without
affecting the overall linker architecture. I don't oppose to that idea, and
if someone wrote a patch, I'm fine with that" is perfectly appropriate.
We need to guide these discussions with use cases, and that's the use case
that's been provided so far.
>
> Longer term, we also should take a serious look at how to unify the
functionality in LLD with that in our JIT runtime linker. Having two linkers in
the LLVM project, one for use with the JIT and one for other things, seems
suboptimal.
In my opinion having a general linker in the JIT is sub optimal. We
should not be desiginig lld around an idea there is not even a consensus
to. And even if it is case that having a general linker is the best way
to write a JIT (I would love to know why a JIT needs something that
handles copy relocations), we will not be able to evaluate the trade
offs with the stand alone lld until we finished it.

Cheers,
Rafael