llvm dev - Dec 2016 - LLD status update and performance chart

On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com>
> wrote:
>
>>
>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> >
>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> >> This will also greatly facilitate certain measurements I'd
like to do
>> >> w.r.t. different strategies for avoiding memory costs for
input files
>> (esp.
>> >> minor faults and dTLB costs). I've almost gotten to the
point of
>> >> implementing this just to do those measurements.
>> >
>> > If you do please keep it local. The bare minimum we have of
library
>> > support is already disproportionately painful and prevents easier
>> sharing
>> > with COFF. We should really not add more until the linker is done.
>>
>> This is so much in contrast with the LLVM development, I find it quite
>> hard to see this as an acceptable position on llvm-dev.
>>
>
> LLD is a subproject of the LLVM project, but as a product, LLD itself is
> not LLVM nor Clang, so some technical decisions that make sense to them are
> not directly be applicable or even inappropriate. As a person who spent
> almost two years on the old LLD and 1.5 years on the new LLD, I can say
> that Rafael's stance on focusing on making a good linker first really
makes
> sense. I can easily imagine that if we didn't focus on that, we
couldn't
> make this much progress over the past 1.5 year and would be stagnated at a
> very basic level. Do you know if I'm a person who worked really hard on
the
> old (and probably "modular" whatever it means) linker so hard?
I'm speaking
> based on the experience. If you have an concrete idea how to construct a
> linker from smaller modules,  please tell me. I still don't get what
you
> want. We can discuss concrete proposals, but "making it (more)
modular" is
> too vague and not really a proposal, so it cannot be a productive
> discussion.
>
> That said, I think the current our "API" to allow users call our
linker's
> main function hit the sweet spot. I know at least a few LLVM-based language
> developers who want to eliminate external dependencies and embed a linker
> to their compilers. That's a reasonable usage, and I think allowing
them to
> pass a map from filename to MemoryBuffer objects makes sense, too. That
> would be done without affecting the overall linker architecture. I
don't
> oppose to that idea, and if someone wrote a patch, I'm fine with that.
>
>
> I’m totally willing to believe you that it is not possible to write the
> fastest ELF linker on earth (or in the universe) with a library based and
> reusable components approach. But clang is not the fastest C/C++ compiler
> available, and LLVM is not the fastest compiler framework either!
>
> So as a project, it seems to me that LLVM has not put the tradeoff on the
> speed/efficiency historically when it was to the detriment of
> layering/component/modularity/reusability/…
>
> Writing the fastest linker possible is nice goal, I regret that a LLVM
> subproject is putting this goal above
layering/component/modularity/reusability/…
> though.
>
I've never mentioned that creating the fastest linker is the only goal.

Medhi, please tell how you would *actually* layer linkers with fine-grained
components. You are just saying that the current LLD is worse than an
imaginary super-beautiful linker.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/16a0aa6f/attachment.html>

> On Dec 13, 2016, at 11:08 AM, Rui Ueyama <ruiu at google.com> wrote:
> 
> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com
<mailto:ruiu at google.com>> wrote:
>> 
>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com <mailto:mehdi.amini at apple.com>> wrote:
>> 
>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> >
>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>> writes:
>> >> This will also greatly facilitate certain measurements I'd
like to do
>> >> w.r.t. different strategies for avoiding memory costs for
input files (esp.
>> >> minor faults and dTLB costs). I've almost gotten to the
point of
>> >> implementing this just to do those measurements.
>> >
>> > If you do please keep it local. The bare minimum we have of
library
>> > support is already disproportionately painful and prevents easier
sharing
>> > with COFF. We should really not add more until the linker is done.
>> 
>> This is so much in contrast with the LLVM development, I find it quite
hard to see this as an acceptable position on llvm-dev.
>> 
>> LLD is a subproject of the LLVM project, but as a product, LLD itself
is not LLVM nor Clang, so some technical decisions that make sense to them are
not directly be applicable or even inappropriate. As a person who spent almost
two years on the old LLD and 1.5 years on the new LLD, I can say that
Rafael's stance on focusing on making a good linker first really makes
sense. I can easily imagine that if we didn't focus on that, we couldn't
make this much progress over the past 1.5 year and would be stagnated at a very
basic level. Do you know if I'm a person who worked really hard on the old
(and probably "modular" whatever it means) linker so hard? I'm
speaking based on the experience. If you have an concrete idea how to construct
a linker from smaller modules,  please tell me. I still don't get what you
want. We can discuss concrete proposals, but "making it (more)
modular" is too vague and not really a proposal, so it cannot be a
productive discussion.
>> 
>> That said, I think the current our "API" to allow users call
our linker's main function hit the sweet spot. I know at least a few
LLVM-based language developers who want to eliminate external dependencies and
embed a linker to their compilers. That's a reasonable usage, and I think
allowing them to pass a map from filename to MemoryBuffer objects makes sense,
too. That would be done without affecting the overall linker architecture. I
don't oppose to that idea, and if someone wrote a patch, I'm fine with
that.
> 
> 
> I’m totally willing to believe you that it is not possible to write the
fastest ELF linker on earth (or in the universe) with a library based and
reusable components approach. But clang is not the fastest C/C++ compiler
available, and LLVM is not the fastest compiler framework either!
> 
> So as a project, it seems to me that LLVM has not put the tradeoff on the
speed/efficiency historically when it was to the detriment of
layering/component/modularity/reusability/…
> 
> Writing the fastest linker possible is nice goal, I regret that a LLVM
subproject is putting this goal above
layering/component/modularity/reusability/… though.
> 
> I've never mentioned that creating the fastest linker is the only goal.
I believe this has clearly been put *ahead* the other design aspects I
mentioned, isn’t it?
> Medhi, please tell how you would *actually* layer linkers with fine-grained
components.
That’s not a bait I’m gonna bite.

> You are just saying that the current LLD is worse than an imaginary
super-beautiful linker.
Using superlative and trying to qualify what I’m writing with "imaginary
super-beautiful linker” is just caricatural and diverting the whole point. LLVM
is quite far from an "imaginary super-beautiful compiler framework”, yet it
is modular and usable as a library.

— 
Mehdi

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/1c9a8f1d/attachment.html>

On Tue, Dec 13, 2016 at 11:23 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Dec 13, 2016, at 11:08 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at
apple.com>
> wrote:
>
>>
>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com>
>> wrote:
>>
>>>
>>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>> >
>>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org>
writes:
>>> >> This will also greatly facilitate certain measurements
I'd like to do
>>> >> w.r.t. different strategies for avoiding memory costs for
input files
>>> (esp.
>>> >> minor faults and dTLB costs). I've almost gotten to
the point of
>>> >> implementing this just to do those measurements.
>>> >
>>> > If you do please keep it local. The bare minimum we have of
library
>>> > support is already disproportionately painful and prevents
easier
>>> sharing
>>> > with COFF. We should really not add more until the linker is
done.
>>>
>>> This is so much in contrast with the LLVM development, I find it
quite
>>> hard to see this as an acceptable position on llvm-dev.
>>>
>>
>> LLD is a subproject of the LLVM project, but as a product, LLD itself
is
>> not LLVM nor Clang, so some technical decisions that make sense to them
are
>> not directly be applicable or even inappropriate. As a person who spent
>> almost two years on the old LLD and 1.5 years on the new LLD, I can say
>> that Rafael's stance on focusing on making a good linker first
really makes
>> sense. I can easily imagine that if we didn't focus on that, we
couldn't
>> make this much progress over the past 1.5 year and would be stagnated
at a
>> very basic level. Do you know if I'm a person who worked really
hard on the
>> old (and probably "modular" whatever it means) linker so
hard? I'm speaking
>> based on the experience. If you have an concrete idea how to construct
a
>> linker from smaller modules,  please tell me. I still don't get
what you
>> want. We can discuss concrete proposals, but "making it (more)
modular" is
>> too vague and not really a proposal, so it cannot be a productive
>> discussion.
>>
>> That said, I think the current our "API" to allow users call
our linker's
>> main function hit the sweet spot. I know at least a few LLVM-based
language
>> developers who want to eliminate external dependencies and embed a
linker
>> to their compilers. That's a reasonable usage, and I think allowing
them to
>> pass a map from filename to MemoryBuffer objects makes sense, too. That
>> would be done without affecting the overall linker architecture. I
don't
>> oppose to that idea, and if someone wrote a patch, I'm fine with
that.
>>
>>
>> I’m totally willing to believe you that it is not possible to write the
>> fastest ELF linker on earth (or in the universe) with a library based
and
>> reusable components approach. But clang is not the fastest C/C++
compiler
>> available, and LLVM is not the fastest compiler framework either!
>>
>> So as a project, it seems to me that LLVM has not put the tradeoff on
the
>> speed/efficiency historically when it was to the detriment of
>> layering/component/modularity/reusability/…
>>
>> Writing the fastest linker possible is nice goal, I regret that a LLVM
>> subproject is putting this goal above
layering/component/modularity/reusability/…
>> though.
>>
>
> I've never mentioned that creating the fastest linker is the only goal.
>
>
> I believe this has clearly been put *ahead* the other design aspects I
> mentioned, isn’t it?
>
> Medhi, please tell how you would *actually* layer linkers with
> fine-grained components.
>
>
> That’s not a bait I’m gonna bite.
>
That's not a bait... I guess you are proposing a different architecture, so
you need to explain it.
> You are just saying that the current LLD is worse than an imaginary
> super-beautiful linker.
>
>
> Using superlative and trying to qualify what I’m writing with
"imaginary
> super-beautiful linker” is just caricatural and diverting the whole point.
> LLVM is quite far from an "imaginary super-beautiful compiler
framework”,
> yet it is modular and usable as a library.
>
> —
> Mehdi
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/14254788/attachment-0001.html>