llvm dev - Dec 2016 - LLD status update and performance chart

On Tue, Dec 13, 2016 at 11:23 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Dec 13, 2016, at 11:08 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at
apple.com>
> wrote:
>
>>
>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com>
>> wrote:
>>
>>>
>>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>> >
>>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org>
writes:
>>> >> This will also greatly facilitate certain measurements
I'd like to do
>>> >> w.r.t. different strategies for avoiding memory costs for
input files
>>> (esp.
>>> >> minor faults and dTLB costs). I've almost gotten to
the point of
>>> >> implementing this just to do those measurements.
>>> >
>>> > If you do please keep it local. The bare minimum we have of
library
>>> > support is already disproportionately painful and prevents
easier
>>> sharing
>>> > with COFF. We should really not add more until the linker is
done.
>>>
>>> This is so much in contrast with the LLVM development, I find it
quite
>>> hard to see this as an acceptable position on llvm-dev.
>>>
>>
>> LLD is a subproject of the LLVM project, but as a product, LLD itself
is
>> not LLVM nor Clang, so some technical decisions that make sense to them
are
>> not directly be applicable or even inappropriate. As a person who spent
>> almost two years on the old LLD and 1.5 years on the new LLD, I can say
>> that Rafael's stance on focusing on making a good linker first
really makes
>> sense. I can easily imagine that if we didn't focus on that, we
couldn't
>> make this much progress over the past 1.5 year and would be stagnated
at a
>> very basic level. Do you know if I'm a person who worked really
hard on the
>> old (and probably "modular" whatever it means) linker so
hard? I'm speaking
>> based on the experience. If you have an concrete idea how to construct
a
>> linker from smaller modules,  please tell me. I still don't get
what you
>> want. We can discuss concrete proposals, but "making it (more)
modular" is
>> too vague and not really a proposal, so it cannot be a productive
>> discussion.
>>
>> That said, I think the current our "API" to allow users call
our linker's
>> main function hit the sweet spot. I know at least a few LLVM-based
language
>> developers who want to eliminate external dependencies and embed a
linker
>> to their compilers. That's a reasonable usage, and I think allowing
them to
>> pass a map from filename to MemoryBuffer objects makes sense, too. That
>> would be done without affecting the overall linker architecture. I
don't
>> oppose to that idea, and if someone wrote a patch, I'm fine with
that.
>>
>>
>> I’m totally willing to believe you that it is not possible to write the
>> fastest ELF linker on earth (or in the universe) with a library based
and
>> reusable components approach. But clang is not the fastest C/C++
compiler
>> available, and LLVM is not the fastest compiler framework either!
>>
>> So as a project, it seems to me that LLVM has not put the tradeoff on
the
>> speed/efficiency historically when it was to the detriment of
>> layering/component/modularity/reusability/…
>>
>> Writing the fastest linker possible is nice goal, I regret that a LLVM
>> subproject is putting this goal above
layering/component/modularity/reusability/…
>> though.
>>
>
> I've never mentioned that creating the fastest linker is the only goal.
>
>
> I believe this has clearly been put *ahead* the other design aspects I
> mentioned, isn’t it?
>
> Medhi, please tell how you would *actually* layer linkers with
> fine-grained components.
>
>
> That’s not a bait I’m gonna bite.
>
That's not a bait... I guess you are proposing a different architecture, so
you need to explain it.
> You are just saying that the current LLD is worse than an imaginary
> super-beautiful linker.
>
>
> Using superlative and trying to qualify what I’m writing with
"imaginary
> super-beautiful linker” is just caricatural and diverting the whole point.
> LLVM is quite far from an "imaginary super-beautiful compiler
framework”,
> yet it is modular and usable as a library.
>
> —
> Mehdi
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/14254788/attachment-0001.html>

> On Dec 13, 2016, at 11:30 AM, Rui Ueyama <ruiu at google.com> wrote:
> 
> On Tue, Dec 13, 2016 at 11:23 AM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> On Dec 13, 2016, at 11:08 AM, Rui Ueyama <ruiu at google.com
<mailto:ruiu at google.com>> wrote:
>> 
>> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at
apple.com <mailto:mehdi.amini at apple.com>> wrote:
>> 
>>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com
<mailto:ruiu at google.com>> wrote:
>>> 
>>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>> 
>>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>> >
>>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>> writes:
>>> >> This will also greatly facilitate certain measurements
I'd like to do
>>> >> w.r.t. different strategies for avoiding memory costs for
input files (esp.
>>> >> minor faults and dTLB costs). I've almost gotten to
the point of
>>> >> implementing this just to do those measurements.
>>> >
>>> > If you do please keep it local. The bare minimum we have of
library
>>> > support is already disproportionately painful and prevents
easier sharing
>>> > with COFF. We should really not add more until the linker is
done.
>>> 
>>> This is so much in contrast with the LLVM development, I find it
quite hard to see this as an acceptable position on llvm-dev.
>>> 
>>> LLD is a subproject of the LLVM project, but as a product, LLD
itself is not LLVM nor Clang, so some technical decisions that make sense to
them are not directly be applicable or even inappropriate. As a person who spent
almost two years on the old LLD and 1.5 years on the new LLD, I can say that
Rafael's stance on focusing on making a good linker first really makes
sense. I can easily imagine that if we didn't focus on that, we couldn't
make this much progress over the past 1.5 year and would be stagnated at a very
basic level. Do you know if I'm a person who worked really hard on the old
(and probably "modular" whatever it means) linker so hard? I'm
speaking based on the experience. If you have an concrete idea how to construct
a linker from smaller modules,  please tell me. I still don't get what you
want. We can discuss concrete proposals, but "making it (more)
modular" is too vague and not really a proposal, so it cannot be a
productive discussion.
>>> 
>>> That said, I think the current our "API" to allow users
call our linker's main function hit the sweet spot. I know at least a few
LLVM-based language developers who want to eliminate external dependencies and
embed a linker to their compilers. That's a reasonable usage, and I think
allowing them to pass a map from filename to MemoryBuffer objects makes sense,
too. That would be done without affecting the overall linker architecture. I
don't oppose to that idea, and if someone wrote a patch, I'm fine with
that.
>> 
>> 
>> I’m totally willing to believe you that it is not possible to write the
fastest ELF linker on earth (or in the universe) with a library based and
reusable components approach. But clang is not the fastest C/C++ compiler
available, and LLVM is not the fastest compiler framework either!
>> 
>> So as a project, it seems to me that LLVM has not put the tradeoff on
the speed/efficiency historically when it was to the detriment of
layering/component/modularity/reusability/…
>> 
>> Writing the fastest linker possible is nice goal, I regret that a LLVM
subproject is putting this goal above
layering/component/modularity/reusability/… though.
>> 
>> I've never mentioned that creating the fastest linker is the only
goal.
> 
> I believe this has clearly been put *ahead* the other design aspects I
mentioned, isn’t it?
> 
>> Medhi, please tell how you would *actually* layer linkers with
fine-grained components.
> 
> That’s not a bait I’m gonna bite.
> 
> That's not a bait... I guess you are proposing a different
architecture, so you need to explain it.
That’s a bait in the sense that I’m not having two months to dig into the Elf
lld and write a design document/proposal for the sole purpose of making a point.
And me not doing this does not impact and is not relevant to the discussion at
stance..

>> You are just saying that the current LLD is worse than an imaginary
super-beautiful linker.
> 
> Using superlative and trying to qualify what I’m writing with
"imaginary super-beautiful linker” is just caricatural and diverting the
whole point. LLVM is quite far from an "imaginary super-beautiful compiler
framework”, yet it is modular and usable as a library.
> 
> — 
> Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/b9907a4c/attachment.html>

On Tue, Dec 13, 2016 at 11:37 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Dec 13, 2016, at 11:30 AM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Tue, Dec 13, 2016 at 11:23 AM, Mehdi Amini <mehdi.amini at
apple.com>
> wrote:
>
>>
>> On Dec 13, 2016, at 11:08 AM, Rui Ueyama <ruiu at google.com>
wrote:
>>
>> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at
apple.com>
>> wrote:
>>
>>>
>>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com>
wrote:
>>>
>>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at
apple.com>
>>> wrote:
>>>
>>>>
>>>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via
llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>> >
>>>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org>
writes:
>>>> >> This will also greatly facilitate certain measurements
I'd like to do
>>>> >> w.r.t. different strategies for avoiding memory costs
for input
>>>> files (esp.
>>>> >> minor faults and dTLB costs). I've almost gotten
to the point of
>>>> >> implementing this just to do those measurements.
>>>> >
>>>> > If you do please keep it local. The bare minimum we have
of library
>>>> > support is already disproportionately painful and prevents
easier
>>>> sharing
>>>> > with COFF. We should really not add more until the linker
is done.
>>>>
>>>> This is so much in contrast with the LLVM development, I find
it quite
>>>> hard to see this as an acceptable position on llvm-dev.
>>>>
>>>
>>> LLD is a subproject of the LLVM project, but as a product, LLD
itself is
>>> not LLVM nor Clang, so some technical decisions that make sense to
them are
>>> not directly be applicable or even inappropriate. As a person who
spent
>>> almost two years on the old LLD and 1.5 years on the new LLD, I can
say
>>> that Rafael's stance on focusing on making a good linker first
really makes
>>> sense. I can easily imagine that if we didn't focus on that, we
couldn't
>>> make this much progress over the past 1.5 year and would be
stagnated at a
>>> very basic level. Do you know if I'm a person who worked really
hard on the
>>> old (and probably "modular" whatever it means) linker so
hard? I'm speaking
>>> based on the experience. If you have an concrete idea how to
construct a
>>> linker from smaller modules,  please tell me. I still don't get
what you
>>> want. We can discuss concrete proposals, but "making it (more)
modular" is
>>> too vague and not really a proposal, so it cannot be a productive
>>> discussion.
>>>
>>> That said, I think the current our "API" to allow users
call our
>>> linker's main function hit the sweet spot. I know at least a
few LLVM-based
>>> language developers who want to eliminate external dependencies and
embed a
>>> linker to their compilers. That's a reasonable usage, and I
think allowing
>>> them to pass a map from filename to MemoryBuffer objects makes
sense, too.
>>> That would be done without affecting the overall linker
architecture. I
>>> don't oppose to that idea, and if someone wrote a patch,
I'm fine with that.
>>>
>>>
>>> I’m totally willing to believe you that it is not possible to write
the
>>> fastest ELF linker on earth (or in the universe) with a library
based and
>>> reusable components approach. But clang is not the fastest C/C++
compiler
>>> available, and LLVM is not the fastest compiler framework either!
>>>
>>> So as a project, it seems to me that LLVM has not put the tradeoff
on
>>> the speed/efficiency historically when it was to the detriment of
>>> layering/component/modularity/reusability/…
>>>
>>> Writing the fastest linker possible is nice goal, I regret that a
LLVM
>>> subproject is putting this goal above
layering/component/modularity/reusability/…
>>> though.
>>>
>>
>> I've never mentioned that creating the fastest linker is the only
goal.
>>
>>
>> I believe this has clearly been put *ahead* the other design aspects I
>> mentioned, isn’t it?
>>
>> Medhi, please tell how you would *actually* layer linkers with
>> fine-grained components.
>>
>>
>> That’s not a bait I’m gonna bite.
>>
>
> That's not a bait... I guess you are proposing a different
architecture,
> so you need to explain it.
>
>
> That’s a bait in the sense that I’m not having two months to dig into the
> Elf lld and write a design document/proposal for the sole purpose of making
> a point. And me not doing this does not impact and is not relevant to the
> discussion at stance..
>
If it takes two months for you to investigate and make a proposal, why are
you so confident about the conclusion of the investigation that what you
think (I still don't get what it is) is doable? I actually dug into the old
LLD for two years (not months) with the hope that there's a good way of
make it work but failed.

You are just saying that the current LLD is worse than an
imaginary
>> super-beautiful linker.
>>
>>
>> Using superlative and trying to qualify what I’m writing with
"imaginary
>> super-beautiful linker” is just caricatural and diverting the whole
point.
>> LLVM is quite far from an "imaginary super-beautiful compiler
framework”,
>> yet it is modular and usable as a library.
>>
>> —
>> Mehdi
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/fe7d2e92/attachment.html>

>> That's not a bait... I guess you are proposing a different
architecture, so you need to explain it.
>
> That’s a bait in the sense that I’m not having two months to dig into the
Elf lld and write a design document/proposal for the sole purpose of making a
point. And me not doing this does not impact and is not relevant to the
discussion at stance..
The people that had 18 months to do it came up with this design. Sorry
if you don't like it.

Cheers,
Rafael