llvm dev - Dec 2016 - LLD status update and performance chart

David Chisnall via llvm-dev <llvm-dev at lists.llvm.org> writes:
> On 12 Dec 2016, at 03:39, Rui Ueyama via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> LLD's driver currently takes only a command line argument strings,
so you need to write ELF files to a filesystem and read them back at the moment.
>> 
>> Theoretically, we could accept a list of command line strings and
MemoryBuffer objects (instead of filenames) so that we can link in-memory ELF
object files. Adding that feature shouldn't be hard.
>
> It would be nice to have a similar interface to the one in libclang, where
the driver is given a map that substitutes in-memory objects for specific files,
so that the same command line can be used whether the inputs are on disk or in
memory.
I would strongly object to expanding our "library" interface at the
moment.

Even what we currently have I think is a bad idea and a distraction from
the current objective: writing an excellent stand alone linker. We
benefit massively from simple error and memory management.

Once the linker is "done", we can discuss other possible uses and
their
costs. By "done" I mean at least having all the features we want and
having replaced bfd ld in at least two freebsd architectures.

Cheers,
Rafael

On Mon, Dec 12, 2016 at 11:13 AM, Rafael Avila de Espindola via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> David Chisnall via llvm-dev <llvm-dev at lists.llvm.org> writes:
>
> > On 12 Dec 2016, at 03:39, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >>
> >> LLD's driver currently takes only a command line argument
strings, so
> you need to write ELF files to a filesystem and read them back at the
> moment.
> >>
> >> Theoretically, we could accept a list of command line strings and
> MemoryBuffer objects (instead of filenames) so that we can link in-memory
> ELF object files. Adding that feature shouldn't be hard.
> >
> > It would be nice to have a similar interface to the one in libclang,
> where the driver is given a map that substitutes in-memory objects for
> specific files, so that the same command line can be used whether the
> inputs are on disk or in memory.
>
> I would strongly object to expanding our "library" interface at
the
> moment.
>
> Even what we currently have I think is a bad idea and a distraction from
> the current objective: writing an excellent stand alone linker. We
> benefit massively from simple error and memory management.
>
> Once the linker is "done", we can discuss other possible uses and
their
> costs. By "done" I mean at least having all the features we want
and
> having replaced bfd ld in at least two freebsd architectures.
>
It seems reasonable to me to focus on this first, but do you agree that
once lld gets to this milestone, the library API is a worthy time
investment?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161212/1e3d7860/attachment.html>

On Mon, Dec 12, 2016 at 11:42 AM, Andrew Kelley <superjoe30 at gmail.com>
wrote:
>
>
> On Mon, Dec 12, 2016 at 11:13 AM, Rafael Avila de Espindola via llvm-dev
<
> llvm-dev at lists.llvm.org> wrote:
>
>> David Chisnall via llvm-dev <llvm-dev at lists.llvm.org> writes:
>>
>> > On 12 Dec 2016, at 03:39, Rui Ueyama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> >>
>> >> LLD's driver currently takes only a command line argument
strings, so
>> you need to write ELF files to a filesystem and read them back at the
>> moment.
>> >>
>> >> Theoretically, we could accept a list of command line strings
and
>> MemoryBuffer objects (instead of filenames) so that we can link
in-memory
>> ELF object files. Adding that feature shouldn't be hard.
>> >
>> > It would be nice to have a similar interface to the one in
libclang,
>> where the driver is given a map that substitutes in-memory objects for
>> specific files, so that the same command line can be used whether the
>> inputs are on disk or in memory.
>>
>> I would strongly object to expanding our "library" interface
at the
>> moment.
>>
>> Even what we currently have I think is a bad idea and a distraction
from
>> the current objective: writing an excellent stand alone linker. We
>> benefit massively from simple error and memory management.
>>
>> Once the linker is "done", we can discuss other possible uses
and their
>> costs. By "done" I mean at least having all the features we
want and
>> having replaced bfd ld in at least two freebsd architectures.
>>
>
> It seems reasonable to me to focus on this first, but do you agree that
> once lld gets to this milestone, the library API is a worthy time
> investment?
>
I'm fine as long as our API is the same as the command line. In that sense,
passing a map from filename to MemoryBuffer objects in addition to command
line arguments is fine to me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161212/110efb55/attachment.html>

> It seems reasonable to me to focus on this first, but do you agree that
> once lld gets to this milestone, the library API is a worthy time
> investment?
It is not something I have an interest on, but I wouldn't object to it
being done if it doesn't regress performance or complicate the linker
too much.

Cheers,
Rafael

On Mon, Dec 12, 2016 at 8:13 AM, Rafael Avila de Espindola via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> David Chisnall via llvm-dev <llvm-dev at lists.llvm.org> writes:
>
> > On 12 Dec 2016, at 03:39, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >>
> >> LLD's driver currently takes only a command line argument
strings, so
> you need to write ELF files to a filesystem and read them back at the
> moment.
> >>
> >> Theoretically, we could accept a list of command line strings and
> MemoryBuffer objects (instead of filenames) so that we can link in-memory
> ELF object files. Adding that feature shouldn't be hard.
> >
> > It would be nice to have a similar interface to the one in libclang,
> where the driver is given a map that substitutes in-memory objects for
> specific files, so that the same command line can be used whether the
> inputs are on disk or in memory.
>
> I would strongly object to expanding our "library" interface at
the
> moment.
>
> Even what we currently have I think is a bad idea and a distraction from
> the current objective: writing an excellent stand alone linker. We
> benefit massively from simple error and memory management.
>
> Once the linker is "done", we can discuss other possible uses and
their
> costs. By "done" I mean at least having all the features we want
and
> having replaced bfd ld in at least two freebsd architectures.
>
What are some of the missing "features we want"? gdb-index I assume?
Any
others?

-- Sean Silva

>
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/ce9a4b50/attachment.html>

>> Once the linker is "done", we can discuss other possible uses
and their
>> costs. By "done" I mean at least having all the features we
want and
>> having replaced bfd ld in at least two freebsd architectures.
>>
>
> What are some of the missing "features we want"? gdb-index I
assume? Any
> others?
I am on PTO, so slow to respond.

Some items that are left:

* Debug fission
* Single file debug fission
* Range extension thunks
* All of freebsd links and works
* Very good performance when all that is in

These have the potential to require changes to the linker and I really don't
want the linker to be locked with other parts of llvm until the above
is done.

If I had shared the first relocation processing design with the JIT,
writing something like r266158 would have been way harder. In other
words, the linker would have been less maintainable. The same is true
for the effort to use synthetic input section. Had the old synthetic
output sections be shared with something, it would have been way harder
to change.

Once the linker is done, it should be possible for someone to see if
there is enough overlap that can be refactored and at what cost (in
readability, maintainability and performance). Before that it is just
guesswork and I am opposed to it.

If someone wants to work on it now, there is always the MachO linker
that should be "modular" already.

Cheers,
Rafael