llvm dev - Dec 2016 - LLD status update and performance chart

Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes:
> On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>>
>> That said, I think the current our "API" to allow users call
our linker's
>> main function hit the sweet spot. I know at least a few LLVM-based
language
>> developers who want to eliminate external dependencies and embed a
linker
>> to their compilers. That's a reasonable usage, and I think allowing
them to
>> pass a map from filename to MemoryBuffer objects makes sense, too. That
>> would be done without affecting the overall linker architecture. I
don't
>> oppose to that idea, and if someone wrote a patch, I'm fine with
that.
>>
>
> As an LLVM-based language developer, this is exactly what I want to do. In
> short:
>
>  * Avoid depending on an external binary
>  * Avoid a fork+exec
>  * Avoid unnecessary use of the filesystem
>
> Does this feature compromise progress on the linker binary?
>
> Would it be reasonable for this feature to exist in the next minor version
> release of lld?
I would please ask for it to wait. It requires designing how the linker
script parser and the thin archive system will fetch new files as they
are found.

Also, please consider what is so bad about writing a file. If your
language has anything like translation unites than most of the program
should already be in .o files.

Cheers,
Rafael

----- Original Message -----
> From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev at
lists.llvm.org>
> To: "Andrew Kelley" <superjoe30 at gmail.com>, "Rui
Ueyama" <ruiu at google.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Tuesday, December 13, 2016 2:30:41 PM
> Subject: Re: [llvm-dev] LLD status update and performance chart
> 
> Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes:
> 
> > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev <
> > llvm-dev at lists.llvm.org> wrote:
> >>
> >> That said, I think the current our "API" to allow users
call our
> >> linker's
> >> main function hit the sweet spot. I know at least a few LLVM-based
> >> language
> >> developers who want to eliminate external dependencies and embed a
> >> linker
> >> to their compilers. That's a reasonable usage, and I think
> >> allowing them to
> >> pass a map from filename to MemoryBuffer objects makes sense, too.
> >> That
> >> would be done without affecting the overall linker architecture. I
> >> don't
> >> oppose to that idea, and if someone wrote a patch, I'm fine
with
> >> that.
> >>
> >
> > As an LLVM-based language developer, this is exactly what I want to
> > do. In
> > short:
> >
> >  * Avoid depending on an external binary
> >  * Avoid a fork+exec
> >  * Avoid unnecessary use of the filesystem
> >
> > Does this feature compromise progress on the linker binary?
> >
> > Would it be reasonable for this feature to exist in the next minor
> > version
> > release of lld?
> 
> I would please ask for it to wait. It requires designing how the
> linker
> script parser and the thin archive system will fetch new files as
> they
> are found.
Instead of just waiting, I think we should discuss design requirements. Clang,
for example, has a VFS layer too (in include/clang/Basic/VirtualFileSystem.h)
which seems independent from Clang. Maybe we want to move it into LLVM and reuse
it from both Clang and LLD. Would this VFS layer meet the requirements for an
LLD VFS layer too?

 -Hal
> 
> Also, please consider what is so bad about writing a file. If your
> language has anything like translation unites than most of the
> program
> should already be in .o files.
> 
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On Tue, Dec 13, 2016 at 3:30 PM, Rafael Avila de Espindola <
rafael.espindola at gmail.com> wrote:
> Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes:
>
> > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev <
> > llvm-dev at lists.llvm.org> wrote:
> >>
> >> That said, I think the current our "API" to allow users
call our
> linker's
> >> main function hit the sweet spot. I know at least a few LLVM-based
> language
> >> developers who want to eliminate external dependencies and embed a
> linker
> >> to their compilers. That's a reasonable usage, and I think
allowing
> them to
> >> pass a map from filename to MemoryBuffer objects makes sense, too.
That
> >> would be done without affecting the overall linker architecture. I
don't
> >> oppose to that idea, and if someone wrote a patch, I'm fine
with that.
> >>
> >
> > As an LLVM-based language developer, this is exactly what I want to
do.
> In
> > short:
> >
> >  * Avoid depending on an external binary
> >  * Avoid a fork+exec
> >  * Avoid unnecessary use of the filesystem
> >
> > Does this feature compromise progress on the linker binary?
> >
> > Would it be reasonable for this feature to exist in the next minor
> version
> > release of lld?
>
> I would please ask for it to wait. It requires designing how the linker
> script parser and the thin archive system will fetch new files as they
> are found.
>
That sounds fair. I'm thankful for all the hard work the LLD team is
putting into the project, and it's not my place to demand any features.

> Also, please consider what is so bad about writing a file. If your
> language has anything like translation unites than most of the program
> should already be in .o files.
>
I see your point. It's true that "Avoid unnecessary use of the
filesystem",
while still a correct way to design software, is the least important of
those three points. "Avoid depending on an external binary" is most of
what
I want to get out of the LLD project.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/83565316/attachment.html>

On Tue, Dec 13, 2016 at 12:40 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> ----- Original Message -----
> > From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev
at lists.llvm.org>
> > To: "Andrew Kelley" <superjoe30 at gmail.com>,
"Rui Ueyama" <
> ruiu at google.com>
> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> > Sent: Tuesday, December 13, 2016 2:30:41 PM
> > Subject: Re: [llvm-dev] LLD status update and performance chart
> >
> > Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes:
> >
> > > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev <
> > > llvm-dev at lists.llvm.org> wrote:
> > >>
> > >> That said, I think the current our "API" to allow
users call our
> > >> linker's
> > >> main function hit the sweet spot. I know at least a few
LLVM-based
> > >> language
> > >> developers who want to eliminate external dependencies and
embed a
> > >> linker
> > >> to their compilers. That's a reasonable usage, and I
think
> > >> allowing them to
> > >> pass a map from filename to MemoryBuffer objects makes sense,
too.
> > >> That
> > >> would be done without affecting the overall linker
architecture. I
> > >> don't
> > >> oppose to that idea, and if someone wrote a patch, I'm
fine with
> > >> that.
> > >>
> > >
> > > As an LLVM-based language developer, this is exactly what I want
to
> > > do. In
> > > short:
> > >
> > >  * Avoid depending on an external binary
> > >  * Avoid a fork+exec
> > >  * Avoid unnecessary use of the filesystem
> > >
> > > Does this feature compromise progress on the linker binary?
> > >
> > > Would it be reasonable for this feature to exist in the next
minor
> > > version
> > > release of lld?
> >
> > I would please ask for it to wait. It requires designing how the
> > linker
> > script parser and the thin archive system will fetch new files as
> > they
> > are found.
>
> Instead of just waiting, I think we should discuss design requirements.
> Clang, for example, has a VFS layer too (in
include/clang/Basic/VirtualFileSystem.h)
> which seems independent from Clang. Maybe we want to move it into LLVM and
> reuse it from both Clang and LLD. Would this VFS layer meet the
> requirements for an LLD VFS layer too?

Sorry my negligence, but what does Clang VFS layer provide? Looking at the
header file, it looks to me that it provides functionalities to hook all
file operations, so that you can run Clang against, well, a virtual file
system that may not exist as a OS-level file system at all, for example. Is
my understanding correct?

As to a JIT linker, I once wrote a small JIT C compiler which creates an
object file in memory and then jump to the entry point of the file after
relocating it in memory. From that experience (and my experience of LLD),
I'd think JIT linker is much easier to write than static linkers, because
JIT linkers don't need to interact with dynamic linkers. Static linkers
have to create data that will be interpreted by the dynamic linker, and
that's one of the most complicated things in LLD (the other is the linker
script, but that's a different story.) So I don't expect that there are
many things that we can share between JIT linkers and static linkers. Of
course there may be something that I don't know well about JIT, but
that's
my feeling at this moment.

>  -Hal
>
> >
> > Also, please consider what is so bad about writing a file. If your
> > language has anything like translation unites than most of the
> > program
> > should already be in .o files.
> >
> > Cheers,
> > Rafael
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/8a1f0848/attachment-0001.html>

> Instead of just waiting, I think we should discuss design requirements.
Clang, for example, has a VFS layer too (in
include/clang/Basic/VirtualFileSystem.h) which seems independent from Clang.
Maybe we want to move it into LLVM and reuse it from both Clang and LLD. Would
this VFS layer meet the requirements for an LLD VFS layer too?
That is not something I will be spending time at the moment.

Cheers,
Rafael