Rui Ueyama via llvm-dev
2016-Dec-13 21:08 UTC
[llvm-dev] LLD status update and performance chart
On Tue, Dec 13, 2016 at 12:40 PM, Hal Finkel <hfinkel at anl.gov> wrote:> ----- Original Message ----- > > From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev at lists.llvm.org> > > To: "Andrew Kelley" <superjoe30 at gmail.com>, "Rui Ueyama" < > ruiu at google.com> > > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > > Sent: Tuesday, December 13, 2016 2:30:41 PM > > Subject: Re: [llvm-dev] LLD status update and performance chart > > > > Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes: > > > > > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev < > > > llvm-dev at lists.llvm.org> wrote: > > >> > > >> That said, I think the current our "API" to allow users call our > > >> linker's > > >> main function hit the sweet spot. I know at least a few LLVM-based > > >> language > > >> developers who want to eliminate external dependencies and embed a > > >> linker > > >> to their compilers. That's a reasonable usage, and I think > > >> allowing them to > > >> pass a map from filename to MemoryBuffer objects makes sense, too. > > >> That > > >> would be done without affecting the overall linker architecture. I > > >> don't > > >> oppose to that idea, and if someone wrote a patch, I'm fine with > > >> that. > > >> > > > > > > As an LLVM-based language developer, this is exactly what I want to > > > do. In > > > short: > > > > > > * Avoid depending on an external binary > > > * Avoid a fork+exec > > > * Avoid unnecessary use of the filesystem > > > > > > Does this feature compromise progress on the linker binary? > > > > > > Would it be reasonable for this feature to exist in the next minor > > > version > > > release of lld? > > > > I would please ask for it to wait. It requires designing how the > > linker > > script parser and the thin archive system will fetch new files as > > they > > are found. > > Instead of just waiting, I think we should discuss design requirements. > Clang, for example, has a VFS layer too (in include/clang/Basic/VirtualFileSystem.h) > which seems independent from Clang. Maybe we want to move it into LLVM and > reuse it from both Clang and LLD. Would this VFS layer meet the > requirements for an LLD VFS layer too?Sorry my negligence, but what does Clang VFS layer provide? Looking at the header file, it looks to me that it provides functionalities to hook all file operations, so that you can run Clang against, well, a virtual file system that may not exist as a OS-level file system at all, for example. Is my understanding correct? As to a JIT linker, I once wrote a small JIT C compiler which creates an object file in memory and then jump to the entry point of the file after relocating it in memory. From that experience (and my experience of LLD), I'd think JIT linker is much easier to write than static linkers, because JIT linkers don't need to interact with dynamic linkers. Static linkers have to create data that will be interpreted by the dynamic linker, and that's one of the most complicated things in LLD (the other is the linker script, but that's a different story.) So I don't expect that there are many things that we can share between JIT linkers and static linkers. Of course there may be something that I don't know well about JIT, but that's my feeling at this moment.> -Hal > > > > > Also, please consider what is so bad about writing a file. If your > > language has anything like translation unites than most of the > > program > > should already be in .o files. > > > > Cheers, > > Rafael > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > -- > Hal Finkel > Lead, Compiler Technology and Programming Languages > Leadership Computing Facility > Argonne National Laboratory >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/8a1f0848/attachment-0001.html>
Rafael Avila de Espindola via llvm-dev
2016-Dec-13 21:24 UTC
[llvm-dev] LLD status update and performance chart
> As to a JIT linker, I once wrote a small JIT C compiler which creates an > object file in memory and then jump to the entry point of the file after > relocating it in memory. From that experience (and my experience of LLD), > I'd think JIT linker is much easier to write than static linkers, because > JIT linkers don't need to interact with dynamic linkers. Static linkers > have to create data that will be interpreted by the dynamic linker, and > that's one of the most complicated things in LLD (the other is the linker > script, but that's a different story.) So I don't expect that there are > many things that we can share between JIT linkers and static linkers. Of > course there may be something that I don't know well about JIT, but that's > my feeling at this moment.My feeling exactly. The existing JIT hurts itself by producing full .o files and then interpreting them. Even with that design it is not clear that we could reasonably share code. If we are wrong, hey, awesome, send the patch. Cheers, Rafael
Hal Finkel via llvm-dev
2016-Dec-13 21:33 UTC
[llvm-dev] LLD status update and performance chart
----- Original Message -----> From: "Rui Ueyama" <ruiu at google.com> > To: "Hal Finkel" <hfinkel at anl.gov> > Cc: "Rafael Avila de Espindola" <rafael.espindola at gmail.com>, > "llvm-dev" <llvm-dev at lists.llvm.org>, "Andrew Kelley" > <superjoe30 at gmail.com> > Sent: Tuesday, December 13, 2016 3:08:15 PM > Subject: Re: [llvm-dev] LLD status update and performance chart> On Tue, Dec 13, 2016 at 12:40 PM, Hal Finkel < hfinkel at anl.gov > > wrote:> > ----- Original Message ----- > > > > From: "Rafael Avila de Espindola via llvm-dev" < > > > llvm-dev at lists.llvm.org > > > > > To: "Andrew Kelley" < superjoe30 at gmail.com >, "Rui Ueyama" < > > > ruiu at google.com > > > > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org > > > > > Sent: Tuesday, December 13, 2016 2:30:41 PM > > > > Subject: Re: [llvm-dev] LLD status update and performance chart > > > > >> > > Andrew Kelley via llvm-dev < llvm-dev at lists.llvm.org > writes: > > > > > > > > > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev < > > > > > llvm-dev at lists.llvm.org > wrote: > > > > >> > > > > >> That said, I think the current our "API" to allow users call > > > >> our > > > > >> linker's > > > > >> main function hit the sweet spot. I know at least a few > > > >> LLVM-based > > > > >> language > > > > >> developers who want to eliminate external dependencies and > > > >> embed > > > >> a > > > > >> linker > > > > >> to their compilers. That's a reasonable usage, and I think > > > > >> allowing them to > > > > >> pass a map from filename to MemoryBuffer objects makes sense, > > > >> too. > > > > >> That > > > > >> would be done without affecting the overall linker > > > >> architecture. > > > >> I > > > > >> don't > > > > >> oppose to that idea, and if someone wrote a patch, I'm fine > > > >> with > > > > >> that. > > > > >> > > > > > > > > > > As an LLVM-based language developer, this is exactly what I > > > > want > > > > to > > > > > do. In > > > > > short: > > > > > > > > > > * Avoid depending on an external binary > > > > > * Avoid a fork+exec > > > > > * Avoid unnecessary use of the filesystem > > > > > > > > > > Does this feature compromise progress on the linker binary? > > > > > > > > > > Would it be reasonable for this feature to exist in the next > > > > minor > > > > > version > > > > > release of lld? > > > > > > > > I would please ask for it to wait. It requires designing how the > > > > linker > > > > script parser and the thin archive system will fetch new files as > > > > they > > > > are found. >> > Instead of just waiting, I think we should discuss design > > requirements. Clang, for example, has a VFS layer too (in > > include/clang/Basic/VirtualFileSystem.h) which seems independent > > from Clang. Maybe we want to move it into LLVM and reuse it from > > both Clang and LLD. Would this VFS layer meet the requirements for > > an LLD VFS layer too? > > Sorry my negligence, but what does Clang VFS layer provide? Looking > at the header file, it looks to me that it provides functionalities > to hook all file operations, so that you can run Clang against, > well, a virtual file system that may not exist as a OS-level file > system at all, for example. Is my understanding correct?Yes, I believe that is correct.> As to a JIT linker, I once wrote a small JIT C compiler which creates > an object file in memory and then jump to the entry point of the > file after relocating it in memory. From that experience (and my > experience of LLD), I'd think JIT linker is much easier to write > than static linkers, because JIT linkers don't need to interact with > dynamic linkers. Static linkers have to create data that will be > interpreted by the dynamic linker, and that's one of the most > complicated things in LLD (the other is the linker script, but > that's a different story.) So I don't expect that there are many > things that we can share between JIT linkers and static linkers. Of > course there may be something that I don't know well about JIT, but > that's my feeling at this moment.Right now, both parts of LLVM have code to process relocations, produces PLTs, etc. Those are target-specific things and it would be nice not to have to code them twice. Obviously, I could be oversimplifying the situation. -Hal> > -Hal >> > > > > > > Also, please consider what is so bad about writing a file. If > > > your > > > > language has anything like translation unites than most of the > > > > program > > > > should already be in .o files. > > > > > > > > Cheers, > > > > Rafael > > > > _______________________________________________ > > > > LLVM Developers mailing list > > > > llvm-dev at lists.llvm.org > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > >> > -- > > > Hal Finkel > > > Lead, Compiler Technology and Programming Languages > > > Leadership Computing Facility > > > Argonne National Laboratory >-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/3aa61fce/attachment.html>
Rui Ueyama via llvm-dev
2016-Dec-13 23:23 UTC
[llvm-dev] LLD status update and performance chart
On Tue, Dec 13, 2016 at 1:33 PM, Hal Finkel <hfinkel at anl.gov> wrote:> > > ------------------------------ > > *From: *"Rui Ueyama" <ruiu at google.com> > *To: *"Hal Finkel" <hfinkel at anl.gov> > *Cc: *"Rafael Avila de Espindola" <rafael.espindola at gmail.com>, > "llvm-dev" <llvm-dev at lists.llvm.org>, "Andrew Kelley" < > superjoe30 at gmail.com> > *Sent: *Tuesday, December 13, 2016 3:08:15 PM > > *Subject: *Re: [llvm-dev] LLD status update and performance chart > > On Tue, Dec 13, 2016 at 12:40 PM, Hal Finkel <hfinkel at anl.gov> wrote: > >> ----- Original Message ----- >> > From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev at lists.llvm.org >> > >> > To: "Andrew Kelley" <superjoe30 at gmail.com>, "Rui Ueyama" < >> ruiu at google.com> >> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> >> > Sent: Tuesday, December 13, 2016 2:30:41 PM >> > Subject: Re: [llvm-dev] LLD status update and performance chart >> > >> > Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes: >> > >> > > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev < >> > > llvm-dev at lists.llvm.org> wrote: >> > >> >> > >> That said, I think the current our "API" to allow users call our >> > >> linker's >> > >> main function hit the sweet spot. I know at least a few LLVM-based >> > >> language >> > >> developers who want to eliminate external dependencies and embed a >> > >> linker >> > >> to their compilers. That's a reasonable usage, and I think >> > >> allowing them to >> > >> pass a map from filename to MemoryBuffer objects makes sense, too. >> > >> That >> > >> would be done without affecting the overall linker architecture. I >> > >> don't >> > >> oppose to that idea, and if someone wrote a patch, I'm fine with >> > >> that. >> > >> >> > > >> > > As an LLVM-based language developer, this is exactly what I want to >> > > do. In >> > > short: >> > > >> > > * Avoid depending on an external binary >> > > * Avoid a fork+exec >> > > * Avoid unnecessary use of the filesystem >> > > >> > > Does this feature compromise progress on the linker binary? >> > > >> > > Would it be reasonable for this feature to exist in the next minor >> > > version >> > > release of lld? >> > >> > I would please ask for it to wait. It requires designing how the >> > linker >> > script parser and the thin archive system will fetch new files as >> > they >> > are found. >> >> Instead of just waiting, I think we should discuss design requirements. >> Clang, for example, has a VFS layer too (in include/clang/Basic/VirtualFileSystem.h) >> which seems independent from Clang. Maybe we want to move it into LLVM and >> reuse it from both Clang and LLD. Would this VFS layer meet the >> requirements for an LLD VFS layer too? > > > Sorry my negligence, but what does Clang VFS layer provide? Looking at the > header file, it looks to me that it provides functionalities to hook all > file operations, so that you can run Clang against, well, a virtual file > system that may not exist as a OS-level file system at all, for example. Is > my understanding correct? > > Yes, I believe that is correct. >I took a look a bit more closely at the VFS layer. I can imagine that a few reasonable use cases of the abstraction. One is obviously what Andrew wants -- in-memory linking. That would be useful for distributed build system too. (You can virtualize a file system using fuse, but that's not cross-platform.) I don't remember who said that, but someone proposed me if LLD can support something like a "linker-driven build system", in which objects are created when a linker needs them. It is experimental so it wouldn't need a support in the main trunk, but a virtual filesystem would allow such system.> > As to a JIT linker, I once wrote a small JIT C compiler which creates an > object file in memory and then jump to the entry point of the file after > relocating it in memory. From that experience (and my experience of LLD), > I'd think JIT linker is much easier to write than static linkers, because > JIT linkers don't need to interact with dynamic linkers. Static linkers > have to create data that will be interpreted by the dynamic linker, and > that's one of the most complicated things in LLD (the other is the linker > script, but that's a different story.) So I don't expect that there are > many things that we can share between JIT linkers and static linkers. Of > course there may be something that I don't know well about JIT, but that's > my feeling at this moment. > > Right now, both parts of LLVM have code to process relocations, produces > PLTs, etc. Those are target-specific things and it would be nice not to > have to code them twice. Obviously, I could be oversimplifying the > situation. > > -Hal > > > >> -Hal >> >> > >> > Also, please consider what is so bad about writing a file. If your >> > language has anything like translation unites than most of the >> > program >> > should already be in .o files. >> > >> > Cheers, >> > Rafael >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >> >> -- >> Hal Finkel >> Lead, Compiler Technology and Programming Languages >> Leadership Computing Facility >> Argonne National Laboratory >> > > > > > -- > Hal Finkel > Lead, Compiler Technology and Programming Languages > Leadership Computing Facility > Argonne National Laboratory >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/69f17bee/attachment.html>
Philip Reames via llvm-dev
2016-Dec-14 04:34 UTC
[llvm-dev] LLD status update and performance chart
On 12/13/2016 01:24 PM, Rafael Avila de Espindola via llvm-dev wrote:>> As to a JIT linker, I once wrote a small JIT C compiler which creates an >> object file in memory and then jump to the entry point of the file after >> relocating it in memory. From that experience (and my experience of LLD), >> I'd think JIT linker is much easier to write than static linkers, because >> JIT linkers don't need to interact with dynamic linkers. Static linkers >> have to create data that will be interpreted by the dynamic linker, and >> that's one of the most complicated things in LLD (the other is the linker >> script, but that's a different story.) So I don't expect that there are >> many things that we can share between JIT linkers and static linkers. Of >> course there may be something that I don't know well about JIT, but that's >> my feeling at this moment. > My feeling exactly. The existing JIT hurts itself by producing full .o > files and then interpreting them.As a user of the JIT infrastructure, I completely and utterly disagree with this position. The "in-memory compiler" architecture has some downsides (e.g. compile time), but overall is one of the designs MCJIT got right from a software design, testability, and flexibility perspective.> Even with that design it is not clear > that we could reasonably share code. If we are wrong, hey, awesome, send > the patch.