I think I have an idea to cover your need and possibly other people's on this thread. It provides the "main() as a library function" feature, input/output files wouldn't go through disks nor file systems, and it doesn't require any major design changes. Sounds too good? That is, we can provide a function that takes command line parameters, do fork, and call the linker's main function. This may sound too simple, but I think that is fairly powerful. Because the child process has a copy(-on-write) of the parent's memory, it can read parent's in-memory object files directly with no overhead. The child can pass the resulting file back to the parent through a shared memory object (which we can obtain using shm_open or something like that). In addition to that, your main process gets a protection from linker's bugs thanks to the operating system's memory protection. But from the user's point of view, that is just a linker's main function that you can call and that works as expected. Even if we want to call exec for whatever reason, we can copy in-memory objects to shared memory objects and exec the linker, so the basic design should work in such case too. The function signature would be something like: bool link(ArrayRef<LinkerArg> CommandLineArgs, MemoryBuffer &OutputFile, std::string &ErrorMsg); where the return value indicates success/failure. LinkerArg is a union type of StringRef and MemoryBufferRef. The result is returned as OutputFile memory buffer. If it prints out any message, ErrorMsg will hold it. (I want to point out that the function "bool link(ArrayRef<const char*> args, raw_ostream& diagnostics, ArrayRef<unique_ptr<MemoryBuffer>> inputs)" doesn't work because the order of command line parameters matters. Some command line parameters, such as --whole-archive/--no-whole-archive, affects how files in between will be interpreted, so you can't separate command line parameters from a list of files.) I think this is a practical solution that we can do now. I can implement this for you. On Thu, Jan 21, 2016 at 9:28 PM, Arseny Kapoulkine < arseny.kapoulkine at gmail.com> wrote:> > In any case, I have simply wasted too much time on a thread with > someone with no patches on the new elf linker. It is really annoying that > you don't put effort into it and seem entitled to dictate its direction. > > Sorry about that. I was initially planning to work on a patch to enhance > the interface for new lld - hence my questions in the original post. Since > I learned that people writing the code for lld are hostile to the idea of > linker-as-a-library, error_code is treated as spaghetti (which would be > fine if LLVM used exceptions which it does not) and patches, even if > submitted, will not actually be reviewed in a timely manner, I'll try to > adapt my code to either not use lld or use lld-as-a-binary. > > I'm disappointed by all of this but obviously it's not my project so I > should not have a say in this. > > Thank you for your time. > > Arseny > > On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola < > rafael.espindola at gmail.com> wrote: > >> > Also, one of the other possible motivations of using LLD directly from >> Clang would be to avoid process overhead on operating systems where that is >> a much more significant part of the compile time cost. We could today >> actually take the fork out of the Clang driver because the Clang frontend >> *is* designed in this way. But we would also need LLD to work in this way. >> >> Then go change clang and send a patch for lld once you are done. It will >> be interested to see if you can measure a single fork in an entire build. >> >> Even better, please finish the new pass manager before working on clang >> forking cc1. >> >> In any case, I have simply wasted too much time on a thread with someone >> with no patches on the new elf linker. It is really annoying that you don't >> put effort into it and seem entitled to dictate its direction. >> >> If you want to kick us out of the llvm project, please start a thread on >> llvm-dev. >> >> If you want lld to be a library, figure out how to do it without >> sacrificing lld's productivity, error reporting and performance (no >> error_code spaghetti) and write a patch. Just don't expect it to be >> reviewed while we have actual missing features. >> >> I will go back to implementing the linker. >> >> Rafael >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/1b61c8cd/attachment.html>
On Windows fork() is not available. If exec() is used instead, process creation time is several times slower than Linux. This may be or not a problem dependeing on how lld is used. In general, on Windows the best solution is multi-threading. 2016-01-22 18:31 GMT+02:00 Rui Ueyama via llvm-dev <llvm-dev at lists.llvm.org> :> I think I have an idea to cover your need and possibly other people's on > this thread. It provides the "main() as a library function" feature, > input/output files wouldn't go through disks nor file systems, and it > doesn't require any major design changes. Sounds too good? > > That is, we can provide a function that takes command line parameters, do > fork, and call the linker's main function. > > This may sound too simple, but I think that is fairly powerful. Because > the child process has a copy(-on-write) of the parent's memory, it can read > parent's in-memory object files directly with no overhead. The child can > pass the resulting file back to the parent through a shared memory object > (which we can obtain using shm_open or something like that). In addition to > that, your main process gets a protection from linker's bugs thanks to the > operating system's memory protection. But from the user's point of view, > that is just a linker's main function that you can call and that works as > expected. > > Even if we want to call exec for whatever reason, we can copy in-memory > objects to shared memory objects and exec the linker, so the basic design > should work in such case too. > > The function signature would be something like: > > bool link(ArrayRef<LinkerArg> CommandLineArgs, MemoryBuffer &OutputFile, > std::string &ErrorMsg); > > where the return value indicates success/failure. LinkerArg is a union > type of StringRef and MemoryBufferRef. The result is returned as OutputFile > memory buffer. If it prints out any message, ErrorMsg will hold it. > > (I want to point out that the function "bool link(ArrayRef<const char*> > args, raw_ostream& diagnostics, ArrayRef<unique_ptr<MemoryBuffer>> > inputs)" doesn't work because the order of command line parameters matters. > Some command line parameters, such as --whole-archive/--no-whole-archive, > affects how files in between will be interpreted, so you can't separate > command line parameters from a list of files.) > > I think this is a practical solution that we can do now. I can implement > this for you. > > On Thu, Jan 21, 2016 at 9:28 PM, Arseny Kapoulkine < > arseny.kapoulkine at gmail.com> wrote: > >> > In any case, I have simply wasted too much time on a thread with >> someone with no patches on the new elf linker. It is really annoying that >> you don't put effort into it and seem entitled to dictate its direction. >> >> Sorry about that. I was initially planning to work on a patch to enhance >> the interface for new lld - hence my questions in the original post. Since >> I learned that people writing the code for lld are hostile to the idea of >> linker-as-a-library, error_code is treated as spaghetti (which would be >> fine if LLVM used exceptions which it does not) and patches, even if >> submitted, will not actually be reviewed in a timely manner, I'll try to >> adapt my code to either not use lld or use lld-as-a-binary. >> >> I'm disappointed by all of this but obviously it's not my project so I >> should not have a say in this. >> >> Thank you for your time. >> >> Arseny >> >> On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola < >> rafael.espindola at gmail.com> wrote: >> >>> > Also, one of the other possible motivations of using LLD directly from >>> Clang would be to avoid process overhead on operating systems where that is >>> a much more significant part of the compile time cost. We could today >>> actually take the fork out of the Clang driver because the Clang frontend >>> *is* designed in this way. But we would also need LLD to work in this way. >>> >>> Then go change clang and send a patch for lld once you are done. It will >>> be interested to see if you can measure a single fork in an entire build. >>> >>> Even better, please finish the new pass manager before working on clang >>> forking cc1. >>> >>> In any case, I have simply wasted too much time on a thread with someone >>> with no patches on the new elf linker. It is really annoying that you don't >>> put effort into it and seem entitled to dictate its direction. >>> >>> If you want to kick us out of the llvm project, please start a thread on >>> llvm-dev. >>> >>> If you want lld to be a library, figure out how to do it without >>> sacrificing lld's productivity, error reporting and performance (no >>> error_code spaghetti) and write a patch. Just don't expect it to be >>> reviewed while we have actual missing features. >>> >>> I will go back to implementing the linker. >>> >>> Rafael >>> >> >> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/7ed13646/attachment.html>
If you want to link ELF object files, you are likely to be using a Unix machine. I'm not trying to address all possible problems but suggesting a practical solution. On Fri, Jan 22, 2016 at 9:49 AM, Yaron Keren <yaron.keren at gmail.com> wrote:> On Windows fork() is not available. If exec() is used instead, process > creation time is several times slower than Linux. This may be or not a > problem dependeing on how lld is used. In general, on Windows the best > solution is multi-threading. > > > 2016-01-22 18:31 GMT+02:00 Rui Ueyama via llvm-dev < > llvm-dev at lists.llvm.org>: > >> I think I have an idea to cover your need and possibly other people's on >> this thread. It provides the "main() as a library function" feature, >> input/output files wouldn't go through disks nor file systems, and it >> doesn't require any major design changes. Sounds too good? >> >> That is, we can provide a function that takes command line parameters, do >> fork, and call the linker's main function. >> >> This may sound too simple, but I think that is fairly powerful. Because >> the child process has a copy(-on-write) of the parent's memory, it can read >> parent's in-memory object files directly with no overhead. The child can >> pass the resulting file back to the parent through a shared memory object >> (which we can obtain using shm_open or something like that). In addition to >> that, your main process gets a protection from linker's bugs thanks to the >> operating system's memory protection. But from the user's point of view, >> that is just a linker's main function that you can call and that works as >> expected. >> >> Even if we want to call exec for whatever reason, we can copy in-memory >> objects to shared memory objects and exec the linker, so the basic design >> should work in such case too. >> >> The function signature would be something like: >> >> bool link(ArrayRef<LinkerArg> CommandLineArgs, MemoryBuffer >> &OutputFile, std::string &ErrorMsg); >> >> where the return value indicates success/failure. LinkerArg is a union >> type of StringRef and MemoryBufferRef. The result is returned as OutputFile >> memory buffer. If it prints out any message, ErrorMsg will hold it. >> >> (I want to point out that the function "bool link(ArrayRef<const char*> >> args, raw_ostream& diagnostics, ArrayRef<unique_ptr<MemoryBuffer>> >> inputs)" doesn't work because the order of command line parameters matters. >> Some command line parameters, such as --whole-archive/--no-whole-archive, >> affects how files in between will be interpreted, so you can't separate >> command line parameters from a list of files.) >> >> I think this is a practical solution that we can do now. I can implement >> this for you. >> >> On Thu, Jan 21, 2016 at 9:28 PM, Arseny Kapoulkine < >> arseny.kapoulkine at gmail.com> wrote: >> >>> > In any case, I have simply wasted too much time on a thread with >>> someone with no patches on the new elf linker. It is really annoying that >>> you don't put effort into it and seem entitled to dictate its direction. >>> >>> Sorry about that. I was initially planning to work on a patch to enhance >>> the interface for new lld - hence my questions in the original post. Since >>> I learned that people writing the code for lld are hostile to the idea of >>> linker-as-a-library, error_code is treated as spaghetti (which would be >>> fine if LLVM used exceptions which it does not) and patches, even if >>> submitted, will not actually be reviewed in a timely manner, I'll try to >>> adapt my code to either not use lld or use lld-as-a-binary. >>> >>> I'm disappointed by all of this but obviously it's not my project so I >>> should not have a say in this. >>> >>> Thank you for your time. >>> >>> Arseny >>> >>> On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola < >>> rafael.espindola at gmail.com> wrote: >>> >>>> > Also, one of the other possible motivations of using LLD directly >>>> from Clang would be to avoid process overhead on operating systems where >>>> that is a much more significant part of the compile time cost. We could >>>> today actually take the fork out of the Clang driver because the Clang >>>> frontend *is* designed in this way. But we would also need LLD to work in >>>> this way. >>>> >>>> Then go change clang and send a patch for lld once you are done. It >>>> will be interested to see if you can measure a single fork in an entire >>>> build. >>>> >>>> Even better, please finish the new pass manager before working on clang >>>> forking cc1. >>>> >>>> In any case, I have simply wasted too much time on a thread with >>>> someone with no patches on the new elf linker. It is really annoying that >>>> you don't put effort into it and seem entitled to dictate its direction. >>>> >>>> If you want to kick us out of the llvm project, please start a thread >>>> on llvm-dev. >>>> >>>> If you want lld to be a library, figure out how to do it without >>>> sacrificing lld's productivity, error reporting and performance (no >>>> error_code spaghetti) and write a patch. Just don't expect it to be >>>> reviewed while we have actual missing features. >>>> >>>> I will go back to implementing the linker. >>>> >>>> Rafael >>>> >>> >>> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/53f7edce/attachment.html>
James Y Knight via llvm-dev
2016-Jan-22 19:24 UTC
[llvm-dev] lld: ELF/COFF main() interface
IMO, this thread has by now gone way beyond its useful life, and at this point it really seems to me like the best thing to do is to just stop, and continue on with development in the same direction as it's been going so far -- the pace of development has thus far been amazing, and it would be a HUGE shame to get in the way of that. But, I do think it would be worthwhile to keep Chandler's 3 points for being "library-compatible" in mind during development, even if the code does not actually adhere to it for now:> - Cannot rely on global state > - Cannot directly call "exit" (but can call "abort" for *programmer* > errors like asserts) > - Cannot leak memoryI'd suggest that it would be reasonable to reopen the discussion about making LLD be library-compatible again after a few months have gone by, preferably in a calmer fashion, and ideally reopened by someone presenting a draft patch/plan that starts towards implementing the necessary changes to accomplish that. It's not really possible to reason about the tradeoffs, without seeing what it might actually look like -- all we have now is "It'd make the code too ugly" vs "No it wouldn't, plus it's really important to do". There's basically no way to resolve that argument via further discussion. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/132433ca/attachment.html>