Arseny Kapoulkine via llvm-dev
2016-Jan-07 16:28 UTC
[llvm-dev] lld: ELF/COFF main() interface
This is really unfortunate. I've read the discussion threads for the atom/chunk controversy and I feel like I understand the reasons for rewriting the linker, but this does not seem to have anything to do with whether the linker is usable as a library or not. As it stands, not only does lld have two completely different linkers (I'm treating COFF/ELF2 as one since they are really two different implementations of the same concept, AFAIU), but one is usable as a library (and even does not require round-tripping generated code through an object file! I was really happy to use that) and the other one isn't. Not sure what the future plans are for Mach-O linker (at this point it seems logical to rewrite that using the new designs but I'm not sure if it ever happens), so maybe at some point we'll just have one linker application instead of a library and an application. Anyway, since linker is the only missing piece for full compilation stack (source language to runnable executable), it's sad to see this specific part of LLVM not working as a library when everything else does. Are there specific concerns in terms of implementation that prevent new lld from being a library? I understand that using global variables and error() function is simpler, but the rest of LLVM does not do that and the codebase there is significantly larger. Am I missing any other issues except for the ones I mentioned in my original e-mail that will come up in a library-like usage scenario? Arseny On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote:> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> In the process of migrating from old lld ELF linker to new (previously >> ELF2) I noticed the interface lost several important features (ordered by >> importance for my use case): >> >> 1. Detecting errors in the first place. New linker seems to call exit(1) >> for any error. >> >> 2. Reporting messages to non-stderr outputs. Previously all link >> functions had a raw_ostream argument so it was possible to delay the error >> output, aggregate it for multiple linked files, output via a different >> format, etc. >> >> 3. Linking multiple outputs in parallel (useful for test drivers) in a >> single process. Not really an interface issue but there are at least two >> global pointers (Config & Driver) that refer to stack variables and are >> used in various places in the code. >> >> All of this seems to indicate a departure from the linker being useable >> as a library. To maintain the previous behavior you'd have to use a linker >> binary & popen. >> >> Is this a conscious design decision or a temporary limitation? >> > > That the new ELF and COFF linkers are designed as commands instead of > libraries is very much an intended design change. > > >> If it's a limitation, how would one go about fixing this? I'm not too >> familiar with the idiomatical error handling in LLVM. Normally in this >> situation I'd just throw from error() but lld is probably compiled without >> exceptions. Is ErrorOr the established practice? How does LLVM generally >> deal with error handling for parsers (like linker script parser) where it's >> a lot of mutually recursive functions where every single one can fail? >> > > Since it's as designed, please run the linker as an external command using > fork/exec (or wrappers of them) instead of trying to use that inside the > same process. > > >> Arseny >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/c28d3f18/attachment.html>
Designing it a command makes things simpler because you can safely assume that most functions always success, or otherwise the entire process terminates. You can defer the operating system to clean up all resources that was used by the process however it fails (with a few exceptions such as temporary files.) I actually like to use the linker as an external command since the operating system provides good isolation between my process and the linker which could fail by an unknown bug or something. Rewriting all of them as functions that return ErrorOr<Something> is technically doable, but it needs strong justification, so I guess that unlikely to change. Why do you want to use that as a library? I don't think "because LLVM and Clang allow that" is not a compelling argument since the linker and the compiler are pretty different programs. I clearly see many reasons to use LLVM and part of Clang as libraries, but they are not directly applicable to LLD. The new linker is more like a command which is built on top of the libraries that LLVM provides. On Thu, Jan 7, 2016 at 8:28 AM, Arseny Kapoulkine < arseny.kapoulkine at gmail.com> wrote:> This is really unfortunate. > > I've read the discussion threads for the atom/chunk controversy and I feel > like I understand the reasons for rewriting the linker, but this does not > seem to have anything to do with whether the linker is usable as a library > or not. > > As it stands, not only does lld have two completely different linkers (I'm > treating COFF/ELF2 as one since they are really two different > implementations of the same concept, AFAIU), but one is usable as a library > (and even does not require round-tripping generated code through an object > file! I was really happy to use that) and the other one isn't. Not sure > what the future plans are for Mach-O linker (at this point it seems logical > to rewrite that using the new designs but I'm not sure if it ever happens), > so maybe at some point we'll just have one linker application instead of a > library and an application. > > Anyway, since linker is the only missing piece for full compilation stack > (source language to runnable executable), it's sad to see this specific > part of LLVM not working as a library when everything else does. > > Are there specific concerns in terms of implementation that prevent new > lld from being a library? I understand that using global variables and > error() function is simpler, but the rest of LLVM does not do that and the > codebase there is significantly larger. Am I missing any other issues > except for the ones I mentioned in my original e-mail that will come up in > a library-like usage scenario? > > Arseny > > On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote: > >> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >>> In the process of migrating from old lld ELF linker to new (previously >>> ELF2) I noticed the interface lost several important features (ordered by >>> importance for my use case): >>> >>> 1. Detecting errors in the first place. New linker seems to call exit(1) >>> for any error. >>> >>> 2. Reporting messages to non-stderr outputs. Previously all link >>> functions had a raw_ostream argument so it was possible to delay the error >>> output, aggregate it for multiple linked files, output via a different >>> format, etc. >>> >>> 3. Linking multiple outputs in parallel (useful for test drivers) in a >>> single process. Not really an interface issue but there are at least two >>> global pointers (Config & Driver) that refer to stack variables and are >>> used in various places in the code. >>> >>> All of this seems to indicate a departure from the linker being useable >>> as a library. To maintain the previous behavior you'd have to use a linker >>> binary & popen. >>> >>> Is this a conscious design decision or a temporary limitation? >>> >> >> That the new ELF and COFF linkers are designed as commands instead of >> libraries is very much an intended design change. >> >> >>> If it's a limitation, how would one go about fixing this? I'm not too >>> familiar with the idiomatical error handling in LLVM. Normally in this >>> situation I'd just throw from error() but lld is probably compiled without >>> exceptions. Is ErrorOr the established practice? How does LLVM generally >>> deal with error handling for parsers (like linker script parser) where it's >>> a lot of mutually recursive functions where every single one can fail? >>> >> >> Since it's as designed, please run the linker as an external command >> using fork/exec (or wrappers of them) instead of trying to use that inside >> the same process. >> >> >>> Arseny >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/67217070/attachment.html>
David Blaikie via llvm-dev
2016-Jan-07 18:26 UTC
[llvm-dev] lld: ELF/COFF main() interface
+Lang for discussions of lld-as-a-library Lang & I have discussed some uses of the linker as a library for reuse within the LLVM JIT, for example, which is being phrased more and more like an in-process version of traditional ahead-of-time compilation (which helps in a number of ways, eg: by having fewer oddities/quirks from a model people are more familiar with) (as for the cleanup issue - LLVM has ways of disabling explicit cleanup when it knows its in-process and is going to go away soon, I think - similar things could be done in LLD) On Thu, Jan 7, 2016 at 9:06 AM, Rui Ueyama via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Designing it a command makes things simpler because you can safely assume > that most functions always success, or otherwise the entire process > terminates. You can defer the operating system to clean up all resources > that was used by the process however it fails (with a few exceptions such > as temporary files.) I actually like to use the linker as an external > command since the operating system provides good isolation between my > process and the linker which could fail by an unknown bug or something. > Rewriting all of them as functions that return ErrorOr<Something> is > technically doable, but it needs strong justification, so I guess that > unlikely to change. > > Why do you want to use that as a library? I don't think "because LLVM and > Clang allow that" is not a compelling argument since the linker and the > compiler are pretty different programs. I clearly see many reasons to use > LLVM and part of Clang as libraries, but they are not directly applicable > to LLD. The new linker is more like a command which is built on top of the > libraries that LLVM provides. > > On Thu, Jan 7, 2016 at 8:28 AM, Arseny Kapoulkine < > arseny.kapoulkine at gmail.com> wrote: > >> This is really unfortunate. >> >> I've read the discussion threads for the atom/chunk controversy and I >> feel like I understand the reasons for rewriting the linker, but this does >> not seem to have anything to do with whether the linker is usable as a >> library or not. >> >> As it stands, not only does lld have two completely different linkers >> (I'm treating COFF/ELF2 as one since they are really two different >> implementations of the same concept, AFAIU), but one is usable as a library >> (and even does not require round-tripping generated code through an object >> file! I was really happy to use that) and the other one isn't. Not sure >> what the future plans are for Mach-O linker (at this point it seems logical >> to rewrite that using the new designs but I'm not sure if it ever happens), >> so maybe at some point we'll just have one linker application instead of a >> library and an application. >> >> Anyway, since linker is the only missing piece for full compilation stack >> (source language to runnable executable), it's sad to see this specific >> part of LLVM not working as a library when everything else does. >> >> Are there specific concerns in terms of implementation that prevent new >> lld from being a library? I understand that using global variables and >> error() function is simpler, but the rest of LLVM does not do that and the >> codebase there is significantly larger. Am I missing any other issues >> except for the ones I mentioned in my original e-mail that will come up in >> a library-like usage scenario? >> >> Arseny >> >> On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote: >> >>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev < >>> llvm-dev at lists.llvm.org> wrote: >>> >>>> In the process of migrating from old lld ELF linker to new (previously >>>> ELF2) I noticed the interface lost several important features (ordered by >>>> importance for my use case): >>>> >>>> 1. Detecting errors in the first place. New linker seems to call >>>> exit(1) for any error. >>>> >>>> 2. Reporting messages to non-stderr outputs. Previously all link >>>> functions had a raw_ostream argument so it was possible to delay the error >>>> output, aggregate it for multiple linked files, output via a different >>>> format, etc. >>>> >>>> 3. Linking multiple outputs in parallel (useful for test drivers) in a >>>> single process. Not really an interface issue but there are at least two >>>> global pointers (Config & Driver) that refer to stack variables and are >>>> used in various places in the code. >>>> >>>> All of this seems to indicate a departure from the linker being useable >>>> as a library. To maintain the previous behavior you'd have to use a linker >>>> binary & popen. >>>> >>>> Is this a conscious design decision or a temporary limitation? >>>> >>> >>> That the new ELF and COFF linkers are designed as commands instead of >>> libraries is very much an intended design change. >>> >>> >>>> If it's a limitation, how would one go about fixing this? I'm not too >>>> familiar with the idiomatical error handling in LLVM. Normally in this >>>> situation I'd just throw from error() but lld is probably compiled without >>>> exceptions. Is ErrorOr the established practice? How does LLVM generally >>>> deal with error handling for parsers (like linker script parser) where it's >>>> a lot of mutually recursive functions where every single one can fail? >>>> >>> >>> Since it's as designed, please run the linker as an external command >>> using fork/exec (or wrappers of them) instead of trying to use that inside >>> the same process. >>> >>> >>>> Arseny >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> llvm-dev at lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>> >>>> >>> >> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/d37164b5/attachment.html>
Eric Christopher via llvm-dev
2016-Jan-07 22:52 UTC
[llvm-dev] lld: ELF/COFF main() interface
There are lots of good reasons to have it as a library ranging from embedding it in processes as an in-process linker (as Dave mentions later in thread) to making it more implicitly testable, to being able to do complete build to executable/library/etc with a single command (this goes back to one, but is a more specific example). There are many good reasons - some of which are being worked on with a library based lld as a prerequisite to have lld be a library based linker. Rewriting every function as ErrorOr sounds terrible and we should avoid that as much as possible, but keeping the general llvm style of "library first" seems to be an important use case. -eric On Thu, Jan 7, 2016 at 9:06 AM Rui Ueyama via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Designing it a command makes things simpler because you can safely assume > that most functions always success, or otherwise the entire process > terminates. You can defer the operating system to clean up all resources > that was used by the process however it fails (with a few exceptions such > as temporary files.) I actually like to use the linker as an external > command since the operating system provides good isolation between my > process and the linker which could fail by an unknown bug or something. > Rewriting all of them as functions that return ErrorOr<Something> is > technically doable, but it needs strong justification, so I guess that > unlikely to change. > > Why do you want to use that as a library? I don't think "because LLVM and > Clang allow that" is not a compelling argument since the linker and the > compiler are pretty different programs. I clearly see many reasons to use > LLVM and part of Clang as libraries, but they are not directly applicable > to LLD. The new linker is more like a command which is built on top of the > libraries that LLVM provides. > > On Thu, Jan 7, 2016 at 8:28 AM, Arseny Kapoulkine < > arseny.kapoulkine at gmail.com> wrote: > >> This is really unfortunate. >> >> I've read the discussion threads for the atom/chunk controversy and I >> feel like I understand the reasons for rewriting the linker, but this does >> not seem to have anything to do with whether the linker is usable as a >> library or not. >> >> As it stands, not only does lld have two completely different linkers >> (I'm treating COFF/ELF2 as one since they are really two different >> implementations of the same concept, AFAIU), but one is usable as a library >> (and even does not require round-tripping generated code through an object >> file! I was really happy to use that) and the other one isn't. Not sure >> what the future plans are for Mach-O linker (at this point it seems logical >> to rewrite that using the new designs but I'm not sure if it ever happens), >> so maybe at some point we'll just have one linker application instead of a >> library and an application. >> >> Anyway, since linker is the only missing piece for full compilation stack >> (source language to runnable executable), it's sad to see this specific >> part of LLVM not working as a library when everything else does. >> >> Are there specific concerns in terms of implementation that prevent new >> lld from being a library? I understand that using global variables and >> error() function is simpler, but the rest of LLVM does not do that and the >> codebase there is significantly larger. Am I missing any other issues >> except for the ones I mentioned in my original e-mail that will come up in >> a library-like usage scenario? >> >> Arseny >> > >> On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote: >> >>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev < >>> llvm-dev at lists.llvm.org> wrote: >>> >>>> In the process of migrating from old lld ELF linker to new (previously >>>> ELF2) I noticed the interface lost several important features (ordered by >>>> importance for my use case): >>>> >>>> 1. Detecting errors in the first place. New linker seems to call >>>> exit(1) for any error. >>>> >>>> 2. Reporting messages to non-stderr outputs. Previously all link >>>> functions had a raw_ostream argument so it was possible to delay the error >>>> output, aggregate it for multiple linked files, output via a different >>>> format, etc. >>>> >>>> 3. Linking multiple outputs in parallel (useful for test drivers) in a >>>> single process. Not really an interface issue but there are at least two >>>> global pointers (Config & Driver) that refer to stack variables and are >>>> used in various places in the code. >>>> >>>> All of this seems to indicate a departure from the linker being useable >>>> as a library. To maintain the previous behavior you'd have to use a linker >>>> binary & popen. >>>> >>>> Is this a conscious design decision or a temporary limitation? >>>> >>> >>> That the new ELF and COFF linkers are designed as commands instead of >>> libraries is very much an intended design change. >>> >>> >>>> If it's a limitation, how would one go about fixing this? I'm not too >>>> familiar with the idiomatical error handling in LLVM. Normally in this >>>> situation I'd just throw from error() but lld is probably compiled without >>>> exceptions. Is ErrorOr the established practice? How does LLVM generally >>>> deal with error handling for parsers (like linker script parser) where it's >>>> a lot of mutually recursive functions where every single one can fail? >>>> >>> >>> Since it's as designed, please run the linker as an external command >>> using fork/exec (or wrappers of them) instead of trying to use that inside >>> the same process. >>> >>> >>>> Arseny >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> llvm-dev at lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>> >>>> >>> >> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/797780f4/attachment.html>
Arseny Kapoulkine via llvm-dev
2016-Jan-07 23:59 UTC
[llvm-dev] lld: ELF/COFF main() interface
> Why do you want to use that as a library? I don't think "because LLVM andClang allow that" is not a compelling argument since the linker and the compiler are pretty different programs. I clearly see many reasons to use LLVM and part of Clang as libraries, but they are not directly applicable to LLD. The new linker is more like a command which is built on top of the libraries that LLVM provides. My specific use-case is that of a compiler with an in-process linker that also runs tests (like Eric mentioned) - so basically a complete ahead-of-time compilation suite in a single process. In my case I *think* it's not prohibitively expensive to invoke a process - it complicates things but does not make them impossible. I started down this path because LLVM and lld allowed that. I had some plans that involved using similar infrastructure in a JIT-like manner (e.g. a REPL that would link the binary and load it), but this is probably better served using the actual JIT. I think there are environments where invoking a process or exit(1)-ing is absolutely prohibitive. I personally can solve these issues by fork/exec-ing the driver to perfom linking, but I don't quite understand how AMDGPU target can be useful in the new ELF linker since I had the impression that AMD basically uses ELF for the shader binary format, and compilation has to happen inside OpenGL driver (where you *want* to run shader compilation from multiple threads and definitely don't want a linking issue to exit the process in case of a user-mode driver). Arseny On Thu, Jan 7, 2016 at 9:06 AM, Rui Ueyama <ruiu at google.com> wrote:> Designing it a command makes things simpler because you can safely assume > that most functions always success, or otherwise the entire process > terminates. You can defer the operating system to clean up all resources > that was used by the process however it fails (with a few exceptions such > as temporary files.) I actually like to use the linker as an external > command since the operating system provides good isolation between my > process and the linker which could fail by an unknown bug or something. > Rewriting all of them as functions that return ErrorOr<Something> is > technically doable, but it needs strong justification, so I guess that > unlikely to change. > > Why do you want to use that as a library? I don't think "because LLVM and > Clang allow that" is not a compelling argument since the linker and the > compiler are pretty different programs. I clearly see many reasons to use > LLVM and part of Clang as libraries, but they are not directly applicable > to LLD. The new linker is more like a command which is built on top of the > libraries that LLVM provides. > > On Thu, Jan 7, 2016 at 8:28 AM, Arseny Kapoulkine < > arseny.kapoulkine at gmail.com> wrote: > >> This is really unfortunate. >> >> I've read the discussion threads for the atom/chunk controversy and I >> feel like I understand the reasons for rewriting the linker, but this does >> not seem to have anything to do with whether the linker is usable as a >> library or not. >> >> As it stands, not only does lld have two completely different linkers >> (I'm treating COFF/ELF2 as one since they are really two different >> implementations of the same concept, AFAIU), but one is usable as a library >> (and even does not require round-tripping generated code through an object >> file! I was really happy to use that) and the other one isn't. Not sure >> what the future plans are for Mach-O linker (at this point it seems logical >> to rewrite that using the new designs but I'm not sure if it ever happens), >> so maybe at some point we'll just have one linker application instead of a >> library and an application. >> >> Anyway, since linker is the only missing piece for full compilation stack >> (source language to runnable executable), it's sad to see this specific >> part of LLVM not working as a library when everything else does. >> >> Are there specific concerns in terms of implementation that prevent new >> lld from being a library? I understand that using global variables and >> error() function is simpler, but the rest of LLVM does not do that and the >> codebase there is significantly larger. Am I missing any other issues >> except for the ones I mentioned in my original e-mail that will come up in >> a library-like usage scenario? >> >> Arseny >> >> On Thu, Jan 7, 2016 at 7:17 AM, Rui Ueyama <ruiu at google.com> wrote: >> >>> On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev < >>> llvm-dev at lists.llvm.org> wrote: >>> >>>> In the process of migrating from old lld ELF linker to new (previously >>>> ELF2) I noticed the interface lost several important features (ordered by >>>> importance for my use case): >>>> >>>> 1. Detecting errors in the first place. New linker seems to call >>>> exit(1) for any error. >>>> >>>> 2. Reporting messages to non-stderr outputs. Previously all link >>>> functions had a raw_ostream argument so it was possible to delay the error >>>> output, aggregate it for multiple linked files, output via a different >>>> format, etc. >>>> >>>> 3. Linking multiple outputs in parallel (useful for test drivers) in a >>>> single process. Not really an interface issue but there are at least two >>>> global pointers (Config & Driver) that refer to stack variables and are >>>> used in various places in the code. >>>> >>>> All of this seems to indicate a departure from the linker being useable >>>> as a library. To maintain the previous behavior you'd have to use a linker >>>> binary & popen. >>>> >>>> Is this a conscious design decision or a temporary limitation? >>>> >>> >>> That the new ELF and COFF linkers are designed as commands instead of >>> libraries is very much an intended design change. >>> >>> >>>> If it's a limitation, how would one go about fixing this? I'm not too >>>> familiar with the idiomatical error handling in LLVM. Normally in this >>>> situation I'd just throw from error() but lld is probably compiled without >>>> exceptions. Is ErrorOr the established practice? How does LLVM generally >>>> deal with error handling for parsers (like linker script parser) where it's >>>> a lot of mutually recursive functions where every single one can fail? >>>> >>> >>> Since it's as designed, please run the linker as an external command >>> using fork/exec (or wrappers of them) instead of trying to use that inside >>> the same process. >>> >>> >>>> Arseny >>>> >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> llvm-dev at lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>> >>>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160107/99b04ae9/attachment.html>