On Wed, Mar 27, 2019 at 2:03 AM Saleem Abdulrasool <compnerd at compnerd.org> wrote:> On Tue, Mar 26, 2019 at 11:13 AM bd1976 llvm <bd1976llvm at gmail.com> wrote: > >> Thanks Saleem/James. >> >> On Tue, Mar 26, 2019 at 5:15 AM Saleem Abdulrasool <compnerd at compnerd.org> >> wrote: >> >>> Yes, I don't think that supporting all options is entirely possible. >>> But, there is certainly some allure to it, as it is a really powerful >>> feature (which you can tell from the behaviour of link and `.drectve` >>> section). The problem with all options is that ELF's model is not as >>> simplistic as link's model, which really complicates things. As an >>> example: `/WHOLEARCHIVE:` rather than `--whole-archive` ... >>> `--no-whole-archive`. I'm also worried about its use in a malicious >>> manner: `--no-pie` being embedded into the executable would disable ASLR, >>> which really is why I think that we would want a subset of options. >>> >> >> I'm not opposed to a linker options scheme; but, I think that it is >> important to separate this from autolinking (as we have proposed here). >> Rationale is upthread. We can support both if required. >> > > I'm not sure I understand how the two are distinct. The linker options > scheme being discussed is in the context of autolinking - aka, how to drive > the autolinking. >I'm imagining: #pragma comment(lib, "foo") => add "foo" to the link in a "general way". This is autolinking. #pragma comment(linker-option, "-l:foo") => Add "-l:foo" to the linker command line. This is linker options. Now it might be that on a specific platform (e.g. MSVC) you can represent #pragma comment(lib, "foo") using linker options; however, in general they are distinct. I would like the following behaviour for ELF:.. #pragma comment(lib, "foo") => add libfoo.a to the link. #pragma comment(lib, "foo.a") => add foo.a to the link #pragma comment(lib, "c:\\foo.a") => add c:\foo.a to the link .. but there is no existing ELF linker command line option that maps to this behaviour; so, it is better to have a separate representation. Also, given that it seems that autolinking is the only usecase currently, the proposed .autolink section is a simpler (e.g. binary tools can dump the section without needing special rules), and more efficient (bytes needed to convey the information) format than .linker-options.> > >> On Mon, Mar 25, 2019 at 8:47 AM Rui Ueyama <ruiu at google.com> wrote: >>> >>>> On Sun, Mar 24, 2019 at 9:51 PM Saleem Abdulrasool < >>>> compnerd at compnerd.org> wrote: >>>> >>>>> Sorry for the late chiming in. >>>>> >>>>> Yes, swift does use autolinking, and I would like to use that on all >>>>> the targets. The only target which does not support this functionality >>>>> currently are ELF based. That said, I think that `#pragma comment(link, >>>>> ...)` is insufficient for my needs. Building Foundation requires framework >>>>> style linking as well. The original design that I had in mind was derived >>>>> from ld64 and link. Personally, I still strongly favour link's behaviour >>>>> of parsing "command line" options from the object files when they are >>>>> loaded. There was strong opposition to that approach from Rui though. >>>>> Would we want to have special pragmas for each "feature"? >>>>> >>>> >> I was hoping that the autolinking proposal here was general enough that >> it could be made to work for you. What if we made the library search >> algorithm even more general than I initially proposed. Something like: >> >> - The linker tries to add a library or relocatable object file from each >> of the strings in a .autolink section by; first, handling the string as if >> it was specified on the commandline; second, by looking for the string in >> each of the library search paths in turn; third, by looking for a >> lib<string>.a or lib<string>.so (depending on the current mode of the >> linker) in each of the library search paths; forth, by looking for the >> string in each of the framework search paths in turn. >> > > What happens when you have Alpha.framework[/libAlpha.dylib] and > libAlpha.dylib? `-framework Alpha` should take > Alpha.framework/libAlpha.dylib and *NOT* libAlpha.dylib. I see no way to > differentiate between the two by mandating a specific ordering. -lAlpha -> > libAlpha.dylib and -framework Alpha -> Alpha.framework/libAlpha.dylib. > This is obviously a contrived case, but, the problem is that there is > ambiguity introduced which you cannot control without having additional > context from the user. So, `#pragma comment(link, ...)` is insufficient > IMO. >> >> So if you have #pragma comment(lib, "foo") in a source file, then on a >> linker that supports framework linking the linker would add to the link the >> first matching file trying: >> >> ./foo >> <first library search path>/foo >> <second library search path>/foo >> .. >> <nth library search path>/foo >> <first library search path>/libfoo.so >> <second library search path>/libfoo.so >> .. >> <nth library search path>/libfoo.so >> >> <first library search path>/libfoo.a >> <second library search path>/libfoo.a >> .. >> <nth library search path>/libfoo.a >> <first framework search path>/foo.framework/foo >> <second library search path>/foo.framework/foo >> .. >> <nth library search path>/foo.framework/foo >> >> What do you think? >> > > I think that it just broke in my example above :-). I think that we need > to realize that this is complex due to the fact that it requires knowledge > of what the user wants to do (which is normally expressed through the > invocation of the tools). So, we need to expose that to the implementation. >The question is how contrived is your case? Upthread, I think that we have established that to keep autolinking reasonably understandable and consistent across linkers you have to accept some limitations. I'm hoping that we can accept this approach of finding libraries as a reasonable limitation. The consequences are that if a codebase has the example situation that you have described, you would either have to restructure the codebase to avoid the ambiguity, or you can't use autolinking for this case. The upside is that no matter if your linker supports frameworks or libraries #pragma comment(lib, "foo") will work in all cases. Otherwise we will be in a situation where you will have to have the equivalent of... #ifdef COFF_LIBRARIES: #pragma comment(linker-option, "/DEFAULTLIB:foo.lib") #'elif ELF_LIBRARIES: #pragma comment(linker-option, "-lfoo") #'elif DARWIN_FRAMEWORKS: #pragma comment(linker-option, "-framework foo") #esle #error Please specify linking model #endif ... somewhere, it would be good to avoid this. OTOH perhaps framework linking is just too different to be lumped in with #pragma comment(lib, ...) perhaps we should restrict #pragma comment(lib, ...) to libraries and implement support for another pragma, e.g. #pragma comment(framework, ...), to support framework linking.> >> I think as you double-quoted the words "command line", I believe we are >>>> on the same page, but what I opposed was to allow arbitrary command line >>>> options in the .linker-options section. That is really a can of worms that >>>> we don't want to open. I didn't oppose allowing a limited set of >>>> command-line-ish directives. That said I'd prefer the proposal of this >>>> thread over the .linker-options though. >>>> >>>> The ELF model doesn't have the simplistic model for processing the >>>>> command line that PE/COFF does. Because ordering is relevant to the model, >>>>> it would be ideal to process them inline, but, since lld already moves far >>>>> enough away from the traditional Unix model, perhaps we can simplify it to >>>>> append the command line directives to the end of the command line. >>>>> >>>>> The other case that is interesting to think about is the autolinking >>>>> support in C++ (and clang) modules. >>>>> >>>> >> When I started this work I looked at the C++ 20 modules proposal: >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf. I >> didn't see anything about autolinking (or linking at all)! Can you explain >> more? >> > > I may have been confusing features from the clang module functionality and > C++20 Modules TS (though, I thought that it included support for > autolinking dependencies for cases where an imported module unit has > available_externally linkage). >Right. So the Clang modules document does mention linking: https://clang.llvm.org/docs/Modules.html#link-declaration. Clang modules are different to the proposed C++20 modules. I assumed that Clang modules support existed only as a prototype to demonstrate how modules "could" work to guide the design/discussions for C++ modules. I don't understand whether Clang modules are in use by anyone? What will happen to Clang modules when C++20 arrives? Even if Clang modules are being used perhaps the linking part of the specification can be changed (as it seems unlikely that this is used as ELF linkers don't have a mechanism to support it)?> > >> >> >>>>> On Thu, Mar 21, 2019 at 9:49 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>> wrote: >>>>> >>>>>> On Thu, Mar 21, 2019 at 12:06 AM Rui Ueyama <ruiu at google.com> wrote: >>>>>> >>>>>>> Perhaps there's no one clean way to solve this issue, because >>>>>>> previously all libraries and object files are explicitly given to the >>>>>>> linker via a command line and the order of files in the command line >>>>>>> matters. That assumes human intervention to work correctly. Now, the >>>>>>> autolinking feature will add libraries implicitly. Since it's implicit, >>>>>>> there will be only one way how that works, so sometimes that works and >>>>>>> sometimes doesn't. >>>>>>> >>>>>>> It feels to me that we should aim for making it work reasonably well >>>>>>> for reasonable use cases. By reasonable use cases, I'm thinking of the >>>>>>> following: >>>>>>> >>>>>>> 1. --static option may or may not be given (i.e. we should allow >>>>>>> that feature for both static linking and dynamic linking.) >>>>>>> 2. There are no competing defined symbols in a given set of >>>>>>> libraries, or if they exist, the program owner doesn't care which is linked >>>>>>> to their program. >>>>>>> 3. There may be circular dependencies between libraries. >>>>>>> >>>>>>> I don't think the above assumption is too odd. If I have to >>>>>>> implement the autolinking feature to GNU linker for the above scenario, I'd >>>>>>> probably use the following scheme: >>>>>>> >>>>>>> 1. While reading object files, memorize libraries that are >>>>>>> autolinked >>>>>>> 2. After linking everything, create a list of files consisting of >>>>>>> autolinked libraries AND libraries given via the command line >>>>>>> 3. Visit each file in the list as if they were wrapped in >>>>>>> --start-group and --end-group. >>>>>>> >>>>>>> I'd think the above scheme should work reasonably well. What do you >>>>>>> think? >>>>>>> >>>>>> >>>>>> Very nice. I agree with your definition of "reasonable" usecaes >>>>>> (actually, as I have said before, I think that restricting autolinking to >>>>>> this "reasonable" set is actually a feature - to avoid developers having >>>>>> source code that only works with a particular linker). I also like the >>>>>> proposal for a GNU implementation - I think this is enough to show that >>>>>> GNU-like linkers could implement this. >>>>>> >>>>>> At this point I will try to prototype this up so that people have an >>>>>> implementation to play with. >>>>>> >>>>>> I am keen to hear from Saleem (compnerd) on this, as he did the >>>>>> original .linker-options work. >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Mar 19, 2019 at 11:02 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Mon, Mar 18, 2019 at 8:02 PM Rui Ueyama <ruiu at google.com> wrote: >>>>>>>> >>>>>>>>> On Thu, Mar 14, 2019 at 1:05 PM bd1976 llvm via llvm-dev < >>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 14, 2019 at 6:27 PM Peter Collingbourne < >>>>>>>>>> peter at pcc.me.uk> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev < >>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> At Sony we offer autolinking as a feature in our ELF toolchain. >>>>>>>>>>>> We would like to see full support for this feature upstream as there is >>>>>>>>>>>> anecdotal evidence that it would find use beyond Sony. >>>>>>>>>>>> >>>>>>>>>>>> In general autolinking ( >>>>>>>>>>>> https://en.wikipedia.org/wiki/Auto-linking) allows developers >>>>>>>>>>>> to specify inputs to the linker in their source code. LLVM and Clang >>>>>>>>>>>> already have support for autolinking on ELF via embedding strings, which >>>>>>>>>>>> specify linker behavior, into a .linker-options section in relocatable >>>>>>>>>>>> object files, see: >>>>>>>>>>>> >>>>>>>>>>>> RFC - >>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html >>>>>>>>>>>> LLVM - >>>>>>>>>>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options, >>>>>>>>>>>> https://reviews.llvm.org/D40849 >>>>>>>>>>>> Clang - >>>>>>>>>>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets, >>>>>>>>>>>> https://reviews.llvm.org/D42758 >>>>>>>>>>>> >>>>>>>>>>>> However, although support was added to Clang and LLVM, no >>>>>>>>>>>> support has been implemented in LLD; and, I get the sense, from reading the >>>>>>>>>>>> reviews, that there wasn't agreement on the implementation when the changes >>>>>>>>>>>> landed. The original motivation seems to have been to remove the >>>>>>>>>>>> "autolink-extract" mechanism used by Swift to workaround the lack of >>>>>>>>>>>> autolinking support for ELF. However, looking at the Swift source code, >>>>>>>>>>>> Swift still seems to be using the "autolink-extract" method. >>>>>>>>>>>> >>>>>>>>>>>> So my first question: Are there any users of the current >>>>>>>>>>>> implementation for ELF? >>>>>>>>>>>> >>>>>>>>>>>> Assuming that no one is using the current code, I would like to >>>>>>>>>>>> suggest a different mechanism for autolinking. >>>>>>>>>>>> >>>>>>>>>>>> For ELF we need limited autolinking support. Specifically, we >>>>>>>>>>>> only need support for "comment lib" pragmas ( >>>>>>>>>>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) >>>>>>>>>>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the >>>>>>>>>>>> implementation as lean as possible. >>>>>>>>>>>> >>>>>>>>>>>> Principles to guide the implementation: >>>>>>>>>>>> - Developers should be able to easily understand autolinking >>>>>>>>>>>> behavior. >>>>>>>>>>>> - Developers should be able to override autolinking from the >>>>>>>>>>>> linker command line. >>>>>>>>>>>> - Inputs specified via pragmas should be handled in a general >>>>>>>>>>>> way to allow the same source code to work in different environments. >>>>>>>>>>>> >>>>>>>>>>>> I would like to propose that we focus on autolinking >>>>>>>>>>>> exclusively and that we divorce the implementation from the idea of "linker >>>>>>>>>>>> options" which, by nature, would tie source code to the vagaries of >>>>>>>>>>>> particular linkers. I don't see much value in supporting other linker >>>>>>>>>>>> operations so I suggest that the binary representation be a mergable string >>>>>>>>>>>> section (SHF_MERGE, SHF_STRINGS), called .autolink, with custom type >>>>>>>>>>>> SHT_LLVM_AUTOLINK (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents >>>>>>>>>>>> appearing in the output). The compiler can form this section by >>>>>>>>>>>> concatenating the arguments of the "comment lib" pragmas in the order they >>>>>>>>>>>> are encountered. Partial (-r, -Ur) links can be handled by concatenating >>>>>>>>>>>> .autolink sections with the normal mergeable string section rules. The >>>>>>>>>>>> current .linker-options can remain (or be removed); but, "comment lib" >>>>>>>>>>>> pragmas for ELF should be lowered to .autolink not to .linker-options. This >>>>>>>>>>>> makes sense as there is no linker option that "comment lib" pragmas map >>>>>>>>>>>> directly to. As an example, #pragma comment(lib, "foo") would result in: >>>>>>>>>>>> >>>>>>>>>>>> .section ".autolink","eMS", at llvm_autolink,1 >>>>>>>>>>>> .asciz "foo" >>>>>>>>>>>> >>>>>>>>>>>> For LTO, equivalent information to the contents of a the >>>>>>>>>>>> .autolink section will be written to the IRSymtab so that it is available >>>>>>>>>>>> to the linker for symbol resolution. >>>>>>>>>>>> >>>>>>>>>>>> The linker will process the .autolink strings in the following >>>>>>>>>>>> way: >>>>>>>>>>>> >>>>>>>>>>>> 1. Inputs from the .autolink sections of a relocatable object >>>>>>>>>>>> file are added when the linker decides to include that file (which could >>>>>>>>>>>> itself be in a library) in the link. Autolinked inputs behave as if they >>>>>>>>>>>> were appended to the command line as a group after all other options. As a >>>>>>>>>>>> consequence the set of autolinked libraries are searched last to resolve >>>>>>>>>>>> symbols. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If we want this to be compatible with GNU linkers, doesn't the >>>>>>>>>>> autolinked input need to appear at the point immediately after the object >>>>>>>>>>> file appears in the link? I'm imagining the case where you have a >>>>>>>>>>> statically linked libc as well as a libbar.a autolinked from a foo.o. The >>>>>>>>>>> link command line would look like this: >>>>>>>>>>> >>>>>>>>>>> ld foo.o -lc >>>>>>>>>>> >>>>>>>>>>> Now foo.o autolinks against bar. The command line becomes: >>>>>>>>>>> >>>>>>>>>>> ld foo.o -lc -lbar >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Actually, I was thinking that on a GNU linker the command line >>>>>>>>>> would become "ld foo.o -lc -( -lbar )-"; but, this doesn't affect your >>>>>>>>>> point. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If libbar.a requires an additional object file from libc.a, it >>>>>>>>>>> will not be added to the link. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> As it stands all the dependencies of an autolinked library must >>>>>>>>>> themselves be autolinked. I had imagined that this is a reasonable >>>>>>>>>> limitation. If not we need another scheme. I try to think about some >>>>>>>>>> motivating examples for this. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 2. It is an error if a file cannot be found for a given string. >>>>>>>>>>>> 3. Any command line options in effect at the end of the command >>>>>>>>>>>> line parsing apply to autolinked inputs, e.g. --whole-archive. >>>>>>>>>>>> 4. Duplicate autolinked inputs are ignored. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This seems like it would work in GNU linkers, as long as the >>>>>>>>>>> autolinked file is added to the link immediately after the last mention, >>>>>>>>>>> rather than the first. Otherwise a command line like: >>>>>>>>>>> >>>>>>>>>>> ld foo1.o foo2.o >>>>>>>>>>> >>>>>>>>>>> (where foo1.o and foo2.o both autolink bar) could end up looking >>>>>>>>>>> like: >>>>>>>>>>> >>>>>>>>>>> ld foo1.o -lbar foo2.o >>>>>>>>>>> >>>>>>>>>>> and you will not link anything from libbar.a that only foo2.o >>>>>>>>>>> requires. It may end up being simpler to not ignore duplicates. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Correct; but, given that the proposal was to handle the libraries >>>>>>>>>> as if they are appended to the link line after everything on the command >>>>>>>>>> line then I think this will work. With deduplication (and the use of >>>>>>>>>> SHF_MERGE) developers get no ordering guarantees. I claim that this is a >>>>>>>>>> feature! My rationale is that the order in which libraries are linked >>>>>>>>>> affects different linkers in different ways (e.g. LLD does not resolve >>>>>>>>>> symbols from archives in a compatible manner with either the Microsoft >>>>>>>>>> linker or the GNU linkers.), by not allowing the user to control the order >>>>>>>>>> I am essentially saying that autolinking is not suitable for libraries that >>>>>>>>>> offer competing copies of the same symbol. This ties into my argument that >>>>>>>>>> "comment lib" pragmas should be handled in as "general" a way as possible. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Right. I think if you need a fine control over the link order, >>>>>>>>> autolinking is not a feature you want to use. Or, in general, if your >>>>>>>>> program is sensitive to a link order because its source object files have >>>>>>>>> competing symbols of the same name, it's perhaps unnecessarily fragile. >>>>>>>>> >>>>>>>>> That being said, I think you need to address the issue that pcc >>>>>>>>> pointed out. If you statically link a program `foo` with the following >>>>>>>>> command line >>>>>>>>> >>>>>>>>> ld -o foo foo.o -lc >>>>>>>>> >>>>>>>>> , `foo.o` auto-imports libbar.a, and libbar.a depends on libc.a, >>>>>>>>> can your proposed feature pull out object files needed for libbar.a? >>>>>>>>> >>>>>>>> >>>>>>>> It won't work on GNU linkers. It will work with LLD as LLD has >>>>>>>> MSVC-like archive handling. However, I would like to make sure that >>>>>>>> whatever we come up with can be supported in the GNU toolchain. >>>>>>>> >>>>>>>> I had thought that it would be acceptable that all the dependencies >>>>>>>> of an autolinked library must themselves be autolinked in order to work on >>>>>>>> GNU style linkers. Having thought more, I don't like this limitation - >>>>>>>> especially as it doesn't exist for Microsoft style linkers. One possible >>>>>>>> resolution could be that GNU linkers might have to implement another >>>>>>>> command line option e.g. --auto-dep=<file> to allow injection into the >>>>>>>> group of autolinked libraries. >>>>>>>> >>>>>>>> i.e In pcc's example you would need to do: "ld foo.o >>>>>>>> --auto-dep=libc.a" which would become "ld --start-group libbar.a libc.a >>>>>>>> --end-group" with autolinking. >>>>>>>> >>>>>>>> I wanted to avoid the approach of inserting autolinked libraries >>>>>>>> after the object that autolinks them. In LLD (and MSVC) it becomes hard to >>>>>>>> reason about "where" the linker is in the command line and it would also >>>>>>>> mean that we can't have the nice separation between parsing the command >>>>>>>> line and doing the rest of the link that we currently have. Also, if you >>>>>>>> give people a way to have a fine grained control over the link order with >>>>>>>> autolinking you risk ending up with source code that will link on GNU style >>>>>>>> linkers but not with LLD (assuming GNU ever implemented support for >>>>>>>> autolinking). >>>>>>>> >>>>>>>> Scenario: >>>>>>>> >>>>>>>> libbar.a(bar.o) - defines symbol bar >>>>>>>> libfoo.a(foo.o) - defines foo and autolinks libbar.a >>>>>>>> main.o - references foo >>>>>>>> another.o - does not reference foo >>>>>>>> No references to bar exist >>>>>>>> >>>>>>>> lld -lfoo another.o --whole-archive main.o with autolinking becomes >>>>>>>> lld -lfoo another.o --whole-archive main.o -lbar result: bar.o gets added >>>>>>>> to the link. >>>>>>>> But, if a change is made so that another.o references bar then the >>>>>>>> link line with autolinking becomes lld -lfoo another.o >>>>>>>> -lbar --whole-archive main.o result: bar.o is not added to the link. >>>>>>>> >>>>>>>> Hopefully the above scenario demonstrates why I think that it >>>>>>>> becomes too complicated to reason about the effects of autolinking with >>>>>>>> pcc's proposed insertion scheme. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> 5. The linker tries to add a library or relocatable object file >>>>>>>>>>>> from each of the strings in a .autolink section by; first, handling the >>>>>>>>>>>> string as if it was specified on the commandline; second, by looking for >>>>>>>>>>>> the string in each of the library search paths in turn; third, by looking >>>>>>>>>>>> for a lib<string>.a or lib<string>.so (depending on the current mode of the >>>>>>>>>>>> linker) in each of the library search paths. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is the second part necessary? "-l:foo" causes the linker to >>>>>>>>>>> search for a file named "foo" in the library search path, so it seems that >>>>>>>>>>> allowing the autolink string to look like ":foo" would satisfy this use >>>>>>>>>>> case. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I worded the proposal to avoid mapping "comment lib" pragmas to >>>>>>>>>> --library command line options. My reasons: >>>>>>>>>> >>>>>>>>>> 1. I find the requirement that the user put ':' in their lib >>>>>>>>>> strings slightly awkward. It means that the source code is now coupled to a >>>>>>>>>> GNU-style linker. So then this isn't merely an ELF linking proposal, it's a >>>>>>>>>> proposal for ELF toolchains with GNU-like linkers (e.g. the arm linker >>>>>>>>>> doesn't support the colon prefix >>>>>>>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0474c/Cjahbdei.html >>>>>>>>>> ). >>>>>>>>>> >>>>>>>>>> 2. The syntax is #pragma comment(lib, ...) not #pragma >>>>>>>>>> linker-option(library, ...) i.e. the only thing this (frankly rather >>>>>>>>>> bizarre) syntax definitely implies is that the argument is related to >>>>>>>>>> libraries (and comments ¯\_(ツ)_/¯); it is a bit of a stretch to interpret >>>>>>>>>> "comment lib" pragmas as mapping directly to "specifying an additional >>>>>>>>>> --library command line option". >>>>>>>>>> >>>>>>>>>> AFAIK all linkers support two ways of specifying inputs; firstly, >>>>>>>>>> directly on the command line; secondly, with an option with very similar >>>>>>>>>> semantics to GNU's --library option. I choose a method of finding a input >>>>>>>>>> files that encompasses both methods of specifying a library on the command >>>>>>>>>> line. I think that this method is actually more intuitive than either the >>>>>>>>>> method used by the linker script INPUT command or by --library. FWIW, I >>>>>>>>>> looked into the history of the colon prefix. It was added in >>>>>>>>>> https://www.sourceware.org/ml/binutils/2007-03/msg00421.html. >>>>>>>>>> Unfortunately, the rationale given is that it was merely a port of a >>>>>>>>>> vxworks linker extension. I couldn't trace the history any further than >>>>>>>>>> that to find the actual design discussion. The linker script command INPUT >>>>>>>>>> uses a different scheme and the command already had this search order 20 >>>>>>>>>> years ago, which is the earliest version of the GNU linker I have history >>>>>>>>>> for; again, the rationale is not available. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 6. A new command line option --no-llvm-autolink will tell LLD to >>>>>>>>>>>> ignore the .autolink sections. >>>>>>>>>>>> >>>>>>>>>>>> Rationale for the above points: >>>>>>>>>>>> >>>>>>>>>>>> 1. Adding the autolinked inputs last makes the process simple >>>>>>>>>>>> to understand from a developers perspective. All linkers are able to >>>>>>>>>>>> implement this scheme. >>>>>>>>>>>> 2. Error-ing for libraries that are not found seems like better >>>>>>>>>>>> behavior than failing the link during symbol resolution. >>>>>>>>>>>> 3. It seems useful for the user to be able to apply command >>>>>>>>>>>> line options which will affect all of the autolinked input files. There is >>>>>>>>>>>> a potential problem of surprise for developers, who might not realize that >>>>>>>>>>>> these options would apply to the "invisible" autolinked input files; >>>>>>>>>>>> however, despite the potential for surprise, this is easy for developers to >>>>>>>>>>>> reason about and gives developers the control that they may require. >>>>>>>>>>>> 4. Unlike on the command line it is probably easy to include >>>>>>>>>>>> the same input file twice via pragmas and might be a pain to fix; think of >>>>>>>>>>>> Third-party libraries supplied as binaries. >>>>>>>>>>>> 5. This algorithm takes into account all of the different ways >>>>>>>>>>>> that ELF linkers find input files. The different search methods are tried >>>>>>>>>>>> by the linker in most obvious to least obvious order. >>>>>>>>>>>> 6. I considered adding finer grained control over which >>>>>>>>>>>> .autolink inputs were ignored (e.g. MSVC has /nodefaultlib:<library>); >>>>>>>>>>>> however, I concluded that this is not necessary: if finer control is >>>>>>>>>>>> required developers can recreate the same effect autolinking would have had >>>>>>>>>>>> using command line options. >>>>>>>>>>>> >>>>>>>>>>>> Thoughts? >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> -- >>>>>>>>>>> Peter >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> LLVM Developers mailing list >>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>> >>>>>>>>> >>>>> >>>>> -- >>>>> Saleem Abdulrasool >>>>> compnerd (at) compnerd (dot) org >>>>> >>>> >>> >>> -- >>> Saleem Abdulrasool >>> compnerd (at) compnerd (dot) org >>> >> > > -- > Saleem Abdulrasool > compnerd (at) compnerd (dot) org >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190327/d4548c34/attachment-0001.html>
The initial post seems to be about ELF only but along the way MSVC was mentioned. So is this suposed to carry over for linking PE-Binaries? In that case: Directly mentioning .dll files as target for autolinking. The PE import section only has pairs/groups of functionnames/dll-file and is handled by the runtimelinker. 2. Error-ing for libraries that are not found seems like better behavior> than failing the link during symbol resolution.Therefore the linker should not error out if the file is not found. There are Compilers for the Windowsplatform that just allow this and it makes the whole thing a lot more comfortable. Am Mi., 27. März 2019 um 13:18 Uhr schrieb bd1976 llvm via llvm-dev < llvm-dev at lists.llvm.org>:> On Wed, Mar 27, 2019 at 2:03 AM Saleem Abdulrasool <compnerd at compnerd.org> > wrote: > >> On Tue, Mar 26, 2019 at 11:13 AM bd1976 llvm <bd1976llvm at gmail.com> >> wrote: >> >>> Thanks Saleem/James. >>> >>> On Tue, Mar 26, 2019 at 5:15 AM Saleem Abdulrasool < >>> compnerd at compnerd.org> wrote: >>> >>>> Yes, I don't think that supporting all options is entirely possible. >>>> But, there is certainly some allure to it, as it is a really powerful >>>> feature (which you can tell from the behaviour of link and `.drectve` >>>> section). The problem with all options is that ELF's model is not as >>>> simplistic as link's model, which really complicates things. As an >>>> example: `/WHOLEARCHIVE:` rather than `--whole-archive` ... >>>> `--no-whole-archive`. I'm also worried about its use in a malicious >>>> manner: `--no-pie` being embedded into the executable would disable ASLR, >>>> which really is why I think that we would want a subset of options. >>>> >>> >>> I'm not opposed to a linker options scheme; but, I think that it is >>> important to separate this from autolinking (as we have proposed here). >>> Rationale is upthread. We can support both if required. >>> >> >> I'm not sure I understand how the two are distinct. The linker options >> scheme being discussed is in the context of autolinking - aka, how to drive >> the autolinking. >> > > I'm imagining: > > #pragma comment(lib, "foo") => add "foo" to the link in a "general way". > This is autolinking. > #pragma comment(linker-option, "-l:foo") => Add "-l:foo" to the linker > command line. This is linker options. > > Now it might be that on a specific platform (e.g. MSVC) you can represent > #pragma comment(lib, "foo") using linker options; however, in general they > are distinct. I would like the following behaviour for ELF:.. > > #pragma comment(lib, "foo") => add libfoo.a to the link. > #pragma comment(lib, "foo.a") => add foo.a to the link > #pragma comment(lib, "c:\\foo.a") => add c:\foo.a to the link > > .. but there is no existing ELF linker command line option that maps to > this behaviour; so, it is better to have a separate representation. Also, > given that it seems that autolinking is the only usecase currently, the > proposed .autolink section is a simpler (e.g. binary tools can dump the > section without needing special rules), and more efficient (bytes needed to > convey the information) format than .linker-options. > > >> >> >>> On Mon, Mar 25, 2019 at 8:47 AM Rui Ueyama <ruiu at google.com> wrote: >>>> >>>>> On Sun, Mar 24, 2019 at 9:51 PM Saleem Abdulrasool < >>>>> compnerd at compnerd.org> wrote: >>>>> >>>>>> Sorry for the late chiming in. >>>>>> >>>>>> Yes, swift does use autolinking, and I would like to use that on all >>>>>> the targets. The only target which does not support this functionality >>>>>> currently are ELF based. That said, I think that `#pragma comment(link, >>>>>> ...)` is insufficient for my needs. Building Foundation requires framework >>>>>> style linking as well. The original design that I had in mind was derived >>>>>> from ld64 and link. Personally, I still strongly favour link's behaviour >>>>>> of parsing "command line" options from the object files when they are >>>>>> loaded. There was strong opposition to that approach from Rui though. >>>>>> Would we want to have special pragmas for each "feature"? >>>>>> >>>>> >>> I was hoping that the autolinking proposal here was general enough that >>> it could be made to work for you. What if we made the library search >>> algorithm even more general than I initially proposed. Something like: >>> >>> - The linker tries to add a library or relocatable object file from each >>> of the strings in a .autolink section by; first, handling the string as if >>> it was specified on the commandline; second, by looking for the string in >>> each of the library search paths in turn; third, by looking for a >>> lib<string>.a or lib<string>.so (depending on the current mode of the >>> linker) in each of the library search paths; forth, by looking for the >>> string in each of the framework search paths in turn. >>> >> >> What happens when you have Alpha.framework[/libAlpha.dylib] and >> libAlpha.dylib? `-framework Alpha` should take >> Alpha.framework/libAlpha.dylib and *NOT* libAlpha.dylib. I see no way to >> differentiate between the two by mandating a specific ordering. -lAlpha -> >> libAlpha.dylib and -framework Alpha -> Alpha.framework/libAlpha.dylib. >> This is obviously a contrived case, but, the problem is that there is >> ambiguity introduced which you cannot control without having additional >> context from the user. So, `#pragma comment(link, ...)` is insufficient >> IMO. >> > >> >>> So if you have #pragma comment(lib, "foo") in a source file, then on a >>> linker that supports framework linking the linker would add to the link the >>> first matching file trying: >>> >>> ./foo >>> <first library search path>/foo >>> <second library search path>/foo >>> .. >>> <nth library search path>/foo >>> <first library search path>/libfoo.so >>> <second library search path>/libfoo.so >>> .. >>> <nth library search path>/libfoo.so >>> >>> <first library search path>/libfoo.a >>> <second library search path>/libfoo.a >>> .. >>> <nth library search path>/libfoo.a >>> <first framework search path>/foo.framework/foo >>> <second library search path>/foo.framework/foo >>> .. >>> <nth library search path>/foo.framework/foo >>> >>> What do you think? >>> >> >> I think that it just broke in my example above :-). I think that we need >> to realize that this is complex due to the fact that it requires knowledge >> of what the user wants to do (which is normally expressed through the >> invocation of the tools). So, we need to expose that to the implementation. >> > > The question is how contrived is your case? Upthread, I think that we have > established that to keep autolinking reasonably understandable and > consistent across linkers you have to accept some limitations. I'm hoping > that we can accept this approach of finding libraries as a reasonable > limitation. The consequences are that if a codebase has the example > situation that you have described, you would either have to restructure the > codebase to avoid the ambiguity, or you can't use autolinking for this > case. The upside is that no matter if your linker supports frameworks or > libraries #pragma comment(lib, "foo") will work in all cases. Otherwise we > will be in a situation where you will have to have the equivalent of... > > #ifdef COFF_LIBRARIES: > #pragma comment(linker-option, "/DEFAULTLIB:foo.lib") > #'elif ELF_LIBRARIES: > #pragma comment(linker-option, "-lfoo") > #'elif DARWIN_FRAMEWORKS: > #pragma comment(linker-option, "-framework foo") > #esle > #error Please specify linking model > #endif > > ... somewhere, it would be good to avoid this. > > OTOH perhaps framework linking is just too different to be lumped in with > #pragma comment(lib, ...) perhaps we should restrict #pragma comment(lib, > ...) to libraries and implement support for another pragma, e.g. #pragma > comment(framework, ...), to support framework linking. > > >> >>> I think as you double-quoted the words "command line", I believe we are >>>>> on the same page, but what I opposed was to allow arbitrary command line >>>>> options in the .linker-options section. That is really a can of worms that >>>>> we don't want to open. I didn't oppose allowing a limited set of >>>>> command-line-ish directives. That said I'd prefer the proposal of this >>>>> thread over the .linker-options though. >>>>> >>>>> The ELF model doesn't have the simplistic model for processing the >>>>>> command line that PE/COFF does. Because ordering is relevant to the model, >>>>>> it would be ideal to process them inline, but, since lld already moves far >>>>>> enough away from the traditional Unix model, perhaps we can simplify it to >>>>>> append the command line directives to the end of the command line. >>>>>> >>>>>> The other case that is interesting to think about is the autolinking >>>>>> support in C++ (and clang) modules. >>>>>> >>>>> >>> When I started this work I looked at the C++ 20 modules proposal: >>> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf. I >>> didn't see anything about autolinking (or linking at all)! Can you explain >>> more? >>> >> >> I may have been confusing features from the clang module functionality >> and C++20 Modules TS (though, I thought that it included support for >> autolinking dependencies for cases where an imported module unit has >> available_externally linkage). >> > > Right. So the Clang modules document does mention linking: > https://clang.llvm.org/docs/Modules.html#link-declaration. Clang modules > are different to the proposed C++20 modules. I assumed that Clang modules > support existed only as a prototype to demonstrate how modules "could" work > to guide the design/discussions for C++ modules. I don't understand whether > Clang modules are in use by anyone? What will happen to Clang modules when > C++20 arrives? Even if Clang modules are being used perhaps the linking > part of the specification can be changed (as it seems unlikely that this is > used as ELF linkers don't have a mechanism to support it)? > > >> >> >>> >>> >>>>>> On Thu, Mar 21, 2019 at 9:49 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Mar 21, 2019 at 12:06 AM Rui Ueyama <ruiu at google.com> wrote: >>>>>>> >>>>>>>> Perhaps there's no one clean way to solve this issue, because >>>>>>>> previously all libraries and object files are explicitly given to the >>>>>>>> linker via a command line and the order of files in the command line >>>>>>>> matters. That assumes human intervention to work correctly. Now, the >>>>>>>> autolinking feature will add libraries implicitly. Since it's implicit, >>>>>>>> there will be only one way how that works, so sometimes that works and >>>>>>>> sometimes doesn't. >>>>>>>> >>>>>>>> It feels to me that we should aim for making it work reasonably >>>>>>>> well for reasonable use cases. By reasonable use cases, I'm thinking of the >>>>>>>> following: >>>>>>>> >>>>>>>> 1. --static option may or may not be given (i.e. we should allow >>>>>>>> that feature for both static linking and dynamic linking.) >>>>>>>> 2. There are no competing defined symbols in a given set of >>>>>>>> libraries, or if they exist, the program owner doesn't care which is linked >>>>>>>> to their program. >>>>>>>> 3. There may be circular dependencies between libraries. >>>>>>>> >>>>>>>> I don't think the above assumption is too odd. If I have to >>>>>>>> implement the autolinking feature to GNU linker for the above scenario, I'd >>>>>>>> probably use the following scheme: >>>>>>>> >>>>>>>> 1. While reading object files, memorize libraries that are >>>>>>>> autolinked >>>>>>>> 2. After linking everything, create a list of files consisting of >>>>>>>> autolinked libraries AND libraries given via the command line >>>>>>>> 3. Visit each file in the list as if they were wrapped in >>>>>>>> --start-group and --end-group. >>>>>>>> >>>>>>>> I'd think the above scheme should work reasonably well. What do you >>>>>>>> think? >>>>>>>> >>>>>>> >>>>>>> Very nice. I agree with your definition of "reasonable" usecaes >>>>>>> (actually, as I have said before, I think that restricting autolinking to >>>>>>> this "reasonable" set is actually a feature - to avoid developers having >>>>>>> source code that only works with a particular linker). I also like the >>>>>>> proposal for a GNU implementation - I think this is enough to show that >>>>>>> GNU-like linkers could implement this. >>>>>>> >>>>>>> At this point I will try to prototype this up so that people have an >>>>>>> implementation to play with. >>>>>>> >>>>>>> I am keen to hear from Saleem (compnerd) on this, as he did the >>>>>>> original .linker-options work. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 19, 2019 at 11:02 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Mon, Mar 18, 2019 at 8:02 PM Rui Ueyama <ruiu at google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 14, 2019 at 1:05 PM bd1976 llvm via llvm-dev < >>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 14, 2019 at 6:27 PM Peter Collingbourne < >>>>>>>>>>> peter at pcc.me.uk> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev < >>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> At Sony we offer autolinking as a feature in our ELF >>>>>>>>>>>>> toolchain. We would like to see full support for this feature upstream as >>>>>>>>>>>>> there is anecdotal evidence that it would find use beyond Sony. >>>>>>>>>>>>> >>>>>>>>>>>>> In general autolinking ( >>>>>>>>>>>>> https://en.wikipedia.org/wiki/Auto-linking) allows developers >>>>>>>>>>>>> to specify inputs to the linker in their source code. LLVM and Clang >>>>>>>>>>>>> already have support for autolinking on ELF via embedding strings, which >>>>>>>>>>>>> specify linker behavior, into a .linker-options section in relocatable >>>>>>>>>>>>> object files, see: >>>>>>>>>>>>> >>>>>>>>>>>>> RFC - >>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html >>>>>>>>>>>>> LLVM - >>>>>>>>>>>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options, >>>>>>>>>>>>> https://reviews.llvm.org/D40849 >>>>>>>>>>>>> Clang - >>>>>>>>>>>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets, >>>>>>>>>>>>> https://reviews.llvm.org/D42758 >>>>>>>>>>>>> >>>>>>>>>>>>> However, although support was added to Clang and LLVM, no >>>>>>>>>>>>> support has been implemented in LLD; and, I get the sense, from reading the >>>>>>>>>>>>> reviews, that there wasn't agreement on the implementation when the changes >>>>>>>>>>>>> landed. The original motivation seems to have been to remove the >>>>>>>>>>>>> "autolink-extract" mechanism used by Swift to workaround the lack of >>>>>>>>>>>>> autolinking support for ELF. However, looking at the Swift source code, >>>>>>>>>>>>> Swift still seems to be using the "autolink-extract" method. >>>>>>>>>>>>> >>>>>>>>>>>>> So my first question: Are there any users of the current >>>>>>>>>>>>> implementation for ELF? >>>>>>>>>>>>> >>>>>>>>>>>>> Assuming that no one is using the current code, I would like >>>>>>>>>>>>> to suggest a different mechanism for autolinking. >>>>>>>>>>>>> >>>>>>>>>>>>> For ELF we need limited autolinking support. Specifically, we >>>>>>>>>>>>> only need support for "comment lib" pragmas ( >>>>>>>>>>>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) >>>>>>>>>>>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the >>>>>>>>>>>>> implementation as lean as possible. >>>>>>>>>>>>> >>>>>>>>>>>>> Principles to guide the implementation: >>>>>>>>>>>>> - Developers should be able to easily understand autolinking >>>>>>>>>>>>> behavior. >>>>>>>>>>>>> - Developers should be able to override autolinking from the >>>>>>>>>>>>> linker command line. >>>>>>>>>>>>> - Inputs specified via pragmas should be handled in a general >>>>>>>>>>>>> way to allow the same source code to work in different environments. >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to propose that we focus on autolinking >>>>>>>>>>>>> exclusively and that we divorce the implementation from the idea of "linker >>>>>>>>>>>>> options" which, by nature, would tie source code to the vagaries of >>>>>>>>>>>>> particular linkers. I don't see much value in supporting other linker >>>>>>>>>>>>> operations so I suggest that the binary representation be a mergable string >>>>>>>>>>>>> section (SHF_MERGE, SHF_STRINGS), called .autolink, with custom type >>>>>>>>>>>>> SHT_LLVM_AUTOLINK (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents >>>>>>>>>>>>> appearing in the output). The compiler can form this section by >>>>>>>>>>>>> concatenating the arguments of the "comment lib" pragmas in the order they >>>>>>>>>>>>> are encountered. Partial (-r, -Ur) links can be handled by concatenating >>>>>>>>>>>>> .autolink sections with the normal mergeable string section rules. The >>>>>>>>>>>>> current .linker-options can remain (or be removed); but, "comment lib" >>>>>>>>>>>>> pragmas for ELF should be lowered to .autolink not to .linker-options. This >>>>>>>>>>>>> makes sense as there is no linker option that "comment lib" pragmas map >>>>>>>>>>>>> directly to. As an example, #pragma comment(lib, "foo") would result in: >>>>>>>>>>>>> >>>>>>>>>>>>> .section ".autolink","eMS", at llvm_autolink,1 >>>>>>>>>>>>> .asciz "foo" >>>>>>>>>>>>> >>>>>>>>>>>>> For LTO, equivalent information to the contents of a the >>>>>>>>>>>>> .autolink section will be written to the IRSymtab so that it is available >>>>>>>>>>>>> to the linker for symbol resolution. >>>>>>>>>>>>> >>>>>>>>>>>>> The linker will process the .autolink strings in the following >>>>>>>>>>>>> way: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Inputs from the .autolink sections of a relocatable object >>>>>>>>>>>>> file are added when the linker decides to include that file (which could >>>>>>>>>>>>> itself be in a library) in the link. Autolinked inputs behave as if they >>>>>>>>>>>>> were appended to the command line as a group after all other options. As a >>>>>>>>>>>>> consequence the set of autolinked libraries are searched last to resolve >>>>>>>>>>>>> symbols. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If we want this to be compatible with GNU linkers, doesn't the >>>>>>>>>>>> autolinked input need to appear at the point immediately after the object >>>>>>>>>>>> file appears in the link? I'm imagining the case where you have a >>>>>>>>>>>> statically linked libc as well as a libbar.a autolinked from a foo.o. The >>>>>>>>>>>> link command line would look like this: >>>>>>>>>>>> >>>>>>>>>>>> ld foo.o -lc >>>>>>>>>>>> >>>>>>>>>>>> Now foo.o autolinks against bar. The command line becomes: >>>>>>>>>>>> >>>>>>>>>>>> ld foo.o -lc -lbar >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Actually, I was thinking that on a GNU linker the command line >>>>>>>>>>> would become "ld foo.o -lc -( -lbar )-"; but, this doesn't affect your >>>>>>>>>>> point. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If libbar.a requires an additional object file from libc.a, it >>>>>>>>>>>> will not be added to the link. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> As it stands all the dependencies of an autolinked library must >>>>>>>>>>> themselves be autolinked. I had imagined that this is a reasonable >>>>>>>>>>> limitation. If not we need another scheme. I try to think about some >>>>>>>>>>> motivating examples for this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 2. It is an error if a file cannot be found for a given string. >>>>>>>>>>>>> 3. Any command line options in effect at the end of the >>>>>>>>>>>>> command line parsing apply to autolinked inputs, e.g. --whole-archive. >>>>>>>>>>>>> 4. Duplicate autolinked inputs are ignored. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This seems like it would work in GNU linkers, as long as the >>>>>>>>>>>> autolinked file is added to the link immediately after the last mention, >>>>>>>>>>>> rather than the first. Otherwise a command line like: >>>>>>>>>>>> >>>>>>>>>>>> ld foo1.o foo2.o >>>>>>>>>>>> >>>>>>>>>>>> (where foo1.o and foo2.o both autolink bar) could end up >>>>>>>>>>>> looking like: >>>>>>>>>>>> >>>>>>>>>>>> ld foo1.o -lbar foo2.o >>>>>>>>>>>> >>>>>>>>>>>> and you will not link anything from libbar.a that only foo2.o >>>>>>>>>>>> requires. It may end up being simpler to not ignore duplicates. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Correct; but, given that the proposal was to handle the >>>>>>>>>>> libraries as if they are appended to the link line after everything on the >>>>>>>>>>> command line then I think this will work. With deduplication (and the use >>>>>>>>>>> of SHF_MERGE) developers get no ordering guarantees. I claim that this is a >>>>>>>>>>> feature! My rationale is that the order in which libraries are linked >>>>>>>>>>> affects different linkers in different ways (e.g. LLD does not resolve >>>>>>>>>>> symbols from archives in a compatible manner with either the Microsoft >>>>>>>>>>> linker or the GNU linkers.), by not allowing the user to control the order >>>>>>>>>>> I am essentially saying that autolinking is not suitable for libraries that >>>>>>>>>>> offer competing copies of the same symbol. This ties into my argument that >>>>>>>>>>> "comment lib" pragmas should be handled in as "general" a way as possible. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Right. I think if you need a fine control over the link order, >>>>>>>>>> autolinking is not a feature you want to use. Or, in general, if your >>>>>>>>>> program is sensitive to a link order because its source object files have >>>>>>>>>> competing symbols of the same name, it's perhaps unnecessarily fragile. >>>>>>>>>> >>>>>>>>>> That being said, I think you need to address the issue that pcc >>>>>>>>>> pointed out. If you statically link a program `foo` with the following >>>>>>>>>> command line >>>>>>>>>> >>>>>>>>>> ld -o foo foo.o -lc >>>>>>>>>> >>>>>>>>>> , `foo.o` auto-imports libbar.a, and libbar.a depends on libc.a, >>>>>>>>>> can your proposed feature pull out object files needed for libbar.a? >>>>>>>>>> >>>>>>>>> >>>>>>>>> It won't work on GNU linkers. It will work with LLD as LLD has >>>>>>>>> MSVC-like archive handling. However, I would like to make sure that >>>>>>>>> whatever we come up with can be supported in the GNU toolchain. >>>>>>>>> >>>>>>>>> I had thought that it would be acceptable that all the >>>>>>>>> dependencies of an autolinked library must themselves be autolinked in >>>>>>>>> order to work on GNU style linkers. Having thought more, I don't like this >>>>>>>>> limitation - especially as it doesn't exist for Microsoft style linkers. >>>>>>>>> One possible resolution could be that GNU linkers might have to implement >>>>>>>>> another command line option e.g. --auto-dep=<file> to allow injection into >>>>>>>>> the group of autolinked libraries. >>>>>>>>> >>>>>>>>> i.e In pcc's example you would need to do: "ld foo.o >>>>>>>>> --auto-dep=libc.a" which would become "ld --start-group libbar.a libc.a >>>>>>>>> --end-group" with autolinking. >>>>>>>>> >>>>>>>>> I wanted to avoid the approach of inserting autolinked libraries >>>>>>>>> after the object that autolinks them. In LLD (and MSVC) it becomes hard to >>>>>>>>> reason about "where" the linker is in the command line and it would also >>>>>>>>> mean that we can't have the nice separation between parsing the command >>>>>>>>> line and doing the rest of the link that we currently have. Also, if you >>>>>>>>> give people a way to have a fine grained control over the link order with >>>>>>>>> autolinking you risk ending up with source code that will link on GNU style >>>>>>>>> linkers but not with LLD (assuming GNU ever implemented support for >>>>>>>>> autolinking). >>>>>>>>> >>>>>>>>> Scenario: >>>>>>>>> >>>>>>>>> libbar.a(bar.o) - defines symbol bar >>>>>>>>> libfoo.a(foo.o) - defines foo and autolinks libbar.a >>>>>>>>> main.o - references foo >>>>>>>>> another.o - does not reference foo >>>>>>>>> No references to bar exist >>>>>>>>> >>>>>>>>> lld -lfoo another.o --whole-archive main.o with autolinking >>>>>>>>> becomes lld -lfoo another.o --whole-archive main.o -lbar result: bar.o gets >>>>>>>>> added to the link. >>>>>>>>> But, if a change is made so that another.o references bar then the >>>>>>>>> link line with autolinking becomes lld -lfoo another.o >>>>>>>>> -lbar --whole-archive main.o result: bar.o is not added to the link. >>>>>>>>> >>>>>>>>> Hopefully the above scenario demonstrates why I think that it >>>>>>>>> becomes too complicated to reason about the effects of autolinking with >>>>>>>>> pcc's proposed insertion scheme. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> 5. The linker tries to add a library or relocatable object file >>>>>>>>>>>>> from each of the strings in a .autolink section by; first, handling the >>>>>>>>>>>>> string as if it was specified on the commandline; second, by looking for >>>>>>>>>>>>> the string in each of the library search paths in turn; third, by looking >>>>>>>>>>>>> for a lib<string>.a or lib<string>.so (depending on the current mode of the >>>>>>>>>>>>> linker) in each of the library search paths. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is the second part necessary? "-l:foo" causes the linker to >>>>>>>>>>>> search for a file named "foo" in the library search path, so it seems that >>>>>>>>>>>> allowing the autolink string to look like ":foo" would satisfy this use >>>>>>>>>>>> case. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I worded the proposal to avoid mapping "comment lib" pragmas to >>>>>>>>>>> --library command line options. My reasons: >>>>>>>>>>> >>>>>>>>>>> 1. I find the requirement that the user put ':' in their lib >>>>>>>>>>> strings slightly awkward. It means that the source code is now coupled to a >>>>>>>>>>> GNU-style linker. So then this isn't merely an ELF linking proposal, it's a >>>>>>>>>>> proposal for ELF toolchains with GNU-like linkers (e.g. the arm linker >>>>>>>>>>> doesn't support the colon prefix >>>>>>>>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0474c/Cjahbdei.html >>>>>>>>>>> ). >>>>>>>>>>> >>>>>>>>>>> 2. The syntax is #pragma comment(lib, ...) not #pragma >>>>>>>>>>> linker-option(library, ...) i.e. the only thing this (frankly rather >>>>>>>>>>> bizarre) syntax definitely implies is that the argument is related to >>>>>>>>>>> libraries (and comments ¯\_(ツ)_/¯); it is a bit of a stretch to interpret >>>>>>>>>>> "comment lib" pragmas as mapping directly to "specifying an additional >>>>>>>>>>> --library command line option". >>>>>>>>>>> >>>>>>>>>>> AFAIK all linkers support two ways of specifying inputs; >>>>>>>>>>> firstly, directly on the command line; secondly, with an option with very >>>>>>>>>>> similar semantics to GNU's --library option. I choose a method of finding a >>>>>>>>>>> input files that encompasses both methods of specifying a library on the >>>>>>>>>>> command line. I think that this method is actually more intuitive than >>>>>>>>>>> either the method used by the linker script INPUT command or by --library. >>>>>>>>>>> FWIW, I looked into the history of the colon prefix. It was added in >>>>>>>>>>> https://www.sourceware.org/ml/binutils/2007-03/msg00421.html. >>>>>>>>>>> Unfortunately, the rationale given is that it was merely a port of a >>>>>>>>>>> vxworks linker extension. I couldn't trace the history any further than >>>>>>>>>>> that to find the actual design discussion. The linker script command INPUT >>>>>>>>>>> uses a different scheme and the command already had this search order 20 >>>>>>>>>>> years ago, which is the earliest version of the GNU linker I have history >>>>>>>>>>> for; again, the rationale is not available. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 6. A new command line option --no-llvm-autolink will tell LLD >>>>>>>>>>>>> to ignore the .autolink sections. >>>>>>>>>>>>> >>>>>>>>>>>>> Rationale for the above points: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Adding the autolinked inputs last makes the process simple >>>>>>>>>>>>> to understand from a developers perspective. All linkers are able to >>>>>>>>>>>>> implement this scheme. >>>>>>>>>>>>> 2. Error-ing for libraries that are not found seems like >>>>>>>>>>>>> better behavior than failing the link during symbol resolution. >>>>>>>>>>>>> 3. It seems useful for the user to be able to apply command >>>>>>>>>>>>> line options which will affect all of the autolinked input files. There is >>>>>>>>>>>>> a potential problem of surprise for developers, who might not realize that >>>>>>>>>>>>> these options would apply to the "invisible" autolinked input files; >>>>>>>>>>>>> however, despite the potential for surprise, this is easy for developers to >>>>>>>>>>>>> reason about and gives developers the control that they may require. >>>>>>>>>>>>> 4. Unlike on the command line it is probably easy to include >>>>>>>>>>>>> the same input file twice via pragmas and might be a pain to fix; think of >>>>>>>>>>>>> Third-party libraries supplied as binaries. >>>>>>>>>>>>> 5. This algorithm takes into account all of the different ways >>>>>>>>>>>>> that ELF linkers find input files. The different search methods are tried >>>>>>>>>>>>> by the linker in most obvious to least obvious order. >>>>>>>>>>>>> 6. I considered adding finer grained control over which >>>>>>>>>>>>> .autolink inputs were ignored (e.g. MSVC has /nodefaultlib:<library>); >>>>>>>>>>>>> however, I concluded that this is not necessary: if finer control is >>>>>>>>>>>>> required developers can recreate the same effect autolinking would have had >>>>>>>>>>>>> using command line options. >>>>>>>>>>>>> >>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> -- >>>>>>>>>>>> Peter >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>> >>>>>>>>>> >>>>>> >>>>>> -- >>>>>> Saleem Abdulrasool >>>>>> compnerd (at) compnerd (dot) org >>>>>> >>>>> >>>> >>>> -- >>>> Saleem Abdulrasool >>>> compnerd (at) compnerd (dot) org >>>> >>> >> >> -- >> Saleem Abdulrasool >> compnerd (at) compnerd (dot) org >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190327/a053a515/attachment-0001.html>
Hi Alexander, thanks for the comments. This thread is specifically about ELF autolinking. COFF and MACHO already have an autolinking scheme. The chatter about MSVC is because we are using that implementation as a reference. On Wed, Mar 27, 2019 at 2:58 PM Alexander Benikowski < sebal007 at googlemail.com> wrote:> The initial post seems to be about ELF only but along the way MSVC was > mentioned. So is this suposed to carry over for linking PE-Binaries? In > that case: > > Directly mentioning .dll files as target for autolinking. The PE import > section only has pairs/groups of functionnames/dll-file and is handled by > the runtimelinker. > > 2. Error-ing for libraries that are not found seems like better behavior >> than failing the link during symbol resolution. > > > Therefore the linker should not error out if the file is not found. There > are Compilers for the Windowsplatform that just allow this and it makes the > whole thing a lot more comfortable. > > Am Mi., 27. März 2019 um 13:18 Uhr schrieb bd1976 llvm via llvm-dev < > llvm-dev at lists.llvm.org>: > >> On Wed, Mar 27, 2019 at 2:03 AM Saleem Abdulrasool <compnerd at compnerd.org> >> wrote: >> >>> On Tue, Mar 26, 2019 at 11:13 AM bd1976 llvm <bd1976llvm at gmail.com> >>> wrote: >>> >>>> Thanks Saleem/James. >>>> >>>> On Tue, Mar 26, 2019 at 5:15 AM Saleem Abdulrasool < >>>> compnerd at compnerd.org> wrote: >>>> >>>>> Yes, I don't think that supporting all options is entirely possible. >>>>> But, there is certainly some allure to it, as it is a really powerful >>>>> feature (which you can tell from the behaviour of link and `.drectve` >>>>> section). The problem with all options is that ELF's model is not as >>>>> simplistic as link's model, which really complicates things. As an >>>>> example: `/WHOLEARCHIVE:` rather than `--whole-archive` ... >>>>> `--no-whole-archive`. I'm also worried about its use in a malicious >>>>> manner: `--no-pie` being embedded into the executable would disable ASLR, >>>>> which really is why I think that we would want a subset of options. >>>>> >>>> >>>> I'm not opposed to a linker options scheme; but, I think that it is >>>> important to separate this from autolinking (as we have proposed here). >>>> Rationale is upthread. We can support both if required. >>>> >>> >>> I'm not sure I understand how the two are distinct. The linker options >>> scheme being discussed is in the context of autolinking - aka, how to drive >>> the autolinking. >>> >> >> I'm imagining: >> >> #pragma comment(lib, "foo") => add "foo" to the link in a "general way". >> This is autolinking. >> #pragma comment(linker-option, "-l:foo") => Add "-l:foo" to the linker >> command line. This is linker options. >> >> Now it might be that on a specific platform (e.g. MSVC) you can represent >> #pragma comment(lib, "foo") using linker options; however, in general they >> are distinct. I would like the following behaviour for ELF:.. >> >> #pragma comment(lib, "foo") => add libfoo.a to the link. >> #pragma comment(lib, "foo.a") => add foo.a to the link >> #pragma comment(lib, "c:\\foo.a") => add c:\foo.a to the link >> >> .. but there is no existing ELF linker command line option that maps to >> this behaviour; so, it is better to have a separate representation. Also, >> given that it seems that autolinking is the only usecase currently, the >> proposed .autolink section is a simpler (e.g. binary tools can dump the >> section without needing special rules), and more efficient (bytes needed to >> convey the information) format than .linker-options. >> >> >>> >>> >>>> On Mon, Mar 25, 2019 at 8:47 AM Rui Ueyama <ruiu at google.com> wrote: >>>>> >>>>>> On Sun, Mar 24, 2019 at 9:51 PM Saleem Abdulrasool < >>>>>> compnerd at compnerd.org> wrote: >>>>>> >>>>>>> Sorry for the late chiming in. >>>>>>> >>>>>>> Yes, swift does use autolinking, and I would like to use that on all >>>>>>> the targets. The only target which does not support this functionality >>>>>>> currently are ELF based. That said, I think that `#pragma comment(link, >>>>>>> ...)` is insufficient for my needs. Building Foundation requires framework >>>>>>> style linking as well. The original design that I had in mind was derived >>>>>>> from ld64 and link. Personally, I still strongly favour link's behaviour >>>>>>> of parsing "command line" options from the object files when they are >>>>>>> loaded. There was strong opposition to that approach from Rui though. >>>>>>> Would we want to have special pragmas for each "feature"? >>>>>>> >>>>>> >>>> I was hoping that the autolinking proposal here was general enough that >>>> it could be made to work for you. What if we made the library search >>>> algorithm even more general than I initially proposed. Something like: >>>> >>>> - The linker tries to add a library or relocatable object file from >>>> each of the strings in a .autolink section by; first, handling the string >>>> as if it was specified on the commandline; second, by looking for the >>>> string in each of the library search paths in turn; third, by looking for a >>>> lib<string>.a or lib<string>.so (depending on the current mode of the >>>> linker) in each of the library search paths; forth, by looking for the >>>> string in each of the framework search paths in turn. >>>> >>> >>> What happens when you have Alpha.framework[/libAlpha.dylib] and >>> libAlpha.dylib? `-framework Alpha` should take >>> Alpha.framework/libAlpha.dylib and *NOT* libAlpha.dylib. I see no way to >>> differentiate between the two by mandating a specific ordering. -lAlpha -> >>> libAlpha.dylib and -framework Alpha -> Alpha.framework/libAlpha.dylib. >>> This is obviously a contrived case, but, the problem is that there is >>> ambiguity introduced which you cannot control without having additional >>> context from the user. So, `#pragma comment(link, ...)` is insufficient >>> IMO. >>> >> >>> >>>> So if you have #pragma comment(lib, "foo") in a source file, then on a >>>> linker that supports framework linking the linker would add to the link the >>>> first matching file trying: >>>> >>>> ./foo >>>> <first library search path>/foo >>>> <second library search path>/foo >>>> .. >>>> <nth library search path>/foo >>>> <first library search path>/libfoo.so >>>> <second library search path>/libfoo.so >>>> .. >>>> <nth library search path>/libfoo.so >>>> >>>> <first library search path>/libfoo.a >>>> <second library search path>/libfoo.a >>>> .. >>>> <nth library search path>/libfoo.a >>>> <first framework search path>/foo.framework/foo >>>> <second library search path>/foo.framework/foo >>>> .. >>>> <nth library search path>/foo.framework/foo >>>> >>>> What do you think? >>>> >>> >>> I think that it just broke in my example above :-). I think that we >>> need to realize that this is complex due to the fact that it requires >>> knowledge of what the user wants to do (which is normally expressed through >>> the invocation of the tools). So, we need to expose that to the >>> implementation. >>> >> >> The question is how contrived is your case? Upthread, I think that we >> have established that to keep autolinking reasonably understandable and >> consistent across linkers you have to accept some limitations. I'm hoping >> that we can accept this approach of finding libraries as a reasonable >> limitation. The consequences are that if a codebase has the example >> situation that you have described, you would either have to restructure the >> codebase to avoid the ambiguity, or you can't use autolinking for this >> case. The upside is that no matter if your linker supports frameworks or >> libraries #pragma comment(lib, "foo") will work in all cases. Otherwise we >> will be in a situation where you will have to have the equivalent of... >> >> #ifdef COFF_LIBRARIES: >> #pragma comment(linker-option, "/DEFAULTLIB:foo.lib") >> #'elif ELF_LIBRARIES: >> #pragma comment(linker-option, "-lfoo") >> #'elif DARWIN_FRAMEWORKS: >> #pragma comment(linker-option, "-framework foo") >> #esle >> #error Please specify linking model >> #endif >> >> ... somewhere, it would be good to avoid this. >> >> OTOH perhaps framework linking is just too different to be lumped in with >> #pragma comment(lib, ...) perhaps we should restrict #pragma comment(lib, >> ...) to libraries and implement support for another pragma, e.g. #pragma >> comment(framework, ...), to support framework linking. >> >> >>> >>>> I think as you double-quoted the words "command line", I believe we are >>>>>> on the same page, but what I opposed was to allow arbitrary command line >>>>>> options in the .linker-options section. That is really a can of worms that >>>>>> we don't want to open. I didn't oppose allowing a limited set of >>>>>> command-line-ish directives. That said I'd prefer the proposal of this >>>>>> thread over the .linker-options though. >>>>>> >>>>>> The ELF model doesn't have the simplistic model for processing the >>>>>>> command line that PE/COFF does. Because ordering is relevant to the model, >>>>>>> it would be ideal to process them inline, but, since lld already moves far >>>>>>> enough away from the traditional Unix model, perhaps we can simplify it to >>>>>>> append the command line directives to the end of the command line. >>>>>>> >>>>>>> The other case that is interesting to think about is the autolinking >>>>>>> support in C++ (and clang) modules. >>>>>>> >>>>>> >>>> When I started this work I looked at the C++ 20 modules proposal: >>>> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf. I >>>> didn't see anything about autolinking (or linking at all)! Can you explain >>>> more? >>>> >>> >>> I may have been confusing features from the clang module functionality >>> and C++20 Modules TS (though, I thought that it included support for >>> autolinking dependencies for cases where an imported module unit has >>> available_externally linkage). >>> >> >> Right. So the Clang modules document does mention linking: >> https://clang.llvm.org/docs/Modules.html#link-declaration. Clang modules >> are different to the proposed C++20 modules. I assumed that Clang modules >> support existed only as a prototype to demonstrate how modules "could" work >> to guide the design/discussions for C++ modules. I don't understand whether >> Clang modules are in use by anyone? What will happen to Clang modules when >> C++20 arrives? Even if Clang modules are being used perhaps the linking >> part of the specification can be changed (as it seems unlikely that this is >> used as ELF linkers don't have a mechanism to support it)? >> >> >>> >>> >>>> >>>> >>>>>>> On Thu, Mar 21, 2019 at 9:49 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Mar 21, 2019 at 12:06 AM Rui Ueyama <ruiu at google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Perhaps there's no one clean way to solve this issue, because >>>>>>>>> previously all libraries and object files are explicitly given to the >>>>>>>>> linker via a command line and the order of files in the command line >>>>>>>>> matters. That assumes human intervention to work correctly. Now, the >>>>>>>>> autolinking feature will add libraries implicitly. Since it's implicit, >>>>>>>>> there will be only one way how that works, so sometimes that works and >>>>>>>>> sometimes doesn't. >>>>>>>>> >>>>>>>>> It feels to me that we should aim for making it work reasonably >>>>>>>>> well for reasonable use cases. By reasonable use cases, I'm thinking of the >>>>>>>>> following: >>>>>>>>> >>>>>>>>> 1. --static option may or may not be given (i.e. we should allow >>>>>>>>> that feature for both static linking and dynamic linking.) >>>>>>>>> 2. There are no competing defined symbols in a given set of >>>>>>>>> libraries, or if they exist, the program owner doesn't care which is linked >>>>>>>>> to their program. >>>>>>>>> 3. There may be circular dependencies between libraries. >>>>>>>>> >>>>>>>>> I don't think the above assumption is too odd. If I have to >>>>>>>>> implement the autolinking feature to GNU linker for the above scenario, I'd >>>>>>>>> probably use the following scheme: >>>>>>>>> >>>>>>>>> 1. While reading object files, memorize libraries that are >>>>>>>>> autolinked >>>>>>>>> 2. After linking everything, create a list of files consisting of >>>>>>>>> autolinked libraries AND libraries given via the command line >>>>>>>>> 3. Visit each file in the list as if they were wrapped in >>>>>>>>> --start-group and --end-group. >>>>>>>>> >>>>>>>>> I'd think the above scheme should work reasonably well. What do >>>>>>>>> you think? >>>>>>>>> >>>>>>>> >>>>>>>> Very nice. I agree with your definition of "reasonable" usecaes >>>>>>>> (actually, as I have said before, I think that restricting autolinking to >>>>>>>> this "reasonable" set is actually a feature - to avoid developers having >>>>>>>> source code that only works with a particular linker). I also like the >>>>>>>> proposal for a GNU implementation - I think this is enough to show that >>>>>>>> GNU-like linkers could implement this. >>>>>>>> >>>>>>>> At this point I will try to prototype this up so that people have >>>>>>>> an implementation to play with. >>>>>>>> >>>>>>>> I am keen to hear from Saleem (compnerd) on this, as he did the >>>>>>>> original .linker-options work. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 19, 2019 at 11:02 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Mon, Mar 18, 2019 at 8:02 PM Rui Ueyama <ruiu at google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 14, 2019 at 1:05 PM bd1976 llvm via llvm-dev < >>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:27 PM Peter Collingbourne < >>>>>>>>>>>> peter at pcc.me.uk> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev < >>>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> At Sony we offer autolinking as a feature in our ELF >>>>>>>>>>>>>> toolchain. We would like to see full support for this feature upstream as >>>>>>>>>>>>>> there is anecdotal evidence that it would find use beyond Sony. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In general autolinking ( >>>>>>>>>>>>>> https://en.wikipedia.org/wiki/Auto-linking) allows >>>>>>>>>>>>>> developers to specify inputs to the linker in their source code. LLVM and >>>>>>>>>>>>>> Clang already have support for autolinking on ELF via embedding strings, >>>>>>>>>>>>>> which specify linker behavior, into a .linker-options section in >>>>>>>>>>>>>> relocatable object files, see: >>>>>>>>>>>>>> >>>>>>>>>>>>>> RFC - >>>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html >>>>>>>>>>>>>> LLVM - >>>>>>>>>>>>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options, >>>>>>>>>>>>>> https://reviews.llvm.org/D40849 >>>>>>>>>>>>>> Clang - >>>>>>>>>>>>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets, >>>>>>>>>>>>>> https://reviews.llvm.org/D42758 >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, although support was added to Clang and LLVM, no >>>>>>>>>>>>>> support has been implemented in LLD; and, I get the sense, from reading the >>>>>>>>>>>>>> reviews, that there wasn't agreement on the implementation when the changes >>>>>>>>>>>>>> landed. The original motivation seems to have been to remove the >>>>>>>>>>>>>> "autolink-extract" mechanism used by Swift to workaround the lack of >>>>>>>>>>>>>> autolinking support for ELF. However, looking at the Swift source code, >>>>>>>>>>>>>> Swift still seems to be using the "autolink-extract" method. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So my first question: Are there any users of the current >>>>>>>>>>>>>> implementation for ELF? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Assuming that no one is using the current code, I would like >>>>>>>>>>>>>> to suggest a different mechanism for autolinking. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For ELF we need limited autolinking support. Specifically, we >>>>>>>>>>>>>> only need support for "comment lib" pragmas ( >>>>>>>>>>>>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) >>>>>>>>>>>>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the >>>>>>>>>>>>>> implementation as lean as possible. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Principles to guide the implementation: >>>>>>>>>>>>>> - Developers should be able to easily understand autolinking >>>>>>>>>>>>>> behavior. >>>>>>>>>>>>>> - Developers should be able to override autolinking from the >>>>>>>>>>>>>> linker command line. >>>>>>>>>>>>>> - Inputs specified via pragmas should be handled in a general >>>>>>>>>>>>>> way to allow the same source code to work in different environments. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would like to propose that we focus on autolinking >>>>>>>>>>>>>> exclusively and that we divorce the implementation from the idea of "linker >>>>>>>>>>>>>> options" which, by nature, would tie source code to the vagaries of >>>>>>>>>>>>>> particular linkers. I don't see much value in supporting other linker >>>>>>>>>>>>>> operations so I suggest that the binary representation be a mergable string >>>>>>>>>>>>>> section (SHF_MERGE, SHF_STRINGS), called .autolink, with custom type >>>>>>>>>>>>>> SHT_LLVM_AUTOLINK (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents >>>>>>>>>>>>>> appearing in the output). The compiler can form this section by >>>>>>>>>>>>>> concatenating the arguments of the "comment lib" pragmas in the order they >>>>>>>>>>>>>> are encountered. Partial (-r, -Ur) links can be handled by concatenating >>>>>>>>>>>>>> .autolink sections with the normal mergeable string section rules. The >>>>>>>>>>>>>> current .linker-options can remain (or be removed); but, "comment lib" >>>>>>>>>>>>>> pragmas for ELF should be lowered to .autolink not to .linker-options. This >>>>>>>>>>>>>> makes sense as there is no linker option that "comment lib" pragmas map >>>>>>>>>>>>>> directly to. As an example, #pragma comment(lib, "foo") would result in: >>>>>>>>>>>>>> >>>>>>>>>>>>>> .section ".autolink","eMS", at llvm_autolink,1 >>>>>>>>>>>>>> .asciz "foo" >>>>>>>>>>>>>> >>>>>>>>>>>>>> For LTO, equivalent information to the contents of a the >>>>>>>>>>>>>> .autolink section will be written to the IRSymtab so that it is available >>>>>>>>>>>>>> to the linker for symbol resolution. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The linker will process the .autolink strings in the >>>>>>>>>>>>>> following way: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Inputs from the .autolink sections of a relocatable object >>>>>>>>>>>>>> file are added when the linker decides to include that file (which could >>>>>>>>>>>>>> itself be in a library) in the link. Autolinked inputs behave as if they >>>>>>>>>>>>>> were appended to the command line as a group after all other options. As a >>>>>>>>>>>>>> consequence the set of autolinked libraries are searched last to resolve >>>>>>>>>>>>>> symbols. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If we want this to be compatible with GNU linkers, doesn't the >>>>>>>>>>>>> autolinked input need to appear at the point immediately after the object >>>>>>>>>>>>> file appears in the link? I'm imagining the case where you have a >>>>>>>>>>>>> statically linked libc as well as a libbar.a autolinked from a foo.o. The >>>>>>>>>>>>> link command line would look like this: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo.o -lc >>>>>>>>>>>>> >>>>>>>>>>>>> Now foo.o autolinks against bar. The command line becomes: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo.o -lc -lbar >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Actually, I was thinking that on a GNU linker the command line >>>>>>>>>>>> would become "ld foo.o -lc -( -lbar )-"; but, this doesn't affect your >>>>>>>>>>>> point. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If libbar.a requires an additional object file from libc.a, it >>>>>>>>>>>>> will not be added to the link. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> As it stands all the dependencies of an autolinked library must >>>>>>>>>>>> themselves be autolinked. I had imagined that this is a reasonable >>>>>>>>>>>> limitation. If not we need another scheme. I try to think about some >>>>>>>>>>>> motivating examples for this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 2. It is an error if a file cannot be found for a given string. >>>>>>>>>>>>>> 3. Any command line options in effect at the end of the >>>>>>>>>>>>>> command line parsing apply to autolinked inputs, e.g. --whole-archive. >>>>>>>>>>>>>> 4. Duplicate autolinked inputs are ignored. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> This seems like it would work in GNU linkers, as long as the >>>>>>>>>>>>> autolinked file is added to the link immediately after the last mention, >>>>>>>>>>>>> rather than the first. Otherwise a command line like: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo1.o foo2.o >>>>>>>>>>>>> >>>>>>>>>>>>> (where foo1.o and foo2.o both autolink bar) could end up >>>>>>>>>>>>> looking like: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo1.o -lbar foo2.o >>>>>>>>>>>>> >>>>>>>>>>>>> and you will not link anything from libbar.a that only foo2.o >>>>>>>>>>>>> requires. It may end up being simpler to not ignore duplicates. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Correct; but, given that the proposal was to handle the >>>>>>>>>>>> libraries as if they are appended to the link line after everything on the >>>>>>>>>>>> command line then I think this will work. With deduplication (and the use >>>>>>>>>>>> of SHF_MERGE) developers get no ordering guarantees. I claim that this is a >>>>>>>>>>>> feature! My rationale is that the order in which libraries are linked >>>>>>>>>>>> affects different linkers in different ways (e.g. LLD does not resolve >>>>>>>>>>>> symbols from archives in a compatible manner with either the Microsoft >>>>>>>>>>>> linker or the GNU linkers.), by not allowing the user to control the order >>>>>>>>>>>> I am essentially saying that autolinking is not suitable for libraries that >>>>>>>>>>>> offer competing copies of the same symbol. This ties into my argument that >>>>>>>>>>>> "comment lib" pragmas should be handled in as "general" a way as possible. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Right. I think if you need a fine control over the link order, >>>>>>>>>>> autolinking is not a feature you want to use. Or, in general, if your >>>>>>>>>>> program is sensitive to a link order because its source object files have >>>>>>>>>>> competing symbols of the same name, it's perhaps unnecessarily fragile. >>>>>>>>>>> >>>>>>>>>>> That being said, I think you need to address the issue that pcc >>>>>>>>>>> pointed out. If you statically link a program `foo` with the following >>>>>>>>>>> command line >>>>>>>>>>> >>>>>>>>>>> ld -o foo foo.o -lc >>>>>>>>>>> >>>>>>>>>>> , `foo.o` auto-imports libbar.a, and libbar.a depends on libc.a, >>>>>>>>>>> can your proposed feature pull out object files needed for libbar.a? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It won't work on GNU linkers. It will work with LLD as LLD has >>>>>>>>>> MSVC-like archive handling. However, I would like to make sure that >>>>>>>>>> whatever we come up with can be supported in the GNU toolchain. >>>>>>>>>> >>>>>>>>>> I had thought that it would be acceptable that all the >>>>>>>>>> dependencies of an autolinked library must themselves be autolinked in >>>>>>>>>> order to work on GNU style linkers. Having thought more, I don't like this >>>>>>>>>> limitation - especially as it doesn't exist for Microsoft style linkers. >>>>>>>>>> One possible resolution could be that GNU linkers might have to implement >>>>>>>>>> another command line option e.g. --auto-dep=<file> to allow injection into >>>>>>>>>> the group of autolinked libraries. >>>>>>>>>> >>>>>>>>>> i.e In pcc's example you would need to do: "ld foo.o >>>>>>>>>> --auto-dep=libc.a" which would become "ld --start-group libbar.a libc.a >>>>>>>>>> --end-group" with autolinking. >>>>>>>>>> >>>>>>>>>> I wanted to avoid the approach of inserting autolinked libraries >>>>>>>>>> after the object that autolinks them. In LLD (and MSVC) it becomes hard to >>>>>>>>>> reason about "where" the linker is in the command line and it would also >>>>>>>>>> mean that we can't have the nice separation between parsing the command >>>>>>>>>> line and doing the rest of the link that we currently have. Also, if you >>>>>>>>>> give people a way to have a fine grained control over the link order with >>>>>>>>>> autolinking you risk ending up with source code that will link on GNU style >>>>>>>>>> linkers but not with LLD (assuming GNU ever implemented support for >>>>>>>>>> autolinking). >>>>>>>>>> >>>>>>>>>> Scenario: >>>>>>>>>> >>>>>>>>>> libbar.a(bar.o) - defines symbol bar >>>>>>>>>> libfoo.a(foo.o) - defines foo and autolinks libbar.a >>>>>>>>>> main.o - references foo >>>>>>>>>> another.o - does not reference foo >>>>>>>>>> No references to bar exist >>>>>>>>>> >>>>>>>>>> lld -lfoo another.o --whole-archive main.o with autolinking >>>>>>>>>> becomes lld -lfoo another.o --whole-archive main.o -lbar result: bar.o gets >>>>>>>>>> added to the link. >>>>>>>>>> But, if a change is made so that another.o references bar then >>>>>>>>>> the link line with autolinking becomes lld -lfoo another.o >>>>>>>>>> -lbar --whole-archive main.o result: bar.o is not added to the link. >>>>>>>>>> >>>>>>>>>> Hopefully the above scenario demonstrates why I think that it >>>>>>>>>> becomes too complicated to reason about the effects of autolinking with >>>>>>>>>> pcc's proposed insertion scheme. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 5. The linker tries to add a library or relocatable object file >>>>>>>>>>>>>> from each of the strings in a .autolink section by; first, handling the >>>>>>>>>>>>>> string as if it was specified on the commandline; second, by looking for >>>>>>>>>>>>>> the string in each of the library search paths in turn; third, by looking >>>>>>>>>>>>>> for a lib<string>.a or lib<string>.so (depending on the current mode of the >>>>>>>>>>>>>> linker) in each of the library search paths. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Is the second part necessary? "-l:foo" causes the linker to >>>>>>>>>>>>> search for a file named "foo" in the library search path, so it seems that >>>>>>>>>>>>> allowing the autolink string to look like ":foo" would satisfy this use >>>>>>>>>>>>> case. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I worded the proposal to avoid mapping "comment lib" pragmas to >>>>>>>>>>>> --library command line options. My reasons: >>>>>>>>>>>> >>>>>>>>>>>> 1. I find the requirement that the user put ':' in their lib >>>>>>>>>>>> strings slightly awkward. It means that the source code is now coupled to a >>>>>>>>>>>> GNU-style linker. So then this isn't merely an ELF linking proposal, it's a >>>>>>>>>>>> proposal for ELF toolchains with GNU-like linkers (e.g. the arm linker >>>>>>>>>>>> doesn't support the colon prefix >>>>>>>>>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0474c/Cjahbdei.html >>>>>>>>>>>> ). >>>>>>>>>>>> >>>>>>>>>>>> 2. The syntax is #pragma comment(lib, ...) not #pragma >>>>>>>>>>>> linker-option(library, ...) i.e. the only thing this (frankly rather >>>>>>>>>>>> bizarre) syntax definitely implies is that the argument is related to >>>>>>>>>>>> libraries (and comments ¯\_(ツ)_/¯); it is a bit of a stretch to interpret >>>>>>>>>>>> "comment lib" pragmas as mapping directly to "specifying an additional >>>>>>>>>>>> --library command line option". >>>>>>>>>>>> >>>>>>>>>>>> AFAIK all linkers support two ways of specifying inputs; >>>>>>>>>>>> firstly, directly on the command line; secondly, with an option with very >>>>>>>>>>>> similar semantics to GNU's --library option. I choose a method of finding a >>>>>>>>>>>> input files that encompasses both methods of specifying a library on the >>>>>>>>>>>> command line. I think that this method is actually more intuitive than >>>>>>>>>>>> either the method used by the linker script INPUT command or by --library. >>>>>>>>>>>> FWIW, I looked into the history of the colon prefix. It was added in >>>>>>>>>>>> https://www.sourceware.org/ml/binutils/2007-03/msg00421.html. >>>>>>>>>>>> Unfortunately, the rationale given is that it was merely a port of a >>>>>>>>>>>> vxworks linker extension. I couldn't trace the history any further than >>>>>>>>>>>> that to find the actual design discussion. The linker script command INPUT >>>>>>>>>>>> uses a different scheme and the command already had this search order 20 >>>>>>>>>>>> years ago, which is the earliest version of the GNU linker I have history >>>>>>>>>>>> for; again, the rationale is not available. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 6. A new command line option --no-llvm-autolink will tell LLD >>>>>>>>>>>>>> to ignore the .autolink sections. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rationale for the above points: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Adding the autolinked inputs last makes the process simple >>>>>>>>>>>>>> to understand from a developers perspective. All linkers are able to >>>>>>>>>>>>>> implement this scheme. >>>>>>>>>>>>>> 2. Error-ing for libraries that are not found seems like >>>>>>>>>>>>>> better behavior than failing the link during symbol resolution. >>>>>>>>>>>>>> 3. It seems useful for the user to be able to apply command >>>>>>>>>>>>>> line options which will affect all of the autolinked input files. There is >>>>>>>>>>>>>> a potential problem of surprise for developers, who might not realize that >>>>>>>>>>>>>> these options would apply to the "invisible" autolinked input files; >>>>>>>>>>>>>> however, despite the potential for surprise, this is easy for developers to >>>>>>>>>>>>>> reason about and gives developers the control that they may require. >>>>>>>>>>>>>> 4. Unlike on the command line it is probably easy to include >>>>>>>>>>>>>> the same input file twice via pragmas and might be a pain to fix; think of >>>>>>>>>>>>>> Third-party libraries supplied as binaries. >>>>>>>>>>>>>> 5. This algorithm takes into account all of the different >>>>>>>>>>>>>> ways that ELF linkers find input files. The different search methods are >>>>>>>>>>>>>> tried by the linker in most obvious to least obvious order. >>>>>>>>>>>>>> 6. I considered adding finer grained control over which >>>>>>>>>>>>>> .autolink inputs were ignored (e.g. MSVC has /nodefaultlib:<library>); >>>>>>>>>>>>>> however, I concluded that this is not necessary: if finer control is >>>>>>>>>>>>>> required developers can recreate the same effect autolinking would have had >>>>>>>>>>>>>> using command line options. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> -- >>>>>>>>>>>>> Peter >>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Saleem Abdulrasool >>>>>>> compnerd (at) compnerd (dot) org >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Saleem Abdulrasool >>>>> compnerd (at) compnerd (dot) org >>>>> >>>> >>> >>> -- >>> Saleem Abdulrasool >>> compnerd (at) compnerd (dot) org >>> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190327/5ed8f484/attachment.html>
On Wed, Mar 27, 2019 at 5:18 AM bd1976 llvm <bd1976llvm at gmail.com> wrote:> On Wed, Mar 27, 2019 at 2:03 AM Saleem Abdulrasool <compnerd at compnerd.org> > wrote: > >> On Tue, Mar 26, 2019 at 11:13 AM bd1976 llvm <bd1976llvm at gmail.com> >> wrote: >> >>> Thanks Saleem/James. >>> >>> On Tue, Mar 26, 2019 at 5:15 AM Saleem Abdulrasool < >>> compnerd at compnerd.org> wrote: >>> >>>> Yes, I don't think that supporting all options is entirely possible. >>>> But, there is certainly some allure to it, as it is a really powerful >>>> feature (which you can tell from the behaviour of link and `.drectve` >>>> section). The problem with all options is that ELF's model is not as >>>> simplistic as link's model, which really complicates things. As an >>>> example: `/WHOLEARCHIVE:` rather than `--whole-archive` ... >>>> `--no-whole-archive`. I'm also worried about its use in a malicious >>>> manner: `--no-pie` being embedded into the executable would disable ASLR, >>>> which really is why I think that we would want a subset of options. >>>> >>> >>> I'm not opposed to a linker options scheme; but, I think that it is >>> important to separate this from autolinking (as we have proposed here). >>> Rationale is upthread. We can support both if required. >>> >> >> I'm not sure I understand how the two are distinct. The linker options >> scheme being discussed is in the context of autolinking - aka, how to drive >> the autolinking. >> > > I'm imagining: > > #pragma comment(lib, "foo") => add "foo" to the link in a "general way". > This is autolinking. > #pragma comment(linker-option, "-l:foo") => Add "-l:foo" to the linker > command line. This is linker options. >What about `#pragma comment(linker-option, "-lfoo")`? Note that `-l:foo` and `-lfoo` have different behaviours.> Now it might be that on a specific platform (e.g. MSVC) you can represent > #pragma comment(lib, "foo") using linker options; however, in general they > are distinct. I would like the following behaviour for ELF:.. > > #pragma comment(lib, "foo") => add libfoo.a to the link. > #pragma comment(lib, "foo.a") => add foo.a to the link > #pragma comment(lib, "c:\\foo.a") => add c:\foo.a to the link >Ah, but the beauty is that they are *NOT* distinct. All three of those have the same semantics if passed on the command line, so that is all the same as: `#pragma comment(lib, ...)`.> .. but there is no existing ELF linker command line option that maps to > this behaviour; so, it is better to have a separate representation. Also, > given that it seems that autolinking is the only usecase currently, the > proposed .autolink section is a simpler (e.g. binary tools can dump the > section without needing special rules), and more efficient (bytes needed to > convey the information) format than .linker-options. >I'm not sure I agree with that. All of those map directly to the semantics of passing that directly on the command line as options (with the exception being in the first one it is spelt `-lfoo`. I am not against using the textual representation, but, am in fact, for that. Rui wanted the "structured" approached.> >> >>> On Mon, Mar 25, 2019 at 8:47 AM Rui Ueyama <ruiu at google.com> wrote: >>>> >>>>> On Sun, Mar 24, 2019 at 9:51 PM Saleem Abdulrasool < >>>>> compnerd at compnerd.org> wrote: >>>>> >>>>>> Sorry for the late chiming in. >>>>>> >>>>>> Yes, swift does use autolinking, and I would like to use that on all >>>>>> the targets. The only target which does not support this functionality >>>>>> currently are ELF based. That said, I think that `#pragma comment(link, >>>>>> ...)` is insufficient for my needs. Building Foundation requires framework >>>>>> style linking as well. The original design that I had in mind was derived >>>>>> from ld64 and link. Personally, I still strongly favour link's behaviour >>>>>> of parsing "command line" options from the object files when they are >>>>>> loaded. There was strong opposition to that approach from Rui though. >>>>>> Would we want to have special pragmas for each "feature"? >>>>>> >>>>> >>> I was hoping that the autolinking proposal here was general enough that >>> it could be made to work for you. What if we made the library search >>> algorithm even more general than I initially proposed. Something like: >>> >>> - The linker tries to add a library or relocatable object file from each >>> of the strings in a .autolink section by; first, handling the string as if >>> it was specified on the commandline; second, by looking for the string in >>> each of the library search paths in turn; third, by looking for a >>> lib<string>.a or lib<string>.so (depending on the current mode of the >>> linker) in each of the library search paths; forth, by looking for the >>> string in each of the framework search paths in turn. >>> >> >> What happens when you have Alpha.framework[/libAlpha.dylib] and >> libAlpha.dylib? `-framework Alpha` should take >> Alpha.framework/libAlpha.dylib and *NOT* libAlpha.dylib. I see no way to >> differentiate between the two by mandating a specific ordering. -lAlpha -> >> libAlpha.dylib and -framework Alpha -> Alpha.framework/libAlpha.dylib. >> This is obviously a contrived case, but, the problem is that there is >> ambiguity introduced which you cannot control without having additional >> context from the user. So, `#pragma comment(link, ...)` is insufficient >> IMO. >> > >> >>> So if you have #pragma comment(lib, "foo") in a source file, then on a >>> linker that supports framework linking the linker would add to the link the >>> first matching file trying: >>> >>> ./foo >>> <first library search path>/foo >>> <second library search path>/foo >>> .. >>> <nth library search path>/foo >>> <first library search path>/libfoo.so >>> <second library search path>/libfoo.so >>> .. >>> <nth library search path>/libfoo.so >>> >>> <first library search path>/libfoo.a >>> <second library search path>/libfoo.a >>> .. >>> <nth library search path>/libfoo.a >>> <first framework search path>/foo.framework/foo >>> <second library search path>/foo.framework/foo >>> .. >>> <nth library search path>/foo.framework/foo >>> >>> What do you think? >>> >> >> I think that it just broke in my example above :-). I think that we need >> to realize that this is complex due to the fact that it requires knowledge >> of what the user wants to do (which is normally expressed through the >> invocation of the tools). So, we need to expose that to the implementation. >> > > The question is how contrived is your case? Upthread, I think that we have > established that to keep autolinking reasonably understandable and > consistent across linkers you have to accept some limitations. I'm hoping > that we can accept this approach of finding libraries as a reasonable > limitation. The consequences are that if a codebase has the example > situation that you have described, you would either have to restructure the > codebase to avoid the ambiguity, or you can't use autolinking for this > case. The upside is that no matter if your linker supports frameworks or > libraries #pragma comment(lib, "foo") will work in all cases. Otherwise we > will be in a situation where you will have to have the equivalent of... >Sure, some limitations are acceptable: like no --no-pie etc. I don't think that it is unreasonable to design for something like frameworks. The problem is the conflicting opinions on having a structured interface without having some sort of structure defined. I don't think that it is that contrived in practicality, just contrived to point out that there is no one resolution that solves all the cases. Had the patches for lld been merged already, we would be now discussing how to add support for something new, so I do think that we have two options: design with this functionality or go with a second means of doing this "simple" autolinking case.> #ifdef COFF_LIBRARIES: > #pragma comment(linker-option, "/DEFAULTLIB:foo.lib") > #'elif ELF_LIBRARIES: > #pragma comment(linker-option, "-lfoo") > #'elif DARWIN_FRAMEWORKS: > #pragma comment(linker-option, "-framework foo") > #esle > #error Please specify linking model > #endif > > ... somewhere, it would be good to avoid this. > > OTOH perhaps framework linking is just too different to be lumped in with > #pragma comment(lib, ...) perhaps we should restrict #pragma comment(lib, > ...) to libraries and implement support for another pragma, e.g. #pragma > comment(framework, ...), to support framework linking. >That is certainly reasonable. So, we could do separate comment types for each bit. Note that "lib" and "linker" are two different types of comments, the former is for autolinking only, the latter is for linker options. Modelling that way seems reasonable to me (where lib only allows you to specify the name of the library to link, no additional functionality like static/shared linking, etc).> > >> >>> I think as you double-quoted the words "command line", I believe we are >>>>> on the same page, but what I opposed was to allow arbitrary command line >>>>> options in the .linker-options section. That is really a can of worms that >>>>> we don't want to open. I didn't oppose allowing a limited set of >>>>> command-line-ish directives. That said I'd prefer the proposal of this >>>>> thread over the .linker-options though. >>>>> >>>>> The ELF model doesn't have the simplistic model for processing the >>>>>> command line that PE/COFF does. Because ordering is relevant to the model, >>>>>> it would be ideal to process them inline, but, since lld already moves far >>>>>> enough away from the traditional Unix model, perhaps we can simplify it to >>>>>> append the command line directives to the end of the command line. >>>>>> >>>>>> The other case that is interesting to think about is the autolinking >>>>>> support in C++ (and clang) modules. >>>>>> >>>>> >>> When I started this work I looked at the C++ 20 modules proposal: >>> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf. I >>> didn't see anything about autolinking (or linking at all)! Can you explain >>> more? >>> >> >> I may have been confusing features from the clang module functionality >> and C++20 Modules TS (though, I thought that it included support for >> autolinking dependencies for cases where an imported module unit has >> available_externally linkage). >> > > Right. So the Clang modules document does mention linking: > https://clang.llvm.org/docs/Modules.html#link-declaration. Clang modules > are different to the proposed C++20 modules. I assumed that Clang modules > support existed only as a prototype to demonstrate how modules "could" work > to guide the design/discussions for C++ modules. I don't understand whether > Clang modules are in use by anyone? What will happen to Clang modules when > C++20 arrives? Even if Clang modules are being used perhaps the linking > part of the specification can be changed (as it seems unlikely that this is > used as ELF linkers don't have a mechanism to support it)? > > >> >> >>> >>> >>>>>> On Thu, Mar 21, 2019 at 9:49 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Mar 21, 2019 at 12:06 AM Rui Ueyama <ruiu at google.com> wrote: >>>>>>> >>>>>>>> Perhaps there's no one clean way to solve this issue, because >>>>>>>> previously all libraries and object files are explicitly given to the >>>>>>>> linker via a command line and the order of files in the command line >>>>>>>> matters. That assumes human intervention to work correctly. Now, the >>>>>>>> autolinking feature will add libraries implicitly. Since it's implicit, >>>>>>>> there will be only one way how that works, so sometimes that works and >>>>>>>> sometimes doesn't. >>>>>>>> >>>>>>>> It feels to me that we should aim for making it work reasonably >>>>>>>> well for reasonable use cases. By reasonable use cases, I'm thinking of the >>>>>>>> following: >>>>>>>> >>>>>>>> 1. --static option may or may not be given (i.e. we should allow >>>>>>>> that feature for both static linking and dynamic linking.) >>>>>>>> 2. There are no competing defined symbols in a given set of >>>>>>>> libraries, or if they exist, the program owner doesn't care which is linked >>>>>>>> to their program. >>>>>>>> 3. There may be circular dependencies between libraries. >>>>>>>> >>>>>>>> I don't think the above assumption is too odd. If I have to >>>>>>>> implement the autolinking feature to GNU linker for the above scenario, I'd >>>>>>>> probably use the following scheme: >>>>>>>> >>>>>>>> 1. While reading object files, memorize libraries that are >>>>>>>> autolinked >>>>>>>> 2. After linking everything, create a list of files consisting of >>>>>>>> autolinked libraries AND libraries given via the command line >>>>>>>> 3. Visit each file in the list as if they were wrapped in >>>>>>>> --start-group and --end-group. >>>>>>>> >>>>>>>> I'd think the above scheme should work reasonably well. What do you >>>>>>>> think? >>>>>>>> >>>>>>> >>>>>>> Very nice. I agree with your definition of "reasonable" usecaes >>>>>>> (actually, as I have said before, I think that restricting autolinking to >>>>>>> this "reasonable" set is actually a feature - to avoid developers having >>>>>>> source code that only works with a particular linker). I also like the >>>>>>> proposal for a GNU implementation - I think this is enough to show that >>>>>>> GNU-like linkers could implement this. >>>>>>> >>>>>>> At this point I will try to prototype this up so that people have an >>>>>>> implementation to play with. >>>>>>> >>>>>>> I am keen to hear from Saleem (compnerd) on this, as he did the >>>>>>> original .linker-options work. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 19, 2019 at 11:02 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Mon, Mar 18, 2019 at 8:02 PM Rui Ueyama <ruiu at google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 14, 2019 at 1:05 PM bd1976 llvm via llvm-dev < >>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 14, 2019 at 6:27 PM Peter Collingbourne < >>>>>>>>>>> peter at pcc.me.uk> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev < >>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> At Sony we offer autolinking as a feature in our ELF >>>>>>>>>>>>> toolchain. We would like to see full support for this feature upstream as >>>>>>>>>>>>> there is anecdotal evidence that it would find use beyond Sony. >>>>>>>>>>>>> >>>>>>>>>>>>> In general autolinking ( >>>>>>>>>>>>> https://en.wikipedia.org/wiki/Auto-linking) allows developers >>>>>>>>>>>>> to specify inputs to the linker in their source code. LLVM and Clang >>>>>>>>>>>>> already have support for autolinking on ELF via embedding strings, which >>>>>>>>>>>>> specify linker behavior, into a .linker-options section in relocatable >>>>>>>>>>>>> object files, see: >>>>>>>>>>>>> >>>>>>>>>>>>> RFC - >>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html >>>>>>>>>>>>> LLVM - >>>>>>>>>>>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options, >>>>>>>>>>>>> https://reviews.llvm.org/D40849 >>>>>>>>>>>>> Clang - >>>>>>>>>>>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets, >>>>>>>>>>>>> https://reviews.llvm.org/D42758 >>>>>>>>>>>>> >>>>>>>>>>>>> However, although support was added to Clang and LLVM, no >>>>>>>>>>>>> support has been implemented in LLD; and, I get the sense, from reading the >>>>>>>>>>>>> reviews, that there wasn't agreement on the implementation when the changes >>>>>>>>>>>>> landed. The original motivation seems to have been to remove the >>>>>>>>>>>>> "autolink-extract" mechanism used by Swift to workaround the lack of >>>>>>>>>>>>> autolinking support for ELF. However, looking at the Swift source code, >>>>>>>>>>>>> Swift still seems to be using the "autolink-extract" method. >>>>>>>>>>>>> >>>>>>>>>>>>> So my first question: Are there any users of the current >>>>>>>>>>>>> implementation for ELF? >>>>>>>>>>>>> >>>>>>>>>>>>> Assuming that no one is using the current code, I would like >>>>>>>>>>>>> to suggest a different mechanism for autolinking. >>>>>>>>>>>>> >>>>>>>>>>>>> For ELF we need limited autolinking support. Specifically, we >>>>>>>>>>>>> only need support for "comment lib" pragmas ( >>>>>>>>>>>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) >>>>>>>>>>>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the >>>>>>>>>>>>> implementation as lean as possible. >>>>>>>>>>>>> >>>>>>>>>>>>> Principles to guide the implementation: >>>>>>>>>>>>> - Developers should be able to easily understand autolinking >>>>>>>>>>>>> behavior. >>>>>>>>>>>>> - Developers should be able to override autolinking from the >>>>>>>>>>>>> linker command line. >>>>>>>>>>>>> - Inputs specified via pragmas should be handled in a general >>>>>>>>>>>>> way to allow the same source code to work in different environments. >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to propose that we focus on autolinking >>>>>>>>>>>>> exclusively and that we divorce the implementation from the idea of "linker >>>>>>>>>>>>> options" which, by nature, would tie source code to the vagaries of >>>>>>>>>>>>> particular linkers. I don't see much value in supporting other linker >>>>>>>>>>>>> operations so I suggest that the binary representation be a mergable string >>>>>>>>>>>>> section (SHF_MERGE, SHF_STRINGS), called .autolink, with custom type >>>>>>>>>>>>> SHT_LLVM_AUTOLINK (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents >>>>>>>>>>>>> appearing in the output). The compiler can form this section by >>>>>>>>>>>>> concatenating the arguments of the "comment lib" pragmas in the order they >>>>>>>>>>>>> are encountered. Partial (-r, -Ur) links can be handled by concatenating >>>>>>>>>>>>> .autolink sections with the normal mergeable string section rules. The >>>>>>>>>>>>> current .linker-options can remain (or be removed); but, "comment lib" >>>>>>>>>>>>> pragmas for ELF should be lowered to .autolink not to .linker-options. This >>>>>>>>>>>>> makes sense as there is no linker option that "comment lib" pragmas map >>>>>>>>>>>>> directly to. As an example, #pragma comment(lib, "foo") would result in: >>>>>>>>>>>>> >>>>>>>>>>>>> .section ".autolink","eMS", at llvm_autolink,1 >>>>>>>>>>>>> .asciz "foo" >>>>>>>>>>>>> >>>>>>>>>>>>> For LTO, equivalent information to the contents of a the >>>>>>>>>>>>> .autolink section will be written to the IRSymtab so that it is available >>>>>>>>>>>>> to the linker for symbol resolution. >>>>>>>>>>>>> >>>>>>>>>>>>> The linker will process the .autolink strings in the following >>>>>>>>>>>>> way: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Inputs from the .autolink sections of a relocatable object >>>>>>>>>>>>> file are added when the linker decides to include that file (which could >>>>>>>>>>>>> itself be in a library) in the link. Autolinked inputs behave as if they >>>>>>>>>>>>> were appended to the command line as a group after all other options. As a >>>>>>>>>>>>> consequence the set of autolinked libraries are searched last to resolve >>>>>>>>>>>>> symbols. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If we want this to be compatible with GNU linkers, doesn't the >>>>>>>>>>>> autolinked input need to appear at the point immediately after the object >>>>>>>>>>>> file appears in the link? I'm imagining the case where you have a >>>>>>>>>>>> statically linked libc as well as a libbar.a autolinked from a foo.o. The >>>>>>>>>>>> link command line would look like this: >>>>>>>>>>>> >>>>>>>>>>>> ld foo.o -lc >>>>>>>>>>>> >>>>>>>>>>>> Now foo.o autolinks against bar. The command line becomes: >>>>>>>>>>>> >>>>>>>>>>>> ld foo.o -lc -lbar >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Actually, I was thinking that on a GNU linker the command line >>>>>>>>>>> would become "ld foo.o -lc -( -lbar )-"; but, this doesn't affect your >>>>>>>>>>> point. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If libbar.a requires an additional object file from libc.a, it >>>>>>>>>>>> will not be added to the link. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> As it stands all the dependencies of an autolinked library must >>>>>>>>>>> themselves be autolinked. I had imagined that this is a reasonable >>>>>>>>>>> limitation. If not we need another scheme. I try to think about some >>>>>>>>>>> motivating examples for this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 2. It is an error if a file cannot be found for a given string. >>>>>>>>>>>>> 3. Any command line options in effect at the end of the >>>>>>>>>>>>> command line parsing apply to autolinked inputs, e.g. --whole-archive. >>>>>>>>>>>>> 4. Duplicate autolinked inputs are ignored. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This seems like it would work in GNU linkers, as long as the >>>>>>>>>>>> autolinked file is added to the link immediately after the last mention, >>>>>>>>>>>> rather than the first. Otherwise a command line like: >>>>>>>>>>>> >>>>>>>>>>>> ld foo1.o foo2.o >>>>>>>>>>>> >>>>>>>>>>>> (where foo1.o and foo2.o both autolink bar) could end up >>>>>>>>>>>> looking like: >>>>>>>>>>>> >>>>>>>>>>>> ld foo1.o -lbar foo2.o >>>>>>>>>>>> >>>>>>>>>>>> and you will not link anything from libbar.a that only foo2.o >>>>>>>>>>>> requires. It may end up being simpler to not ignore duplicates. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Correct; but, given that the proposal was to handle the >>>>>>>>>>> libraries as if they are appended to the link line after everything on the >>>>>>>>>>> command line then I think this will work. With deduplication (and the use >>>>>>>>>>> of SHF_MERGE) developers get no ordering guarantees. I claim that this is a >>>>>>>>>>> feature! My rationale is that the order in which libraries are linked >>>>>>>>>>> affects different linkers in different ways (e.g. LLD does not resolve >>>>>>>>>>> symbols from archives in a compatible manner with either the Microsoft >>>>>>>>>>> linker or the GNU linkers.), by not allowing the user to control the order >>>>>>>>>>> I am essentially saying that autolinking is not suitable for libraries that >>>>>>>>>>> offer competing copies of the same symbol. This ties into my argument that >>>>>>>>>>> "comment lib" pragmas should be handled in as "general" a way as possible. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Right. I think if you need a fine control over the link order, >>>>>>>>>> autolinking is not a feature you want to use. Or, in general, if your >>>>>>>>>> program is sensitive to a link order because its source object files have >>>>>>>>>> competing symbols of the same name, it's perhaps unnecessarily fragile. >>>>>>>>>> >>>>>>>>>> That being said, I think you need to address the issue that pcc >>>>>>>>>> pointed out. If you statically link a program `foo` with the following >>>>>>>>>> command line >>>>>>>>>> >>>>>>>>>> ld -o foo foo.o -lc >>>>>>>>>> >>>>>>>>>> , `foo.o` auto-imports libbar.a, and libbar.a depends on libc.a, >>>>>>>>>> can your proposed feature pull out object files needed for libbar.a? >>>>>>>>>> >>>>>>>>> >>>>>>>>> It won't work on GNU linkers. It will work with LLD as LLD has >>>>>>>>> MSVC-like archive handling. However, I would like to make sure that >>>>>>>>> whatever we come up with can be supported in the GNU toolchain. >>>>>>>>> >>>>>>>>> I had thought that it would be acceptable that all the >>>>>>>>> dependencies of an autolinked library must themselves be autolinked in >>>>>>>>> order to work on GNU style linkers. Having thought more, I don't like this >>>>>>>>> limitation - especially as it doesn't exist for Microsoft style linkers. >>>>>>>>> One possible resolution could be that GNU linkers might have to implement >>>>>>>>> another command line option e.g. --auto-dep=<file> to allow injection into >>>>>>>>> the group of autolinked libraries. >>>>>>>>> >>>>>>>>> i.e In pcc's example you would need to do: "ld foo.o >>>>>>>>> --auto-dep=libc.a" which would become "ld --start-group libbar.a libc.a >>>>>>>>> --end-group" with autolinking. >>>>>>>>> >>>>>>>>> I wanted to avoid the approach of inserting autolinked libraries >>>>>>>>> after the object that autolinks them. In LLD (and MSVC) it becomes hard to >>>>>>>>> reason about "where" the linker is in the command line and it would also >>>>>>>>> mean that we can't have the nice separation between parsing the command >>>>>>>>> line and doing the rest of the link that we currently have. Also, if you >>>>>>>>> give people a way to have a fine grained control over the link order with >>>>>>>>> autolinking you risk ending up with source code that will link on GNU style >>>>>>>>> linkers but not with LLD (assuming GNU ever implemented support for >>>>>>>>> autolinking). >>>>>>>>> >>>>>>>>> Scenario: >>>>>>>>> >>>>>>>>> libbar.a(bar.o) - defines symbol bar >>>>>>>>> libfoo.a(foo.o) - defines foo and autolinks libbar.a >>>>>>>>> main.o - references foo >>>>>>>>> another.o - does not reference foo >>>>>>>>> No references to bar exist >>>>>>>>> >>>>>>>>> lld -lfoo another.o --whole-archive main.o with autolinking >>>>>>>>> becomes lld -lfoo another.o --whole-archive main.o -lbar result: bar.o gets >>>>>>>>> added to the link. >>>>>>>>> But, if a change is made so that another.o references bar then the >>>>>>>>> link line with autolinking becomes lld -lfoo another.o >>>>>>>>> -lbar --whole-archive main.o result: bar.o is not added to the link. >>>>>>>>> >>>>>>>>> Hopefully the above scenario demonstrates why I think that it >>>>>>>>> becomes too complicated to reason about the effects of autolinking with >>>>>>>>> pcc's proposed insertion scheme. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> 5. The linker tries to add a library or relocatable object file >>>>>>>>>>>>> from each of the strings in a .autolink section by; first, handling the >>>>>>>>>>>>> string as if it was specified on the commandline; second, by looking for >>>>>>>>>>>>> the string in each of the library search paths in turn; third, by looking >>>>>>>>>>>>> for a lib<string>.a or lib<string>.so (depending on the current mode of the >>>>>>>>>>>>> linker) in each of the library search paths. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is the second part necessary? "-l:foo" causes the linker to >>>>>>>>>>>> search for a file named "foo" in the library search path, so it seems that >>>>>>>>>>>> allowing the autolink string to look like ":foo" would satisfy this use >>>>>>>>>>>> case. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I worded the proposal to avoid mapping "comment lib" pragmas to >>>>>>>>>>> --library command line options. My reasons: >>>>>>>>>>> >>>>>>>>>>> 1. I find the requirement that the user put ':' in their lib >>>>>>>>>>> strings slightly awkward. It means that the source code is now coupled to a >>>>>>>>>>> GNU-style linker. So then this isn't merely an ELF linking proposal, it's a >>>>>>>>>>> proposal for ELF toolchains with GNU-like linkers (e.g. the arm linker >>>>>>>>>>> doesn't support the colon prefix >>>>>>>>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0474c/Cjahbdei.html >>>>>>>>>>> ). >>>>>>>>>>> >>>>>>>>>>> 2. The syntax is #pragma comment(lib, ...) not #pragma >>>>>>>>>>> linker-option(library, ...) i.e. the only thing this (frankly rather >>>>>>>>>>> bizarre) syntax definitely implies is that the argument is related to >>>>>>>>>>> libraries (and comments ¯\_(ツ)_/¯); it is a bit of a stretch to interpret >>>>>>>>>>> "comment lib" pragmas as mapping directly to "specifying an additional >>>>>>>>>>> --library command line option". >>>>>>>>>>> >>>>>>>>>>> AFAIK all linkers support two ways of specifying inputs; >>>>>>>>>>> firstly, directly on the command line; secondly, with an option with very >>>>>>>>>>> similar semantics to GNU's --library option. I choose a method of finding a >>>>>>>>>>> input files that encompasses both methods of specifying a library on the >>>>>>>>>>> command line. I think that this method is actually more intuitive than >>>>>>>>>>> either the method used by the linker script INPUT command or by --library. >>>>>>>>>>> FWIW, I looked into the history of the colon prefix. It was added in >>>>>>>>>>> https://www.sourceware.org/ml/binutils/2007-03/msg00421.html. >>>>>>>>>>> Unfortunately, the rationale given is that it was merely a port of a >>>>>>>>>>> vxworks linker extension. I couldn't trace the history any further than >>>>>>>>>>> that to find the actual design discussion. The linker script command INPUT >>>>>>>>>>> uses a different scheme and the command already had this search order 20 >>>>>>>>>>> years ago, which is the earliest version of the GNU linker I have history >>>>>>>>>>> for; again, the rationale is not available. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 6. A new command line option --no-llvm-autolink will tell LLD >>>>>>>>>>>>> to ignore the .autolink sections. >>>>>>>>>>>>> >>>>>>>>>>>>> Rationale for the above points: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Adding the autolinked inputs last makes the process simple >>>>>>>>>>>>> to understand from a developers perspective. All linkers are able to >>>>>>>>>>>>> implement this scheme. >>>>>>>>>>>>> 2. Error-ing for libraries that are not found seems like >>>>>>>>>>>>> better behavior than failing the link during symbol resolution. >>>>>>>>>>>>> 3. It seems useful for the user to be able to apply command >>>>>>>>>>>>> line options which will affect all of the autolinked input files. There is >>>>>>>>>>>>> a potential problem of surprise for developers, who might not realize that >>>>>>>>>>>>> these options would apply to the "invisible" autolinked input files; >>>>>>>>>>>>> however, despite the potential for surprise, this is easy for developers to >>>>>>>>>>>>> reason about and gives developers the control that they may require. >>>>>>>>>>>>> 4. Unlike on the command line it is probably easy to include >>>>>>>>>>>>> the same input file twice via pragmas and might be a pain to fix; think of >>>>>>>>>>>>> Third-party libraries supplied as binaries. >>>>>>>>>>>>> 5. This algorithm takes into account all of the different ways >>>>>>>>>>>>> that ELF linkers find input files. The different search methods are tried >>>>>>>>>>>>> by the linker in most obvious to least obvious order. >>>>>>>>>>>>> 6. I considered adding finer grained control over which >>>>>>>>>>>>> .autolink inputs were ignored (e.g. MSVC has /nodefaultlib:<library>); >>>>>>>>>>>>> however, I concluded that this is not necessary: if finer control is >>>>>>>>>>>>> required developers can recreate the same effect autolinking would have had >>>>>>>>>>>>> using command line options. >>>>>>>>>>>>> >>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> -- >>>>>>>>>>>> Peter >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>> >>>>>>>>>> >>>>>> >>>>>> -- >>>>>> Saleem Abdulrasool >>>>>> compnerd (at) compnerd (dot) org >>>>>> >>>>> >>>> >>>> -- >>>> Saleem Abdulrasool >>>> compnerd (at) compnerd (dot) org >>>> >>> >> >> -- >> Saleem Abdulrasool >> compnerd (at) compnerd (dot) org >> >-- Saleem Abdulrasool compnerd (at) compnerd (dot) org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190327/c254ded7/attachment-0001.html>
Sorry that is took a while to come back on this. I wanted to make sure that I understood the situation with modules before proceeding. I have a patchset that I think is faithful to this RFC taking into account Saleems comments in particular; hopefully, having a set of changes can focus this discussion. Given that clang modules already have a concept of autolinking I feel that the definition of autolinking that I have offered so far has been too narrow. I now feel that autolinking is in common usage to mean any embedding of linker directives in source files. This clearly encompasses both "comment lib" and "comment linker" pragmas. Therefore, I have renamed by proposal to be about "dependent libraries" rather than autolinking https://reviews.llvm.org/D60274. Please take a look. On Thu, Mar 28, 2019 at 3:04 AM Saleem Abdulrasool <compnerd at compnerd.org> wrote:> On Wed, Mar 27, 2019 at 5:18 AM bd1976 llvm <bd1976llvm at gmail.com> wrote: > >> On Wed, Mar 27, 2019 at 2:03 AM Saleem Abdulrasool <compnerd at compnerd.org> >> wrote: >> >>> On Tue, Mar 26, 2019 at 11:13 AM bd1976 llvm <bd1976llvm at gmail.com> >>> wrote: >>> >>>> Thanks Saleem/James. >>>> >>>> On Tue, Mar 26, 2019 at 5:15 AM Saleem Abdulrasool < >>>> compnerd at compnerd.org> wrote: >>>> >>>>> Yes, I don't think that supporting all options is entirely possible. >>>>> But, there is certainly some allure to it, as it is a really powerful >>>>> feature (which you can tell from the behaviour of link and `.drectve` >>>>> section). The problem with all options is that ELF's model is not as >>>>> simplistic as link's model, which really complicates things. As an >>>>> example: `/WHOLEARCHIVE:` rather than `--whole-archive` ... >>>>> `--no-whole-archive`. I'm also worried about its use in a malicious >>>>> manner: `--no-pie` being embedded into the executable would disable ASLR, >>>>> which really is why I think that we would want a subset of options. >>>>> >>>> >>>> I'm not opposed to a linker options scheme; but, I think that it is >>>> important to separate this from autolinking (as we have proposed here). >>>> Rationale is upthread. We can support both if required. >>>> >>> >>> I'm not sure I understand how the two are distinct. The linker options >>> scheme being discussed is in the context of autolinking - aka, how to drive >>> the autolinking. >>> >> >> I'm imagining: >> >> #pragma comment(lib, "foo") => add "foo" to the link in a "general way". >> This is autolinking. >> #pragma comment(linker-option, "-l:foo") => Add "-l:foo" to the linker >> command line. This is linker options. >> > > What about `#pragma comment(linker-option, "-lfoo")`? Note that `-l:foo` > and `-lfoo` have different behaviours. > > >> Now it might be that on a specific platform (e.g. MSVC) you can represent >> #pragma comment(lib, "foo") using linker options; however, in general they >> are distinct. I would like the following behaviour for ELF:.. >> >> #pragma comment(lib, "foo") => add libfoo.a to the link. >> #pragma comment(lib, "foo.a") => add foo.a to the link >> #pragma comment(lib, "c:\\foo.a") => add c:\foo.a to the link >> > > Ah, but the beauty is that they are *NOT* distinct. All three of those > have the same semantics if passed on the command line, so that is all the > same as: `#pragma comment(lib, ...)`. > > >> .. but there is no existing ELF linker command line option that maps to >> this behaviour; so, it is better to have a separate representation. Also, >> given that it seems that autolinking is the only usecase currently, the >> proposed .autolink section is a simpler (e.g. binary tools can dump the >> section without needing special rules), and more efficient (bytes needed to >> convey the information) format than .linker-options. >> > > I'm not sure I agree with that. All of those map directly to the > semantics of passing that directly on the command line as options (with the > exception being in the first one it is spelt `-lfoo`. I am not against > using the textual representation, but, am in fact, for that. Rui wanted > the "structured" approached. >I accept that a structured approach might not be helpful in the case of "linker options"; however, I think that it is important to avoid the compiler and linker having to do string processing on the options to account for e.g. tokenization by the command line parser.> > >> >>> >>>> On Mon, Mar 25, 2019 at 8:47 AM Rui Ueyama <ruiu at google.com> wrote: >>>>> >>>>>> On Sun, Mar 24, 2019 at 9:51 PM Saleem Abdulrasool < >>>>>> compnerd at compnerd.org> wrote: >>>>>> >>>>>>> Sorry for the late chiming in. >>>>>>> >>>>>>> Yes, swift does use autolinking, and I would like to use that on all >>>>>>> the targets. The only target which does not support this functionality >>>>>>> currently are ELF based. That said, I think that `#pragma comment(link, >>>>>>> ...)` is insufficient for my needs. Building Foundation requires framework >>>>>>> style linking as well. The original design that I had in mind was derived >>>>>>> from ld64 and link. Personally, I still strongly favour link's behaviour >>>>>>> of parsing "command line" options from the object files when they are >>>>>>> loaded. There was strong opposition to that approach from Rui though. >>>>>>> Would we want to have special pragmas for each "feature"? >>>>>>> >>>>>> >>>> I was hoping that the autolinking proposal here was general enough that >>>> it could be made to work for you. What if we made the library search >>>> algorithm even more general than I initially proposed. Something like: >>>> >>>> - The linker tries to add a library or relocatable object file from >>>> each of the strings in a .autolink section by; first, handling the string >>>> as if it was specified on the commandline; second, by looking for the >>>> string in each of the library search paths in turn; third, by looking for a >>>> lib<string>.a or lib<string>.so (depending on the current mode of the >>>> linker) in each of the library search paths; forth, by looking for the >>>> string in each of the framework search paths in turn. >>>> >>> >>> What happens when you have Alpha.framework[/libAlpha.dylib] and >>> libAlpha.dylib? `-framework Alpha` should take >>> Alpha.framework/libAlpha.dylib and *NOT* libAlpha.dylib. I see no way to >>> differentiate between the two by mandating a specific ordering. -lAlpha -> >>> libAlpha.dylib and -framework Alpha -> Alpha.framework/libAlpha.dylib. >>> This is obviously a contrived case, but, the problem is that there is >>> ambiguity introduced which you cannot control without having additional >>> context from the user. So, `#pragma comment(link, ...)` is insufficient >>> IMO. >>> >> >>> >>>> So if you have #pragma comment(lib, "foo") in a source file, then on a >>>> linker that supports framework linking the linker would add to the link the >>>> first matching file trying: >>>> >>>> ./foo >>>> <first library search path>/foo >>>> <second library search path>/foo >>>> .. >>>> <nth library search path>/foo >>>> <first library search path>/libfoo.so >>>> <second library search path>/libfoo.so >>>> .. >>>> <nth library search path>/libfoo.so >>>> >>>> <first library search path>/libfoo.a >>>> <second library search path>/libfoo.a >>>> .. >>>> <nth library search path>/libfoo.a >>>> <first framework search path>/foo.framework/foo >>>> <second library search path>/foo.framework/foo >>>> .. >>>> <nth library search path>/foo.framework/foo >>>> >>>> What do you think? >>>> >>> >>> I think that it just broke in my example above :-). I think that we >>> need to realize that this is complex due to the fact that it requires >>> knowledge of what the user wants to do (which is normally expressed through >>> the invocation of the tools). So, we need to expose that to the >>> implementation. >>> >> >> The question is how contrived is your case? Upthread, I think that we >> have established that to keep autolinking reasonably understandable and >> consistent across linkers you have to accept some limitations. I'm hoping >> that we can accept this approach of finding libraries as a reasonable >> limitation. The consequences are that if a codebase has the example >> situation that you have described, you would either have to restructure the >> codebase to avoid the ambiguity, or you can't use autolinking for this >> case. The upside is that no matter if your linker supports frameworks or >> libraries #pragma comment(lib, "foo") will work in all cases. Otherwise we >> will be in a situation where you will have to have the equivalent of... >> > > Sure, some limitations are acceptable: like no --no-pie etc. I don't > think that it is unreasonable to design for something like frameworks. The > problem is the conflicting opinions on having a structured interface > without having some sort of structure defined. I don't think that it is > that contrived in practicality, just contrived to point out that there is > no one resolution that solves all the cases. Had the patches for lld been > merged already, we would be now discussing how to add support for something > new, so I do think that we have two options: design with this functionality > or go with a second means of doing this "simple" autolinking case. >Alternatively, we don't need to choose as both approaches can coexist. In fact I can't remove the .linker-options as it is needed for clang modules.> > >> #ifdef COFF_LIBRARIES: >> #pragma comment(linker-option, "/DEFAULTLIB:foo.lib") >> #'elif ELF_LIBRARIES: >> #pragma comment(linker-option, "-lfoo") >> #'elif DARWIN_FRAMEWORKS: >> #pragma comment(linker-option, "-framework foo") >> #esle >> #error Please specify linking model >> #endif >> >> ... somewhere, it would be good to avoid this. >> >> OTOH perhaps framework linking is just too different to be lumped in with >> #pragma comment(lib, ...) perhaps we should restrict #pragma comment(lib, >> ...) to libraries and implement support for another pragma, e.g. #pragma >> comment(framework, ...), to support framework linking. >> > > That is certainly reasonable. So, we could do separate comment types for > each bit. Note that "lib" and "linker" are two different types of > comments, the former is for autolinking only, the latter is for linker > options. Modelling that way seems reasonable to me (where lib only allows > you to specify the name of the library to link, no additional functionality > like static/shared linking, etc). >Agreed.> > >> >> >>> >>>> I think as you double-quoted the words "command line", I believe we are >>>>>> on the same page, but what I opposed was to allow arbitrary command line >>>>>> options in the .linker-options section. That is really a can of worms that >>>>>> we don't want to open. I didn't oppose allowing a limited set of >>>>>> command-line-ish directives. That said I'd prefer the proposal of this >>>>>> thread over the .linker-options though. >>>>>> >>>>>> The ELF model doesn't have the simplistic model for processing the >>>>>>> command line that PE/COFF does. Because ordering is relevant to the model, >>>>>>> it would be ideal to process them inline, but, since lld already moves far >>>>>>> enough away from the traditional Unix model, perhaps we can simplify it to >>>>>>> append the command line directives to the end of the command line. >>>>>>> >>>>>>> The other case that is interesting to think about is the autolinking >>>>>>> support in C++ (and clang) modules. >>>>>>> >>>>>> >>>> When I started this work I looked at the C++ 20 modules proposal: >>>> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf. I >>>> didn't see anything about autolinking (or linking at all)! Can you explain >>>> more? >>>> >>> >>> I may have been confusing features from the clang module functionality >>> and C++20 Modules TS (though, I thought that it included support for >>> autolinking dependencies for cases where an imported module unit has >>> available_externally linkage). >>> >> >> Right. So the Clang modules document does mention linking: >> https://clang.llvm.org/docs/Modules.html#link-declaration. Clang modules >> are different to the proposed C++20 modules. I assumed that Clang modules >> support existed only as a prototype to demonstrate how modules "could" work >> to guide the design/discussions for C++ modules. I don't understand whether >> Clang modules are in use by anyone? What will happen to Clang modules when >> C++20 arrives? Even if Clang modules are being used perhaps the linking >> part of the specification can be changed (as it seems unlikely that this is >> used as ELF linkers don't have a mechanism to support it)? >> >> >>> >>> >>>> >>>> >>>>>>> On Thu, Mar 21, 2019 at 9:49 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Mar 21, 2019 at 12:06 AM Rui Ueyama <ruiu at google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Perhaps there's no one clean way to solve this issue, because >>>>>>>>> previously all libraries and object files are explicitly given to the >>>>>>>>> linker via a command line and the order of files in the command line >>>>>>>>> matters. That assumes human intervention to work correctly. Now, the >>>>>>>>> autolinking feature will add libraries implicitly. Since it's implicit, >>>>>>>>> there will be only one way how that works, so sometimes that works and >>>>>>>>> sometimes doesn't. >>>>>>>>> >>>>>>>>> It feels to me that we should aim for making it work reasonably >>>>>>>>> well for reasonable use cases. By reasonable use cases, I'm thinking of the >>>>>>>>> following: >>>>>>>>> >>>>>>>>> 1. --static option may or may not be given (i.e. we should allow >>>>>>>>> that feature for both static linking and dynamic linking.) >>>>>>>>> 2. There are no competing defined symbols in a given set of >>>>>>>>> libraries, or if they exist, the program owner doesn't care which is linked >>>>>>>>> to their program. >>>>>>>>> 3. There may be circular dependencies between libraries. >>>>>>>>> >>>>>>>>> I don't think the above assumption is too odd. If I have to >>>>>>>>> implement the autolinking feature to GNU linker for the above scenario, I'd >>>>>>>>> probably use the following scheme: >>>>>>>>> >>>>>>>>> 1. While reading object files, memorize libraries that are >>>>>>>>> autolinked >>>>>>>>> 2. After linking everything, create a list of files consisting of >>>>>>>>> autolinked libraries AND libraries given via the command line >>>>>>>>> 3. Visit each file in the list as if they were wrapped in >>>>>>>>> --start-group and --end-group. >>>>>>>>> >>>>>>>>> I'd think the above scheme should work reasonably well. What do >>>>>>>>> you think? >>>>>>>>> >>>>>>>> >>>>>>>> Very nice. I agree with your definition of "reasonable" usecaes >>>>>>>> (actually, as I have said before, I think that restricting autolinking to >>>>>>>> this "reasonable" set is actually a feature - to avoid developers having >>>>>>>> source code that only works with a particular linker). I also like the >>>>>>>> proposal for a GNU implementation - I think this is enough to show that >>>>>>>> GNU-like linkers could implement this. >>>>>>>> >>>>>>>> At this point I will try to prototype this up so that people have >>>>>>>> an implementation to play with. >>>>>>>> >>>>>>>> I am keen to hear from Saleem (compnerd) on this, as he did the >>>>>>>> original .linker-options work. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 19, 2019 at 11:02 AM bd1976 llvm <bd1976llvm at gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Mon, Mar 18, 2019 at 8:02 PM Rui Ueyama <ruiu at google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 14, 2019 at 1:05 PM bd1976 llvm via llvm-dev < >>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:27 PM Peter Collingbourne < >>>>>>>>>>>> peter at pcc.me.uk> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 14, 2019 at 6:08 AM bd1976 llvm via llvm-dev < >>>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> At Sony we offer autolinking as a feature in our ELF >>>>>>>>>>>>>> toolchain. We would like to see full support for this feature upstream as >>>>>>>>>>>>>> there is anecdotal evidence that it would find use beyond Sony. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In general autolinking ( >>>>>>>>>>>>>> https://en.wikipedia.org/wiki/Auto-linking) allows >>>>>>>>>>>>>> developers to specify inputs to the linker in their source code. LLVM and >>>>>>>>>>>>>> Clang already have support for autolinking on ELF via embedding strings, >>>>>>>>>>>>>> which specify linker behavior, into a .linker-options section in >>>>>>>>>>>>>> relocatable object files, see: >>>>>>>>>>>>>> >>>>>>>>>>>>>> RFC - >>>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-January/120101.html >>>>>>>>>>>>>> LLVM - >>>>>>>>>>>>>> https://llvm.org/docs/Extensions.html#linker-options-section-linker-options, >>>>>>>>>>>>>> https://reviews.llvm.org/D40849 >>>>>>>>>>>>>> Clang - >>>>>>>>>>>>>> https://clang.llvm.org/docs/LanguageExtensions.html#specifying-linker-options-on-elf-targets, >>>>>>>>>>>>>> https://reviews.llvm.org/D42758 >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, although support was added to Clang and LLVM, no >>>>>>>>>>>>>> support has been implemented in LLD; and, I get the sense, from reading the >>>>>>>>>>>>>> reviews, that there wasn't agreement on the implementation when the changes >>>>>>>>>>>>>> landed. The original motivation seems to have been to remove the >>>>>>>>>>>>>> "autolink-extract" mechanism used by Swift to workaround the lack of >>>>>>>>>>>>>> autolinking support for ELF. However, looking at the Swift source code, >>>>>>>>>>>>>> Swift still seems to be using the "autolink-extract" method. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So my first question: Are there any users of the current >>>>>>>>>>>>>> implementation for ELF? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Assuming that no one is using the current code, I would like >>>>>>>>>>>>>> to suggest a different mechanism for autolinking. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For ELF we need limited autolinking support. Specifically, we >>>>>>>>>>>>>> only need support for "comment lib" pragmas ( >>>>>>>>>>>>>> https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) >>>>>>>>>>>>>> in C/C++ e.g. #pragma comment(lib, "foo"). My suggestion that we keep the >>>>>>>>>>>>>> implementation as lean as possible. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Principles to guide the implementation: >>>>>>>>>>>>>> - Developers should be able to easily understand autolinking >>>>>>>>>>>>>> behavior. >>>>>>>>>>>>>> - Developers should be able to override autolinking from the >>>>>>>>>>>>>> linker command line. >>>>>>>>>>>>>> - Inputs specified via pragmas should be handled in a general >>>>>>>>>>>>>> way to allow the same source code to work in different environments. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would like to propose that we focus on autolinking >>>>>>>>>>>>>> exclusively and that we divorce the implementation from the idea of "linker >>>>>>>>>>>>>> options" which, by nature, would tie source code to the vagaries of >>>>>>>>>>>>>> particular linkers. I don't see much value in supporting other linker >>>>>>>>>>>>>> operations so I suggest that the binary representation be a mergable string >>>>>>>>>>>>>> section (SHF_MERGE, SHF_STRINGS), called .autolink, with custom type >>>>>>>>>>>>>> SHT_LLVM_AUTOLINK (0x6fff4c04), and SHF_EXCLUDE set (to avoid the contents >>>>>>>>>>>>>> appearing in the output). The compiler can form this section by >>>>>>>>>>>>>> concatenating the arguments of the "comment lib" pragmas in the order they >>>>>>>>>>>>>> are encountered. Partial (-r, -Ur) links can be handled by concatenating >>>>>>>>>>>>>> .autolink sections with the normal mergeable string section rules. The >>>>>>>>>>>>>> current .linker-options can remain (or be removed); but, "comment lib" >>>>>>>>>>>>>> pragmas for ELF should be lowered to .autolink not to .linker-options. This >>>>>>>>>>>>>> makes sense as there is no linker option that "comment lib" pragmas map >>>>>>>>>>>>>> directly to. As an example, #pragma comment(lib, "foo") would result in: >>>>>>>>>>>>>> >>>>>>>>>>>>>> .section ".autolink","eMS", at llvm_autolink,1 >>>>>>>>>>>>>> .asciz "foo" >>>>>>>>>>>>>> >>>>>>>>>>>>>> For LTO, equivalent information to the contents of a the >>>>>>>>>>>>>> .autolink section will be written to the IRSymtab so that it is available >>>>>>>>>>>>>> to the linker for symbol resolution. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The linker will process the .autolink strings in the >>>>>>>>>>>>>> following way: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Inputs from the .autolink sections of a relocatable object >>>>>>>>>>>>>> file are added when the linker decides to include that file (which could >>>>>>>>>>>>>> itself be in a library) in the link. Autolinked inputs behave as if they >>>>>>>>>>>>>> were appended to the command line as a group after all other options. As a >>>>>>>>>>>>>> consequence the set of autolinked libraries are searched last to resolve >>>>>>>>>>>>>> symbols. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If we want this to be compatible with GNU linkers, doesn't the >>>>>>>>>>>>> autolinked input need to appear at the point immediately after the object >>>>>>>>>>>>> file appears in the link? I'm imagining the case where you have a >>>>>>>>>>>>> statically linked libc as well as a libbar.a autolinked from a foo.o. The >>>>>>>>>>>>> link command line would look like this: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo.o -lc >>>>>>>>>>>>> >>>>>>>>>>>>> Now foo.o autolinks against bar. The command line becomes: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo.o -lc -lbar >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Actually, I was thinking that on a GNU linker the command line >>>>>>>>>>>> would become "ld foo.o -lc -( -lbar )-"; but, this doesn't affect your >>>>>>>>>>>> point. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If libbar.a requires an additional object file from libc.a, it >>>>>>>>>>>>> will not be added to the link. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> As it stands all the dependencies of an autolinked library must >>>>>>>>>>>> themselves be autolinked. I had imagined that this is a reasonable >>>>>>>>>>>> limitation. If not we need another scheme. I try to think about some >>>>>>>>>>>> motivating examples for this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 2. It is an error if a file cannot be found for a given string. >>>>>>>>>>>>>> 3. Any command line options in effect at the end of the >>>>>>>>>>>>>> command line parsing apply to autolinked inputs, e.g. --whole-archive. >>>>>>>>>>>>>> 4. Duplicate autolinked inputs are ignored. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> This seems like it would work in GNU linkers, as long as the >>>>>>>>>>>>> autolinked file is added to the link immediately after the last mention, >>>>>>>>>>>>> rather than the first. Otherwise a command line like: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo1.o foo2.o >>>>>>>>>>>>> >>>>>>>>>>>>> (where foo1.o and foo2.o both autolink bar) could end up >>>>>>>>>>>>> looking like: >>>>>>>>>>>>> >>>>>>>>>>>>> ld foo1.o -lbar foo2.o >>>>>>>>>>>>> >>>>>>>>>>>>> and you will not link anything from libbar.a that only foo2.o >>>>>>>>>>>>> requires. It may end up being simpler to not ignore duplicates. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Correct; but, given that the proposal was to handle the >>>>>>>>>>>> libraries as if they are appended to the link line after everything on the >>>>>>>>>>>> command line then I think this will work. With deduplication (and the use >>>>>>>>>>>> of SHF_MERGE) developers get no ordering guarantees. I claim that this is a >>>>>>>>>>>> feature! My rationale is that the order in which libraries are linked >>>>>>>>>>>> affects different linkers in different ways (e.g. LLD does not resolve >>>>>>>>>>>> symbols from archives in a compatible manner with either the Microsoft >>>>>>>>>>>> linker or the GNU linkers.), by not allowing the user to control the order >>>>>>>>>>>> I am essentially saying that autolinking is not suitable for libraries that >>>>>>>>>>>> offer competing copies of the same symbol. This ties into my argument that >>>>>>>>>>>> "comment lib" pragmas should be handled in as "general" a way as possible. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Right. I think if you need a fine control over the link order, >>>>>>>>>>> autolinking is not a feature you want to use. Or, in general, if your >>>>>>>>>>> program is sensitive to a link order because its source object files have >>>>>>>>>>> competing symbols of the same name, it's perhaps unnecessarily fragile. >>>>>>>>>>> >>>>>>>>>>> That being said, I think you need to address the issue that pcc >>>>>>>>>>> pointed out. If you statically link a program `foo` with the following >>>>>>>>>>> command line >>>>>>>>>>> >>>>>>>>>>> ld -o foo foo.o -lc >>>>>>>>>>> >>>>>>>>>>> , `foo.o` auto-imports libbar.a, and libbar.a depends on libc.a, >>>>>>>>>>> can your proposed feature pull out object files needed for libbar.a? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It won't work on GNU linkers. It will work with LLD as LLD has >>>>>>>>>> MSVC-like archive handling. However, I would like to make sure that >>>>>>>>>> whatever we come up with can be supported in the GNU toolchain. >>>>>>>>>> >>>>>>>>>> I had thought that it would be acceptable that all the >>>>>>>>>> dependencies of an autolinked library must themselves be autolinked in >>>>>>>>>> order to work on GNU style linkers. Having thought more, I don't like this >>>>>>>>>> limitation - especially as it doesn't exist for Microsoft style linkers. >>>>>>>>>> One possible resolution could be that GNU linkers might have to implement >>>>>>>>>> another command line option e.g. --auto-dep=<file> to allow injection into >>>>>>>>>> the group of autolinked libraries. >>>>>>>>>> >>>>>>>>>> i.e In pcc's example you would need to do: "ld foo.o >>>>>>>>>> --auto-dep=libc.a" which would become "ld --start-group libbar.a libc.a >>>>>>>>>> --end-group" with autolinking. >>>>>>>>>> >>>>>>>>>> I wanted to avoid the approach of inserting autolinked libraries >>>>>>>>>> after the object that autolinks them. In LLD (and MSVC) it becomes hard to >>>>>>>>>> reason about "where" the linker is in the command line and it would also >>>>>>>>>> mean that we can't have the nice separation between parsing the command >>>>>>>>>> line and doing the rest of the link that we currently have. Also, if you >>>>>>>>>> give people a way to have a fine grained control over the link order with >>>>>>>>>> autolinking you risk ending up with source code that will link on GNU style >>>>>>>>>> linkers but not with LLD (assuming GNU ever implemented support for >>>>>>>>>> autolinking). >>>>>>>>>> >>>>>>>>>> Scenario: >>>>>>>>>> >>>>>>>>>> libbar.a(bar.o) - defines symbol bar >>>>>>>>>> libfoo.a(foo.o) - defines foo and autolinks libbar.a >>>>>>>>>> main.o - references foo >>>>>>>>>> another.o - does not reference foo >>>>>>>>>> No references to bar exist >>>>>>>>>> >>>>>>>>>> lld -lfoo another.o --whole-archive main.o with autolinking >>>>>>>>>> becomes lld -lfoo another.o --whole-archive main.o -lbar result: bar.o gets >>>>>>>>>> added to the link. >>>>>>>>>> But, if a change is made so that another.o references bar then >>>>>>>>>> the link line with autolinking becomes lld -lfoo another.o >>>>>>>>>> -lbar --whole-archive main.o result: bar.o is not added to the link. >>>>>>>>>> >>>>>>>>>> Hopefully the above scenario demonstrates why I think that it >>>>>>>>>> becomes too complicated to reason about the effects of autolinking with >>>>>>>>>> pcc's proposed insertion scheme. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 5. The linker tries to add a library or relocatable object file >>>>>>>>>>>>>> from each of the strings in a .autolink section by; first, handling the >>>>>>>>>>>>>> string as if it was specified on the commandline; second, by looking for >>>>>>>>>>>>>> the string in each of the library search paths in turn; third, by looking >>>>>>>>>>>>>> for a lib<string>.a or lib<string>.so (depending on the current mode of the >>>>>>>>>>>>>> linker) in each of the library search paths. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Is the second part necessary? "-l:foo" causes the linker to >>>>>>>>>>>>> search for a file named "foo" in the library search path, so it seems that >>>>>>>>>>>>> allowing the autolink string to look like ":foo" would satisfy this use >>>>>>>>>>>>> case. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I worded the proposal to avoid mapping "comment lib" pragmas to >>>>>>>>>>>> --library command line options. My reasons: >>>>>>>>>>>> >>>>>>>>>>>> 1. I find the requirement that the user put ':' in their lib >>>>>>>>>>>> strings slightly awkward. It means that the source code is now coupled to a >>>>>>>>>>>> GNU-style linker. So then this isn't merely an ELF linking proposal, it's a >>>>>>>>>>>> proposal for ELF toolchains with GNU-like linkers (e.g. the arm linker >>>>>>>>>>>> doesn't support the colon prefix >>>>>>>>>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0474c/Cjahbdei.html >>>>>>>>>>>> ). >>>>>>>>>>>> >>>>>>>>>>>> 2. The syntax is #pragma comment(lib, ...) not #pragma >>>>>>>>>>>> linker-option(library, ...) i.e. the only thing this (frankly rather >>>>>>>>>>>> bizarre) syntax definitely implies is that the argument is related to >>>>>>>>>>>> libraries (and comments ¯\_(ツ)_/¯); it is a bit of a stretch to interpret >>>>>>>>>>>> "comment lib" pragmas as mapping directly to "specifying an additional >>>>>>>>>>>> --library command line option". >>>>>>>>>>>> >>>>>>>>>>>> AFAIK all linkers support two ways of specifying inputs; >>>>>>>>>>>> firstly, directly on the command line; secondly, with an option with very >>>>>>>>>>>> similar semantics to GNU's --library option. I choose a method of finding a >>>>>>>>>>>> input files that encompasses both methods of specifying a library on the >>>>>>>>>>>> command line. I think that this method is actually more intuitive than >>>>>>>>>>>> either the method used by the linker script INPUT command or by --library. >>>>>>>>>>>> FWIW, I looked into the history of the colon prefix. It was added in >>>>>>>>>>>> https://www.sourceware.org/ml/binutils/2007-03/msg00421.html. >>>>>>>>>>>> Unfortunately, the rationale given is that it was merely a port of a >>>>>>>>>>>> vxworks linker extension. I couldn't trace the history any further than >>>>>>>>>>>> that to find the actual design discussion. The linker script command INPUT >>>>>>>>>>>> uses a different scheme and the command already had this search order 20 >>>>>>>>>>>> years ago, which is the earliest version of the GNU linker I have history >>>>>>>>>>>> for; again, the rationale is not available. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 6. A new command line option --no-llvm-autolink will tell LLD >>>>>>>>>>>>>> to ignore the .autolink sections. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rationale for the above points: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Adding the autolinked inputs last makes the process simple >>>>>>>>>>>>>> to understand from a developers perspective. All linkers are able to >>>>>>>>>>>>>> implement this scheme. >>>>>>>>>>>>>> 2. Error-ing for libraries that are not found seems like >>>>>>>>>>>>>> better behavior than failing the link during symbol resolution. >>>>>>>>>>>>>> 3. It seems useful for the user to be able to apply command >>>>>>>>>>>>>> line options which will affect all of the autolinked input files. There is >>>>>>>>>>>>>> a potential problem of surprise for developers, who might not realize that >>>>>>>>>>>>>> these options would apply to the "invisible" autolinked input files; >>>>>>>>>>>>>> however, despite the potential for surprise, this is easy for developers to >>>>>>>>>>>>>> reason about and gives developers the control that they may require. >>>>>>>>>>>>>> 4. Unlike on the command line it is probably easy to include >>>>>>>>>>>>>> the same input file twice via pragmas and might be a pain to fix; think of >>>>>>>>>>>>>> Third-party libraries supplied as binaries. >>>>>>>>>>>>>> 5. This algorithm takes into account all of the different >>>>>>>>>>>>>> ways that ELF linkers find input files. The different search methods are >>>>>>>>>>>>>> tried by the linker in most obvious to least obvious order. >>>>>>>>>>>>>> 6. I considered adding finer grained control over which >>>>>>>>>>>>>> .autolink inputs were ignored (e.g. MSVC has /nodefaultlib:<library>); >>>>>>>>>>>>>> however, I concluded that this is not necessary: if finer control is >>>>>>>>>>>>>> required developers can recreate the same effect autolinking would have had >>>>>>>>>>>>>> using command line options. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> -- >>>>>>>>>>>>> Peter >>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> LLVM Developers mailing list >>>>>>>>>>>> llvm-dev at lists.llvm.org >>>>>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>>>>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Saleem Abdulrasool >>>>>>> compnerd (at) compnerd (dot) org >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Saleem Abdulrasool >>>>> compnerd (at) compnerd (dot) org >>>>> >>>> >>> >>> -- >>> Saleem Abdulrasool >>> compnerd (at) compnerd (dot) org >>> >> > > -- > Saleem Abdulrasool > compnerd (at) compnerd (dot) org >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190404/91dbc1f8/attachment-0001.html>