llvm dev - Jan 2018 - Linker Option support for ELF

> Given how ELF works I would expect an unknown section to simply end up
> in the output, but we can use SHF_EXCLUDE to avoid that.
Yes, gold currently treats unknown section types pretty much the same
as PROGBITS sections. The SHF_ALLOCATE and SHF_EXCLUDE flags would
control where and whether the section goes into the output file.

Another thing that we need to work out is the link order. The link
order is basically a topologically-ordered list of objects, ordered so
that if A depends on B, A precedes B in the link order. Today, in the
absence of any dependency information at all, we rely on the user and
compiler to come up with a reasonably correct link order and pass a
linear list of files and libraries to the linker. In an ideal world
(e.g., one where you can just type "ld main.o"), we'd have
explicit
dependencies for every object, and we could construct a topological
order automatically. But with this feature, we will have a partial
list of explicit dependencies, and without a complete list, we have no
good way of adding new objects into the link order.

One way to approximate a proper link order would be to place each
added object immediately after the last object that requests it. For
example, if you run "ld a.o b.o c.o -lc", and both a.o and b.o request
libm, you would insert libm (i.e., any and all objects extracted from
libm if it's an archive library) after b.o and before c.o. But this
approach wouldn't work -- we'd have to read and process the directive
section from every object before establishing the final link order,
which means we can't start building our symbol table until we've read
all the objects, which means we can't search archive libraries.

I think what would work is to insert each requested object or shared
library into the link order immediately after the object that requests
it, but only if the object hasn't already been inserted and isn't
already listed on the command line (i.e., we won't try to load the
same file twice); and to search each requested archive library
immediately after each object that requests it (of course, because of
how library searching works, we would load a given archive member once
at most). With this method, libm would be searched after both a.o and
b.o, so we'd load any members needed by a.o before b.o, and any
remaining members needed by b.o before c.o.

The difference between this and a proper topological ordering would be
small, but would have a subtle effect on symbol interposition. I'm
willing to require anyone who depends on symbol interposition to
control their link order explicitly via the command line.

In my ideal world, archive libraries would carry dependency
information rather than the individual objects within them. I suspect
that's too much to ask.

I see no need for shared libraries to carry any dependency information
beyond the DT_NEEDED entries they already have.

(It would be so much easier to build a self-driving car if we could
immediately jump to the point where all cars are self-driving, right?)

-cary

On Fri, Jan 12, 2018 at 5:56 PM, Cary Coutant via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> > Given how ELF works I would expect an unknown section to simply end up
> > in the output, but we can use SHF_EXCLUDE to avoid that.
>
> Yes, gold currently treats unknown section types pretty much the same
> as PROGBITS sections. The SHF_ALLOCATE and SHF_EXCLUDE flags would
> control where and whether the section goes into the output file.
>
> Another thing that we need to work out is the link order. The link
> order is basically a topologically-ordered list of objects, ordered so
> that if A depends on B, A precedes B in the link order. Today, in the
> absence of any dependency information at all, we rely on the user and
> compiler to come up with a reasonably correct link order and pass a
> linear list of files and libraries to the linker. In an ideal world
> (e.g., one where you can just type "ld main.o"), we'd have
explicit
> dependencies for every object, and we could construct a topological
> order automatically. But with this feature, we will have a partial
> list of explicit dependencies, and without a complete list, we have no
> good way of adding new objects into the link order.
>
> One way to approximate a proper link order would be to place each
> added object immediately after the last object that requests it. For
> example, if you run "ld a.o b.o c.o -lc", and both a.o and b.o
request
> libm, you would insert libm (i.e., any and all objects extracted from
> libm if it's an archive library) after b.o and before c.o. But this
> approach wouldn't work -- we'd have to read and process the
directive
> section from every object before establishing the final link order,
> which means we can't start building our symbol table until we've
read
> all the objects, which means we can't search archive libraries.
>
I'm little confused. Even if the final command line becomes something like
this:

  ld -static a.o -lm b.o -lm c.o -lc

it links without any problem, no? I mean no files will be pulled out from
libm.a more than once.

FWIW the notion of the link order is very different in lld because our
linking order is similar to Windows linkers rather than the traditional
Unix linker.

I think what would work is to insert each requested object or
shared
> library into the link order immediately after the object that requests
> it, but only if the object hasn't already been inserted and isn't
> already listed on the command line (i.e., we won't try to load the
> same file twice); and to search each requested archive library
> immediately after each object that requests it (of course, because of
> how library searching works, we would load a given archive member once
> at most). With this method, libm would be searched after both a.o and
> b.o, so we'd load any members needed by a.o before b.o, and any
> remaining members needed by b.o before c.o.
>
> The difference between this and a proper topological ordering would be
> small, but would have a subtle effect on symbol interposition. I'm
> willing to require anyone who depends on symbol interposition to
> control their link order explicitly via the command line.
>
> In my ideal world, archive libraries would carry dependency
> information rather than the individual objects within them. I suspect
> that's too much to ask.
>
> I see no need for shared libraries to carry any dependency information
> beyond the DT_NEEDED entries they already have.
>
> (It would be so much easier to build a self-driving car if we could
> immediately jump to the point where all cars are self-driving, right?)
>
> -cary
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180112/01eab71a/attachment.html>

For those interested, I've updated the patch on D40849.  It is
significantly simpler, emitting the options as pairs of C-style strings in
a section named `.linker-options` as the agreement here was.

On Fri, Jan 12, 2018 at 8:41 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Fri, Jan 12, 2018 at 5:56 PM, Cary Coutant via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> > Given how ELF works I would expect an unknown section to simply
end up
>> > in the output, but we can use SHF_EXCLUDE to avoid that.
>>
>> Yes, gold currently treats unknown section types pretty much the same
>> as PROGBITS sections. The SHF_ALLOCATE and SHF_EXCLUDE flags would
>> control where and whether the section goes into the output file.
>>
>> Another thing that we need to work out is the link order. The link
>> order is basically a topologically-ordered list of objects, ordered so
>> that if A depends on B, A precedes B in the link order. Today, in the
>> absence of any dependency information at all, we rely on the user and
>> compiler to come up with a reasonably correct link order and pass a
>> linear list of files and libraries to the linker. In an ideal world
>> (e.g., one where you can just type "ld main.o"), we'd
have explicit
>> dependencies for every object, and we could construct a topological
>> order automatically. But with this feature, we will have a partial
>> list of explicit dependencies, and without a complete list, we have no
>> good way of adding new objects into the link order.
>>
>> One way to approximate a proper link order would be to place each
>> added object immediately after the last object that requests it. For
>> example, if you run "ld a.o b.o c.o -lc", and both a.o and
b.o request
>> libm, you would insert libm (i.e., any and all objects extracted from
>> libm if it's an archive library) after b.o and before c.o. But this
>> approach wouldn't work -- we'd have to read and process the
directive
>> section from every object before establishing the final link order,
>> which means we can't start building our symbol table until
we've read
>> all the objects, which means we can't search archive libraries.
>>
>
> I'm little confused. Even if the final command line becomes something
like
> this:
>
>   ld -static a.o -lm b.o -lm c.o -lc
>
> it links without any problem, no? I mean no files will be pulled out from
> libm.a more than once.
>
> FWIW the notion of the link order is very different in lld because our
> linking order is similar to Windows linkers rather than the traditional
> Unix linker.
>
> I think what would work is to insert each requested object or shared
>> library into the link order immediately after the object that requests
>> it, but only if the object hasn't already been inserted and
isn't
>> already listed on the command line (i.e., we won't try to load the
>> same file twice); and to search each requested archive library
>> immediately after each object that requests it (of course, because of
>> how library searching works, we would load a given archive member once
>> at most). With this method, libm would be searched after both a.o and
>> b.o, so we'd load any members needed by a.o before b.o, and any
>> remaining members needed by b.o before c.o.
>>
>> The difference between this and a proper topological ordering would be
>> small, but would have a subtle effect on symbol interposition. I'm
>> willing to require anyone who depends on symbol interposition to
>> control their link order explicitly via the command line.
>>
>> In my ideal world, archive libraries would carry dependency
>> information rather than the individual objects within them. I suspect
>> that's too much to ask.
>>
>> I see no need for shared libraries to carry any dependency information
>> beyond the DT_NEEDED entries they already have.
>>
>> (It would be so much easier to build a self-driving car if we could
>> immediately jump to the point where all cars are self-driving, right?)
>>
>> -cary
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
Saleem Abdulrasool
compnerd (at) compnerd (dot) org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180118/46c9c10d/attachment.html>