llvm dev - Jul 2015 - [LLVMdev] Some thought on handling ELF shared libraries in lld

Most ELF shared libraries can be sliced in two ways. One is following
the information in the program headers (e_phoff). The other is
following the information in the section headers (e_shoff).

Regular relocatable objects only have the section header.

At runtime, the dynamic linker only uses the program headers. In fact,
the section headers is optional.

When given a shared library, how should the static linker handle it?

Note that, unlike the dynamic linker, the static one has to find all
the defined symbol is a shared library. It is not enough to just look
up the currently undefined symbols. To see that, consider

$ cat test.c
void f(void) { }
$ cat test2.c
void f(void);
void g(void) {
  f();
}

$ clang -c test.c test2.c -fPIC
$ clang -shared test.o -o test.so
$ rm -f test.a
$ ar rc test.a test.o
$ clang test.so test2.o test.a -o t.so -Wl,-t -shared
$ clang         test2.o test.a -o t.so -Wl,-t -shared

The second link will include the archive member, the first one will not.

It is tempting to use the program headers in the static linker. Doing
so would let us support linking with shared libraries with no section
headers, but there are a few issues:

* The intention of the spec seems to be for everything static to use
the section headers and everything dynamic to use the program headers.
* Finding the number of symbols with the program header in a
traditional ELF file is a hack. One has to read the nchain field of
the hash table.
* It doesn't seem even possible to find that information in files
using the newer gnu hash format
(https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections).

Given that, it looks like we should use the sections. For what it is
worth, it looks like that is what every other ELF linker does.

Cheers,
Rafael

----- Original Message -----
> From: "Rafael Espíndola" <rafael.espindola at gmail.com>
> To: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Tuesday, July 21, 2015 7:46:58 AM
> Subject: [LLVMdev] Some thought on handling ELF shared libraries in lld
> 
> Most ELF shared libraries can be sliced in two ways. One is following
> the information in the program headers (e_phoff). The other is
> following the information in the section headers (e_shoff).
> 
> Regular relocatable objects only have the section header.
> 
> At runtime, the dynamic linker only uses the program headers. In
> fact,
> the section headers is optional.
> 
> When given a shared library, how should the static linker handle it?
> 
> Note that, unlike the dynamic linker, the static one has to find all
> the defined symbol is a shared library. It is not enough to just look
> up the currently undefined symbols. To see that, consider
> 
> $ cat test.c
> void f(void) { }
> $ cat test2.c
> void f(void);
> void g(void) {
>   f();
> }
> 
> $ clang -c test.c test2.c -fPIC
> $ clang -shared test.o -o test.so
> $ rm -f test.a
> $ ar rc test.a test.o
> $ clang test.so test2.o test.a -o t.so -Wl,-t -shared
> $ clang         test2.o test.a -o t.so -Wl,-t -shared
> 
> The second link will include the archive member, the first one will
> not.
> 
> It is tempting to use the program headers in the static linker. Doing
> so would let us support linking with shared libraries with no section
> headers, but there are a few issues:
> 
> * The intention of the spec seems to be for everything static to use
> the section headers and everything dynamic to use the program
> headers.
> * Finding the number of symbols with the program header in a
> traditional ELF file is a hack. One has to read the nchain field of
> the hash table.
> * It doesn't seem even possible to find that information in files
> using the newer gnu hash format
> (https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections).
Why do you need the total number of symbols?

 -Hal
> 
> Given that, it looks like we should use the sections. For what it is
> worth, it looks like that is what every other ELF linker does.
> 
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

Sounds reasonable. The sstrip tool that performs stripping of the
section header does state that this makes shared libraries unsuitable
for static linking. From the documentation:
> A shared-object library stripped in this fashion will still be usable by
the dynamic linker, but not by the static linker.
That said, it should be technically possible to statically link a
shared library with no section header table if we chose to support
that use case. Can't you use the symbol indices of dynamic relocations
to achieve this instead of counting the number of symbols?

On Tue, Jul 21, 2015 at 1:46 PM, Rafael Espíndola
<rafael.espindola at gmail.com> wrote:
> Most ELF shared libraries can be sliced in two ways. One is following
> the information in the program headers (e_phoff). The other is
> following the information in the section headers (e_shoff).
>
> Regular relocatable objects only have the section header.
>
> At runtime, the dynamic linker only uses the program headers. In fact,
> the section headers is optional.
>
> When given a shared library, how should the static linker handle it?
>
> Note that, unlike the dynamic linker, the static one has to find all
> the defined symbol is a shared library. It is not enough to just look
> up the currently undefined symbols. To see that, consider
>
> $ cat test.c
> void f(void) { }
> $ cat test2.c
> void f(void);
> void g(void) {
>   f();
> }
>
> $ clang -c test.c test2.c -fPIC
> $ clang -shared test.o -o test.so
> $ rm -f test.a
> $ ar rc test.a test.o
> $ clang test.so test2.o test.a -o t.so -Wl,-t -shared
> $ clang         test2.o test.a -o t.so -Wl,-t -shared
>
> The second link will include the archive member, the first one will not.
>
> It is tempting to use the program headers in the static linker. Doing
> so would let us support linking with shared libraries with no section
> headers, but there are a few issues:
>
> * The intention of the spec seems to be for everything static to use
> the section headers and everything dynamic to use the program headers.
> * Finding the number of symbols with the program header in a
> traditional ELF file is a hack. One has to read the nchain field of
> the hash table.
> * It doesn't seem even possible to find that information in files
> using the newer gnu hash format
> (https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections).
>
> Given that, it looks like we should use the sections. For what it is
> worth, it looks like that is what every other ELF linker does.
>
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On 21 July 2015 at 09:08, Hal Finkel <hfinkel at anl.gov>
wrote:
> ----- Original Message -----
>> From: "Rafael Espíndola" <rafael.espindola at
gmail.com>
>> To: "LLVM Developers Mailing List" <llvmdev at
cs.uiuc.edu>
>> Sent: Tuesday, July 21, 2015 7:46:58 AM
>> Subject: [LLVMdev] Some thought on handling ELF shared libraries in lld
>>
>> Most ELF shared libraries can be sliced in two ways. One is following
>> the information in the program headers (e_phoff). The other is
>> following the information in the section headers (e_shoff).
>>
>> Regular relocatable objects only have the section header.
>>
>> At runtime, the dynamic linker only uses the program headers. In
>> fact,
>> the section headers is optional.
>>
>> When given a shared library, how should the static linker handle it?
>>
>> Note that, unlike the dynamic linker, the static one has to find all
>> the defined symbol is a shared library. It is not enough to just look
>> up the currently undefined symbols. To see that, consider
>>
>> $ cat test.c
>> void f(void) { }
>> $ cat test2.c
>> void f(void);
>> void g(void) {
>>   f();
>> }
>>
>> $ clang -c test.c test2.c -fPIC
>> $ clang -shared test.o -o test.so
>> $ rm -f test.a
>> $ ar rc test.a test.o
>> $ clang test.so test2.o test.a -o t.so -Wl,-t -shared
>> $ clang         test2.o test.a -o t.so -Wl,-t -shared
>>
>> The second link will include the archive member, the first one will
>> not.
>>
>> It is tempting to use the program headers in the static linker. Doing
>> so would let us support linking with shared libraries with no section
>> headers, but there are a few issues:
>>
>> * The intention of the spec seems to be for everything static to use
>> the section headers and everything dynamic to use the program
>> headers.
>> * Finding the number of symbols with the program header in a
>> traditional ELF file is a hack. One has to read the nchain field of
>> the hash table.
>> * It doesn't seem even possible to find that information in files
>> using the newer gnu hash format
>> (https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections).
>
> Why do you need the total number of symbols?
To find all the defined ones. See the above example.

Cheers,
Rafael

On 21 July 2015 at 09:34, Dave Bozier <seifsta at gmail.com>
wrote:
> Sounds reasonable. The sstrip tool that performs stripping of the
> section header does state that this makes shared libraries unsuitable
> for static linking. From the documentation:
>
>> A shared-object library stripped in this fashion will still be usable
by the dynamic linker, but not by the static linker.
>
> That said, it should be technically possible to statically link a
> shared library with no section header table if we chose to support
> that use case. Can't you use the symbol indices of dynamic relocations
> to achieve this instead of counting the number of symbols?
It is technically possible. But we have to find the total symbols
early (symbol resolution) and so we would need to do an early scan of
another table just to find that number.

Cheers,
Rafael

On Tue, Jul 21, 2015 at 5:46 AM, Rafael Espíndola
<rafael.espindola at gmail.com> wrote:
> Most ELF shared libraries can be sliced in two ways. One is following
> the information in the program headers (e_phoff). The other is
> following the information in the section headers (e_shoff).
>
> Regular relocatable objects only have the section header.
>
> At runtime, the dynamic linker only uses the program headers. In fact,
> the section headers is optional.
>
> When given a shared library, how should the static linker handle it?
>
> Note that, unlike the dynamic linker, the static one has to find all
> the defined symbol is a shared library. It is not enough to just look
> up the currently undefined symbols. To see that, consider
>
> $ cat test.c
> void f(void) { }
> $ cat test2.c
> void f(void);
> void g(void) {
>   f();
> }
>
> $ clang -c test.c test2.c -fPIC
> $ clang -shared test.o -o test.so
> $ rm -f test.a
> $ ar rc test.a test.o
> $ clang test.so test2.o test.a -o t.so -Wl,-t -shared
> $ clang         test2.o test.a -o t.so -Wl,-t -shared
>
> The second link will include the archive member, the first one will not.
>
> It is tempting to use the program headers in the static linker. Doing
> so would let us support linking with shared libraries with no section
> headers, but there are a few issues:
>
> * The intention of the spec seems to be for everything static to use
> the section headers and everything dynamic to use the program headers.
> * Finding the number of symbols with the program header in a
> traditional ELF file is a hack. One has to read the nchain field of
> the hash table.
> * It doesn't seem even possible to find that information in files
> using the newer gnu hash format
> (https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections).
>
> Given that, it looks like we should use the sections. For what it is
> worth, it looks like that is what every other ELF linker does.
>
I agree we should use sections. Also, there's no reason we diverge
from what other linker do unless there's a real reason.


-- 
Davide

"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare