llvm dev - Dec 2017 - [lld] bug detecting undefined symbols in shared libraries

I have a test case where lld-5.0 fails to detect an undefined symbol (this
bug also happens with clang/lld 4.0).  I haven't narrowed down exactly all
the circumstances when this can occur, but in this case it was for a
virtual method in a class defined in a shared library.  If I build the
executable with the raw object files, the linker notices the missing method
(see test1 vs test2).  This was tested on Ubuntu 14.04 using the
llvm/clang-5.0 downloaded from the official apt repository.

If I use gold (from binutils 2.26), it finds the error during link time.


********** DEMO OUTPUT
$ make clean && make
rm -f main.o virtual.o virtual.so test1 test2
clang++-5.0 -fPIC -c main.cc
clang++-5.0 -fPIC -c virtual.cc
clang++-5.0 -fPIC -fuse-ld=lld-5.0 -shared -o virtual.so virtual.o
clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test1 main.o virtual.so
clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test2 main.o virtual.o
/usr/bin/ld.lld-5.0: error: undefined symbol: foo::bar()
>>> referenced by virtual.cc
>>>               virtual.o:(vtable for foo)
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make: *** [test2] Error 1

Note that running test1 results in an error:
$ ./test1
./test1: symbol lookup error: ./virtual.so: undefined symbol: _ZN3foo3barEv

********** CODE main.cc
#include "virtual.h"
int main() {
        foo baz;
        return 0;
}

********** CODE virtual.h
struct foo {
        virtual ~foo();
        virtual void bar();
};

********** CODE virtual.cc
#include "virtual.h"
foo::~foo() { }

********* CODE Makefile
all: test1 test2

CC = clang++-5.0 -fPIC
LD = ${CC} -fuse-ld=lld-5.0

test1: main.o virtual.so
    ${LD} -Wl,-R$$ORIGIN. -o test1 main.o virtual.so

test2: main.o virtual.o
    ${LD} -Wl,-R$$ORIGIN. -o test2 main.o virtual.o

virtual.o: virtual.cc virtual.h
    ${CC} -c virtual.cc

virtual.so: virtual.o
    ${LD} -shared -o virtual.so virtual.o

main.o: main.cc virtual.h
    ${CC} -c main.cc

clean:
    rm -f main.o virtual.o virtual.so test1 test2
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171211/9ae46b83/attachment.html>

Hi Scott,

I think you are saying that lld doesn't report an error for undefined
symbols that exist in a .so which cannot be resolved at link-time. That's
intentional. Trying to resolve undefined symbols in a .so at link-time
doesn't make sense because we can just link fine without resolving such
symbols, and the static linker doesn't really know whether it's actually
an
error condition or not. The real failure could only happen at load-time
when the undefined symbols cannot be resolved using other DSOs. So we do
not care about undefined symbols in DSOs in most cases.

On Mon, Dec 11, 2017 at 3:39 PM, Scott Smith via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I have a test case where lld-5.0 fails to detect an undefined symbol (this
> bug also happens with clang/lld 4.0).  I haven't narrowed down exactly
all
> the circumstances when this can occur, but in this case it was for a
> virtual method in a class defined in a shared library.  If I build the
> executable with the raw object files, the linker notices the missing method
> (see test1 vs test2).  This was tested on Ubuntu 14.04 using the
> llvm/clang-5.0 downloaded from the official apt repository.
>
> If I use gold (from binutils 2.26), it finds the error during link time.
>
>
> ********** DEMO OUTPUT
> $ make clean && make
> rm -f main.o virtual.o virtual.so test1 test2
> clang++-5.0 -fPIC -c main.cc
> clang++-5.0 -fPIC -c virtual.cc
> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -shared -o virtual.so virtual.o
> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test1 main.o
> virtual.so
> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test2 main.o virtual.o
> /usr/bin/ld.lld-5.0: error: undefined symbol: foo::bar()
> >>> referenced by virtual.cc
> >>>               virtual.o:(vtable for foo)
> clang: error: linker command failed with exit code 1 (use -v to see
> invocation)
> make: *** [test2] Error 1
>
> Note that running test1 results in an error:
> $ ./test1
> ./test1: symbol lookup error: ./virtual.so: undefined symbol: _ZN3foo3barEv
>
> ********** CODE main.cc
> #include "virtual.h"
> int main() {
>         foo baz;
>         return 0;
> }
>
> ********** CODE virtual.h
> struct foo {
>         virtual ~foo();
>         virtual void bar();
> };
>
> ********** CODE virtual.cc
> #include "virtual.h"
> foo::~foo() { }
>
> ********* CODE Makefile
> all: test1 test2
>
> CC = clang++-5.0 -fPIC
> LD = ${CC} -fuse-ld=lld-5.0
>
> test1: main.o virtual.so
>     ${LD} -Wl,-R$$ORIGIN. -o test1 main.o virtual.so
>
> test2: main.o virtual.o
>     ${LD} -Wl,-R$$ORIGIN. -o test2 main.o virtual.o
>
> virtual.o: virtual.cc virtual.h
>     ${CC} -c virtual.cc
>
> virtual.so: virtual.o
>     ${LD} -shared -o virtual.so virtual.o
>
> main.o: main.cc virtual.h
>     ${CC} -c main.cc
>
> clean:
>     rm -f main.o virtual.o virtual.so test1 test2
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171211/26063326/attachment.html>

With shared libraries, it is always possible that you end up running with a
different set of libraries than you actually linked with.  However, how
often is that actually the case?  It seems at the very least there should
be an option to tell the user that unless they switch libraries, their code
won't be runnable.  In most cases, everything is available to the linker to
detect the situation (as both ld and gold do).

gold has a set of options to vary their behavior -
--[no-]allow-shlib-undefined, --no-undefined, -z defs,
--warn-unresolved-symbols, --error-unresolved-symbols, --unresolved-symbols
ignore-all/report-all/ignore-in-object-files/ignore-in-shared-libs.  It
seems that lld is implementing --allow-shlib-undefined (test1 builds, test2
fails to build).  Is it possible for lld to implement
--[no-]allow-shlib-undefined to switch the behavior for undefined symbols
in shared libraries?

$ make clean && make -k
rm -f main.o virtual.o virtual.so test1 test2
clang++-5.0 -fPIC -c main.cc
clang++-5.0 -fPIC -c virtual.cc
clang++-5.0 -fPIC -fuse-ld=gold -Wl,--allow-shlib-undefined -shared -o
virtual.so virtual.o
clang++-5.0 -fPIC -fuse-ld=gold -Wl,--allow-shlib-undefined -Wl,-R$ORIGIN.
-o test1 main.o virtual.so
clang++-5.0 -fPIC -fuse-ld=gold -Wl,--allow-shlib-undefined -Wl,-R$ORIGIN.
-o test2 main.o virtual.o
virtual.o:virtual.cc:vtable for foo: error: undefined reference to
'foo::bar()'
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make: *** [test2] Error 1
make: Target `all' not remade because of errors.


On Mon, Dec 11, 2017 at 3:54 PM, Rui Ueyama <ruiu at google.com> wrote:
> Hi Scott,
>
> I think you are saying that lld doesn't report an error for undefined
> symbols that exist in a .so which cannot be resolved at link-time.
That's
> intentional. Trying to resolve undefined symbols in a .so at link-time
> doesn't make sense because we can just link fine without resolving such
> symbols, and the static linker doesn't really know whether it's
actually an
> error condition or not. The real failure could only happen at load-time
> when the undefined symbols cannot be resolved using other DSOs. So we do
> not care about undefined symbols in DSOs in most cases.
>
> On Mon, Dec 11, 2017 at 3:39 PM, Scott Smith via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> I have a test case where lld-5.0 fails to detect an undefined symbol
>> (this bug also happens with clang/lld 4.0).  I haven't narrowed
down
>> exactly all the circumstances when this can occur, but in this case it
was
>> for a virtual method in a class defined in a shared library.  If I
build
>> the executable with the raw object files, the linker notices the
missing
>> method (see test1 vs test2).  This was tested on Ubuntu 14.04 using the
>> llvm/clang-5.0 downloaded from the official apt repository.
>>
>> If I use gold (from binutils 2.26), it finds the error during link
time.
>>
>>
>> ********** DEMO OUTPUT
>> $ make clean && make
>> rm -f main.o virtual.o virtual.so test1 test2
>> clang++-5.0 -fPIC -c main.cc
>> clang++-5.0 -fPIC -c virtual.cc
>> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -shared -o virtual.so virtual.o
>> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test1 main.o
>> virtual.so
>> clang++-5.0 -fPIC -fuse-ld=lld-5.0 -Wl,-R$ORIGIN. -o test2 main.o
>> virtual.o
>> /usr/bin/ld.lld-5.0: error: undefined symbol: foo::bar()
>> >>> referenced by virtual.cc
>> >>>               virtual.o:(vtable for foo)
>> clang: error: linker command failed with exit code 1 (use -v to see
>> invocation)
>> make: *** [test2] Error 1
>>
>> Note that running test1 results in an error:
>> $ ./test1
>> ./test1: symbol lookup error: ./virtual.so: undefined symbol:
>> _ZN3foo3barEv
>>
>> ********** CODE main.cc
>> #include "virtual.h"
>> int main() {
>>         foo baz;
>>         return 0;
>> }
>>
>> ********** CODE virtual.h
>> struct foo {
>>         virtual ~foo();
>>         virtual void bar();
>> };
>>
>> ********** CODE virtual.cc
>> #include "virtual.h"
>> foo::~foo() { }
>>
>> ********* CODE Makefile
>> all: test1 test2
>>
>> CC = clang++-5.0 -fPIC
>> LD = ${CC} -fuse-ld=lld-5.0
>>
>> test1: main.o virtual.so
>>     ${LD} -Wl,-R$$ORIGIN. -o test1 main.o virtual.so
>>
>> test2: main.o virtual.o
>>     ${LD} -Wl,-R$$ORIGIN. -o test2 main.o virtual.o
>>
>> virtual.o: virtual.cc virtual.h
>>     ${CC} -c virtual.cc
>>
>> virtual.so: virtual.o
>>     ${LD} -shared -o virtual.so virtual.o
>>
>> main.o: main.cc virtual.h
>>     ${CC} -c main.cc
>>
>> clean:
>>     rm -f main.o virtual.o virtual.so test1 test2
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171211/b8aad5d8/attachment.html>