llvm dev - Oct 2019 - llvm-strip creates unloadable shared objects on linux-armv7hf

Hello Rui,

Thanks for your reply. I tried with the keep-section argument and that
made the shared library work.

Should these sections be kept around by default maybe?

-- Tobias

On Thu, Oct 17, 2019 at 11:06 AM Rui Ueyama <ruiu at google.com>
wrote:
>
> One thing I noticed is that llvm-strip seemed to remove a .ARM.attributes
section. Can you try --keep-section=.ARM.attributes to tell to the command to
keep the section?
>
> On Thu, Oct 17, 2019 at 5:37 PM Tobias Hieta via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> Hello,
>>
>> Recently we tried to streamline our toolchain by removing some GNU
>> tools with LLVM tools to avoid having multiple copies of strip, nm, ar
>> and similar tools. Today we ran into a really strange issue where
>> shared objects where not loadable.
>>
>> We where able to track this down to llvm-strip with default arguments
>> creating a shared object that doesn't load correctly. It works if
we
>> switch from not passing args to llvm-strip to pass -strip-all-gnu
>>
>> I built bzip2 and libbz2.so to show this case here:
>> https://1drv.ms/u/s!Amjta5FRkBbbkiz14aHTRJe03LlL?e=2mGvTN
>>
>> This is the error we got by using the stripped shared object with
>> llvm-strip -strip-all:
>>
>> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all ./bzip2
>> ERROR: ld.so: object './libbz2.so.all' from LD_PRELOAD cannot
be
>> preloaded (cannot open shared object file): ignored.
>> ./bzip2: error while loading shared libraries: libbz2.so: cannot open
>> shared object file: No such file or directory
>>
>> And with strip-all-gnu it works:
>> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all_gnu ./bzip2
>> bzip2: I won't write compressed data to a terminal.
>> bzip2: For help, type: `bzip2 --help'.
>>
>> This only seems to happen on Linux-armv7hf - we haven't seen this
on
>> any of the intel platforms.
>>
>> Thanks,
>> Tobias
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

I'm not ARM expert, but yes I guess so.

On Thu, Oct 17, 2019 at 6:16 PM Tobias Hieta <tobias at plexapp.com>
wrote:
> Hello Rui,
>
> Thanks for your reply. I tried with the keep-section argument and that
> made the shared library work.
>
> Should these sections be kept around by default maybe?
>
> -- Tobias
>
> On Thu, Oct 17, 2019 at 11:06 AM Rui Ueyama <ruiu at google.com>
wrote:
> >
> > One thing I noticed is that llvm-strip seemed to remove a
> .ARM.attributes section. Can you try --keep-section=.ARM.attributes to tell
> to the command to keep the section?
> >
> > On Thu, Oct 17, 2019 at 5:37 PM Tobias Hieta via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >>
> >> Hello,
> >>
> >> Recently we tried to streamline our toolchain by removing some GNU
> >> tools with LLVM tools to avoid having multiple copies of strip,
nm, ar
> >> and similar tools. Today we ran into a really strange issue where
> >> shared objects where not loadable.
> >>
> >> We where able to track this down to llvm-strip with default
arguments
> >> creating a shared object that doesn't load correctly. It works
if we
> >> switch from not passing args to llvm-strip to pass -strip-all-gnu
> >>
> >> I built bzip2 and libbz2.so to show this case here:
> >> https://1drv.ms/u/s!Amjta5FRkBbbkiz14aHTRJe03LlL?e=2mGvTN
> >>
> >> This is the error we got by using the stripped shared object with
> >> llvm-strip -strip-all:
> >>
> >> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all ./bzip2
> >> ERROR: ld.so: object './libbz2.so.all' from LD_PRELOAD
cannot be
> >> preloaded (cannot open shared object file): ignored.
> >> ./bzip2: error while loading shared libraries: libbz2.so: cannot
open
> >> shared object file: No such file or directory
> >>
> >> And with strip-all-gnu it works:
> >> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all_gnu ./bzip2
> >> bzip2: I won't write compressed data to a terminal.
> >> bzip2: For help, type: `bzip2 --help'.
> >>
> >> This only seems to happen on Linux-armv7hf - we haven't seen
this on
> >> any of the intel platforms.
> >>
> >> Thanks,
> >> Tobias
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191017/b3ce48b5/attachment.html>

Hello Tobias,

Does your system happen to be using eglibc? I ran into this problem on
an Ubuntu 14.04 system and it was down to the eglibc dynamic loader
using the .ARM.attributes section when performing dlopen. Strictly
speaking the .ARM.attributes section is only defined for relocatable
objects, the ABI says that their presence in executables or dynamic
loaders is neither required or forbidden, so it is somewhat risky for
any portable program to depend on their presence.

Since eglibc was merged back into glibc this dependency on
.ARM.attributes went away. For LLD we took the position that it was
worth keeping the .ARM.attributes to placate eglibc, as this was more
likely to be encountered 2 years ago. I've not got a strong position
on llvm-strip, in theory it should be strippable from executable and
shared libraries, but there may be a case that eglibc is important
enough to not strip by default.

Peter

On Thu, 17 Oct 2019 at 10:16, Tobias Hieta via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hello Rui,
>
> Thanks for your reply. I tried with the keep-section argument and that
> made the shared library work.
>
> Should these sections be kept around by default maybe?
>
> -- Tobias
>
> On Thu, Oct 17, 2019 at 11:06 AM Rui Ueyama <ruiu at google.com>
wrote:
> >
> > One thing I noticed is that llvm-strip seemed to remove a
.ARM.attributes section. Can you try --keep-section=.ARM.attributes to tell to
the command to keep the section?
> >
> > On Thu, Oct 17, 2019 at 5:37 PM Tobias Hieta via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
> >>
> >> Hello,
> >>
> >> Recently we tried to streamline our toolchain by removing some GNU
> >> tools with LLVM tools to avoid having multiple copies of strip,
nm, ar
> >> and similar tools. Today we ran into a really strange issue where
> >> shared objects where not loadable.
> >>
> >> We where able to track this down to llvm-strip with default
arguments
> >> creating a shared object that doesn't load correctly. It works
if we
> >> switch from not passing args to llvm-strip to pass -strip-all-gnu
> >>
> >> I built bzip2 and libbz2.so to show this case here:
> >> https://1drv.ms/u/s!Amjta5FRkBbbkiz14aHTRJe03LlL?e=2mGvTN
> >>
> >> This is the error we got by using the stripped shared object with
> >> llvm-strip -strip-all:
> >>
> >> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all ./bzip2
> >> ERROR: ld.so: object './libbz2.so.all' from LD_PRELOAD
cannot be
> >> preloaded (cannot open shared object file): ignored.
> >> ./bzip2: error while loading shared libraries: libbz2.so: cannot
open
> >> shared object file: No such file or directory
> >>
> >> And with strip-all-gnu it works:
> >> admin at Netgear-RN212:~/b$ LD_PRELOAD=./libbz2.so.all_gnu ./bzip2
> >> bzip2: I won't write compressed data to a terminal.
> >> bzip2: For help, type: `bzip2 --help'.
> >>
> >> This only seems to happen on Linux-armv7hf - we haven't seen
this on
> >> any of the intel platforms.
> >>
> >> Thanks,
> >> Tobias
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hello Peter,

I was able to fix this issue with this simple patch against llvm-objcopy:

diff --git a/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
b/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
index dd6a7d7e14b..c0dfd3a9838 100644
--- a/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
+++ b/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
@@ -503,6 +503,8 @@ static Error replaceAndRemoveSections(const
CopyConfig &Config, Object &Obj) {
return false;
if (StringRef(Sec.Name).startswith(".gnu.warning"))
return false;
+ if (StringRef(Sec.Name).startswith(".ARM.attributes"))
+ return false;
if (Sec.ParentSegment != nullptr)
return false;
return (Sec.Flags & SHF_ALLOC) == 0;

Is this a good way of fixing the issue? Happy to read up on how to
submit patches if you think this is the way to go.

-- Tobias


On Thu, Oct 17, 2019 at 12:27 PM Peter Smith <peter.smith at linaro.org>
wrote:
>
> It is possible that it was present in older versions of glibc, with
> the limited time I've got for archaeology I found
> https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1712938
> My change in LLD was https://reviews.llvm.org/D27718 although it
> doesn't help much.
>
> You may be able to follow the bread crumb trail from the launchpad
> bug. Good luck!
>
> Peter
>
> On Thu, 17 Oct 2019 at 10:59, Tobias Hieta <tobias at plexapp.com>
wrote:
> >
> > Peter,
> >
> > Thanks for your detailed response. I am not 100% sure on what libc we
> > are using here, it's a netgear NAS device. We publish Plex for
> > multiple NAS vendors devices and their toolchain always seems to lag
> > behind a bit, which is why we invested in building our own toolchain
> > based on LLVM tools.
> >
> > Does seem to be glibc though:
> > ii libc-bin 2.19-18+deb8u10.netgear1 armel GNU C Library: Binaries
> >
> > Maybe it's just because it's older why this happens?
> >
> > Thanks,
> > Tobias
> >
> > On Thu, Oct 17, 2019 at 11:25 AM Peter Smith <peter.smith at
linaro.org> wrote:
> > >
> > > Hello Tobias,
> > >
> > > Does your system happen to be using eglibc? I ran into this
problem on
> > > an Ubuntu 14.04 system and it was down to the eglibc dynamic
loader
> > > using the .ARM.attributes section when performing dlopen.
Strictly
> > > speaking the .ARM.attributes section is only defined for
relocatable
> > > objects, the ABI says that their presence in executables or
dynamic
> > > loaders is neither required or forbidden, so it is somewhat risky
for
> > > any portable program to depend on their presence.
> > >
> > > Since eglibc was merged back into glibc this dependency on
> > > .ARM.attributes went away. For LLD we took the position that it
was
> > > worth keeping the .ARM.attributes to placate eglibc, as this was
more
> > > likely to be encountered 2 years ago. I've not got a strong
position
> > > on llvm-strip, in theory it should be strippable from executable
and
> > > shared libraries, but there may be a case that eglibc is
important
> > > enough to not strip by default.
> > >
> > > Peter
> > >
> > > On Thu, 17 Oct 2019 at 10:16, Tobias Hieta via llvm-dev
> > > <llvm-dev at lists.llvm.org> wrote:
> > > >
> > > > Hello Rui,
> > > >
> > > > Thanks for your reply. I tried with the keep-section
argument and that
> > > > made the shared library work.
> > > >
> > > > Should these sections be kept around by default maybe?
> > > >
> > > > -- Tobias
> > > >
> > > > On Thu, Oct 17, 2019 at 11:06 AM Rui Ueyama <ruiu at
google.com> wrote:
> > > > >
> > > > > One thing I noticed is that llvm-strip seemed to remove
a .ARM.attributes section. Can you try --keep-section=.ARM.attributes to tell to
the command to keep the section?
> > > > >
> > > > > On Thu, Oct 17, 2019 at 5:37 PM Tobias Hieta via
llvm-dev <llvm-dev at lists.llvm.org> wrote:
> > > > >>
> > > > >> Hello,
> > > > >>
> > > > >> Recently we tried to streamline our toolchain by
removing some GNU
> > > > >> tools with LLVM tools to avoid having multiple
copies of strip, nm, ar
> > > > >> and similar tools. Today we ran into a really
strange issue where
> > > > >> shared objects where not loadable.
> > > > >>
> > > > >> We where able to track this down to llvm-strip with
default arguments
> > > > >> creating a shared object that doesn't load
correctly. It works if we
> > > > >> switch from not passing args to llvm-strip to pass
-strip-all-gnu
> > > > >>
> > > > >> I built bzip2 and libbz2.so to show this case here:
> > > > >>
https://1drv.ms/u/s!Amjta5FRkBbbkiz14aHTRJe03LlL?e=2mGvTN
> > > > >>
> > > > >> This is the error we got by using the stripped
shared object with
> > > > >> llvm-strip -strip-all:
> > > > >>
> > > > >> admin at Netgear-RN212:~/b$
LD_PRELOAD=./libbz2.so.all ./bzip2
> > > > >> ERROR: ld.so: object './libbz2.so.all' from
LD_PRELOAD cannot be
> > > > >> preloaded (cannot open shared object file):
ignored.
> > > > >> ./bzip2: error while loading shared libraries:
libbz2.so: cannot open
> > > > >> shared object file: No such file or directory
> > > > >>
> > > > >> And with strip-all-gnu it works:
> > > > >> admin at Netgear-RN212:~/b$
LD_PRELOAD=./libbz2.so.all_gnu ./bzip2
> > > > >> bzip2: I won't write compressed data to a
terminal.
> > > > >> bzip2: For help, type: `bzip2 --help'.
> > > > >>
> > > > >> This only seems to happen on Linux-armv7hf - we
haven't seen this on
> > > > >> any of the intel platforms.
> > > > >>
> > > > >> Thanks,
> > > > >> Tobias
> > > > >> _______________________________________________
> > > > >> LLVM Developers mailing list
> > > > >> llvm-dev at lists.llvm.org
> > > > >>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > llvm-dev at lists.llvm.org
> > > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hello Tobias,

I think that looks reasonable to me, I think it will be down to the
llvm-objcopy team whether they want to make .ARM.attributes a special
case or not. The best way to find out is to submit a patch, citing the
problems with old versions of libc, I'd expect that you'll need to add
a test case for the patch to be accepted. To do that it is probably
best to look at the existing tests for llvm-strip and try and copy
them. The test could be as simple as generating a binary with a
section of type SHT_ARM_ATTRIBUTES and checking that strip didn't
remove it. These tests sometimes use yaml2obj to generate an ELF file
without needing a compiler and linker. Running the tests should be as
simple as ninja check-llvm or make check-llvm depending on whether you
used ninja or make when building llvm. If you want to run just one
test then you can use bin/llvm-lit -v -a /path/to/test.s (from your
build directory).

The instructions on how to contribute are in
https://llvm.org/docs/DeveloperPolicy.html the people that I know have
been active in llvm-objdump are MaskRay (Fangrui Song), rupprect
(Jordan Rupprecht), grimar (George Rimar). If you include these people
on the reviewers then I'm sure they'll be able to add anyone else that
they think would be interested.

Hope this helps

Peter

On Thu, 17 Oct 2019 at 12:53, Tobias Hieta <tobias at plexapp.com>
wrote:
>
> Hello Peter,
>
> I was able to fix this issue with this simple patch against llvm-objcopy:
>
> diff --git a/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
> b/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
> index dd6a7d7e14b..c0dfd3a9838 100644
> --- a/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
> +++ b/llvm/tools/llvm-objcopy/ELF/ELFObjcopy.cpp
> @@ -503,6 +503,8 @@ static Error replaceAndRemoveSections(const
> CopyConfig &Config, Object &Obj) {
> return false;
> if (StringRef(Sec.Name).startswith(".gnu.warning"))
> return false;
> + if (StringRef(Sec.Name).startswith(".ARM.attributes"))
> + return false;
> if (Sec.ParentSegment != nullptr)
> return false;
> return (Sec.Flags & SHF_ALLOC) == 0;
>
> Is this a good way of fixing the issue? Happy to read up on how to
> submit patches if you think this is the way to go.
>
> -- Tobias
>
>
> On Thu, Oct 17, 2019 at 12:27 PM Peter Smith <peter.smith at
linaro.org> wrote:
> >
> > It is possible that it was present in older versions of glibc, with
> > the limited time I've got for archaeology I found
> > https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1712938
> > My change in LLD was https://reviews.llvm.org/D27718 although it
> > doesn't help much.
> >
> > You may be able to follow the bread crumb trail from the launchpad
> > bug. Good luck!
> >
> > Peter
> >
> > On Thu, 17 Oct 2019 at 10:59, Tobias Hieta <tobias at
plexapp.com> wrote:
> > >
> > > Peter,
> > >
> > > Thanks for your detailed response. I am not 100% sure on what
libc we
> > > are using here, it's a netgear NAS device. We publish Plex
for
> > > multiple NAS vendors devices and their toolchain always seems to
lag
> > > behind a bit, which is why we invested in building our own
toolchain
> > > based on LLVM tools.
> > >
> > > Does seem to be glibc though:
> > > ii libc-bin 2.19-18+deb8u10.netgear1 armel GNU C Library:
Binaries
> > >
> > > Maybe it's just because it's older why this happens?
> > >
> > > Thanks,
> > > Tobias
> > >
> > > On Thu, Oct 17, 2019 at 11:25 AM Peter Smith <peter.smith at
linaro.org> wrote:
> > > >
> > > > Hello Tobias,
> > > >
> > > > Does your system happen to be using eglibc? I ran into this
problem on
> > > > an Ubuntu 14.04 system and it was down to the eglibc dynamic
loader
> > > > using the .ARM.attributes section when performing dlopen.
Strictly
> > > > speaking the .ARM.attributes section is only defined for
relocatable
> > > > objects, the ABI says that their presence in executables or
dynamic
> > > > loaders is neither required or forbidden, so it is somewhat
risky for
> > > > any portable program to depend on their presence.
> > > >
> > > > Since eglibc was merged back into glibc this dependency on
> > > > .ARM.attributes went away. For LLD we took the position that
it was
> > > > worth keeping the .ARM.attributes to placate eglibc, as this
was more
> > > > likely to be encountered 2 years ago. I've not got a
strong position
> > > > on llvm-strip, in theory it should be strippable from
executable and
> > > > shared libraries, but there may be a case that eglibc is
important
> > > > enough to not strip by default.
> > > >
> > > > Peter
> > > >
> > > > On Thu, 17 Oct 2019 at 10:16, Tobias Hieta via llvm-dev
> > > > <llvm-dev at lists.llvm.org> wrote:
> > > > >
> > > > > Hello Rui,
> > > > >
> > > > > Thanks for your reply. I tried with the keep-section
argument and that
> > > > > made the shared library work.
> > > > >
> > > > > Should these sections be kept around by default maybe?
> > > > >
> > > > > -- Tobias
> > > > >
> > > > > On Thu, Oct 17, 2019 at 11:06 AM Rui Ueyama <ruiu at
google.com> wrote:
> > > > > >
> > > > > > One thing I noticed is that llvm-strip seemed to
remove a .ARM.attributes section. Can you try --keep-section=.ARM.attributes to
tell to the command to keep the section?
> > > > > >
> > > > > > On Thu, Oct 17, 2019 at 5:37 PM Tobias Hieta via
llvm-dev <llvm-dev at lists.llvm.org> wrote:
> > > > > >>
> > > > > >> Hello,
> > > > > >>
> > > > > >> Recently we tried to streamline our toolchain
by removing some GNU
> > > > > >> tools with LLVM tools to avoid having multiple
copies of strip, nm, ar
> > > > > >> and similar tools. Today we ran into a really
strange issue where
> > > > > >> shared objects where not loadable.
> > > > > >>
> > > > > >> We where able to track this down to llvm-strip
with default arguments
> > > > > >> creating a shared object that doesn't load
correctly. It works if we
> > > > > >> switch from not passing args to llvm-strip to
pass -strip-all-gnu
> > > > > >>
> > > > > >> I built bzip2 and libbz2.so to show this case
here:
> > > > > >>
https://1drv.ms/u/s!Amjta5FRkBbbkiz14aHTRJe03LlL?e=2mGvTN
> > > > > >>
> > > > > >> This is the error we got by using the stripped
shared object with
> > > > > >> llvm-strip -strip-all:
> > > > > >>
> > > > > >> admin at Netgear-RN212:~/b$
LD_PRELOAD=./libbz2.so.all ./bzip2
> > > > > >> ERROR: ld.so: object './libbz2.so.all'
from LD_PRELOAD cannot be
> > > > > >> preloaded (cannot open shared object file):
ignored.
> > > > > >> ./bzip2: error while loading shared libraries:
libbz2.so: cannot open
> > > > > >> shared object file: No such file or directory
> > > > > >>
> > > > > >> And with strip-all-gnu it works:
> > > > > >> admin at Netgear-RN212:~/b$
LD_PRELOAD=./libbz2.so.all_gnu ./bzip2
> > > > > >> bzip2: I won't write compressed data to a
terminal.
> > > > > >> bzip2: For help, type: `bzip2 --help'.
> > > > > >>
> > > > > >> This only seems to happen on Linux-armv7hf -
we haven't seen this on
> > > > > >> any of the intel platforms.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Tobias
> > > > > >>
_______________________________________________
> > > > > >> LLVM Developers mailing list
> > > > > >> llvm-dev at lists.llvm.org
> > > > > >>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > > > > _______________________________________________
> > > > > LLVM Developers mailing list
> > > > > llvm-dev at lists.llvm.org
> > > > >
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev