Konstantin Belousov
2019-Feb-21 08:54 UTC
Strange rtld-elf failure on stable/12 [Was: libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)]
On Thu, Feb 21, 2019 at 09:24:43AM +0100, Harry Schmalzbauer wrote:> Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer: > > Hello, > > > ? > > gdb shows: > > Core was generated by `/usr/sbin/auditdistd'. > > Program terminated with signal 11, Segmentation fault. > > Reading symbols from /lib/libutil.so.9...Reading symbols from > > /usr/lib/debug//lib/libutil.so.9.debug...done. > > done. > > Loaded symbols for /lib/libutil.so.9 > > Reading symbols from /libexec/ld-elf.so.1...Reading symbols from > > /usr/lib/debug//libexec/ld-elf.so.1.debug...done. > > done. > > Loaded symbols for /libexec/ld-elf.so.1 > > #0? memset (dest=0x80056f790, c=0, len=<value optimized out>) > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 > > 5624??????????????????? ((char *)dest)[i] = c; > > (gdb) bt > > #0? memset (dest=0x80056f790, c=0, len=<value optimized out>) > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 > > #1? 0x0000000800235b07 in map_object (fd=3, path=0x800246140 > > "/lib/libcrypto.so.111", > > ??? sb=0x7fffffffd4a8) > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249 > > #2? 0x0000000800230806 in load_object (name=0x201dba > > "libcrypto.so.111", fd_u=-1, > > ??? refobj=0x800248000, flags=<value optimized out>) > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493 > > #3? 0x0000000800229972 in _rtld (sp=<value optimized out>, > > exit_proc=0x7fffffffea30, > > ??? objp=0x7fffffffea38) > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315 > > #4? 0x0000000800228019 in .rtld_start () > > ??? at > > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39 > > #5? 0x0000000000000000 in ?? () > > Current language:? auto; currently minimal > > > > Any help highly appreciated. > > > > This is with a live CD (amd64), compiled with stable/12 from today (so > > clang 7.01). > > The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which > > compiled the live CD. > > bhyve host is 11.2.? But that shouldn't play a role, does it? > > I'm really interested what happens here. > I built stable/11 in that bhyve guest and updated that guest to > stable/11 from yesterday. > To my surpise llvm 7.01 was also merged to stable/11.? Thank you for > that great supprt! > No problems with any binary in the stable/11 bhyve guest. > > Then I built stable/12 in that re-built stable/11 guest. > As result, again all binaries linked to /lib/libcrypto.so.111 crash > (signal 11) with the stable/12 iso in the same bhyve guest. > > Here the example from ntpq: > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libedit.so.7...Reading symbols from > /usr/lib/debug//lib/libedit.so.7.debug...done. > done. > Loaded symbols for /lib/libedit.so.7 > Reading symbols from /lib/libm.so.5...Reading symbols from > /usr/lib/debug//lib/libm.so.5.debug...done. > done. > Loaded symbols for /lib/libm.so.5 > Reading symbols from /libexec/ld-elf.so.1...Reading symbols from > /usr/lib/debug//libexec/ld-elf.so.1.debug...done. > done. > #0? memset (dest=0x8005ef790, c=0, len=<value optimized out>) at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 > 5624??????????????????? ((char *)dest)[i] = c; > (gdb) bt > #0? memset (dest=0x8005ef790, c=0, len=<value optimized out>) at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 > #1? 0x000000080025db07 in map_object (fd=3, path=0x80026e1a0 > "/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249 > #2? 0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111", > fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493 > #3? 0x0000000800251972 in _rtld (sp=<value optimized out>, > exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315 > #4? 0x0000000800250019 in .rtld_start () at > /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39 > #5? 0x0000000000000000 in ?? () > > So please correct me if I'm comletely wrong, but the problem here seems > to be reproducably rtld-elf related. > Unfortunately I don't know anything about object files and linkers and > the related fundamental stuff.If you do not know about linkers, why do you claim that the problem is related to rtld ?> But maybe someone else has an idea what's going wrong here?The fault happens during zeroing of bss. Most likely it is due to some strangeness of the object being loaded. For diagnostic, show the output of "readelf -a libcrypto.so.111".
Harry Schmalzbauer
2019-Feb-21 09:03 UTC
libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)
Am 21.02.2019 um 09:54 schrieb Konstantin Belousov:> On Thu, Feb 21, 2019 at 09:24:43AM +0100, Harry Schmalzbauer wrote: >> Am 20.02.2019 um 17:51 schrieb Harry Schmalzbauer: >>> Hello, >>> >> ? >>> gdb shows: >>> Core was generated by `/usr/sbin/auditdistd'. >>> Program terminated with signal 11, Segmentation fault. >>> Reading symbols from /lib/libutil.so.9...Reading symbols from >>> /usr/lib/debug//lib/libutil.so.9.debug...done. >>> done. >>> Loaded symbols for /lib/libutil.so.9 >>> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from >>> /usr/lib/debug//libexec/ld-elf.so.1.debug...done. >>> done. >>> Loaded symbols for /libexec/ld-elf.so.1 >>> #0? memset (dest=0x80056f790, c=0, len=<value optimized out>) >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 >>> 5624??????????????????? ((char *)dest)[i] = c; >>> (gdb) bt >>> #0? memset (dest=0x80056f790, c=0, len=<value optimized out>) >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 >>> #1? 0x0000000800235b07 in map_object (fd=3, path=0x800246140 >>> "/lib/libcrypto.so.111", >>> ??? sb=0x7fffffffd4a8) >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249 >>> #2? 0x0000000800230806 in load_object (name=0x201dba >>> "libcrypto.so.111", fd_u=-1, >>> ??? refobj=0x800248000, flags=<value optimized out>) >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493 >>> #3? 0x0000000800229972 in _rtld (sp=<value optimized out>, >>> exit_proc=0x7fffffffea30, >>> ??? objp=0x7fffffffea38) >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315 >>> #4? 0x0000000800228019 in .rtld_start () >>> ??? at >>> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39 >>> #5? 0x0000000000000000 in ?? () >>> Current language:? auto; currently minimal >>> >>> Any help highly appreciated. >>> >>> This is with a live CD (amd64), compiled with stable/12 from today (so >>> clang 7.01). >>> The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which >>> compiled the live CD. >>> bhyve host is 11.2.? But that shouldn't play a role, does it? >> >> I'm really interested what happens here. >> I built stable/11 in that bhyve guest and updated that guest to >> stable/11 from yesterday. >> To my surpise llvm 7.01 was also merged to stable/11.? Thank you for >> that great supprt! >> No problems with any binary in the stable/11 bhyve guest. >> >> Then I built stable/12 in that re-built stable/11 guest. >> As result, again all binaries linked to /lib/libcrypto.so.111 crash >> (signal 11) with the stable/12 iso in the same bhyve guest. >> >> Here the example from ntpq: >> Program terminated with signal 11, Segmentation fault. >> Reading symbols from /lib/libedit.so.7...Reading symbols from >> /usr/lib/debug//lib/libedit.so.7.debug...done. >> done. >> Loaded symbols for /lib/libedit.so.7 >> Reading symbols from /lib/libm.so.5...Reading symbols from >> /usr/lib/debug//lib/libm.so.5.debug...done. >> done. >> Loaded symbols for /lib/libm.so.5 >> Reading symbols from /libexec/ld-elf.so.1...Reading symbols from >> /usr/lib/debug//libexec/ld-elf.so.1.debug...done. >> done. >> #0? memset (dest=0x8005ef790, c=0, len=<value optimized out>) at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 >> 5624??????????????????? ((char *)dest)[i] = c; >> (gdb) bt >> #0? memset (dest=0x8005ef790, c=0, len=<value optimized out>) at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624 >> #1? 0x000000080025db07 in map_object (fd=3, path=0x80026e1a0 >> "/lib/libcrypto.so.111", sb=0x7fffffffd4c8) at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249 >> #2? 0x0000000800258806 in load_object (name=0x201b40 "libcrypto.so.111", >> fd_u=-1, refobj=0x800270000, flags=<value optimized out>) at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493 >> #3? 0x0000000800251972 in _rtld (sp=<value optimized out>, >> exit_proc=0x7fffffffea50, objp=0x7fffffffea58) at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315 >> #4? 0x0000000800250019 in .rtld_start () at >> /usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39 >> #5? 0x0000000000000000 in ?? () >> >> So please correct me if I'm comletely wrong, but the problem here seems >> to be reproducably rtld-elf related. >> Unfortunately I don't know anything about object files and linkers and >> the related fundamental stuff. > If you do not know about linkers, why do you claim that the problem > is related to rtld ? > >> But maybe someone else has an idea what's going wrong here? > > The fault happens during zeroing of bss. Most likely it is due to some > strangeness of the object being loaded. For diagnostic, show > the output of "readelf -a libcrypto.so.111".Thanks for your help! I just guess it's rtld related, since I obviously misinterpreted the backtrace. Reverting topic change? ELF Header: Magic: 7f 45 4c 46 02 01 01 09 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: FreeBSD ABI Version: 0 Type: DYN (Shared object file) Machine: Advanced Micro Devices x86-64 Version: 0x1 Entry point address: 0x116000 Start of program headers: 64 (bytes into file) Start of section headers: 3090864 (bytes into file) Flags: 0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 8 Size of section headers: 64 (bytes) Number of section headers: 29 Section header string table index: 28 Elf file type is DYN (Shared object file) Entry point 0x116000 There are 8 program headers, starting at offset 64 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040 0x00000000000001c0 0x00000000000001c0 R 0x8 LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000115a7c 0x0000000000115a7c R 0x1000 LOAD 0x0000000000116000 0x0000000000116000 0x0000000000116000 0x00000000001acb20 0x00000000001acb20 R E 0x1000 LOAD 0x00000000002c3000 0x00000000002c3000 0x00000000002c3000 0x000000000002f790 0x00000000000325e0 RW 0x1000 DYNAMIC 0x00000000002f1a80 0x00000000002f1a80 0x00000000002f1a80 0x0000000000000190 0x0000000000000190 RW 0x8 GNU_RELRO 0x00000000002c9000 0x00000000002c9000 0x00000000002c9000 0x0000000000029790 0x0000000000029790 R 0x1 GNU_EH_FRAME 0x00000000000d0050 0x00000000000d0050 0x00000000000d0050 0x000000000000bc74 0x000000000000bc74 R 0x4 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RW 0 Section to Segment mapping: Segment Sections... 00 01 (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) 02 03 04 05 06 07 (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) (null) There are 29 section headers, starting at offset 0x2f29b0: Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align [ 0] (null) NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 : : : [28] (null) NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings) I (info), L (link order), G (group), x (unknown) O (extra OS processing required) o (OS specific), p (processor specific) There is no dynamic section in this file. Thanks a lot, -harry