On Sat, Jun 13, 2020 at 10:24:49AM +1000, Dewayne Geraghty wrote:> After upgrading to 12.1Stable as of June 11: > 1) squid - fails with segmentation fault, ldd "Cannot load PIE binary" > 2) gcc9 - suffers a cc1 internal compiler error > 3) pkg-static - issues "failed" messages, unable to package or install > > Environment Xeon E3, ufs2 only, previously running FreeBSD 12.1 dated > 1st May (from kernel.old). > Prior to the upgrade all ports were rebuilt without issue, but NOT > installed as they were a fall-back, in the event that clang 10 caused > issues (the concern). There are multiple jails on this system, both > amd64 and i386 - some for building, testing and production use. One of > the production i386 jails runs squid, unchanged since Sept 2019. > > /etc/src.conf contains > WITH_PIE=YES > WITH_BIND_NOW=YES > > Most of our 1400+ ports are built and run with relro, now, pie and where > possible with noexecstack &/or no-common. These functioned in an ASLR > environment. (ASLR is only disabled during builds (gcc9 complains), or > when there's a problem, now). > > Note: NONE of the ports were rebuilt after the upgrade. However as part > of resolution, beep and squid were rebuilt. > > === Sequence of thigns ==> > Upgrade performed. System rebooted without incident to > FreeBSD 12.1-STABLE #0 r362003M: Thu Jun 11 23:07:00 AEST 2020 i386 > hqdev-amd64-smp-vga 1201517 1201517 > but some port/application failures: > > Problem 1 > --------- > > i386 jail demonstrated: > > # /usr/local/etc/rc.d/squid start > Starting squid. > Segmentation faultSegmentation fault means that image was activated and kernel handed control to usermode. Try to debug it some, for instance use ktrace and LD_DEBUG=1 to see how much things progressed.> > # ldd /usr/local/sbin/squid > /usr/local/sbin/squid: > ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid > as DSO > /usr/local/sbin/squid: exit status 1This is cosmetics, the problem is in ldd(1) mis-detecting PIE binary as DSO. Before some changes in rtld it was innocent, but since from recent times rtld refuses to dlopen(3) PIE binaries, method that ldd uses for DSO no longer works. It will take some time to fix ldd, because it needs to start parsing dynamic segment for DF_1_FLAGS.
Hi Konstantin,
I did try ktrace/kdump but kdump complained of "data too short".
Using
your suggestion about LD_DEBUG nearly caused me to fall off my chair. I
think this is most relevant:
# setenv LD_DEBUG 1
# ktrace -f /tmp/sq3.kt /usr/local/sbin/squid start
/libexec/ld-elf.so.1 is initialized, base address = 0x8aded000
RTLD dynamic = 0x8ae0f6c0
RTLD pltgot = 0
initializing thread locks
_rtld_thread_init: done
processing main program's program header
note osrel 1201517
note fctl0 0
note crt_no_init
AT_EXECPATH 0xffffdfe0 /usr/bin/ktrace
obj_main path /usr/bin/ktrace
Filling in DT_DEBUG entry
Ignoring d_tag 1879048186 = 0x6ffffffa
/usr/bin/ktrace valid_hash_sysv 1 valid_hash_gnu 0 dynsymcount 30
lm_init("(null)")
loading LD_PRELOAD libraries
loading needed objects
Searching for "libc.so.7"
search_library_pathfds('libc.so.7', '(null)', fdp)
lm_find("/usr/bin/ktrace", "/lib")
lmp_find("/usr/bin/ktrace")
lmp_find("$DEFAULT$")
Trying "/lib/libc.so.7"
Opened "/lib/libc.so.7", fd 3
loading "/lib/libc.so.7"
Ignoring d_tag 1879048186 = 0x6ffffffa
/lib/libc.so.7 valid_hash_sysv 1 valid_hash_gnu 1 dynsymcount 3126
0x6d20f000 .. 0x6d3eafff: /lib/libc.so.7
checking for required versions
initializing initial thread local storage offsets
relocating "/usr/bin/ktrace"
reloc_jmpslot: *0x4b177c04 = 0x6d39bac0
reloc_jmpslot: *0x4b177c08 = 0x6d3b6480
... [ Lots of these ]
reloc_jmpslot: *0x9d4859fc = 0x9d46e560
reloc_jmpslot: *0x9d485a00 = 0x9d47a428
reloc_jmpslot: *0x9d485a04 = 0x9d47a448
relocating "/usr/local/lib/heimdal/libasn1.so.8"
relocating "/usr/local/lib/heimdal/libwind.so.0"
relocating "/usr/local/lib/heimdal/libheimbase.so.1"
relocating "/usr/local/lib/heimdal/libhx509.so.5"
relocating "/usr/local/lib/heimdal/libhcrypto.so.4"
relocating "/usr/local/lib/heimdal/libheimsqlite.so.0"
relocating "/usr/local/lib/heimdal/libcom_err.so.1"
relocating "/usr/local/lib/heimdal/libroken.so.18"
relocating "/usr/local/lib/libintl.so.8"
relocating "/usr/local/lib/heimdal/libheimntlm.so.0"
doing copy relocations
initializing initial thread local storage
initializing key program variables
"__progname": *0x73be38b8 <-- 0xffffdd44
"environ": *0x73cbbc08 <-- 0xfffc69cc
"__elf_aux_vector": *0x9d4a6a5c <-- 0xfffc6a4c
resolving ifuncs
reloc_jmpslot: *0x9d484d88 = 0x9d352cc0
reloc_jmpslot: *0x9d484d8c = 0x9d352c70
reloc_jmpslot: *0x9cb16e04 = 0x9d352c70
reloc_jmpslot: *0x9cb16e0c = 0x9d352cc0
calling init function for /lib/libc.so.7 at 0x9d47c440
calling init function for /lib/libc.so.7 at 0x9d40fc20
calling init function for /lib/libc.so.7 at 0x9d3f0f40
calling init function for /lib/libthr.so.3 at 0x9cb15210
calling init function for /lib/libthr.so.3 at 0x9cb10cb0
_rtld_thread_init: done
calling init function for /lib/libgcc_s.so.1 at 0x9bfad540
calling init function for /lib/libgcc_s.so.1 at 0x9bfa3de0
calling init function for /lib/libcxxrt.so.1 at 0x9b698480
calling init function for /usr/lib/libc++.so.1 at 0x9ab43d90
calling init function for /usr/lib/libc++.so.1 at 0x9aae3e40
calling init function for /lib/libcrypt.so.5 at 0x965d3210
calling init function for /usr/local/lib/libcrypto.so.11 at 0x96e9f420
calling init function for /usr/local/lib/heimdal/libroken.so.18 at
0xa32b6f60
calling init function for /usr/local/lib/libintl.so.8 at 0xa23cb104
calling init function for /usr/local/lib/heimdal/libheimbase.so.1 at
0x9ed15de0
calling init function for /usr/local/lib/heimdal/libcom_err.so.1 at
0x9fec3768
calling init function for /usr/local/lib/heimdal/libasn1.so.8 at 0x9e23f054
calling init function for /usr/local/lib/heimdal/libhcrypto.so.4 at
0x9fb5641c
calling init function for /usr/local/lib/heimdal/libheimsqlite.so.0 at
0xa1b9603c
calling init function for /usr/local/lib/heimdal/libwind.so.0 at 0x9f77a990
calling init function for /usr/local/lib/heimdal/libhx509.so.5 at 0xa0c4e3b4
calling init function for /usr/local/lib/heimdal/libkrb5.so.26 at 0x99527188
calling init function for /usr/local/lib/heimdal/libheimntlm.so.0 at
0xa498c814
calling init function for /usr/local/lib/heimdal/libgssapi.so.3 at
0x9a45bfd0
calling init function for /usr/local/lib/libpcre.so.1 at 0x97550ca8
calling init function for /usr/local/lib/libpcreposix.so.0 at 0x9728e518
calling init function for /lib/libm.so.5 at 0x97139f10
calling init function for /usr/local/lib/libssl.so.11 at 0x98579078
calling init function for /usr/lib/libregex.so.1 at 0x95e8f0a0
calling init function for /usr/lib/librt.so.1 at 0x953ca240
loading filtees
enforcing main obj relro
transferring control to program entry point = 0x73757360
"atexit" in "squid" ==> 0x9d457ac0 in
"libc.so.7"
reloc_jmpslot: *0x73be32b4 = 0x9d457ac0
"_ZSt13set_terminatePFvvE" in "squid" ==> 0x9b697a30 in
"libcxxrt.so.1"
reloc_jmpslot: *0x73be3794 = 0x9b697a30
"strlen" in "squid" ==> 0x9d4750b0 in
"libc.so.7"
reloc_jmpslot: *0x73be3108 = 0x9d4750b0
"malloc" in "squid" ==> 0x9d3e6620 in
"libc.so.7"
reloc_jmpslot: *0x73be3820 = 0x9d3e6620
"memcpy" in "squid" ==> 0x9d47aa20 in
"libc.so.7"
reloc_jmpslot: *0x73be3500 = 0x9d47aa20
"_Znwj" in "squid" ==> 0x9ab437c0 in
"libc++.so.1"
reloc_jmpslot: *0x73be3578 = 0x9ab437c0
"free" in "squid" ==> 0x9d3e9ea0 in "libc.so.7"
reloc_jmpslot: *0x73be3790 = 0x9d3e9ea0
"_ZdlPv" in "squid" ==> 0x9ab438d0 in
"libc++.so.1"
reloc_jmpslot: *0x73be32b8 = 0x9ab438d0
"getopt_long" in "squid" ==> 0x9d3515f0 in
"libc.so.7"
reloc_jmpslot: *0x73be3614 = 0x9d3515f0
Segmentation fault
I've placed the ktrace (sq3.kt.xz) and LD_DEBUG output (sq.deb.xz) at
http://www.heuristicsystems.com/squid4K
Pity about ldd, but I appreciate the complexity & the effort. I'm also
comforted that it isn't an error, rather the evolution of ldd :)