freebsd stable - Jun 2015 - Many core dumps in pthread_getspecific.

On Wed, Jun 03, 2015 at 01:34:04PM +0200, Andre Meiser
wrote:
> Hi,
> 
> I'm new to FreeBSD and I'm running into a lot of core dumps (mainly
from Xorg, but also from vim, firefox, ...).
> That's why I asked on freebsd-questions:
> 
> https://lists.freebsd.org/pipermail/freebsd-questions/2015-June/266138.html
> https://lists.freebsd.org/pipermail/freebsd-questions/2015-June/266141.html
> 
> The last Xorg crash was less than 2h ago. :(
> 
> It looks like that there is a missing check for NULL in
lib/libkse/thread/thr_spec.c (line 211-214):
> 
>     pthread = _get_curthread();
> 
>     /* Check if there is specific data: */
>     if (pthread->specific != NULL && (unsigned int)key <
PTHREAD_KEYS_MAX) {
> 
> If pthread is NULL it will core dump. So maybe something like this would
help:
> 
>     if (pthread != NULL && pthread->specific != NULL &&
(unsigned int)key < PTHREAD_KEYS_MAX) {
> 
> But as I'm said, I'm new to FreeBSD I can't evaluate it yet.
> 
> I'm running the most recent stable:
> 
> % uname -a
> FreeBSD andre 10.1-RELEASE-p10 FreeBSD 10.1-RELEASE-p10 #0: Wed May 13
06:54:13 UTC 2015
> 
> I haven't compiled the base system so far. Maybe a more experienced
FreeBSD guy can verify this and decide if it makes any sense or if I have to dig
any deeper into it.
The backtraces you posted do not make any sense, e.g. pthread_mutex_unlock()
does not call pthread_getspecific(), and pthread_getspecific() is not
recursive.

You should recompile both libc and libthr with debugging symbols, like
cd /usr/src
(cd lib/libc && make all install DEBUG_FLAGS=-g)
(cd lib/libthr && make all install DEBUG_FLAGS=-g)
then obtain the core dump and post backtraces.

On Wed, Jun 03, 2015 at 16:58 +0200, Konstantin Belousov
wrote:
> You should recompile both libc and libthr with debugging symbols, like
> cd /usr/src
> (cd lib/libc && make all install DEBUG_FLAGS=-g)
> (cd lib/libthr && make all install DEBUG_FLAGS=-g)
> then obtain the core dump and post backtraces.
Thank you, that was exactly what I was looking for. Now I like FreeBSD even
more. ;)

I made this short after and also rebooted the entire system to make all
programmes use those debug libs. Since than I had not a single core dump.

I experienced something similar in the past, that with activated debugging some
errors can't be triggered any longer.

At the moment I'm happy without crashes and I can work with this system. As
soon as I'm getting a new core dump, I'll post the backtrace. If this
won't happen for weeks, I may recompile the libs again, try to find a way to
trigger the bug on purpose, enable the debug flag again and than provide the
info.

In the meantime, maybe a core developer may think about the lines of code
I'd provided. Why is _get_curthread() compared to NULL at thr_kern.c (line
601) but not at thr_spec.c (line 224)? Either _get_curthread() never ever
returns NULL, than it's pointless to test it or it's missing [not only]
at thr_spec.c.

I'm using activated hyperthreading:

% sysctl hw.model hw.machine hw.ncpu
hw.model: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz
hw.machine: amd64
hw.ncpu: 8

Sincerely yours Andre.

Hi,

On Wed, Jun 03, 2015 at 16:58 +0200, Konstantin Belousov
wrote:
> You should recompile both libc and libthr with debugging symbols, like
> cd /usr/src
> (cd lib/libc && make all install DEBUG_FLAGS=-g)
> (cd lib/libthr && make all install DEBUG_FLAGS=-g)
> then obtain the core dump and post backtraces.
still no new core dump from Xorg, but one from vim.
As far as I recall, vim used to crash at pthread_getspecific, too, but this time
it was different:

Core was generated by `vim'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libncurses.so.8...(no debugging symbols found)...done.
Loaded symbols for /lib/libncurses.so.8
Reading symbols from /usr/local/lib/libintl.so.8...(no debugging symbols
found)...done.
Loaded symbols for /usr/local/lib/libintl.so.8
Reading symbols from /usr/local/lib/libpython2.7.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/local/lib/libpython2.7.so.1
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libutil.so.9...done.
Loaded symbols for /lib/libutil.so.9
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x000000080149d6a2 in check_deferred_signal (curthread=0x802406400) at
/usr/src/lib/libthr/thread/thr_sig.c:331
331             getcontext(uc);
[New Thread 802406400 (LWP 101373/vim)]
Current language:  auto; currently minimal
(gdb) bt
#0  0x000000080149d6a2 in check_deferred_signal (curthread=0x802406400) at
/usr/src/lib/libthr/thread/thr_sig.c:331
#1  0x000000080149d5ed in _thr_ast (curthread=0x802406400) at
/usr/src/lib/libthr/thread/thr_sig.c:264
#2  0x00000008014a23c7 in _thr_rtld_lock_release (lock=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_rtld.c:162
#3  0x000000080083c94d in _r_debug_postinit () from /libexec/ld-elf.so.1
#4  0x000000080083a15d in .text () from /libexec/ld-elf.so.1
#5  0x00000000004e3d83 in preserve_exit ()
#6  0x000000000051e808 in mch_libcall ()
#7  0x000000080149e47a in handle_signal (actp=<value optimized out>,
sig=<value optimized out>, info=<value optimized out>, ucp=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_sig.c:240
#8  0x000000080149e062 in thr_sighandler (sig=<value optimized out>,
info=<value optimized out>, _ucp=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_sig.c:183
#9  <signal handler called>
#10 0x000000080149d6a2 in check_deferred_signal (curthread=0x802406400) at
/usr/src/lib/libthr/thread/thr_sig.c:331
#11 0x000000080149d5ed in _thr_ast (curthread=0x802406400) at
/usr/src/lib/libthr/thread/thr_sig.c:264
#12 0x00000008014a23c7 in _thr_rtld_lock_release (lock=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_rtld.c:162
#13 0x000000080083c94d in _r_debug_postinit () from /libexec/ld-elf.so.1
#14 0x000000080083a15d in .text () from /libexec/ld-elf.so.1
#15 0x000000080149e4e2 in handle_signal (actp=<value optimized out>,
sig=<value optimized out>, info=<value optimized out>, ucp=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_sig.c:256
#16 0x000000080149e062 in thr_sighandler (sig=<value optimized out>,
info=<value optimized out>, _ucp=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_sig.c:183
#17 <signal handler called>
#18 0x00000008017f9b7a in select () from /lib/libc.so.7
#19 0x000000080149bb32 in __select (numfds=1, readfds=0x7fffffffdfb0,
writefds=0x0, exceptfds=0x7fffffffdf30, timeout=0x0) at
/usr/src/lib/libthr/thread/thr_syscalls.c:561
#20 0x000000000051a33b in mch_write ()
#21 0x000000000051a5ac in mch_inchar ()
#22 0x00000000005b7a17 in ui_inchar ()
#23 0x00000000004ae86a in inchar ()
#24 0x00000000004b1cdb in vgetc ()
#25 0x00000000004b0bda in vgetc ()
#26 0x00000000004b2499 in safe_vgetc ()
#27 0x00000000004f560f in normal_cmd ()
#28 0x00000000005df297 in main_loop ()
#29 0x00000000005de908 in main ()

It's still a thread problem, but I've no idea if it's related with
the former pthread_getspecific problem.

Sincerely yours Andre.