freebsd stable - Jun 2016 - 10.3-RELEASE amd64 segmentation faults in wc, sh...

Hello,

I'm building a pair of firewall with 10.3 and I see some rare
segmentation faults (5 in a week) in processes like wc, sh or ifstated.

wc, sh are used by some scripts who check the state of the interfaces
and the state of bgp sessions. Ifstated called theses scripts too.

I thought there was a bug in ifstated so I made a small program in
perl to do the same thing but the problem is not in ifstated.

The machines are some Dell R730, I've checked the memory with
memtest86 for one week without error. The problem occurs on both
firewalls so i don't think this is a hardware problem.

Any idea? 

Thanks, regards.

ifstated
--------
Core was generated by `ifstated'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/lib/libevent-2.0.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/local/lib/libevent-2.0.so.5
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000800e23a67 in pthread_getspecific () from /lib/libthr.so.3
Cannot find new threads: generic error

sh
--
Core was generated by `sh'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x000000080063351b in _rtld_atfork_post () from /libexec/ld-elf.so.1

wc
--
Core was generated by `wc'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000800612524 in _rtld_atfork_post () from /libexec/ld-elf.so.1

On Thu, Jun 30, 2016 at 01:57:32PM +0200, Patrick Lamaiziere
wrote:
> Hello,
> 
> I'm building a pair of firewall with 10.3 and I see some rare
> segmentation faults (5 in a week) in processes like wc, sh or ifstated.
> 
> wc, sh are used by some scripts who check the state of the interfaces
> and the state of bgp sessions. Ifstated called theses scripts too.
> 
> I thought there was a bug in ifstated so I made a small program in
> perl to do the same thing but the problem is not in ifstated.
> 
> The machines are some Dell R730, I've checked the memory with
> memtest86 for one week without error. The problem occurs on both
> firewalls so i don't think this is a hardware problem.
> 
> Any idea? 
> 
> Thanks, regards.
> 
> ifstated
> --------
> Core was generated by `ifstated'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/local/lib/libevent-2.0.so.5...(no debugging
symbols found)...done.
> Loaded symbols for /usr/local/lib/libevent-2.0.so.5
> Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /lib/libthr.so.3...(no debugging symbols
found)...done.
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x0000000800e23a67 in pthread_getspecific () from /lib/libthr.so.3
> Cannot find new threads: generic error
> 
> sh
> --
> Core was generated by `sh'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x000000080063351b in _rtld_atfork_post () from /libexec/ld-elf.so.1
> 
> wc
> --
> Core was generated by `wc'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x0000000800612524 in _rtld_atfork_post () from /libexec/ld-elf.so.1
This is most likely the problems, reported and fixed in r300758,
PR 204764 r302063, and PR 204426 r302236.  First and second commits
are already in stable/10, the third one will be merged in several days.

Le Thu, 30 Jun 2016 15:52:04 +0300,
Konstantin Belousov <kostikbel at gmail.com> a ?crit :

Hello,
> On Thu, Jun 30, 2016 at 01:57:32PM +0200, Patrick Lamaiziere wrote:
> > Hello,
> > 
> > I'm building a pair of firewall with 10.3 and I see some rare
> > segmentation faults (5 in a week) in processes like wc, sh or
> > ifstated.
...
> This is most likely the problems, reported and fixed in r300758,
> PR 204764 r302063, and PR 204426 r302236.  First and second commits
> are already in stable/10, the third one will be merged in several
> days.
For the record, it looks like the first and second commits fixed
this problem here. I've not yet tested the third one (don't know if it
is already in 10/stable).

Thanks again Konstantin.
Regards,