openssh unix dev - Dec 2009 - Static build segfaults on x86_64

Hello everyone,

I would like to ask you for advice on how to approach (or solve) this
particular problem.

I use Slackware Linux and compile Openssh from source. I prefer to
compile it statically so it doesn't get messed up if I update openssl
libraries. Up until now this approach was working OK for me.

Lately I have been challenged with Slackware64 installations and I
have come across a problem with Openssh version (5.3p1, but result is
the same with 5.2p1). What happens is that sshd daemon keeps accepting
connections as long as no one disconnects. On the first DISconnection
the server daemon dies with segfault message, of which a strace output
I have included below. This only happens on 64-bit platform, on 32-bit
it is still working hanky-dory. Non-static build works ok on both
platforms.

Sum:
32-bit shared: OK
32-bit static: OK
64-bit shared: OK
64-bit static: SEGFAULT on first client disconnect

How should I start solving this problem? I am proficient in PHP and
other Untyped languages but I am only moderately familiar with C
programming.

Thank you for your responses,
Bostjan Skufca



--------------[ Segfault output from strace ]-------------------------------
# strace -p 21976
Process 21976 attached - interrupt to quit
select(6, [3], NULL, NULL, NULL)        = ? ERESTARTNOHAND (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], WNOHANG, NULL) =
21984
wait4(-1, 0x7fffc6243994, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, NULL, {0x400be0, [], SA_RESTORER,
0xf0000000fc0c748}, 8) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 21976 detached
-----------------------------------------------------------------------------------



--------------[ Environment description ]-------------------------------
- kernel: 2.6.32.2 (64 bit)
- OS: Slackware64-13
- gcc: from distro (whole toolchain actually)
- libs: glibc is from distro
- libs: openssl is customly compiled 0.9.8l (./config
--prefix=$PDESTDIR_OPENSSL shared)
- openssh: 5.3p1 configured like this:

./configure --prefix=$PDESTDIR \
  --sysconfdir=/etc/ssh \
  --with-ssl-dir=$PDESTDIR_OPENSSL \
  --with-ldflags=-static \
  --with-privsep-path=/var/empty \
  --with-default-path=...

Using "LDFLAGS=-static" produces the same results.
------------------------------------------------------------------------------------

Bostjan Skufca wrote:
> Hello everyone,
> 
> I would like to ask you for advice on how to approach (or solve) this
> particular problem.
> 
> I use Slackware Linux and compile Openssh from source. I prefer to
> compile it statically so it doesn't get messed up if I update openssl
> libraries. Up until now this approach was working OK for me.
As an aside: you don't need to statically link the whole binary just to 
get the crypto libs.  If you build OpenSSL with just the static library 
(libcrypto.a) and configure OpenSSH --with-ssl-dir=thatdir then the 
linker will pick up the crypto functions from the static library and the 
remainder from the system dynamic libraries.
> Lately I have been challenged with Slackware64 installations and I
> have come across a problem with Openssh version (5.3p1, but result is
> the same with 5.2p1). What happens is that sshd daemon keeps accepting
> connections as long as no one disconnects. On the first DISconnection
> the server daemon dies with segfault message, of which a strace output
> I have included below. 
Unfortunately the strace does not show anything useful, but from your 
description it sounds like it's crashing in the SIGCHLD handler, 
although I don't know why it would show up only with static linking.

You might get something more useful from enabling sshd's debugging, eg

# /path/to/sshd -o LogLevel=debug3 -D

[...]
> How should I start solving this problem? I am proficient in PHP and
> other Untyped languages but I am only moderately familiar with C
> programming.
Did it generate a core dump?  if so, your first step is to feed it into 
gdb and generate a backtrace, which will tell you where it crashed.  You do:

$ gdb /path/to/sshd core
(gdb) bt

If it doesn't generate a core dump (eg because the ulimit prevents it) 
then you can attach gdb to the process you expect to crash, wait for it 
to crash then generate the backtrace

$ gdb /path/to/sshd [pid of sshd]
[wait for crash]
(gdb) bt

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.