freebsd stable - Dec 2009 - Passenger hangs on live and SEGV on tests possible threading / kernel bug?

We're having an issue with Passenger on FreeBSD where it will hang
and stop processing any more requests the details are attach to
the following bug report:
http://code.google.com/p/phusion-passenger/issues/detail?id=318#c14

In addition the test suite crashes in what seems to be a very
basic test, which I'm at a loss with.
http://code.google.com/p/phusion-passenger/issues/detail?id=441

I'm thinking this may be a bugs in the FreeBSD either kernel or
thread library as the crashes don't make any sense from the
application side.

Any advise on debugging or feedback on the stack traces would
be much appreciated.

    Regards
    Steve

===============================================This e.mail is private and
confidential between Multiplay (UK) Ltd. and the person or entity to whom it is
addressed. In the event of misdirection, the recipient is prohibited from using,
copying, printing or otherwise disseminating it or any information contained in
it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.

On Thursday 17 December 2009 6:12:07 am Steven Hartland
wrote:
> We're having an issue with Passenger on FreeBSD where it will hang
> and stop processing any more requests the details are attach to
> the following bug report:
> http://code.google.com/p/phusion-passenger/issues/detail?id=318#c14
> 
> In addition the test suite crashes in what seems to be a very
> basic test, which I'm at a loss with.
> http://code.google.com/p/phusion-passenger/issues/detail?id=441
> 
> I'm thinking this may be a bugs in the FreeBSD either kernel or
> thread library as the crashes don't make any sense from the
> application side.
> 
> Any advise on debugging or feedback on the stack traces would
> be much appreciated.
For the hang it seems you have a thread waiting in a blocking read(), a thread 
waiting in a blocking accept(), and lots of threads creating condition 
variables.  However, the pthread_cond_init() in libpthread (libthr on FreeBSD) 
doesn't call pthread_cleanup_push(), so your stack trace doesn't make
sense to
me.  However, that may be gdb getting confused.  The pthread_cleanup_push() 
frame may be cond_init().  However, it doesn't call umtx_op() (the 
_thr_umutex_init() call it makes just initializes the structure, it doesn't 
make a _umtx_op() system call).  You might try posting on threads@ to try to 
get more info on this, but your pthread_cond_init() stack traces don't
really
make sense.  Can you rebuild libc and libthr with debug symbols?

For example:

# cd /usr/src/lib/libc
# make clean 
# make DEBUG_FLAGS=-g
# make DEBUG_FLAGS=-g install

However, if you are hanging in read(), that usually means you have a socket 
that just doesn't have data.  That might be an application bug of some sort.

The segv trace doesn't include the first part of GDB messages which show
which
thread actually had a seg fault.  It looks like it was the thread that was 
throwing an exception.  However, nanosleep() doesn't throw exceptions, so
that
stack trace doesn't really make sense either.  Perhaps that stack is hosed
by
the exception handling code?

-- 
John Baldwin