thr3ads.net - freebsd stable - nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0? [Dec 2009]

If this information is useful, please help other people find it:
Share via:

Steven Hartland

2009-Dec-09 01:59 UTC

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm seeing a
strange
segv which seems to indicate a core library error in _rtld_error. Could this
be the case or is the stack just badly corrupted?

(gdb) bt
#0  0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
#1  0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
#2  0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
#3  0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
#4  0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
#5  0x000000080055576d in ?? () from /libexec/ld-elf.so.1
#6  0x0000000000000001 in ?? ()
#7  0x00000000004117f8 in
boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose
(this=0x800768980) at
sp_counted_impl.hpp:78
Previous frame inner to this frame (corrupt stack?)

    Regards
    Steve


===============================================This e.mail is private and
confidential between Multiplay (UK) Ltd. and the person or entity to whom it is
addressed. In the event of misdirection, the recipient is prohibited from using,
copying, printing or otherwise disseminating it or any information contained in
it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.

Kostik Belousov

2009-Dec-09 10:21 UTC

head link

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

On Wed, Dec 09, 2009 at 01:48:38AM -0000, Steven Hartland
wrote:> I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm
seeing a
> strange
> segv which seems to indicate a core library error in _rtld_error. Could
this
> be the case or is the stack just badly corrupted?
> 
> (gdb) bt
> #0  0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
> #1  0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
> #2  0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
> #3  0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
> #4  0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
> #5  0x000000080055576d in ?? () from /libexec/ld-elf.so.1
> #6  0x0000000000000001 in ?? ()
> #7  0x00000000004117f8 in 
>
boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose
(this=0x800768980) at
> sp_counted_impl.hpp:78
> Previous frame inner to this frame (corrupt stack?)
You need to rebuild rtld with debugging information. Ideally, all shared
objects should have valid debug info.

At least, enter src/libexec/rtld-elf and do
make obj
make depend
make all install DEBUG_FLAGS=-g
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20091209/7c8224e2/attachment-0001.pgp

Steven Hartland

2009-Dec-09 11:21 UTC

head link

nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

----- Original Message ----- 
From: "Kostik Belousov" <kostikbel@gmail.com>
To: "Steven Hartland" <killing@multiplay.co.uk>
Cc: <freebsd-hackers@freebsd.org>; <freebsd-stable@freebsd.org>
Sent: Wednesday, December 09, 2009 10:21 AM
Subject: Re: nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

This is the trace once world had been recompiled with:-
CFLAGS=-pipe
WITH_CTF=1
DEBUG_FLAGS=-g


#0  0x0000000800c95eec in thr_kill () at thr_kill.S:3
#1  0x0000000800b22e9e in _thr_send_sig (thread=0x800f06600, sig=6) at
/usr/src/lib/libthr/thread/thr_kern.c:92
#2  0x0000000800b1f878 in _raise (sig=6) at
/usr/src/lib/libthr/thread/thr_sig.c:187
#3  0x0000000800d74003 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#4  0x000000000043b8a7 in Client::threadMain (this=0x800f9cf40) at
ext/nginx/HelperServer.cpp:516
#5  0x0000000000411302 in boost::_mfi::mf0<void, Client>::operator()
(this=0x7fffffa45ea8, p=0x800f9cf40) at
mem_fn_template.hpp:49
#6  0x0000000000411651 in boost::_bi::list1<boost::_bi::value<Client*>
>::operator()<boost::_mfi::mf0<void, Client>,
boost::_bi::list0> (this=0x7fffffa45eb8, f=@0x7fffffa45ea8,
a=@0x7fffffa45d7f) at bind.hpp:232
#7  0x0000000000411696 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
Client>, boost::_bi::list1<boost::_bi::value<Client*> >
 >::operator() (this=0x7fffffa45ea8) at bind_template.hpp:20
#8  0x00000000004116bd in
boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
boost::_mfi::mf0<void,
Client>, boost::_bi::list1<boost::_bi::value<Client*> > >,
void>::invoke (
    function_obj_ptr=@0x7fffffa45ea8) at function_template.hpp:158
#9  0x000000000042e73a in boost::function0<void, std::allocator<void>
>::operator() (this=0x7fffffa45ea0) at
function_template.hpp:825
#10 0x0000000000435760 in oxt::thread::thread_main (func=@0x7fffffa45ea0,
data=@0x7fffffa45e90) at thread.hpp:107
#11 0x000000000041310e in
boost::_bi::list2<boost::_bi::value<boost::function<void ()(),
std::allocator<void> > >,
boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> >
>::operator()<void (*)(boost::function<void ()(),
std::allocator<void> >,
boost::shared_ptr<oxt::thread::thread_data>), boost::_bi::list0>
(this=0x800f3ee80, f=@0x800f3ee78,
a=@0x7fffffa45f0f) at bind.hpp:289
#12 0x0000000000413196 in boost::_bi::bind_t<void, void
(*)(boost::function<void ()(), std::allocator<void> >,
boost::shared_ptr<oxt::thread::thread_data>),
boost::_bi::list2<boost::_bi::value<boost::function<void ()(),
std::allocator<void>
 > >,
boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> > >
>::operator() (this=0x800f3ee78) at bind_template.hpp:20
#13 0x00000000004131b9 in
boost::thread::thread_data<boost::_bi::bind_t<void, void
(*)(boost::function<void ()(),
std::allocator<void> >,
boost::shared_ptr<oxt::thread::thread_data>),
boost::_bi::list2<boost::_bi::value<boost::function<void
()(), std::allocator<void> > >,
boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> > >
> >::run (this=0x800f3ee00) at
thread.hpp:130
#14 0x0000000000443259 in thread_proxy (param=0x800f3ee00) at
ext/boost/src/pthread/thread.cpp:127
#15 0x0000000800b1badd in thread_start (curthread=0x800f06600) at
/usr/src/lib/libthr/thread/thr_create.c:288
#16 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffffa46000
Current language:  auto; currently asm

It seems that in the passenger client threads it calls closeStream which errors
when
the socket close errors with ENOTCONN
        virtual void closeStream() {
            TRACE_POINT();
            if (fd != -1) {
                int ret = syscalls::close(fd);
                fd = -1;
                if (ret == -1) {
                    if (errno == EIO) {
                        throw SystemException("A write operation on the
session stream failed",
                            errno);
                    } else {
                        throw SystemException("Cannot close the session
stream",
                            errno);
                    }
                }
            }
        }

This causes it to call abort on the the thread which then crashes the app with
the above stack trace, which seems really weird. Anyone got any ideas?

    Regards
    steve 


===============================================This e.mail is private and
confidential between Multiplay (UK) Ltd. and the person or entity to whom it is
addressed. In the event of misdirection, the recipient is prohibited from using,
copying, printing or otherwise disseminating it or any information contained in
it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.

Matt Reimer

2010-May-07 17:36 UTC

head link

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

On Tue, Dec 8, 2009 at 6:48 PM, Steven Hartland <killing@multiplay.co.uk>
wrote:> I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm
seeing a
> strange
> segv which seems to indicate a core library error in _rtld_error. Could
this
> be the case or is the stack just badly corrupted?
>
> (gdb) bt
> #0 ?0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
> #1 ?0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
> #2 ?0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
> #3 ?0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
> #4 ?0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
> #5 ?0x000000080055576d in ?? () from /libexec/ld-elf.so.1
> #6 ?0x0000000000000001 in ?? ()
> #7 ?0x00000000004117f8 in
>
boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose
> (this=0x800768980) at sp_counted_impl.hpp:78
> Previous frame inner to this frame (corrupt stack?)
>
> ? Regards
> ? Steve
Steve,

Did you figure this out? We're seeing something very similar with
nginx + passenger + FreeBSD 8.0.

Matt

Matt Reimer

2010-May-07 21:33 UTC

head link

nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

On Wed, Dec 9, 2009 at 4:20 AM, Steven Hartland <killing@multiplay.co.uk>
wrote:>
> ----- Original Message ----- From: "Kostik Belousov"
<kostikbel@gmail.com>
> To: "Steven Hartland" <killing@multiplay.co.uk>
> Cc: <freebsd-hackers@freebsd.org>; <freebsd-stable@freebsd.org>
> Sent: Wednesday, December 09, 2009 10:21 AM
> Subject: Re: nginx + passenger = segv in _rtld_error on restart on
> FreeBSD8.0?
>
> This is the trace once world had been recompiled with:-
> CFLAGS=-pipe
> WITH_CTF=1
> DEBUG_FLAGS=-g
>
>
> #0 ?0x0000000800c95eec in thr_kill () at thr_kill.S:3
> #1 ?0x0000000800b22e9e in _thr_send_sig (thread=0x800f06600, sig=6) at
> /usr/src/lib/libthr/thread/thr_kern.c:92
> #2 ?0x0000000800b1f878 in _raise (sig=6) at
> /usr/src/lib/libthr/thread/thr_sig.c:187
> #3 ?0x0000000800d74003 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
> #4 ?0x000000000043b8a7 in Client::threadMain (this=0x800f9cf40) at
> ext/nginx/HelperServer.cpp:516
> #5 ?0x0000000000411302 in boost::_mfi::mf0<void, Client>::operator()
> (this=0x7fffffa45ea8, p=0x800f9cf40) at mem_fn_template.hpp:49
> #6 ?0x0000000000411651 in
boost::_bi::list1<boost::_bi::value<Client*>
>>::operator()<boost::_mfi::mf0<void, Client>,
boost::_bi::list0>
> (this=0x7fffffa45eb8, f=@0x7fffffa45ea8, a=@0x7fffffa45d7f) at bind.hpp:232
> #7 ?0x0000000000411696 in boost::_bi::bind_t<void,
boost::_mfi::mf0<void,
> Client>, boost::_bi::list1<boost::_bi::value<Client*> >
>::operator()
> (this=0x7fffffa45ea8) at bind_template.hpp:20
> #8 ?0x00000000004116bd in
>
boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf0<void, Client>,
boost::_bi::list1<boost::_bi::value<Client*>
>> >, void>::invoke (
> ? function_obj_ptr=@0x7fffffa45ea8) at function_template.hpp:158
> #9 ?0x000000000042e73a in boost::function0<void,
std::allocator<void>
>>::operator() (this=0x7fffffa45ea0) at function_template.hpp:825
> #10 0x0000000000435760 in oxt::thread::thread_main (func=@0x7fffffa45ea0,
> data=@0x7fffffa45e90) at thread.hpp:107
> #11 0x000000000041310e in
> boost::_bi::list2<boost::_bi::value<boost::function<void ()(),
> std::allocator<void> > >,
> boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> >
>>::operator()<void (*)(boost::function<void ()(),
std::allocator<void> >,
> boost::shared_ptr<oxt::thread::thread_data>), boost::_bi::list0>
> (this=0x800f3ee80, f=@0x800f3ee78, a=@0x7fffffa45f0f) at bind.hpp:289
> #12 0x0000000000413196 in boost::_bi::bind_t<void, void
> (*)(boost::function<void ()(), std::allocator<void> >,
> boost::shared_ptr<oxt::thread::thread_data>),
> boost::_bi::list2<boost::_bi::value<boost::function<void ()(),
> std::allocator<void> > >,
> boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> >
>
>>::operator() (this=0x800f3ee78) at bind_template.hpp:20
> #13 0x00000000004131b9 in
> boost::thread::thread_data<boost::_bi::bind_t<void, void
> (*)(boost::function<void ()(), std::allocator<void> >,
> boost::shared_ptr<oxt::thread::thread_data>),
> boost::_bi::list2<boost::_bi::value<boost::function<void ()(),
> std::allocator<void> > >,
> boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> >
> > >::run
> (this=0x800f3ee00) at thread.hpp:130
> #14 0x0000000000443259 in thread_proxy (param=0x800f3ee00) at
> ext/boost/src/pthread/thread.cpp:127
> #15 0x0000000800b1badd in thread_start (curthread=0x800f06600) at
> /usr/src/lib/libthr/thread/thr_create.c:288
> #16 0x0000000000000000 in ?? ()
> Cannot access memory at address 0x7fffffa46000
> Current language: ?auto; currently asm
>
> It seems that in the passenger client threads it calls closeStream which
> errors when
> the socket close errors with ENOTCONN
> ? ? ? virtual void closeStream() {
> ? ? ? ? ? TRACE_POINT();
> ? ? ? ? ? if (fd != -1) {
> ? ? ? ? ? ? ? int ret = syscalls::close(fd);
> ? ? ? ? ? ? ? fd = -1;
> ? ? ? ? ? ? ? if (ret == -1) {
> ? ? ? ? ? ? ? ? ? if (errno == EIO) {
> ? ? ? ? ? ? ? ? ? ? ? throw SystemException("A write operation on the
> session stream failed",
> ? ? ? ? ? ? ? ? ? ? ? ? ? errno);
> ? ? ? ? ? ? ? ? ? } else {
> ? ? ? ? ? ? ? ? ? ? ? throw SystemException("Cannot close the session
> stream",
> ? ? ? ? ? ? ? ? ? ? ? ? ? errno);
> ? ? ? ? ? ? ? ? ? }
> ? ? ? ? ? ? ? }
> ? ? ? ? ? }
> ? ? ? }
>
> This causes it to call abort on the the thread which then crashes the app
> with
> the above stack trace, which seems really weird. Anyone got any ideas?
>
> ? Regards
> ? steve
Steve,

The patch for PR 144061 works for us.

http://lists.freebsd.org/pipermail/freebsd-hackers/2010-February/030741.html
http://www.freebsd.org/cgi/query-pr.cgi?pr=144061

Matt

Steven Hartland

2010-May-08 16:23 UTC

head link

nginx + passenger = segv in _rtld_error on restart onFreeBSD8.0?

Thanks Matt most appreciated!
----- Original Message ----- 
From: "Matt Reimer" <mattjreimer@gmail.com>
Steve,

The patch for PR 144061 works for us.

http://lists.freebsd.org/pipermail/freebsd-hackers/2010-February/030741.html
http://www.freebsd.org/cgi/query-pr.cgi?pr=144061

===============================================This e.mail is private and
confidential between Multiplay (UK) Ltd. and the person or entity to whom it is
addressed. In the event of misdirection, the recipient is prohibited from using,
copying, printing or otherwise disseminating it or any information contained in
it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.

freebsd stable - Dec 2009 - nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

nginx + passenger = segv in _rtld_error on restart onFreeBSD8.0?