I am writing in regard to PR at http://www.freebsd.org/cgi/query-pr.cgi?pr=bin%2F59552 . I am experiencing behavior on 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #11: Sun Mar 26 00:03:52 EST 2006 which looks a lot like something that would be caused by this PR. This happens when apache-1.3 processes that run with Mason code receive a SIGUSR1 (when newsyslog does log rotation) and apache gracefully kills off all processes when restarting. The following is the stack trace that lead me to this PR:> #0 0x00000000 in ?? () > #1 0x2949dad8 in ?? () > #2 0x294a02c4 in ?? () > #3 0x281ca708 in ?? () from /lib/libc.so.6 > #4 0xbfbfe708 in ?? () > #5 0x28196556 in endhostent () from /lib/libc.so.6 > #6 0x2949ed4c in ?? () > #7 0x280a9980 in ?? () from /libexec/ld-elf.so.1 > #8 0x280a9878 in ?? () from /libexec/ld-elf.so.1 > #9 0xbfbfe718 in ?? () > #10 0x2808d5ef in find_symdef () from /libexec/ld- elf.so.1 > #11 0x2808da7c in find_symdef () from /libexec/ld-elf.so.1 > #12 0x281ba09a in __cxa_finalize () from /lib/libc.so.6 > #13 0x281b9d9a in exit () from /lib/libc.so.6 > #14 0x0805cd5b in clean_child_exit (code=692716228) at http_main.c:522 > #15 0x0805e707 in just_die (sig=30) at http_main.c:3202 > #16 0x0805e72a in usr1_handler (sig=692716228) at http_main.c:3212 > #17 0xbfbfffb4 in ?? () > #18 0x0000001e in ?? () > #19 0x00000000 in ?? () > #20 0xbfbfe7e0 in ?? () > #21 0x00000002 in ?? () > #22 0x0805e710 in just_die () at http_main.c:3202 > #23 0x0805fdae in child_main (child_num_arg=692716228) at http_main.c:4571 > #24 0x0806046e in make_child (s=0x294a02c4, slot=0, now=0) at > http_main.c:5051 > #25 0x080604f8 in startup_children (number_to_start=5) at http_main.c:5078 > #26 0x08060b11 in standalone_main (argc=2, argv=0xbfbfeca0) at > http_main.c:5410 > #27 0x08061392 in main (argc=2, argv=0xbfbfeca0) at http_main.c:5767Please let me know if you have any thoughts on the subject or if this is completely unrelated problem. Thanks!
On 5/11/06, m m <needacoder@gmail.com> wrote:> I am writing in regard to PR at > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin%2F59552 . I am experiencing > behavior on 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #11: Sun Mar 26 00:03:52 > EST 2006 which looks a lot like something that would be caused by this PR. > This happens when apache-1.3 processes that run with Mason code receive a > SIGUSR1 (when newsyslog does log rotation) and apache gracefully kills off > all processes when restarting. The following is the stack trace that lead > me to this PR: >Are you able to reproduce this panic on 6.1-RELEASE? -pete -- ~~o0OO0o~~ Pete Wright www.nycbug.org NYC's *BSD User Group
m m wrote:> I am writing in regard to PR at > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin%2F59552 . I am > experiencing > behavior on 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #11: Sun Mar 26 > 00:03:52 > EST 2006 which looks a lot like something that would be caused by this > PR. > This happens when apache-1.3 processes that run with Mason code receive a > SIGUSR1 (when newsyslog does log rotation) and apache gracefully kills > off > all processes when restarting.Would this explain the following segfaults on 6.1-stable? /var/log/messages.0.bz2:Apr 26 00:00:04 mail kernel: pid 87211 (httpd), uid 80: exited on signal 6 /var/log/messages.0.bz2:Apr 26 00:00:04 mail kernel: pid 87208 (httpd), uid 80: exited on signal 6 /var/log/messages.0.bz2:Apr 26 00:00:04 mail kernel: pid 87210 (httpd), uid 80: exited on signal 6 /var/log/messages.0.bz2:Apr 26 00:00:04 mail kernel: pid 87209 (httpd), uid 80: exited on signal 6 -- Michael Butler, CISSP Security Architect Protected Networks http://www.protected-networks.net
On Thu, 11 May 2006 20:57:20 -0400 "m m" <needacoder@gmail.com> wrote:> I am writing in regard to PR at > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin%2F59552 . I am > experiencing behavior on 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #11: > Sun Mar 26 00:03:52 EST 2006 which looks a lot like something that > would be caused by this PR. This happens when apache-1.3 processes > that run with Mason code receive a SIGUSR1 (when newsyslog does log > rotation) and apache gracefully kills off all processes when > restarting. The following is the stack trace that lead me to this PR: > > > #0 0x00000000 in ?? () > > #1 0x2949dad8 in ?? () > > #2 0x294a02c4 in ?? () > > #3 0x281ca708 in ?? () from /lib/libc.so.6 > > #4 0xbfbfe708 in ?? () > > #5 0x28196556 in endhostent () from /lib/libc.so.6 > > #6 0x2949ed4c in ?? () > > #7 0x280a9980 in ?? () from /libexec/ld-elf.so.1 > > #8 0x280a9878 in ?? () from /libexec/ld-elf.so.1 > > #9 0xbfbfe718 in ?? () > > #10 0x2808d5ef in find_symdef () from /libexec/ld- elf.so.1 > > #11 0x2808da7c in find_symdef () from /libexec/ld-elf.so.1 > > #12 0x281ba09a in __cxa_finalize () from /lib/libc.so.6 > > #13 0x281b9d9a in exit () from /lib/libc.so.6 > > #14 0x0805cd5b in clean_child_exit (code=692716228) at > > http_main.c:522 #15 0x0805e707 in just_die (sig=30) at > > http_main.c:3202 #16 0x0805e72a in usr1_handler (sig=692716228) at > > http_main.c:3212 #17 0xbfbfffb4 in ?? () > > #18 0x0000001e in ?? () > > #19 0x00000000 in ?? () > > #20 0xbfbfe7e0 in ?? () > > #21 0x00000002 in ?? () > > #22 0x0805e710 in just_die () at http_main.c:3202 > > #23 0x0805fdae in child_main (child_num_arg=692716228) at > > http_main.c:4571 #24 0x0806046e in make_child (s=0x294a02c4, > > slot=0, now=0) at http_main.c:5051 > > #25 0x080604f8 in startup_children (number_to_start=5) at > > http_main.c:5078 #26 0x08060b11 in standalone_main (argc=2, > > argv=0xbfbfeca0) at http_main.c:5410 > > #27 0x08061392 in main (argc=2, argv=0xbfbfeca0) at http_main.c:5767 > > Please let me know if you have any thoughts on the subject or if this > is completely unrelated problem. Thanks! > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org"You'll need to build ld-elf.so.1 and libc.so.6 to get a sensible backtrace. -- Alexander Kabaev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060514/6ad8e327/signature.pgp
Konstantin Belousov
2006-May-22 03:51 UTC
improper handling of dlpened's C++/atexit() code?
On Sun, May 21, 2006 at 06:22:34PM -0400, m m wrote:> n 5/21/06, Konstantin Belousov <kostikbel@gmail.com> wrote: > >> >Program received signal SIGSEGV, Segmentation fault. > >> >0x00000000 in ?? () > >> >(gdb) bt > >> >#0 0x00000000 in ?? () > >> >#1 0x294c0ad8 in __do_global_dtors_aux () from > >> >/usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >> >#2 0x294c1d4c in _fini () from > >> >/usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >> >#3 0x280b4c80 in ?? () > >> >#4 0x280aaab8 in ?? () from /libexec/ld-elf.so.1 > >> >#5 0xbfbfe6e8 in ?? () > >> >#6 0x2808dca6 in objlist_call_fini (list=0x280a96d8) at > >> >/usr/src/libexec/rtld-elf/rtld.c:1336 > >> >#7 0x2808e1d4 in rtld_exit () at /usr/src/libexec/rtld-elf/rtld.c:1528 > >> >#8 0x281d58ea in __cxa_finalize (dso=0x0) at > >> >/usr/src/lib/libc/stdlib/atexit.c:184 > >> >#9 0x281d55ba in exit (status=0) at /usr/src/lib/libc/stdlib/exit.c:69 > >> >#10 0x0805d0cb in clean_child_exit () > >> >#11 0x0805ea77 in just_die () > >> >#12 0x0805ea9a in usr1_handler () > >> >#13 0xbfbfffb4 in ?? () > >> >#14 0x0000001e in ?? () > >> >#15 0x00000000 in ?? () > >> >#16 0xbfbfe7c0 in ?? () > >> >#17 0x00000002 in ?? () > >> >#18 0x0805ea80 in just_die () > >> >#19 0x0806011e in child_main () > >> >#20 0x080607de in make_child () > >> >#21 0x08060868 in startup_children () > >> >#22 0x08060e81 in standalone_main () > >> >#23 0x08061702 in main () > > > >Could you, please, put somewhere: > >1. /usr/local/lib/perl5/5.8.8/mach/auto/Sys/Syslog/Syslog.so > >2. output of lsof -p <some apache child process pid> for apache running > >in your usual configuration. > > > >Also, could you run the apache with LD_PRELOAD=/usr/lib/libstdc++.so.5 > >and report whether the problem persists ? > > Konstantin, > Thank you for looking into this. > > lsof: http://www.savefile.com/files/6494253 > Syslog.so: http://www.savefile.com/files/2163369 > > Although it's not an indicator of certainty (I have had it exit > cleanly in the past), it appears that running Apache with LD_PRELOAD > of libstdc++ does allow it to exit cleanly. Please let me know how I > can further assist. If it would make things easier - I can provide > access to a jail on this machine which exhibits the same behavior.Ok, I have a theory how it happens. Investigation of your instance of Syslog.so shows that crash happens at the following code of /usr/lib/crtbeginS.o: 282: if (__deregister_frame_info) 283: __deregister_frame_info (__EH_FRAME_BEGIN__); (this comes in from contrib/gcc/crtstuff.c, lines 282-283). Symbol __deregister_frame_info is weak and undefined in all your DSOs except libstdc++.so.5. This symbol provides part of the C++ runtime support for exception handling, and reasonably included from c++ runtime support library. Both lines 282 and 283 produce dynamic relocations in final DSO, but line 282 implies R_386_GLOB_DAT, and 283 - R386_JUMP_SLOT (for PLT). First relocation is resolved immediately on DSO load, second one is resolved on demand. My theory is that, at the time of loading Syslog.so, libstdc++.so.5 is loaded in the process, resulting in first relocation being satisfied by rtld immediately. But, at the time exit() processing comes to _fini() function of Syslog.so, libstdc++.so.5 is unloaded. And weak PLT relocation is resolved to 0. As result we got the frame #0 from your trace. This theory is confirmed by presence of libstdc++ in lsof output. Please, check that it does not show up at the time of crash dump by using "show shared" gdb command on crash dump. Short-time fix is to use LD_PRELOAD hack. The real solution would be to mark the libstdc++ DSO as unloadable and implement support for unloadable DSO in rtld (BTW, I think this is also needed for threading libraries libpthread and libthr for the same reason). I know that glibc dynamic loader has support for this feature. P.S. Apache seems to call exit(3) from the signal handler. This is wrong. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060522/a48ae46d/attachment.pgp