Fletcher Cocquyt
2006-Aug-22 23:16 UTC
[dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after118855-15)
I need some help using Dtrace to debug what in 118855-15 is breaking my web server - from the pstack in the thread below, can anyone suggest a starting point? Thanks, Fletcher. -----Original Message----- From: Fletcher Cocquyt [mailto:fcocquyt at stanford.edu] Sent: Monday, August 14, 2006 12:16 PM To: ''Matty''; ''Solaris Dtrace List'' Subject: RE: [dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after patches) The plot thickens - the hangs re-appeared after applying the recommended patches to sol10u1 At this point I have re-installed sol10u1 and checkpointed the non-hanging webserver (no patches) with a Flash archive and I have a raidctl mirror sync in progress. Once the mirror is synced up, my plan is to 1) break the mirror 2) apply the kernel patch, or next in recommended list 3) reboot, re-run the web test 4) if HANG report to SUN (on the open case) - this patch breaks web server 5) else repeat 2-4 with next patch in the list Hopefully I will isolate what I suspect is a bad patch Stay tuned... -----Original Message----- From: dtrace-discuss-bounces at opensolaris.org [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Fletcher Cocquyt Sent: Wednesday, August 09, 2006 3:28 PM To: ''Matty''; ''Solaris Dtrace List'' Subject: RE: [dtrace-discuss] apache hangs on sol10u2 not sol10u1 So I tarred up the apache install and dropped it on a sol10u1 server - it runs perfectly clean - no hanging web/svn operations Maybe I''m guilty of jumping on U2 too quickly before the first round of patches? Still have the Sun case open - but I think I''ll need to downgrade to U1 for operational time constraints. -----Original Message----- From: dtrace-discuss-bounces at opensolaris.org [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Fletcher Cocquyt Sent: Monday, August 07, 2006 2:05 PM To: ''Matty''; ''Solaris Dtrace List'' Subject: RE: [dtrace-discuss] dtrace apache hang on sol10u2 Apache hang - pstack below: god at irt-web-xyz:httpd-2.0.59 2:03pm 129 # pstack 6092 6092: /opt/httpd-2.0.59/bin/httpd -k start bfb906d7 pollsys (8044ad0, 1, 8044ab0, 0) bfb3a722 poll (8044ad0, 1, 493e0, 493e0, 7, 40004) + 52 bfdba1e7 apr_poll (8044b40, 1, 8044b3c, 493e0, 0, 0) + ab bfdba7f7 apr_wait_for_io_or_timeout (0, 8242718, 0, 8046d30) + 6f bfdb0ec2 apr_socket_sendv (8242718, 8046d30, 2, 8044be8) + 4a bfdb125a apr_sendv (8242718, 8046d30, 2, 8044be8, 8271a18, 8242718) + 22 080b426e writev_it_all (4, 2050f, 8044c64, 8271a18, 0, 0) + 3e 080b5832 core_output_filter (8242bc8, 8271a18, 82466f0, bfdcb9b8) + 9a6 0808da31 chunk_filter (8313848, 8271a18, 852cbc0, 1000) + 169 080b0126 ap_content_length_filter (8265e38, 8271a18, 8046ea8, bfdb8aab) + aa 08091353 ap_byterange_filter (8265e20, 8271a18, 1f91b, bf816b47) + 73 080adf81 ap_filter_flush (8271a18, 8265e20, 82466f0, 1355) + 11 bfe16126 apr_brigade_write (8271a18, 80adf70, 8265e20, 921f0e0, 1f91b, 92d7aa8) + 7a bf8dd62b brigade_write_fn (85a7270, 921f0e0, 8046fa8, 0) + 2f bf813a92 svn_stream_write (85a7278, 921f0e0, 8046fa8, 1f91b) + 26 bf8161a7 encode_data (862dda8, 92c04c0, 8046ffc, 8577ad0) + 8f bf813a92 svn_stream_write (85a7288, 92c04c0, 8046ffc, fffffffe, 8490800, 84907e8) + 26 bf863080 window_handler (92e34c8, 8577ad0, 0, bf8de3d3) + 28c bf8de416 window_handler (92e34c8, 85a7260, 84ae838, bf8c88fc) + 56 bf864c11 svn_txdelta_send_txstream (84c3c48, bf8de3c0, 85a7260, 836d578) + 51 bf8ae87f update_entry (0, 0, 836d5c0, 84fbdc0, 8329550, 836d5b0) + 533 bf8ae28e delta_dirs (0, 83294a8, 8329550, 83294a0, 0, 8329468) + 3f6 bf8ae727 update_entry (0, 0, 83294a8, 832d408, 82b8798, 83294a0) + 3db bf8ae28e delta_dirs (824bac0, 827a5a0, 82b8798, bf8b1ce9, 1, 8265158) + 3f6 bf8aee76 svn_repos_finish_report (827a518, 8265158, 8282240, 827a6b0) + 3c2 bf8defa6 dav_svn__update_report (8271710, 82713e0, 8265e20, 827149a) + 976 bf8e1443 dav_svn_deliver_report (8265190, 8271710, 82713e0, 8265e20, 8265190, bf99d50c) + 20b bf996c76 dav_method_report (ffffffff, 0, 0, bfdb8aab, 8271348, 8271170) + be bf997f52 dav_handler (8265190, 82664e8, 0, 82664b8) + 906 080a24ea ap_run_handler (8265190, 8265190, 8047838, 80a2847, 82651a8, 64) + 32 080a28b1 ap_invoke_handler (8265190, 0, 8047868, 80ae861) + ad 080920b5 ap_process_request (8265190, 4, 8265190, 0) + 135 0808db9d ap_process_http_connection (82427f0, 82466f0, 80478c8, 80abe85) + e9 080abbf6 ap_run_process_connection (82427f0, 8242718, 82426e0, 82427f0, 0, 0) + 32 080a0cfd child_main (10, 1, 1, 0) + 345 080a0f27 make_child (8047a00, 8047a08, 8047a38, 80a164f, 3f1, 8235440) + d7 080a0fb8 startup_children (3f1, 8235440, 8047978, 8074d65, fc8, 5) + 40 080a164f ap_mpm_run (80f9150, 8133238, 80fe880, 80fe880) + 68f 080a6963 main (3, 8047ad8, 8047ae8) + 5cb 0806d63c _start (3, 8047bd8, 8047bf4, 8047bf7, 0, 8047bfd) + 80 -----Original Message----- From: dtrace-discuss-bounces at opensolaris.org [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Matty Sent: Monday, August 07, 2006 1:57 PM To: Solaris Dtrace List Subject: Re: [dtrace-discuss] dtrace apache hang on sol10u2 On Mon, 7 Aug 2006, Fletcher Cocquyt wrote:> I have a reproducible hang on my sol10u2 apache 2.0.59 web server.Is this a system or web server hang? Assuming the web server is hung, what does pstack print when you run it against a hung process? - Ryan -- UNIX Administrator http://prefetch.net _______________________________________________ dtrace-discuss mailing list dtrace-discuss at opensolaris.org _______________________________________________ dtrace-discuss mailing list dtrace-discuss at opensolaris.org _______________________________________________ dtrace-discuss mailing list dtrace-discuss at opensolaris.org
Ronald Kuehn
2006-Aug-22 23:27 UTC
[dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after118855-15)
On Wednesday, August 23, 2006 at 01:16:01 CEST, Fletcher Cocquyt wrote: Hi,> I need some help using Dtrace to debug what in 118855-15 is breaking my web > server - from the pstack in the thread below, can anyone suggest a starting > point?[...]> -----Original Message----- > From: dtrace-discuss-bounces at opensolaris.org > [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Fletcher > Cocquyt > Sent: Monday, August 07, 2006 2:05 PM > To: ''Matty''; ''Solaris Dtrace List'' > Subject: RE: [dtrace-discuss] dtrace apache hang on sol10u2 > > Apache hang - pstack below: > > god at irt-web-xyz:httpd-2.0.59 2:03pm 129 # pstack 6092 > > 6092: /opt/httpd-2.0.59/bin/httpd -k start > bfb906d7 pollsys (8044ad0, 1, 8044ab0, 0) > bfb3a722 poll (8044ad0, 1, 493e0, 493e0, 7, 40004) + 52 > bfdba1e7 apr_poll (8044b40, 1, 8044b3c, 493e0, 0, 0) + ab > bfdba7f7 apr_wait_for_io_or_timeout (0, 8242718, 0, 8046d30) + 6f > bfdb0ec2 apr_socket_sendv (8242718, 8046d30, 2, 8044be8) + 4a > bfdb125a apr_sendv (8242718, 8046d30, 2, 8044be8, 8271a18, 8242718) + 22 > 080b426e writev_it_all (4, 2050f, 8044c64, 8271a18, 0, 0) + 3e > 080b5832 core_output_filter (8242bc8, 8271a18, 82466f0, bfdcb9b8) + 9a6 > 0808da31 chunk_filter (8313848, 8271a18, 852cbc0, 1000) + 169 > 080b0126 ap_content_length_filter (8265e38, 8271a18, 8046ea8, bfdb8aab) +[...] since there have been some issues involving the new TCP fusion code, could you please test the following: add set ip:do_tcp_fusion = 0x0 to /etc/system and reboot. Do these hangs still happen after this change? Ronald
Fletcher Cocquyt
2006-Aug-23 00:06 UTC
[dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after118855-15)
Thanks for the suggestion - I tried it and it still hangs on pollsys Please let me know if you have other ideas, Fletcher. -----Original Message----- From: Ronald Kuehn [mailto:ronald.kuehn at sun.com] Sent: Tuesday, August 22, 2006 4:27 PM To: Fletcher Cocquyt Cc: ''Matty''; ''Solaris Dtrace List'' Subject: Re: [dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after118855-15) On Wednesday, August 23, 2006 at 01:16:01 CEST, Fletcher Cocquyt wrote: Hi,> I need some help using Dtrace to debug what in 118855-15 is breaking myweb> server - from the pstack in the thread below, can anyone suggest astarting> point?[...]> -----Original Message----- > From: dtrace-discuss-bounces at opensolaris.org > [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Fletcher > Cocquyt > Sent: Monday, August 07, 2006 2:05 PM > To: ''Matty''; ''Solaris Dtrace List'' > Subject: RE: [dtrace-discuss] dtrace apache hang on sol10u2 > > Apache hang - pstack below: > > god at irt-web-xyz:httpd-2.0.59 2:03pm 129 # pstack 6092 > > 6092: /opt/httpd-2.0.59/bin/httpd -k start > bfb906d7 pollsys (8044ad0, 1, 8044ab0, 0) > bfb3a722 poll (8044ad0, 1, 493e0, 493e0, 7, 40004) + 52 > bfdba1e7 apr_poll (8044b40, 1, 8044b3c, 493e0, 0, 0) + ab > bfdba7f7 apr_wait_for_io_or_timeout (0, 8242718, 0, 8046d30) + 6f > bfdb0ec2 apr_socket_sendv (8242718, 8046d30, 2, 8044be8) + 4a > bfdb125a apr_sendv (8242718, 8046d30, 2, 8044be8, 8271a18, 8242718) + 22 > 080b426e writev_it_all (4, 2050f, 8044c64, 8271a18, 0, 0) + 3e > 080b5832 core_output_filter (8242bc8, 8271a18, 82466f0, bfdcb9b8) + 9a6 > 0808da31 chunk_filter (8313848, 8271a18, 852cbc0, 1000) + 169 > 080b0126 ap_content_length_filter (8265e38, 8271a18, 8046ea8, bfdb8aab) +[...] since there have been some issues involving the new TCP fusion code, could you please test the following: add set ip:do_tcp_fusion = 0x0 to /etc/system and reboot. Do these hangs still happen after this change? Ronald
Matty
2006-Aug-24 19:34 UTC
[dtrace-discuss] apache hangs on sol10u2 and sol10u1 (after118855-15)
On Tue, 22 Aug 2006, Fletcher Cocquyt wrote:> Thanks for the suggestion - I tried it and it still hangs on pollsys>> god at irt-web-xyz:httpd-2.0.59 2:03pm 129 # pstack 6092 >> >> 6092: /opt/httpd-2.0.59/bin/httpd -k start >> bfb906d7 pollsys (8044ad0, 1, 8044ab0, 0) >> bfb3a722 poll (8044ad0, 1, 493e0, 493e0, 7, 40004) + 52 >> bfdba1e7 apr_poll (8044b40, 1, 8044b3c, 493e0, 0, 0) + ab >> bfdba7f7 apr_wait_for_io_or_timeout (0, 8242718, 0, 8046d30) + 6f >> bfdb0ec2 apr_socket_sendv (8242718, 8046d30, 2, 8044be8) + 4a >> bfdb125a apr_sendv (8242718, 8046d30, 2, 8044be8, 8271a18, 8242718) + 22 >> 080b426e writev_it_all (4, 2050f, 8044c64, 8271a18, 0, 0) + 3e >> 080b5832 core_output_filter (8242bc8, 8271a18, 82466f0, bfdcb9b8) + 9a6 >> 0808da31 chunk_filter (8313848, 8271a18, 852cbc0, 1000) + 169 >> 080b0126 ap_content_length_filter (8265e38, 8271a18, 8046ea8, bfdb8aab) +This could be caused by a variety of things. To see what Apache is waiting on, I would recommend building Apache in maintainer mode (add "--enable-maintainer-mode" to the configure line to enable it) to get debugging symbols. Once you have debugging symbols, attach to the httpd process when it hangs, skip back to the stack frame for apr_poll, and print out the contents of the apr_pollfd_t structure passed to apr_poll() (You will be most interested in the contents of the apr_datatype_e and apr_descriptor members. Check apr_poll.h for a detailed description of the apr_pollfd_t structure.). The contents of the structure should tell you what Apache is waiting on, and should help with determining the next steps to take. You could also dig up similar information by digging through the structures passed to poll(). Maybe one of the Solaris kernel folks can provide some guidance for doing so. Thanks, - Ryan -- UNIX Administrator http://prefetch.net
Fletcher Cocquyt
2006-Nov-12 18:07 UTC
[dtrace-discuss] apache hangs on sol10u2 and sol10u1(after118855-15) - UPDATE
Well, below is the Sun gcore analysis of the hang - note the timeout value
in the apr_poll call below - does 30000LL seem like a legit value??
thanks
=>[3] libapr-0.so.0.9.12:apr_poll(aprset = 0x8044df0, num = 1, nsds
0x8044db8, timeout = 300000LL), line 130 in "poll.c"
Reading httpd
core file header read successfully
Reading ld.so.1
Reading libaprutil-0.so.0.9.12
Reading libexpat.so.0.1.0
Reading libapr-0.so.0.9.12
Reading mod_proxy.so
Reading mod_proxy_connect.so
Reading mod_proxy_ftp.so
Reading mod_proxy_http.so
Reading mod_dav.so
Reading mod_info.so
Reading mod_dav_fs.so
Reading mod_vhost_alias.so
Reading mod_rewrite.so
Reading mod_dav_svn.so
Reading libsvn_repos-1.so.0.0.0
Reading libsvn_fs-1.so.0.0.0
Reading libsvn_delta-1.so.0.0.0
Reading libsvn_subr-1.so.0.0.0
Reading libz.so.1
Reading libiconv.so.2.1.0
Reading libsvn_fs_fs-1.so.0.0.0
Reading libz.so.1.2.1
Reading mod_authz_svn.so
Reading mod_webauth.so
Reading libwebauth.so.1.1.2
Reading libkrb5.so.3
Reading libk5crypto.so.3
Reading libcom_err.so.3
Reading libcurl.so.2.0.2
Reading libgen.so.1
Reading libphp4.so
Reading libmysqlclient.so.14.0.0
Reading libnetsnmp.so.5.2.1
Reading mod_auth_kerb.so
Reading libgssapi_krb5.so.2
Reading mod_extract_forwarded.so
Reading nss_files.so.1
Reading UTF-8%646.so
Reading 646%UTF-8.so
program terminated by signal 0 (UNKNOWN SIGNAL)
0xffffffffffffffff: <bad address 0xffffffffffffffff>
Current function is apr_poll
130 i = poll(pollset, num_to_poll, timeout);
[1] 0xbfb906c7(0x8044d80, 0x1, 0x8044d60, 0x0), at 0xbfb906c7
[2] 0xbfb3a722(0x8044d80, 0x1, 0x493e0, 0x493e0, 0x7, 0x40004), at
0xbfb3a722
=>[3] libapr-0.so.0.9.12:apr_poll(aprset = 0x8044df0, num = 1, nsds
0x8044db8, timeout = 300000LL), line 130 in "poll.c"
[4] libapr-0.so.0.9.12:apr_wait_for_io_or_timeout(f = (nil), s = (nil),
for_read = 0), line 54 in "waitio.c"
[5] libapr-0.so.0.9.12:apr_socket_sendv(sock = 0x82454e8, vec = 0x8046fe0,
nvec = 0, len = 0x8044e98), line 208 in "sendrecv.c"
[6] libapr-0.so.0.9.12:apr_sendv(sock = 0x82454e8, vec = 0x8046fe0, nvec 2,
len = 0x8044e98), line 991 in "sendrecv.c"
[7] httpd:writev_it_all(s = 0x82454e8, vec = (nil), nvec = 4, len 133173U,
nbytes = 0x8044f08), line 2928 in "core.c"
[8] httpd:core_output_filter(f = 0x8245998, b = 0x826b860), line 4295 in
"core.c"
[9] httpd:chunk_filter(f = 0x886e880, b = 0x826b860), line 220 in
"http_core.c"
[10] httpd:ap_content_length_filter(f = 0x8e6eff0, b = 0x826b860), line
1268 in "protocol.c"
[11] httpd:ap_byterange_filter(f = 0x8e6efd8, bb = 0x826b860), line 3134
in "http_protocol.c"
[12] httpd:ap_filter_flush(bb = 0x826b860, ctx = 0x8e6efd8), line 565 in
"util_filter.c"
[13] libaprutil-0.so.0.9.12:apr_brigade_write(b = (nil), flush = 0x80ae864
= &ap_filter_flush(apr_bucket_brigade *bb, void *ctx), ctx = 0x8e6efd8, str
= 0x8ec4468
"XFBLAwQK\nALy4Gy8JAAAATUVUQS1JTkYvFAAIAAgUTUFOSUZFU1QuTUaNj8sKwjAQRfeF/EN+I
EEFN93VoqCo\nm0r3QzrVYDMp6Yj69/aFiHXhes6Ze+8ByJbYsMoxNNZTLOd6JqI0IDAWavWMZVK
DuaBMiNvbUkTr\nByN1rDqCw1ga75ynRlVAZxFlNRpbWgPcESfLVYvs4AqBQaYDKvc/0Byp8OEdl
/mS7xBQbvyNih6Z\nGmPlRVd56+oKHRL//+5LGcv6cNbQW3qcpodpk4TPeBG9AFBLBwjMe5/VqgA
AAEsBAACwBAAAAG9y\nZwthcGFjaGUTY29tbW9uc7IYbGFuHWVudW1tYXRodGltInhjZXB0aW9uI
GJ1aWxkZXIsRW51bVV0\naWxzLmNsYXNzlZPbbtNAEIb/aQ6uHZMegHIotKSFNklpDQIJiVBuKhC
RXLgoRNxuHCt15diVD30H\nHoJ3gCsCF4hrHgox66xCmkRN8cWsZ/T/n3dnvL///" ...,
nbyte
= 130126U), line 393 in "apr_brigade.c"
[14] mod_dav_svn.so:brigade_write_fn(baton = (nil), data = 0x8ec4468
"XFBLAwQK\nALy4Gy8JAAAATUVUQS1JTkYvFAAIAAgUTUFOSUZFU1QuTUaNj8sKwjAQRfeF/EN+I
EEFN93VoqCo\nm0r3QzrVYDMp6Yj69/aFiHXhes6Ze+8ByJbYsMoxNNZTLOd6JqI0IDAWavWMZVK
DuaBMiNvbUkTr\nByN1rDqCw1ga75ynRlVAZxFlNRpbWgPcESfLVYvs4AqBQaYDKvc/0Byp8OEdl
/mS7xBQbvyNih6Z\nGmPlRVd56+oKHRL//+5LGcv6cNbQW3qcpodpk4TPeBG9AFBLBwjMe5/VqgA
AAEsBAACwBAAAAG9y\nZwthcGFjaGUTY29tbW9uc7IYbGFuHWVudW1tYXRodGltInhjZXB0aW9uI
GJ1aWxkZXIsRW51bVV0\naWxzLmNsYXNzlZPbbtNAEIb/aQ6uHZMegHIotKSFNklpDQIJiVBuKhC
RXLgoRNxuHCt15diVD30H\nHoJ3gCsCF4hrHgox66xCmkRN8cWsZ/T/n3dnvL///" ..., len
0x8047218), line 333 in "update.c"
[15] libsvn_subr-1.so.0.0.0:svn_stream_write(stream = (nil), data 0x8ec4468
"XFBLAwQK\nALy4Gy8JAAAATUVUQS1JTkYvFAAIAAgUTUFOSUZFU1QuTUaNj8sKwjAQRfeF/EN+I
EEFN93VoqCo\nm0r3QzrVYDMp6Yj69/aFiHXhes6Ze+8ByJbYsMoxNNZTLOd6JqI0IDAWavWMZVK
DuaBMiNvbUkTr\nByN1rDqCw1ga75ynRlVAZxFlNRpbWgPcESfLVYvs4AqBQaYDKvc/0Byp8OEdl
/mS7xBQbvyNih6Z\nGmPlRVd56+oKHRL//+5LGcv6cNbQW3qcpodpk4TPeBG9AFBLBwjMe5/VqgA
AAEsBAACwBAAAAG9y\nZwthcGFjaGUTY29tbW9uc7IYbGFuHWVudW1tYXRodGltInhjZXB0aW9uI
GJ1aWxkZXIsRW51bVV0\naWxzLmNsYXNzlZPbbtNAEIb/aQ6uHZMegHIotKSFNklpDQIJiVBuKhC
RXLgoRNxuHCt15diVD30H\nHoJ3gCsCF4hrHgox66xCmkRN8cWsZ/T/n3dnvL///" ..., len
0x8047258), line 107 in "stream.c"
[16] libsvn_subr-1.so.0.0.0:encode_data(baton = (nil), data = 0x8fbbec8
"PK^C^D\n", len = 0x8047268), line 136 in "svn_base64.c"
[17] libsvn_subr-1.so.0.0.0:svn_stream_write(stream = (nil), data 0x8fbbec8
"PK^C^D\n", len = 0x80472b8), line 107 in "stream.c"
[18] libsvn_delta-1.so.0.0.0:window_handler(window = 0x9191a60, baton
0x82bb0c8), line 257 in "svndiff.c"
[19] mod_dav_svn.so:window_handler(window = (nil), baton = (nil)), line
920 in "update.c"
[20] libsvn_delta-1.so.0.0.0:svn_txdelta_send_txstream(txstream 0x911f248,
handler = 0xbf8cf850 &`mod_dav_svn.so`update.c`window_handler(register
svn_txdelta_window_t
*window, register void *baton), handler_baton = 0x832a960, pool 0x8a4caf8), line
791 in "text_delta.c"
[21] libsvn_repos-1.so.0.0.0:update_entry(b = 0x84dda48, s_rev = 25281,
s_path = (nil), s_entry = (nil), t_path = 0x8a4cb50
"/intranet/MeSA2/MeSA2-tags/v_2_1_6/lib/commons-lang-2.0.jar", t_entry
0x83bd408, dir_baton = 0x90bb268, e_path = 0x8a4cb30
"lib/commons-lang-2.0.jar", info = (nil), recurse = 1, pool =
0x8a4caf8),
line 490 in "reporter.c"
[22] libsvn_repos-1.so.0.0.0:delta_dirs(b = 0x84dda48, s_rev = 25281,
s_path = (nil), t_path = 0x90bb1c0
"/intranet/MeSA2/MeSA2-tags/v_2_1_6/lib",
dir_baton = 0x90bb268, e_path = 0x90bb1b8 "lib", start_empty = 0, pool
0x90bb180), line 796 in "reporter.c"
[23] libsvn_repos-1.so.0.0.0:update_entry(b = 0x84dda48, s_rev = 25281,
s_path = (nil), s_entry = (nil), t_path = 0x90bb1c0
"/intranet/MeSA2/MeSA2-tags/v_2_1_6/lib", t_entry = 0x8cae340,
dir_baton 0x82a0518, e_path = 0x90bb1b8 "lib", info = (nil), recurse =
1, pool 0x90bb180), line 658 in "reporter.c"
[24] libsvn_repos-1.so.0.0.0:delta_dirs(b = 0x84dda48, s_rev = 25281,
s_path = 0x866b008 "/intranet/MeSA2/MeSA2-tags/v_2_1_6", t_path =
0x84ddad0
"/intranet/MeSA2/MeSA2-tags/v_2_1_6", dir_baton = 0x82a0518, e_path
0xbf8a2902 "", start_empty = 0, pool = 0x8e6e310), line 796 in
"reporter.c"
[25] libsvn_repos-1.so.0.0.0:svn_repos_finish_report(baton = (nil), pool
0x8e6e310), line 857 in "reporter.c"
[26] mod_dav_svn.so:dav_svn__update_report(resource = 0x826b4d8, doc
0x826b1a8, output = 0x8e6efd8), line 1442 in "update.c"
[27] mod_dav_svn.so:dav_svn_deliver_report(r = 0x8e6e348, resource 0x826b4d8,
doc = 0x826b1a8, output = 0x8e6efd8), line 1344 in "version.c"
[28] mod_dav.so:dav_method_report(r = (nil)), line 4042 in
"mod_dav.c"
[29] mod_dav.so:dav_handler(r = 0x8e6e348), line 4678 in "mod_dav.c"
[30] httpd:ap_run_handler(r = (nil)), line 152 in "config.c"
[31] httpd:ap_invoke_handler(r = (nil)), line 364 in "config.c"
[32] httpd:ap_process_request(r = (nil)), line 249 in
"http_request.c"
[33] httpd:ap_process_http_connection(c = (nil)), line 253 in
"http_core.c"
[34] httpd:ap_run_process_connection(c = (nil)), line 43 in
"connection.c"
[35] httpd:child_main(child_num_arg = 134511576), line 610 in
"prefork.c"
[36] httpd:make_child(s = (nil), slot = 0), line 704 in "prefork.c"
[37] httpd:ap_mpm_run(_pconf = (nil), plog = 0x8133f38, s = (nil)), line
839 in "prefork.c"
[38] httpd:main(argc = 3, argv = 0x8047d94), line 623 in "main.c"
Loadobject mappings for current core file:
0x08050000 /opt/httpd-2.0.59/bin/httpd
0xbff60000 /usr/local/stow/openssl-0.9.7e/lib/libssl.so.0.9.7
0xbfe50000 /usr/local/stow/openssl-0.9.7e/lib/libcrypto.so.0.9.7
0xbffb0000 /lib/libdl.so.1
is being filtered by: /usr/lib/ld.so.1
0xbfe10000 /opt/httpd-2.0.59/lib/libaprutil-0.so.0.9.12
0xbfdd0000 /opt/httpd-2.0.59/lib/libexpat.so.0.1.0
0xbfda0000 /opt/httpd-2.0.59/lib/libapr-0.so.0.9.12
0xbfd80000 /lib/libsendfile.so.1
0xbfd60000 /lib/librt.so.1
0xbfcf0000 /lib/libm.so.2
0xbfcd0000 /lib/libsocket.so.1
0xbfc40000 /lib/libnsl.so.1
0xbfbf0000 /lib/libresolv.so.2
0xbfbd0000 /lib/libpthread.so.1
is being filtered by: libc.so.1
0xbfaf0000 /lib/libc.so.1
0xbfad0000 /usr/sfw/lib/libgcc_s.so.1
0xbfaa0000 /lib/libaio.so.1
0xbfa80000 /lib/libmd5.so.1
0xbf630000 /lib/libscf.so.1
0xbf600000 /lib/libdoor.so.1
0xbf5e0000 /lib/libuutil.so.1
0xbf5c0000 /lib/libmp.so.2
0xbfa30000 /opt/httpd-2.0.59/modules/mod_proxy.so
0xbfa10000 /opt/httpd-2.0.59/modules/mod_proxy_connect.so
0xbf9f0000 /opt/httpd-2.0.59/modules/mod_proxy_ftp.so
0xbf9d0000 /opt/httpd-2.0.59/modules/mod_proxy_http.so
0xbf9a0000 /opt/httpd-2.0.59/modules/mod_dav.so
0xbf980000 /opt/httpd-2.0.59/modules/mod_info.so
0xbf960000 /opt/httpd-2.0.59/modules/mod_dav_fs.so
0xbf920000 /opt/httpd-2.0.59/modules/mod_vhost_alias.so
0xbf900000 /opt/httpd-2.0.59/modules/mod_rewrite.so
0xbf8c0000 /opt/httpd-2.0.59/modules/mod_dav_svn.so
0xbf890000 /usr/local/stow/subversion-1.4.0/lib/libsvn_repos-1.so.0.0.0
0xbf870000 /usr/local/stow/subversion-1.4.0/lib/libsvn_fs-1.so.0.0.0
0xbf840000 /usr/local/stow/subversion-1.4.0/lib/libsvn_delta-1.so.0.0.0
0xbf800000 /usr/local/stow/subversion-1.4.0/lib/libsvn_subr-1.so.0.0.0
0xbf7e0000 /usr/lib/libz.so.1
0xbf6e0000 /usr/local/lib/libiconv.so.2.1.0
0xbf6a0000 /usr/local/stow/subversion-1.4.0/lib/libsvn_fs_fs-1.so.0.0.0
0xbf670000 /usr/local/stow/zlib-1.2.1/lib/libz.so.1.2.1
0xbf5a0000 /opt/httpd-2.0.59/modules/mod_authz_svn.so
0xbf560000 /opt/httpd-2.0.59/modules/mod_webauth.so
0xbf540000 /usr/local/lib/libwebauth.so.1.1.2
0xbf4b0000 /opt/httpd-2.0.59/lib/libkrb5.so.3
0xbf480000 /opt/httpd-2.0.59/lib/libk5crypto.so.3
0xbf460000 /opt/httpd-2.0.59/lib/libcom_err.so.3
0xbf350000 /usr/local/stow/curl-7.11.0/lib/libcurl.so.2.0.2
0xbf330000 /lib/libgen.so.1
0xbea00000 /opt/httpd-2.0.59/modules/libphp4.so
0xbf2a0000 /usr/local/stow/mysql-4.1.9/lib/mysql/libmysqlclient.so.14.0.0
0xbf280000 /lib/libm.so.1
is being filtered by: libm.so.2
0xbf1d0000 /usr/local/stow/net-snmp-5.2.1/lib/libnetsnmp.so.5.2.1
0xbf1b0000 /lib/libkstat.so.1
0xbf150000 /lib/libadm.so.1
0xbf130000 /usr/lib/libcrypt_i.so.1
0xbf110000 /opt/httpd-2.0.59/modules/mod_auth_kerb.so
0xbf0d0000 /opt/httpd-2.0.59/lib/libgssapi_krb5.so.2
0xbf0b0000 /opt/httpd-2.0.59/modules/mod_extract_forwarded.so
0xbf060000 /opt/httpd-2.0.59/lib/nss_files.so.1
0xbf020000 /usr/lib/iconv/UTF-8%646.so
0xbf000000 /usr/lib/iconv/646%UTF-8.so
0xbffca000 /lib/ld.so.1 [LM_ID_LDSO]
-----Original Message-----
From: dtrace-discuss-bounces at opensolaris.org
[mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Matty
Sent: Thursday, August 24, 2006 12:34 PM
To: Solaris Dtrace List
Subject: RE: [dtrace-discuss] apache hangs on sol10u2 and
sol10u1(after118855-15)
On Tue, 22 Aug 2006, Fletcher Cocquyt wrote:
> Thanks for the suggestion - I tried it and it still hangs on pollsys
>> god at irt-web-xyz:httpd-2.0.59 2:03pm 129 # pstack 6092
>>
>> 6092: /opt/httpd-2.0.59/bin/httpd -k start
>> bfb906d7 pollsys (8044ad0, 1, 8044ab0, 0)
>> bfb3a722 poll (8044ad0, 1, 493e0, 493e0, 7, 40004) + 52
>> bfdba1e7 apr_poll (8044b40, 1, 8044b3c, 493e0, 0, 0) + ab
>> bfdba7f7 apr_wait_for_io_or_timeout (0, 8242718, 0, 8046d30) + 6f
>> bfdb0ec2 apr_socket_sendv (8242718, 8046d30, 2, 8044be8) + 4a
>> bfdb125a apr_sendv (8242718, 8046d30, 2, 8044be8, 8271a18, 8242718) +
>> 22 080b426e writev_it_all (4, 2050f, 8044c64, 8271a18, 0, 0) + 3e
>> 080b5832 core_output_filter (8242bc8, 8271a18, 82466f0, bfdcb9b8) +
>> 9a6
>> 0808da31 chunk_filter (8313848, 8271a18, 852cbc0, 1000) + 169
>> 080b0126 ap_content_length_filter (8265e38, 8271a18, 8046ea8,
>> bfdb8aab) +
This could be caused by a variety of things. To see what Apache is waiting
on, I would recommend building Apache in maintainer mode (add
"--enable-maintainer-mode" to the configure line to enable it) to get
debugging symbols. Once you have debugging symbols, attach to the httpd
process when it hangs, skip back to the stack frame for apr_poll, and print
out the contents of the apr_pollfd_t structure passed to apr_poll() (You
will be most interested in the contents of the apr_datatype_e and
apr_descriptor members. Check apr_poll.h for a detailed description of the
apr_pollfd_t structure.). The contents of the structure should tell you what
Apache is waiting on, and should help with determining the next steps to
take. You could also dig up similar information by digging through the
structures passed to poll(). Maybe one of the Solaris kernel folks can
provide some guidance for doing so.
Thanks,
- Ryan
--
UNIX Administrator
http://prefetch.net
_______________________________________________
dtrace-discuss mailing list
dtrace-discuss at opensolaris.org