Alfred Peng
2008-Sep-19 02:57 UTC
[dtrace-discuss] How to analyze the core stack like this?
Hi guys, Here is a core stack from running "Songbird --version" on JDS Nevada b99: bash-3.2$ songbird --version POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. Segmentation Fault (core dumped) bash-3.2$ pstack core core ''core'' of 2911: songbird --version fecb617b findenv (808f324, fef7aa18, 1, 8046fb4) + 54 fecb6623 getenv (fef7aa18) + 31 fef78c17 dprintf (1, fef7a864, 0) + 27 fef78f54 dtrace_dof_fini (feffb7dc, fefb0620, fef79074, 8047044, fefd2cc5, feffde88) + 58 fef790b6 _fini (feffde88, feffb178, feffb7dc, fedb7e40, 0, fc3b0028) + 42 fefd2cc5 call_fini (feffb178, fc3b0018) + f5 fefd2e57 atexit_fini (80471ac, 8047084, fedb2000, fefc7064, feffde88, 8) + 63 fecb0094 _exithandle (feffb7dc, 80566c3, 0, 8056630, 805d6fc, fefd2df4) + 53 feca28e2 exit (2, 8047214, 0, 0, 8047227, 804724d) + 12 Songbird is a desktop media application based on Mozilla''s XULRunner platform. As BrendanG''s JavaScript DTrace has been integrated in Mozilla''s code, the DTrace feature is also enabled in Songbird. Well, Songbird''s GUI works well, but the CLI has this core. Any help for this is appreciated. Thanks, -Alfred
Mike Gerdts
2008-Sep-19 04:41 UTC
[dtrace-discuss] How to analyze the core stack like this?
On Thu, Sep 18, 2008 at 9:57 PM, Alfred Peng <Alfred.Peng at sun.com> wrote:> Hi guys, > > Here is a core stack from running "Songbird --version" on JDS Nevada b99: > > bash-3.2$ songbird --version > POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. > Segmentation Fault (core dumped) > bash-3.2$ pstack core > core ''core'' of 2911: songbird --version > fecb617b findenv (808f324, fef7aa18, 1, 8046fb4) + 54 > fecb6623 getenv (fef7aa18) + 31 > fef78c17 dprintf (1, fef7a864, 0) + 27 > fef78f54 dtrace_dof_fini (feffb7dc, fefb0620, fef79074, 8047044,For this to happen, it looks as though songbird must have whacked the environment - perhaps unintentionally. http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/drti.c#66 It would seem to make sense for drti.c to read all of the environment variables during dof_init() and store them in static global variables. It doesn''t change the fact that the app is quite likely buggy but it does keep the blame from being put in the wrong place. -- Mike Gerdts http://mgerdts.blogspot.com/
Alfred Peng
2008-Sep-19 05:22 UTC
[dtrace-discuss] How to analyze the core stack like this?
Mike Gerdts wrote:> On Thu, Sep 18, 2008 at 9:57 PM, Alfred Peng <Alfred.Peng at sun.com> wrote: > >> Hi guys, >> >> Here is a core stack from running "Songbird --version" on JDS Nevada b99: >> >> bash-3.2$ songbird --version >> POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. >> Segmentation Fault (core dumped) >> bash-3.2$ pstack core >> core ''core'' of 2911: songbird --version >> fecb617b findenv (808f324, fef7aa18, 1, 8046fb4) + 54 >> fecb6623 getenv (fef7aa18) + 31 >> fef78c17 dprintf (1, fef7a864, 0) + 27 >> fef78f54 dtrace_dof_fini (feffb7dc, fefb0620, fef79074, 8047044, >> > > For this to happen, it looks as though songbird must have whacked the > environment - perhaps unintentionally. > > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/drti.c#66 > > It would seem to make sense for drti.c to read all of the environment > variables during dof_init() and store them in static global variables. > It doesn''t change the fact that the app is quite likely buggy but it > does keep the blame from being put in the wrong place.Hi Mike, Thanks for the reply. Just tried to set DTRACE_DOF_INIT_DEBUG and rerun the command: bash-3.2$ songbird --version dtrace DOF libCrun.so.1: DTrace ioctl succeeded for DOF at fef7aa68 dtrace DOF libmozjs.so: DTrace ioctl succeeded for DOF at fec0eb30 POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. dtrace DOF libmozjs.so: DTrace ioctl removed DOF (0) Segmentation Fault (core dumped) However, it works fine for Firefox on the same box(different XULRunner) bash-3.2$ firefox --version dtrace DOF libCrun.so.1: DTrace ioctl succeeded for DOF at fef7aa68 dtrace DOF libmozjs.so: DTrace ioctl succeeded for DOF at f8a7aad0 Mozilla Firefox 3.0.1, Copyright (c) 1998 - 2008 mozilla.org dtrace DOF libmozjs.so: DTrace ioctl removed DOF (0) dtrace DOF libCrun.so.1: DTrace ioctl removed DOF (0) -Alfred
Mike Gerdts
2008-Sep-19 14:22 UTC
[dtrace-discuss] How to analyze the core stack like this?
On Fri, Sep 19, 2008 at 12:22 AM, Alfred Peng <Alfred.Peng at sun.com> wrote:> Mike Gerdts wrote: >> On Thu, Sep 18, 2008 at 9:57 PM, Alfred Peng <Alfred.Peng at sun.com> wrote: >>> Hi guys, >>> >>> Here is a core stack from running "Songbird --version" on JDS Nevada b99: >>> >>> bash-3.2$ songbird --version >>> POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. >>> Segmentation Fault (core dumped) >>> bash-3.2$ pstack core >>> core ''core'' of 2911: songbird --version >>> fecb617b findenv (808f324, fef7aa18, 1, 8046fb4) + 54 >>> fecb6623 getenv (fef7aa18) + 31 >>> fef78c17 dprintf (1, fef7a864, 0) + 27 >>> fef78f54 dtrace_dof_fini (feffb7dc, fefb0620, fef79074, 8047044, >>> >> >> For this to happen, it looks as though songbird must have whacked the >> environment - perhaps unintentionally. >> >> >> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/drti.c#66 >> >> It would seem to make sense for drti.c to read all of the environment >> variables during dof_init() and store them in static global variables. >> It doesn''t change the fact that the app is quite likely buggy but it >> does keep the blame from being put in the wrong place. > > Hi Mike, > > Thanks for the reply. > > Just tried to set DTRACE_DOF_INIT_DEBUG and rerun the command: > > bash-3.2$ songbird --version > dtrace DOF libCrun.so.1: DTrace ioctl succeeded for DOF at fef7aa68 > dtrace DOF libmozjs.so: DTrace ioctl succeeded for DOF at fec0eb30 > POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. > dtrace DOF libmozjs.so: DTrace ioctl removed DOF (0) > Segmentation Fault (core dumped)What does "pargs -ae core" say? It would probably be easiest to read if you can reproduce with the following: $ env -i PATH=$PATH songbird --version -- Mike Gerdts http://mgerdts.blogspot.com/
Alfred Peng
2008-Sep-19 14:44 UTC
[dtrace-discuss] How to analyze the core stack like this?
Mike Gerdts wrote:> On Fri, Sep 19, 2008 at 12:22 AM, Alfred Peng <Alfred.Peng at sun.com> wrote: > >> Mike Gerdts wrote: >> >>> On Thu, Sep 18, 2008 at 9:57 PM, Alfred Peng <Alfred.Peng at sun.com> wrote: >>> >>>> Hi guys, >>>> >>>> Here is a core stack from running "Songbird --version" on JDS Nevada b99: >>>> >>>> bash-3.2$ songbird --version >>>> POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. >>>> Segmentation Fault (core dumped) >>>> bash-3.2$ pstack core >>>> core ''core'' of 2911: songbird --version >>>> fecb617b findenv (808f324, fef7aa18, 1, 8046fb4) + 54 >>>> fecb6623 getenv (fef7aa18) + 31 >>>> fef78c17 dprintf (1, fef7a864, 0) + 27 >>>> fef78f54 dtrace_dof_fini (feffb7dc, fefb0620, fef79074, 8047044, >>>> >>>> >>> For this to happen, it looks as though songbird must have whacked the >>> environment - perhaps unintentionally. >>> >>> >>> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/drti.c#66 >>> >>> It would seem to make sense for drti.c to read all of the environment >>> variables during dof_init() and store them in static global variables. >>> It doesn''t change the fact that the app is quite likely buggy but it >>> does keep the blame from being put in the wrong place. >>> >> Hi Mike, >> >> Thanks for the reply. >> >> Just tried to set DTRACE_DOF_INIT_DEBUG and rerun the command: >> >> bash-3.2$ songbird --version >> dtrace DOF libCrun.so.1: DTrace ioctl succeeded for DOF at fef7aa68 >> dtrace DOF libmozjs.so: DTrace ioctl succeeded for DOF at fec0eb30 >> POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. >> dtrace DOF libmozjs.so: DTrace ioctl removed DOF (0) >> Segmentation Fault (core dumped) >> > > What does "pargs -ae core" say? It would probably be easiest to read > if you can reproduce with the following: > > $ env -i PATH=$PATH songbird --versionbash-3.2$ env -i PATH=$PATH songbird --version POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. Segmentation Fault (core dumped) bash-3.2$ pargs -ae core core ''core'' of 3606: songbird --version argv[0]: songbird argv[1]: <NULL> envp[0]: <0xfe38aea9> envp[1]: PATH=/usr/bin:/usr/openwin/bin:/usr/ucb Thanks, -Alfred
Mike Gerdts
2008-Sep-20 02:31 UTC
[dtrace-discuss] How to analyze the core stack like this?
On Fri, Sep 19, 2008 at 9:44 AM, Alfred Peng <Alfred.Peng at sun.com> wrote:> bash-3.2$ env -i PATH=$PATH songbird --version > POTI, Inc. Songbird 0.7.0, Copyright(c) 2005-2008 POTI, Inc. > Segmentation Fault (core dumped) > bash-3.2$ pargs -ae core > core ''core'' of 3606: songbird --version > argv[0]: songbird > argv[1]: <NULL> > > envp[0]: <0xfe38aea9> > envp[1]: PATH=/usr/bin:/usr/openwin/bin:/usr/ucbAs I suspected, kinda. I have never seen pargs print an address - presumably to an address that is outside of its address space. ("pmap -x core" may help confirm) I think that there is a bug in Songbird that is being exposed by the dtrace probes. It would probably be most fruitful to take this up the Songbird developers. I still think that it would be a good idea for dtri.o to not read environment variables after the application has had a chance to mess with the environment. Even if this is fixed, existing applications would need to be relinked with the new drti.o to see the fix. I have filed a bug report - perhaps someone at Sun can post the bug ID to the list as I will not see it until someone at Sun starts to update it. Here''s the reproducible test case I included in the bug report (special thanks to Alan, http://blogs.sun.com/tpenta/entry/dtrace_using_placing_sdt_probes). === hello.c ==#include <stdio.h> #include <unistd.h> #include <sys/sdt.h> int main(int argc, char **argv, char **envp) { DTRACE_PROBE(world, loop); if ( argc != 1 ) { envp[0] = 0xff; } printf("Hello World\n"); } =============== === myserv.d ==provider world { probe loop(); }; #pragma D attributes Evolving/Evolving/Common provider world provider #pragma D attributes Private/Private/Common provider world module #pragma D attributes Private/Private/Common provider world function #pragma D attributes Evolving/Evolving/Common provider world name #pragma D attributes Evolving/Evolving/Common provider world args =============== $ cc -c hello.c $ dtrace -32 -G -s myserv.d hello.o $ cc -o hello -ldtrace myserv.o hello.o $ ./hello $ ./hello a Hello World Segmentation Fault (core dumped) $ pstack core core ''core'' of 2025: ./hello a d1a0608b findenv (8047810, 80515e4, 1, 8047764) + 54 d1a06533 getenv (80515e4) + 31 08050fbf dprintf (1, 8051430, 0) + 27 080512fc dtrace_dof_fini (d1b01000, 80477d8, d19fffa4, 80478d0, 80477c0, d1b01000) + 58 08051406 _fini (80478d0, 80477c0, d1b01000, d1bfee58, 10, 8050e46) + 26 d19fffa4 _exithandle (d1bfc7dc, 8050e46, c, c, 8050dde, 80513e0) + 53 d19f2802 exit (2, 8047938, 8047940, 0, ff, 8047a1c) + 12 $ env -i FOO=foo BAR=bar ./hello a Hello World Segmentation Fault (core dumped) $ pstack core core ''core'' of 2056: ./hello a d1a0608b findenv (8047f60, 80515e4, 1, 8047eb4) + 54 d1a06533 getenv (80515e4) + 31 08050fbf dprintf (1, 8051430, 0) + 27 080512fc dtrace_dof_fini (d1b01000, 8047f28, d19fffa4, 8047f6c, 8047f10, d1b01000) + 58 08051406 _fini (8047f6c, 8047f10, d1b01000, d1bfee58, 10, 8050e46) + 26 d19fffa4 _exithandle (d1bfc7dc, 8050e46, c, c, 8050dde, 80513e0) + 53 d19f2802 exit (2, 8047fd4, 8047fdc, 0, ff, 8047fe6) + 12 $ pargs -e core core ''core'' of 2056: ./hello a envp[0]: <0x000000ff> envp[1]: FOO=foo -- Mike Gerdts http://mgerdts.blogspot.com/
Mike Gerdts
2008-Sep-20 13:57 UTC
[dtrace-discuss] How to analyze the core stack like this?
On Fri, Sep 19, 2008 at 9:31 PM, Mike Gerdts <mgerdts at gmail.com> wrote:> I still think that it would be a good idea for dtri.o to not read > environment variables after the application has had a chance to mess > with the environment. Even if this is fixed, existing applications > would need to be relinked with the new drti.o to see the fix. I have > filed a bug report - perhaps someone at Sun can post the bug ID to the > list as I will not see it until someone at Sun starts to update it.I have developed and tested a fix (attached). If someone can get me the CR ID, I will file drop some mail to request-sponsor to get it integrated. -- Mike Gerdts http://mgerdts.blogspot.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: drti-getenv.patch Type: application/octet-stream Size: 1199 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080920/3edfd7de/attachment.obj>
Chad Lewis
2008-Sep-20 16:11 UTC
[dtrace-discuss] How to analyze the core stack like this?
Bug ID: 6750659 Synopsis: drti.o crashes app due to corrupt environment On Sep 20, 2008, at 6:57 AM, Mike Gerdts wrote:> On Fri, Sep 19, 2008 at 9:31 PM, Mike Gerdts <mgerdts at gmail.com> > wrote: >> I still think that it would be a good idea for dtri.o to not read >> environment variables after the application has had a chance to mess >> with the environment. Even if this is fixed, existing applications >> would need to be relinked with the new drti.o to see the fix. I have >> filed a bug report - perhaps someone at Sun can post the bug ID to >> the >> list as I will not see it until someone at Sun starts to update it. > > I have developed and tested a fix (attached). If someone can get me > the CR ID, I will file drop some mail to request-sponsor to get it > integrated. > > -- > Mike Gerdts > http://mgerdts.blogspot.com/<drti- > getenv.patch>_______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
Mike Gerdts
2008-Nov-09 01:04 UTC
[dtrace-discuss] How to analyze the core stack like this?
On Sat, Sep 20, 2008 at 7:57 AM, Mike Gerdts <mgerdts at gmail.com> wrote:> On Fri, Sep 19, 2008 at 9:31 PM, Mike Gerdts <mgerdts at gmail.com> wrote: >> I still think that it would be a good idea for dtri.o to not read >> environment variables after the application has had a chance to mess >> with the environment. Even if this is fixed, existing applications >> would need to be relinked with the new drti.o to see the fix. I have >> filed a bug report - perhaps someone at Sun can post the bug ID to the >> list as I will not see it until someone at Sun starts to update it. > > I have developed and tested a fix (attached). If someone can get me > the CR ID, I will file drop some mail to request-sponsor to get it > integrated.A short while after I sent the above message, I lost the zpool that had my build environment in it. Hint: don''t use VirtualBox''s SAS driver. As Chad Lewis replied, a bug ID was assigned: http://bugs.opensolaris.org/view_bug.do?bug_id=6750659 Anyway, I noticed that this was bumped to Priority 2 (High) a couple days ago. Since I had the fix ready to go and I had been wanting to play with the OpenSolaris build farm[1], I figured it would be a good thing for me to go after. I have sent mail to request-sponsor[2] and am awaiting a reply. 1.http://test.opensolaris.org/testfarm/index.jsp 2.http://mail.opensolaris.org/pipermail/request-sponsor/2008-November/003989.html I''ve done a build of onnv_102 (in just 72 minutes!) and tested via a bfu from sxce 101. As soon as I can get past some problems uploading to cr.opensolaris.org[3], I''ll get a webrev up. 3.http://opensolaris.org/jive/thread.jspa?messageID=304165&tstart=0 -- Mike Gerdts http://mgerdts.blogspot.com/
Jon Haslam
2008-Nov-12 00:27 UTC
[dtrace-discuss] How to analyze the core stack like this?
Hi Mike, If nobody has picked this up, I''ll sponsor you. Contact me offline and we''ll take it from there. Jon.> On Sat, Sep 20, 2008 at 7:57 AM, Mike Gerdts <mgerdts at gmail.com> wrote: > >> On Fri, Sep 19, 2008 at 9:31 PM, Mike Gerdts <mgerdts at gmail.com> wrote: >> >>> I still think that it would be a good idea for dtri.o to not read >>> environment variables after the application has had a chance to mess >>> with the environment. Even if this is fixed, existing applications >>> would need to be relinked with the new drti.o to see the fix. I have >>> filed a bug report - perhaps someone at Sun can post the bug ID to the >>> list as I will not see it until someone at Sun starts to update it. >>> >> I have developed and tested a fix (attached). If someone can get me >> the CR ID, I will file drop some mail to request-sponsor to get it >> integrated. >> > > A short while after I sent the above message, I lost the zpool that > had my build environment in it. Hint: don''t use VirtualBox''s SAS > driver. > > As Chad Lewis replied, a bug ID was assigned: > > http://bugs.opensolaris.org/view_bug.do?bug_id=6750659 > > Anyway, I noticed that this was bumped to Priority 2 (High) a couple > days ago. Since I had the fix ready to go and I had been wanting to > play with the OpenSolaris build farm[1], I figured it would be a good > thing for me to go after. I have sent mail to request-sponsor[2] and > am awaiting a reply. > > 1.http://test.opensolaris.org/testfarm/index.jsp > 2.http://mail.opensolaris.org/pipermail/request-sponsor/2008-November/003989.html > > I''ve done a build of onnv_102 (in just 72 minutes!) and tested via a > bfu from sxce 101. As soon as I can get past some problems uploading > to cr.opensolaris.org[3], I''ll get a webrev up. > > 3.http://opensolaris.org/jive/thread.jspa?messageID=304165&tstart=0 > >