dtrace discuss - Dec 2006 - Enabling anonymous probes to fire immediately after install, without a previous login

Hi,

We have a tricky intermittent gnome-session crash we need to debug in 
Solaris nevada b55, that we are only seeing after an initial install 
from DVD media and the first logon by root. Not on subsequent logons by 
root or any other created user.

What I''d like to do is use the app_crash.d script in the article below 
to catch the sigsegv signal from gnome-session and dump out a pstack. 
However to setup anonymous tracing I seem to need to login first, enable 
it for a given script and then reboot. This will be too late unfortunately.

Would it be possible to just replace the config and system files created 
by running "dtrace -A app_crash.d" in the install image, to have 
anonymous tracing for these probes enabled on install and first login by 
root? Has anyone done this before?
# dtrace -AFs app_crash.d
dtrace: saved anonymous enabling in /kernel/drv/dtrace.conf
dtrace: added forceload directives to /etc/system
dtrace: run update_drv(1M) or reboot to enable changes
#

Here is the article describing the app_crash.d script:
http://developers.sun.com/solaris/articles/app_crash/app_crash.html#DTsolution 

DTrace script: 
http://developers.sun.com/solaris/articles/app_crash/app_crash.d.txt
User Script: 
http://developers.sun.com/solaris/articles/app_crash/runme_on_app_crash.txt

JR

John Rice stated:
< Hi,
< 
< We have a tricky intermittent gnome-session crash we need to debug in 
< Solaris nevada b55, that we are only seeing after an initial install 
< from DVD media and the first logon by root. Not on subsequent logons by 
< root or any other created user.
< 
< What I''d like to do is use the app_crash.d script in the article
below
< to catch the sigsegv signal from gnome-session and dump out a pstack. 
< However to setup anonymous tracing I seem to need to login first, enable 
< it for a given script and then reboot. This will be too late unfortunately.
< 
< Would it be possible to just replace the config and system files created 
< by running "dtrace -A app_crash.d" in the install image, to have 
< anonymous tracing for these probes enabled on install and first login by 
< root? Has anyone done this before?

  You can''t use dtrace directly from an install image as it
isn''t there :)
  Theres an old thread in either the internal dtrace archives or the 
  opensolaris one where there''s talk of how to get it in though, adding
  the dtrace packages to the install boot image.  Though not necessarily
  needed here.

  You should be able to use take an already generated dtrace.conf and
  /etc/system settings to get what you want.  You''ll just need to copy
  them down the /a root dir on the install partition before you reboot
  after install.

< # dtrace -AFs app_crash.d

  dtrace -o d.out -AFs app_crash.d

  The d.out would then  be appended to the /a/kernel/drv/dtrace.conf
  Similarly for the /etc/system settings.

  How you''d do this from install off a DVD ?  Perhaps have the d.out
and
  the generated /etc/system files on a perserved partition or before reboot
  plumb a nic etc (unless already plumbed :) and get it from another machine.

< dtrace: saved anonymous enabling in /kernel/drv/dtrace.conf
< dtrace: added forceload directives to /etc/system
< dtrace: run update_drv(1M) or reboot to enable changes
< #
< 
< Here is the article describing the app_crash.d script:
<
http://developers.sun.com/solaris/articles/app_crash/app_crash.html#DTsolution
< 
< DTrace script: 
< http://developers.sun.com/solaris/articles/app_crash/app_crash.d.txt
< User Script: 
< http://developers.sun.com/solaris/articles/app_crash/runme_on_app_crash.txt

 Interesting script to try and run just after an install :)

 You''ve tried this the usual way already and it works ?  i.e. not from
 an install, but the usual ''dtrace -AFs app_crash.d'' and
reboot ?

< 
< JR
< 
< _______________________________________________
< dtrace-discuss mailing list
< dtrace-discuss at opensolaris.org


Regards,
-- 
Sean.
.

Thanks Sean for all the suggestions.

Darren and I got it sorted and got the stack traces :)

Turns out to be a race condition between the fc-cache setting up the 
font cash and any Gnome app accessing the fonts before the cache was 
ready. The fix is just to introduce a dependency on the fc-cache in 
DTlogin''s SMF manifest file.

We did the following:
Created a simplified appcrash.d script [below]
Ran on my own nevada b54 machine:
dtrace -AFs appcrash.d
saved:
/kernel/drv/dtrace.conf
/etc/system

Got the test system to a point where we had just finished install.
Log''ed into Failsafe mode - mounted local file system to /a
Mounted usb stick with the above files on it:
Copied over the files to the mounted /a drive
/a/kernel/drv/dtrace.conf
/a/etc/system

Modified the Gnome settings to stop it intercepting the segv on reboot:
Edit /a/usr/dt/config/Xinitrc.jds and add:
export GNOME_DISABLE_CRASH_DIALOG=1

Reboot.

Gnome-session crashed and the logon hangs. Just killed the X Server to 
get back to the login prompt, logged in and got the dtrace anonymous 
ouput. Odd thing here is that the system("prun ... didn''t work for
us to
cleanly close down gnome-session. It was stopped when we logged in. No 
big deal as we still got the stack data - is this a difference when 
running anonymous scripts I wonder? Didn''t see this effect when running
the scripts on my own machine.

dtrace -ae -o dtrace.log

Happy days as they say.

JR


appcrash.d
-------------
#!/usr/sbin/dtrace -qws
#pragma D option bufpolicy=ring
#pragma D option bufsize=4m

proc:::signal-send
/(args[2] == SIGBUS || args[2] == SIGSEGV) &&
pid == args[1]->pr_pid/
{
@[execname, pid, ustack(100,1000)] = count();

stop();
printf("\n=================\n");
printa(@);
trunc(@);
printf("=================\n");
system("prun %d", pid); <--- Didn''t seem to be fired ??
}

Log:
-------
0 | sigtoproc:signal-send
================
gnome-session 3215
libfontconfig.so.1`FcPatternPosition+0x6b
libfontconfig.so.1`FcPatternFindElt+0x2e
libfontconfig.so.1`FcPatternGet+0x2e
libfontconfig.so.1`FcFontSetSort+0x498
libfontconfig.so.1`FcFontSort+0xaa
libpangoft2-1.0.so.0.1400.7`pango_fc_font_map_get_patterns+0x1ba
libpangoft2-1.0.so.0.1400.7`pango_fc_font_map_load_fontset+0x36


Sean McGrath - Sun Microsystems Ireland wrote:
> John Rice stated:
> < Hi,
> < 
> < We have a tricky intermittent gnome-session crash we need to debug in 
> < Solaris nevada b55, that we are only seeing after an initial install 
> < from DVD media and the first logon by root. Not on subsequent logons
by
> < root or any other created user.
> < 
> < What I''d like to do is use the app_crash.d script in the
article below
> < to catch the sigsegv signal from gnome-session and dump out a pstack. 
> < However to setup anonymous tracing I seem to need to login first,
enable
> < it for a given script and then reboot. This will be too late
unfortunately.
> < 
> < Would it be possible to just replace the config and system files
created
> < by running "dtrace -A app_crash.d" in the install image, to
have
> < anonymous tracing for these probes enabled on install and first login
by
> < root? Has anyone done this before?
>
>   You can''t use dtrace directly from an install image as it
isn''t there :)
>   Theres an old thread in either the internal dtrace archives or the 
>   opensolaris one where there''s talk of how to get it in though,
adding
>   the dtrace packages to the install boot image.  Though not necessarily
>   needed here.
>
>   You should be able to use take an already generated dtrace.conf and
>   /etc/system settings to get what you want.  You''ll just need to
copy
>   them down the /a root dir on the install partition before you reboot
>   after install.
>
> < # dtrace -AFs app_crash.d
>
>   dtrace -o d.out -AFs app_crash.d
>
>   The d.out would then  be appended to the /a/kernel/drv/dtrace.conf
>   Similarly for the /etc/system settings.
>
>   How you''d do this from install off a DVD ?  Perhaps have the
d.out and
>   the generated /etc/system files on a perserved partition or before reboot
>   plumb a nic etc (unless already plumbed :) and get it from another
machine.
>
> < dtrace: saved anonymous enabling in /kernel/drv/dtrace.conf
> < dtrace: added forceload directives to /etc/system
> < dtrace: run update_drv(1M) or reboot to enable changes
> < #
> < 
> < Here is the article describing the app_crash.d script:
> <
http://developers.sun.com/solaris/articles/app_crash/app_crash.html#DTsolution
> < 
> < DTrace script: 
> < http://developers.sun.com/solaris/articles/app_crash/app_crash.d.txt
> < User Script: 
> <
http://developers.sun.com/solaris/articles/app_crash/runme_on_app_crash.txt
>
>  Interesting script to try and run just after an install :)
>
>  You''ve tried this the usual way already and it works ?  i.e. not
from
>  an install, but the usual ''dtrace -AFs app_crash.d'' and
reboot ?
>
> < 
> < JR
> < 
> < _______________________________________________
> < dtrace-discuss mailing list
> < dtrace-discuss at opensolaris.org
>
>
> Regards,
>