[dtrace-discuss] dtrace wired issue

Adam Leventhal
Mon, 14 Jan 2008 16:55:15 -0000

On Mon, Jan 14, 2008 at 06:46:49AM -0800, Bryan Cantrill
wrote:
> > So, I failed to understand the modules loading needs 4 minutes.
>
> Yes, this is definitely fishy.  Is this a highly memory constrained system?
> If you "modunload -i 0" enough times to get dtrace(7D) unloaded
(that
> is, "dtrace" doesn''t appear in modinfo), does it again
take 4 minutes?
> As you can imagine, it''s a little tough to investigate this
problem because
> we can''t use DTrace to do it! ;)

Kicking the dtrace kernel module out is a bit trickier than back in the day
due to this bug:

  6282201 USDT probes can prevent dtrace(7D) from being unloaded

In order to unload dtrace, you''ll need to make sure there are no
processes
with USDT probes. This is done as part of one of the DTrace test suite
tests:

--8<-- usr/src/cmd/dtrace/test/tst/common/predicates/tst.predcache.ksh
--8<--
28 unload()
29 {
30 	#
31 	# Get the list of services whose processes have USDT probes.  Ideally
32 	# it would be possible to unload the fasttrap provider while USDT
33 	# probes exist -- once that fix is integrated, this hack can go away
34 	# We create two lists -- one of regular SMF services and one of legacy
35 	# services -- since each must be enabled and disabled using a specific
36 	# mechanism.
37 	#
38 	pids=$(dtrace -l | \
39 	    perl -ne ''print "$1\n" if
(/^\s*\S+\s+\S*\D(\d+)\s+/);'' | \
40 	    sort | uniq | tr ''\n'' '','')
41
42 	ctids=$(ps -p $pids -o ctid | tail +2 | sort | uniq)
43 	svcs44 	lrcs45
46 	for ct in $ctids
47 	do
48 		line=$(svcs -o fmri,ctid | grep " $ct\$")
49 		svc=$(echo $line | cut -d'' '' -f1)
50
51 		if [[ $(svcs -Ho STA $svc) == "LRC" ]]; then
52 			lrc=$(svcs -Ho SVC $svc | tr _ ''?'')
53 			lrcs="$lrcs $lrc"
54 		else
55 			svcs="$svcs $svc"
56 	fi
57 	done
58
59 	for svc in $svcs
60 	do
61 		svcadm disable -ts $svc
62 	done
63
64 	for lrc in $lrcs
65 	do
66 		#
67 		# Does it seem a little paternalistic that lsvcrun requires
68 		# this environment variable to be set? I''d say so...
69 		#
70 		SMF_RESTARTER=svc:/system/svc/restarter:default \
71 		    /lib/svc/bin/lsvcrun $lrc stop
72 	done
73
74 	modunload -i 0
75 	modunload -i 0
76 	modunload -i 0
77 	modinfo | grep dtrace
78 	success=$?
79
80 	for svc in $svcs
81 	do
82 		svcadm enable -ts $svc
83 	done
84
85 	for lrc in $lrcs
86 	do
87 		SMF_RESTARTER=svc:/system/svc/restarter:default \
88 		    /lib/svc/bin/lsvcrun $lrc start
89 	done
90
91 	if [ ! $success ]; then
92 		echo $tst: could not unload dtrace
93 		exit 1
94 	fi
95 }
--8<-- usr/src/cmd/dtrace/test/tst/common/predicates/tst.predcache.ksh
--8<--

Adam

--
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl