All,
from reading of the DTrace documentation, eg. at
http://wikis.sun.com/display/DTrace/pid+Provider, I got the impression that
I could do something like
dtrace -n ''pid$target:<mylib>::entry{}'' -c ...
for any existing mylib.so{.1} ... which does not seem to be the case; for
the library I''m creating as well as libdladm, which I tried for sanity
check, I get a message like
dtrace: invalid probe specifier pid$target:libdladm::entry{}: probe
description pid102461:libdladm::entry does not match any probes
(again, for sanity check, I used the probe specifier
"pid$target:libc:str*:entry", which *did* work)
If I''m not doing something wrong that''s just too obvious to
notice myself
;-), I suspect I need to add some spice to the library''s makefile
(it''s on
the ON tree under usr/src/lib) - can someone point me either to a good (ie
simple enough for a bear of little brain) example (I don''t think libc
is
one...) or to documentation on this?
(in "Dtrace Tips, Tricks and Gotchas"
[http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf] pg. 42 I read:
Gotcha: Unsporting libraries
[...]
Library writers: don''t do this!
? gcc: Don''t use -fomit-frame-pointer!
? Sun compilers: avoid -xO4; it does this by
default!
which, AFAICT, I''m not doing [I tweaked the options for development])
TIA
Michael
--
Michael Schuster http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''
Pavan Chandrashekar - Sun Microsystems
2009-Apr-01 17:05 UTC
[dtrace-discuss] DTracing one''s own libraries
On 04/01/09 22:11, Michael Schuster wrote:> All, > > from reading of the DTrace documentation, eg. at > http://wikis.sun.com/display/DTrace/pid+Provider, I got the impression > that I could do something like > > dtrace -n ''pid$target:<mylib>::entry{}'' -c ... > > for any existing mylib.so{.1} ... which does not seem to be the case; > for the library I''m creating as well as libdladm, which I tried for > sanity check, I get a message like > > dtrace: invalid probe specifier pid$target:libdladm::entry{}: probe > description pid102461:libdladm::entry does not match any probes >Add -Z to your dtrace command line. There is a good possibility that the library that you were trying to trace had not been loaded. So, dtrace -Z -n ''pid$target:<mylib>::entry{}'' -c ... should work. Pavan> (again, for sanity check, I used the probe specifier > "pid$target:libc:str*:entry", which *did* work) > > If I''m not doing something wrong that''s just too obvious to notice > myself ;-), I suspect I need to add some spice to the library''s makefile > (it''s on the ON tree under usr/src/lib) - can someone point me either to > a good (ie simple enough for a bear of little brain) example (I don''t > think libc is one...) or to documentation on this? > > (in "Dtrace Tips, Tricks and Gotchas" > [http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf] pg. 42 I read: > Gotcha: Unsporting libraries > [...] > Library writers: don''t do this! > ? gcc: Don''t use -fomit-frame-pointer! > ? Sun compilers: avoid -xO4; it does this by > default! > which, AFAICT, I''m not doing [I tweaked the options for development]) > > TIA > Michael
On Wed, Apr 1, 2009 at 12:41 PM, Michael Schuster <Michael.Schuster at sun.com> wrote:> All, > > from reading of the DTrace documentation, eg. at > http://wikis.sun.com/display/DTrace/pid+Provider, I got the impression that > I could do something like > > ? ? ? ?dtrace -n ''pid$target:<mylib>::entry{}'' -c ... > > for any existing mylib.so{.1} ... which does not seem to be the case; for > the library I''m creating as well as libdladm, which I tried for sanity > check, I get a message like > > ? ? ? ?dtrace: invalid probe specifier pid$target:libdladm::entry{}: probe > description pid102461:libdladm::entry does not match any probes > > (again, for sanity check, I used the probe specifier > "pid$target:libc:str*:entry", which *did* work)My first thought was that the library had been stripped, but that''s not the case: # file /lib/libdladm.so.1 /lib/libdladm.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1 [FPU], dynamically linked, not stripped, no debugging information available # And I can see these probes on my system: # fuser /lib/libdladm.so.1 /lib/libdladm.so.1: 101318m 100998m 100993m 100522m 100077m 100075m 100016m # dtrace -ln ''pid$target:libdladm::entry'' -p 100016 | head ID PROVIDER MODULE FUNCTION NAME 82336 pid100016 libdladm.so.1 stat entry 82337 pid100016 libdladm.so.1 lstat entry 82338 pid100016 libdladm.so.1 fstat entry 82339 pid100016 libdladm.so.1 mknod entry 82340 pid100016 libdladm.so.1 dladm_open entry 82341 pid100016 libdladm.so.1 dladm_close entry 82342 pid100016 libdladm.so.1 dladm_dld_fd entry 82343 pid100016 libdladm.so.1 dladm_door_fd entry 82344 pid100016 libdladm.so.1 dladm_status2str entry # I assume ldd shows the application linked against your library and libdladm. Is the application explicitly loading the libraries? In that case the probe points won''t exist when dtrace tries to instrument them. Chad
On 04/01/09 10:06, Chad Mynhier wrote:> On Wed, Apr 1, 2009 at 12:41 PM, Michael Schuster > <Michael.Schuster at sun.com> wrote: >> All, >> >> from reading of the DTrace documentation, eg. at >> http://wikis.sun.com/display/DTrace/pid+Provider, I got the impression that >> I could do something like >> >> dtrace -n ''pid$target:<mylib>::entry{}'' -c ... >> >> for any existing mylib.so{.1} ... which does not seem to be the case; for >> the library I''m creating as well as libdladm, which I tried for sanity >> check, I get a message like >> >> dtrace: invalid probe specifier pid$target:libdladm::entry{}: probe >> description pid102461:libdladm::entry does not match any probes >> >> (again, for sanity check, I used the probe specifier >> "pid$target:libc:str*:entry", which *did* work) > > My first thought was that the library had been stripped, but that''s > not the case: > > # file /lib/libdladm.so.1 > /lib/libdladm.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1 > [FPU], dynamically linked, not stripped, no debugging information > available > # > > And I can see these probes on my system: > > # fuser /lib/libdladm.so.1 > /lib/libdladm.so.1: 101318m 100998m 100993m 100522m 100077m > 100075m 100016m > # dtrace -ln ''pid$target:libdladm::entry'' -p 100016 | head > ID PROVIDER MODULE FUNCTION NAME > 82336 pid100016 libdladm.so.1 stat entry > 82337 pid100016 libdladm.so.1 lstat entry > 82338 pid100016 libdladm.so.1 fstat entry > 82339 pid100016 libdladm.so.1 mknod entry > 82340 pid100016 libdladm.so.1 dladm_open entry > 82341 pid100016 libdladm.so.1 dladm_close entry > 82342 pid100016 libdladm.so.1 dladm_dld_fd entry > 82343 pid100016 libdladm.so.1 dladm_door_fd entry > 82344 pid100016 libdladm.so.1 dladm_status2str entry > #I see similar stuff for my library ... so why would dtrace fail to show calls into libdladm for a process that''s started by dtrace (ie using -c) (using -Z as someone suggested) $ dtrace -Z -n ''pid$target:libdladm::entry{}'' -c "dladm show-link" dtrace: description ''pid$target:libdladm::entry'' matched 0 probes LINK CLASS MTU STATE OVER iprb1 phys 1500 unknown -- ce2 phys 1500 down -- bge0 phys 1500 up -- bge1 phys 1500 up -- iprb0 phys 1500 unknown -- bge2 phys 1500 up -- bge3 phys 1500 up -- dtrace: pid 102624 has exited> I assume ldd shows the application linked against your library and > libdladm.the applications I''m running are each linked to the resp. libraries I''m trying to inspect, yes (dladm is just an example, but I''m not using it in my tool).> Is the application explicitly loading the libraries? In > that case the probe points won''t exist when dtrace tries to instrument > them.not that I know of - mine certainly isn''t, and someone from the Crossbow team just confirmed that dladm shoulnd''t be doing that either. thx Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
On Wed, Apr 1, 2009 at 1:42 PM, Michael Schuster <Michael.Schuster at sun.com> wrote:> > I see similar stuff for my library ... so why would dtrace fail to show > calls into libdladm for a process that''s started by dtrace (ie using -c) > (using -Z as someone suggested) > > $ dtrace -Z -n ''pid$target:libdladm::entry{}'' -c "dladm show-link" > dtrace: description ''pid$target:libdladm::entry'' matched 0 probes > LINK ? ? ? ?CLASS ? ?MTU ? ?STATE ? ?OVER > iprb1 ? ? ? phys ? ? 1500 ? unknown ?-- > ce2 ? ? ? ? phys ? ? 1500 ? down ? ? -- > bge0 ? ? ? ?phys ? ? 1500 ? up ? ? ? -- > bge1 ? ? ? ?phys ? ? 1500 ? up ? ? ? -- > iprb0 ? ? ? phys ? ? 1500 ? unknown ?-- > bge2 ? ? ? ?phys ? ? 1500 ? up ? ? ? -- > bge3 ? ? ? ?phys ? ? 1500 ? up ? ? ? -- > dtrace: pid 102624 has exited >Hmm, what''s odd is that I see this working on an update 6 system: # dtrace -ln ''pid$target:::entry'' -c "dladm show-link" | awk ''{print $3}'' | uniq MODULE dladm LM1`ld.so.1 libdladm.so.1 libmacadm.so.1 libdlpi.so.1 liblaadm.so.1 libkstat.so.1 libc.so.1 libdevinfo.so.1 libsocket.so.1 libnvpair.so.1 libsec.so.1 libgen.so.1 libnsl.so.1 libavl.so.1 legacy # But here''s what I see on a recent Nevada build: # dtrace -ln ''pid$target:::entry'' -c "dladm show-link" | awk ''{print $3}'' | uniq MODULE dladm LM1`ld.so.1 libc.so.1 MTU 1500 # So I''d say this is likely a bug. I''ll try to figure out what''s going on. Chad
On 04/01/09 11:27, Chad Mynhier wrote:> Hmm, what''s odd is that I see this working on an update 6 system:[...]> So I''d say this is likely a bug. I''ll try to figure out what''s going on.cool, thx. Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
As Pavan suggested DTrace isn''t preloading the libraries. Try using the LD_PRELOAD, LD_PRELOAD_32, or LD_PRELOAD_64 environmental variables to preload the libraries you need. There''s an example on my blog http://blogs.sun.com/madlab/entry/dtrace_the_pid_provider_and Hope this helps. -- This message posted from opensolaris.org
On Wed, Apr 1, 2009 at 4:02 PM, Steve Scargall <steve.scargall at sun.com> wrote:> As Pavan suggested DTrace isn''t preloading the libraries. ?Try using the LD_PRELOAD, LD_PRELOAD_32, or LD_PRELOAD_64 environmental variables to preload the libraries you need. ?There''s an example on my blog http://blogs.sun.com/madlab/entry/dtrace_the_pid_provider_and >While that might be a workaround, it''s not the solution. You shouldn''t need to know all of the libraries an application is linked against in order to use DTrace against it. This is a deeper bug. For example, if I stop dladm on entry to its first strcmp(), I see this: # pldd -F `pgrep dladm` 8967: dladm show-link /usr/lib/libc/libc_hwcap2.so.1 # # pmap -F `pgrep dladm` 8967: dladm show-link 08046000 8K rw--- [ stack ] 08050000 80K r-x-- /sbin/dladm 08074000 8K rw--- /sbin/dladm 08076000 4K rw--- [ heap ] FEE30000 1276K r-x-- /usr/lib/libc/libc_hwcap2.so.1 FEF7F000 28K rwx-- /usr/lib/libc/libc_hwcap2.so.1 FEF86000 8K rwx-- /usr/lib/libc/libc_hwcap2.so.1 FEF90000 4K rwx-- [ anon ] FEFA0000 4K rw--- [ anon ] FEFB0000 4K rw--- [ anon ] FEFBE000 180K r-x-- /lib/ld.so.1 FEFFB000 8K rwx-- /lib/ld.so.1 FEFFD000 4K rwx-- /lib/ld.so.1 total 1616K # But if I look at dlmgmtd, I see the right thing: # pldd `pgrep dlmgmtd` 15: /sbin/dlmgmtd /usr/lib/libc/libc_hwcap2.so.1 /lib/libdladm.so.1 /lib/libcurses.so.1 /lib/libavl.so.1 /lib/libdlpi.so.1 /lib/libinetutil.so.1 /lib/libnvpair.so.1 /lib/libnsl.so.1 /lib/libsysevent.so.1 # This suggests it''s a libproc/procfs bug, but I haven''t figured it out yet. Chad
Pavan Chandrashekar - Sun Microsystems
2009-Apr-02 11:39 UTC
[dtrace-discuss] DTracing one''s own libraries
[..]> > I see similar stuff for my library ... so why would dtrace fail to show > calls into libdladm for a process that''s started by dtrace (ie using -c) > (using -Z as someone suggested) > > $ dtrace -Z -n ''pid$target:libdladm::entry{}'' -c "dladm show-link" > dtrace: description ''pid$target:libdladm::entry'' matched 0 probes > LINK CLASS MTU STATE OVER > iprb1 phys 1500 unknown -- > ce2 phys 1500 down -- > bge0 phys 1500 up -- > bge1 phys 1500 up -- > iprb0 phys 1500 unknown -- > bge2 phys 1500 up -- > bge3 phys 1500 up -- > dtrace: pid 102624 has exitedIt was looking for the exact module name (libdladm* would not work, which I cannot explain why. Looks like - for lazy loads, you need to specify the exact module name) bash-3.2# dtrace -Z -n ''pid$target:libdladm.so.1::entry{}'' -c "dladm show-link" dtrace: description ''pid$target:libdladm.so.1::entry'' matched 0 probes LINK CLASS MTU STATE OVER bge0 phys 1500 up -- bge1 phys 1500 unknown -- dtrace: pid 9032 has exited CPU ID FUNCTION:NAME 0 64966 dladm_walk_datalink_id:entry 0 64959 dladm_door_call:entry 0 64739 dladm_errno2status:entry 0 64965 i_dladm_phys_status:entry 0 64839 dladm_phys_info:entry 0 64737 i_dladm_ioctl:entry 0 64971 dladm_datalink_id2info:entry 0 64959 dladm_door_call:entry 0 64739 dladm_errno2status:entry 0 64965 i_dladm_phys_status:entry ... ... ... Pavan> > >> I assume ldd shows the application linked against your library and >> libdladm. > > the applications I''m running are each linked to the resp. libraries I''m > trying to inspect, yes (dladm is just an example, but I''m not using it > in my tool). > >> Is the application explicitly loading the libraries? In >> that case the probe points won''t exist when dtrace tries to instrument >> them. > > not that I know of - mine certainly isn''t, and someone from the Crossbow > team just confirmed that dladm shoulnd''t be doing that either. > > thx > Michael
Pavan Chandrashekar - Sun Microsystems
2009-Apr-02 11:39 UTC
[dtrace-discuss] DTracing one''s own libraries
On 04/02/09 02:10, Chad Mynhier wrote:> On Wed, Apr 1, 2009 at 4:02 PM, Steve Scargall <steve.scargall at sun.com> wrote: >> As Pavan suggested DTrace isn''t preloading the libraries. Try using the LD_PRELOAD, LD_PRELOAD_32, or LD_PRELOAD_64 environmental variables to preload the libraries you need. There''s an example on my blog http://blogs.sun.com/madlab/entry/dtrace_the_pid_provider_and >> > > While that might be a workaround, it''s not the solution. You > shouldn''t need to know all of the libraries an application is linked > against in order to use DTrace against it. > > This is a deeper bug. For example, if I stop dladm on entry to its > first strcmp(), I see this: > > # pldd -F `pgrep dladm` > 8967: dladm show-link > /usr/lib/libc/libc_hwcap2.so.1 > # > # pmap -F `pgrep dladm` > 8967: dladm show-link > 08046000 8K rw--- [ stack ] > 08050000 80K r-x-- /sbin/dladm > 08074000 8K rw--- /sbin/dladm > 08076000 4K rw--- [ heap ] > FEE30000 1276K r-x-- /usr/lib/libc/libc_hwcap2.so.1 > FEF7F000 28K rwx-- /usr/lib/libc/libc_hwcap2.so.1 > FEF86000 8K rwx-- /usr/lib/libc/libc_hwcap2.so.1 > FEF90000 4K rwx-- [ anon ] > FEFA0000 4K rw--- [ anon ] > FEFB0000 4K rw--- [ anon ] > FEFBE000 180K r-x-- /lib/ld.so.1 > FEFFB000 8K rwx-- /lib/ld.so.1 > FEFFD000 4K rwx-- /lib/ld.so.1 > total 1616K > # > > But if I look at dlmgmtd, I see the right thing:This could mean that at the time this process'' map was seen, some or the other symbol from these libraries were used (especially, given that it is a daemon), and in case of dladm, they were not. There is no necessity of loading a library till it''s first symbol is accessed. In your example, when the first strcmp in dladm was hit, maybe, none of the libdladm symbols were accessed yet. Pavan> > # pldd `pgrep dlmgmtd` > 15: /sbin/dlmgmtd > /usr/lib/libc/libc_hwcap2.so.1 > /lib/libdladm.so.1 > /lib/libcurses.so.1 > /lib/libavl.so.1 > /lib/libdlpi.so.1 > /lib/libinetutil.so.1 > /lib/libnvpair.so.1 > /lib/libnsl.so.1 > /lib/libsysevent.so.1 > # > > This suggests it''s a libproc/procfs bug, but I haven''t figured it out yet. > > Chad > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
Pavan Chandrashekar - Sun Microsystems
2009-Apr-02 12:34 UTC
[dtrace-discuss] DTracing one''s own libraries
On 04/01/09 23:57, Chad Mynhier wrote:> On Wed, Apr 1, 2009 at 1:42 PM, Michael Schuster > <Michael.Schuster at sun.com> wrote: >> I see similar stuff for my library ... so why would dtrace fail to show >> calls into libdladm for a process that''s started by dtrace (ie using -c) >> (using -Z as someone suggested) >> >> $ dtrace -Z -n ''pid$target:libdladm::entry{}'' -c "dladm show-link" >> dtrace: description ''pid$target:libdladm::entry'' matched 0 probes >> LINK CLASS MTU STATE OVER >> iprb1 phys 1500 unknown -- >> ce2 phys 1500 down -- >> bge0 phys 1500 up -- >> bge1 phys 1500 up -- >> iprb0 phys 1500 unknown -- >> bge2 phys 1500 up -- >> bge3 phys 1500 up -- >> dtrace: pid 102624 has exited >> > > Hmm, what''s odd is that I see this working on an update 6 system: > > # dtrace -ln ''pid$target:::entry'' -c "dladm show-link" | awk ''{print $3}'' | uniq > MODULE > dladm > LM1`ld.so.1 > libdladm.so.1 > libmacadm.so.1 > libdlpi.so.1 > liblaadm.so.1 > libkstat.so.1 > libc.so.1 > libdevinfo.so.1 > libsocket.so.1 > libnvpair.so.1 > libsec.so.1 > libgen.so.1 > libnsl.so.1 > libavl.so.1 > legacy > # > > But here''s what I see on a recent Nevada build: > > # dtrace -ln ''pid$target:::entry'' -c "dladm show-link" | awk ''{print $3}'' | uniq > MODULE > dladm > LM1`ld.so.1 > libc.so.1 > MTU > 1500 > #This observation of yours is correct but there is more to the story. I did the following: truss -u a.out -u libc -o dladm.log dladm show-link on both a s10u6 machine and a Nevada build 92 machine. Looks like ld behaves differently on both. On s10u6, a resolvepath was called on all the libraries that the executable depended on right away, even before making any calls to any of the APIs. But on Nevada, they were done on a true "on-demand" basis. So, that explains your difference in behaviour, and also the one with the strcmp that you used as a check point to see if the libraries were loaded. Anyone with more knowledge on the changes that have gone into the process loading part should comment. Pavan> > So I''d say this is likely a bug. I''ll try to figure out what''s going on. > > Chad > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
On Thu, Apr 2, 2009 at 8:34 AM, Pavan Chandrashekar - Sun Microsystems <Pavan.Chandrashekar at sun.com> wrote:> > This observation of yours is correct but there is more to the story. > I did the following: > ?truss -u a.out -u libc -o dladm.log dladm show-link > > on both a s10u6 machine and a Nevada build 92 machine. Looks like ld behaves > differently on both. On s10u6, a resolvepath was called on all the libraries > that the executable depended on right away, even before making any calls to > any of the APIs. > But on Nevada, they were done on a true "on-demand" basis. So, that explains > your difference in behaviour, and also the one with the strcmp that you used > as a check point to see if the libraries were loaded.Yeah, this jibes with what I''ve been seeing, and I''d tracked it down to lazy loading by the linker. (I used dtrace to stop the executable both before and after the first libdladm call, and then I ran pldd on the stopped binary.) So something has changed so that everything is lazily loaded. Regardless of that, dtrace should still be doing the right thing. I''m looking into why it''s not doing the right thing. Chad
Chad Mynhier wrote:> On Wed, Apr 1, 2009 at 4:02 PM, Steve Scargall <steve.scargall at sun.com> wrote: >> As Pavan suggested DTrace isn''t preloading the libraries. Try using the LD_PRELOAD, LD_PRELOAD_32, or LD_PRELOAD_64 environmental variables to preload the libraries you need. There''s an example on my blog http://blogs.sun.com/madlab/entry/dtrace_the_pid_provider_and >> > > While that might be a workaround, it''s not the solution. You > shouldn''t need to know all of the libraries an application is linked > against in order to use DTrace against it. > > This is a deeper bug. For example, if I stop dladm on entry to its > first strcmp(), I see this: > > # pldd -F `pgrep dladm` > 8967: dladm show-link > /usr/lib/libc/libc_hwcap2.so.1 > # > # pmap -F `pgrep dladm` > 8967: dladm show-link > 08046000 8K rw--- [ stack ] > 08050000 80K r-x-- /sbin/dladm > 08074000 8K rw--- /sbin/dladm > 08076000 4K rw--- [ heap ] > FEE30000 1276K r-x-- /usr/lib/libc/libc_hwcap2.so.1 > FEF7F000 28K rwx-- /usr/lib/libc/libc_hwcap2.so.1 > FEF86000 8K rwx-- /usr/lib/libc/libc_hwcap2.so.1 > FEF90000 4K rwx-- [ anon ] > FEFA0000 4K rw--- [ anon ] > FEFB0000 4K rw--- [ anon ] > FEFBE000 180K r-x-- /lib/ld.so.1 > FEFFB000 8K rwx-- /lib/ld.so.1 > FEFFD000 4K rwx-- /lib/ld.so.1 > total 1616K > # > > But if I look at dlmgmtd, I see the right thing: > > # pldd `pgrep dlmgmtd` > 15: /sbin/dlmgmtd > /usr/lib/libc/libc_hwcap2.so.1 > /lib/libdladm.so.1 > /lib/libcurses.so.1 > /lib/libavl.so.1 > /lib/libdlpi.so.1 > /lib/libinetutil.so.1 > /lib/libnvpair.so.1 > /lib/libnsl.so.1 > /lib/libsysevent.so.1 > # > > This suggests it''s a libproc/procfs bug, but I haven''t figured it out yet.I exchanged some email with Rod Evans and then Jonathan Adams; as it turns out, if the complete library name (libmylib.so.1) is specified, DTrace does its job; apparently, the matching code has suffered since it was introduced. thx all for your help Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''