dtrace discuss - Dec 2007 - How to trace all libnsl:svc_register

We have a system where rpcbind is consuming a lot of CPU and we believe one of
our application process is registering transient program numbers but it is not
unregistering them i.e. rpcinfo output shows increasing program numbers.

How would I check which process is doing this?

Thanks in advance.
-- prasad


--
This message posted from opensolaris.org

It''s not possible to trace all calls to a function accross all
processes as
they come and go, but what you could do is examine what system calls are
made by svc_register() and then use the syscall provider to gather the data
you need.

Adam

On Mon, Dec 03, 2007 at 08:11:00AM -0800, prasad wrote:
> We have a system where rpcbind is consuming a lot of CPU and we believe one
of our application process is registering transient program numbers but it is
not unregistering them i.e. rpcinfo output shows increasing program numbers.
> 
> How would I check which process is doing this?
> 
> Thanks in advance.
> -- prasad
> 
> 
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

On Dec 3, 2007 11:54 AM, Adam Leventhal <ahl at eng.sun.com>
wrote:
> It''s not possible to trace all calls to a function accross all
processes as
> they come and go, but what you could do is examine what system calls are
> made by svc_register() and then use the syscall provider to gather the data
> you need.
Would it be reasonable to consider adding probes to RPC library
functions (usr/src/lib/libnsl/rpc/*.c) to provide generic methods for
tracing RPC''s?  While probes that aren''t enabled have no
impact to a
running process, I don''t think I understand how the existence of the
probes gets registered with the kernel to understand the impact on
process startup.  That is, if libnsl had some USDT probes in it, would
it have a measurable impact on the startup time of programs that link
against libnsl?

My real area of concern is more around name services, but the same
issues would seem to apply.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On Mon, Dec 03, 2007 at 02:41:20PM -0600, Mike Gerdts
wrote:
> Would it be reasonable to consider adding probes to RPC library
> functions (usr/src/lib/libnsl/rpc/*.c) to provide generic methods for
> tracing RPC''s?  While probes that aren''t enabled have no
impact to a
> running process, I don''t think I understand how the existence of
the
> probes gets registered with the kernel to understand the impact on
> process startup.  That is, if libnsl had some USDT probes in it, would
> it have a measurable impact on the startup time of programs that link
> against libnsl?
Absolutely. Adding USDT probes to libnsl would be a great idea.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

On Dec 3, 2007 3:12 PM, Adam Leventhal <ahl at eng.sun.com>
wrote:
> On Mon, Dec 03, 2007 at 02:41:20PM -0600, Mike Gerdts wrote:
> > Would it be reasonable to consider adding probes to RPC library
> > functions (usr/src/lib/libnsl/rpc/*.c) to provide generic methods for
> > tracing RPC''s?  While probes that aren''t enabled
have no impact to a
> > running process, I don''t think I understand how the existence
of the
> > probes gets registered with the kernel to understand the impact on
> > process startup.  That is, if libnsl had some USDT probes in it, would
> > it have a measurable impact on the startup time of programs that link
> > against libnsl?
>
> Absolutely. Adding USDT probes to libnsl would be a great idea.
Is there some updated documentation that helps out with less trivial
USDT probes?  The wikified dtrace manual[1] isn''t too helpful when you
want to provide access to data structures.

I figured that there must be an example to learn from, so I went
looking through the ON source and under usr/src/cmd iscsitgtd seems to
be the only command with USDT probes[2].  Based upon the iscsitgtd
source, it looks like I would need to use translators[3].  However,
in usr/src/cmd/iscsi/iscsitgtd/*.c I don''t see any DTRACE_PROBE*()
calls so I suspect that something more advanced (less labor
intensive?) is being done.

1.http://wikis.sun.com/display/DTrace/Statically+Defined+Tracing+for+User+Applications
2.http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/iscsi/iscsitgtd/iscsi_provider.d
3.http://wikis.sun.com/display/DTrace/Translators

Any clues you can throw my way would be most appreciated.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On Mon, Dec 03, 2007 at 11:07:15PM -0600, Mike Gerdts
wrote:
> Is there some updated documentation that helps out with less trivial
> USDT probes?  The wikified dtrace manual[1] isn''t too helpful when
you
> want to provide access to data structures.
I don''t think you''ll need translated types for a libnsl
provider. The new
mechanism for USDT can be found here:

  http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better

If anyone is interested in updating the wiki chapter, please let me know.
Otherwise, I hope to get an update done sometime within the next few months.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

> > It''s not possible to trace all calls to a function
> > accross all processes as
> > they come and go, but what you could do is examine
> > what system calls are
> > made by svc_register() and then use the syscall
> > provider to gather the data
> > you need.
> > 
> > Adam
> > 
> 
> How do I determine the system calls that are made by
> svc_register()? 
> 
> -- prasad

You could always write a program that calls it, and truss that program.
if you put a write(1,"-----",5)  or something recognizable like that
right before the call, it should be easy to spot in the output and ignore
everything before that (i.e. all the dynamic linking at startup and such).


--
This message posted from opensolaris.org

On Tue, Dec 04, 2007 at 12:16:32AM -0800, Richard L. Hamilton
wrote:
> > How do I determine the system calls that are made by
> > svc_register()? 
> 
> You could always write a program that calls it, and truss that program.
> if you put a write(1,"-----",5)  or something recognizable like
that
> right before the call, it should be easy to spot in the output and ignore
> everything before that (i.e. all the dynamic linking at startup and such).
Truss: feh. Use DTrace:

pid$target::svc_register:entry
{
	self->follow = 1;
}

syscall:::entry
/self->follow/
{
	@[probefunc] = count();
}

pid$target::svc_register:return
/self->follow/
{
	self->follow = 0;
}

It looks like there''s an ioctl in there that you could use to uniquely
identify such calls:

syscall::ioctl:entry
/self->follow/
{
	@ioctl[arg0] = count();
}

Then verify with another DTrace invocation:

# dtrace -n ''syscall::ioctl:entry/arg0 == <value>/{ @[ustack()] =
count(); }''

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

On Dec 4, 2007 1:49 AM, Adam Leventhal <ahl at eng.sun.com>
wrote:
> On Mon, Dec 03, 2007 at 11:07:15PM -0600, Mike Gerdts wrote:
> > Is there some updated documentation that helps out with less trivial
> > USDT probes?  The wikified dtrace manual[1] isn''t too helpful
when you
> > want to provide access to data structures.
>
> I don''t think you''ll need translated types for a libnsl
provider. The new
> mechanism for USDT can be found here:
>
>   http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better
This got me started.  I''ve added a couple probes to clnt_create with
some success.  The first problem I ran into was with ctfmerge:

ERROR: ctfmerge: Input file pics/rpc_provider.o was partially built
from C sources, but no CTF data was present
Removing libnsl.so.1

A bit of hackery (not sure it is long-term right) in ../Makefile.lib
got me past this.

I was rather surprised when I next ran into this:

check_fnames: error: forbidden names found in pics/rpc_provider.o
    close
    dlinfo
    fprintf
    ioctl
    open64
    strerror
*** Error code 1
dmake: Fatal error: Command failed for target `fnamecheck''
Current working directory
/ws/2007-11-rpc-dtrace/onnv-gate/usr/src/lib/libnsl/sparc
*** Error code 1
The following command caused the error:
cd sparc; pwd; dmake all
dmake: Fatal error: Command failed for target `sparc''
Current working directory /ws/2007-11-rpc-dtrace/onnv-gate/usr/src/lib/libnsl

Does this mean that lint rules need to tamed or that rpc_provider.o
needs to be generated in a different way?

FWIW, pics/rpc_provider.o is generated from rpc/rpc_provider.c:

provider rpc {
        probe clnt_create_start(char*, int, int, char*);
        probe clnt_create_done();
        probe clnt_tli_create_start();
};


Using the following in Makefile.com:

../rpc/%.h: ../rpc/%.d
        $(DTRACE) -h -s $< -o $@

pics/%.o: ../rpc/%.d
        @echo "COMPILE.d=<$(COMPILE.d)>"
        $(COMPILE.d) -o $@ -s $< $(RPC:%=pics/%)


If I then do:

LD_LIBRARY_PATH=`pwd`/sparc mconnect

Then dtrace -l lists the probes I added and I see calls to
clnt_tli_create() when I run rpcinfo.
> If anyone is interested in updating the wiki chapter, please let me know.
> Otherwise, I hope to get an update done sometime within the next few
months.
Assuming I get to a point where this all works out, I may be willing
to take it on.  If someone else with more ability steps up before I
get a draft out, my feelings won''t be hurt either.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

>This got me started.  I''ve added a couple probes to clnt_create
with
>some success.  The first problem I ran into was with ctfmerge:
>
>ERROR: ctfmerge: Input file pics/rpc_provider.o was partially built
>from C sources, but no CTF data was present
>Removing libnsl.so.1
The standard compiler rules should provide the ctf commands.
>A bit of hackery (not sure it is long-term right) in ../Makefile.lib
>got me past this.
>
>I was rather surprised when I next ran into this:
>
>check_fnames: error: forbidden names found in pics/rpc_provider.o
>    close
>    dlinfo
>    fprintf
>    ioctl
    open64
>    strerror

Libraries should not call the "naked" version of these symbols as
those
can also be defined in user programs and those will then be called instead.


You will need to call _close etc instead; this is done through including
"synonyms.h" before you include anything else.  (See your typical
other
library source file)



Casper

On Dec 5, 2007 1:24 AM,  <Casper.Dik at sun.com>
wrote:
> >check_fnames: error: forbidden names found in pics/rpc_provider.o
> >    close
> >    dlinfo
> >    fprintf
> >    ioctl
>     open64
> >    strerror
>
>
> Libraries should not call the "naked" version of these symbols as
those
> can also be defined in user programs and those will then be called instead.
>
>
> You will need to call _close etc instead; this is done through including
> "synonyms.h" before you include anything else.  (See your typical
other
> library source file)
The problem is that I did not generate these from C.  I used (roughly)
the following, as suggested at
http://blogs.sun.com/ahl/entry/user_land_tracing_gets_better and
referenced by Adam earlier in this thread.

dtrace -G -o rpc_provider.o -s rpc_provider.d

My previous post listed the actual d provider source and the bulk of
the Makefile.* updates.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

> On Dec 5, 2007 1:24 AM,  <Casper.Dik at sun.com> wrote:
> > >check_fnames: error: forbidden names found in
> pics/rpc_provider.o
> > >    close
> > >    dlinfo
> > >    fprintf
> > >    ioctl
> >     open64
> > >    strerror
> >
> >
> > Libraries should not call the "naked" version of
> these symbols as those
> > can also be defined in user programs and those will
> then be called instead.
> >
> >
> > You will need to call _close etc instead; this is
> done through including
> > "synonyms.h" before you include anything else.
>  (See your typical other
>  library source file)
> The problem is that I did not generate these from C.
>  I used (roughly)
> he following, as suggested at
> http://blogs.sun.com/ahl/entry/user_land_tracing_gets_
> better and
> referenced by Adam earlier in this thread.
> 
> dtrace -G -o rpc_provider.o -s rpc_provider.d
> 
> My previous post listed the actual d provider source
> and the bulk of
> the Makefile.* updates.
> 
HI,

After a big discussion last week on the SPARKS mailinglist on how to handle the
debugging in
libsldap I''m currently trying to create USDT probes to address the
current way of debugging
and to provide better debugging for that particular library. I have read quite a
lot the last
days and started making my first small steps into just inserting a very basic
probe
which outputs a string debug message (as described by Adam Leventhal in 2006 in
one of
his blogs) this is just a first test to see if I get the hang of it all.

But I run into exactly the same problems described here with the CTF symbols
missing when doing a simple

dtrace -G -o libsldap_provider.o -s ../common/libsldap_provider.d pics/*.o

e.g.

ERROR: ctfmerge: Input file pics/libsldap_provider.o was partially built from C
sources, but no CTF data was present.

And I probably run into the next problem mentioned here. So before trying to
solve
this myself (have been searching for the last day or so) are we somewhat stuck
here.

It seems we are missing some functionality in dtrace -G or am I missing
something
very obviously.

Marco van Wieringen


--
This message posted from opensolaris.org

On Sun, Feb 03, 2008 at 02:07:30PM -0800, Marco van Wieringen
wrote:
> But I run into exactly the same problems described here with the CTF
symbols missing when doing a simple
> 
> dtrace -G -o libsldap_provider.o -s ../common/libsldap_provider.d pics/*.o
> 
> e.g.
> 
> ERROR: ctfmerge: Input file pics/libsldap_provider.o was partially built
from C sources, but no CTF data was present.
You should not be running ctfmerge on object files generated by dtrace(1M).
You''re probably running ctfmerge as part of the standard Makefiles --
you''ll
just need to work around it by adding a different category of object files
and overriding some of the default targets.

Adam

-- 
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl