dtrace discuss - Nov 2009 - Status of DTrace NFSv3/v4 client providers

We recently had a strange NFS performance anomaly between a V880 running
snv_124 and two NetApp filers.  To investigate, a DTrace NFSv4 (and
eventually NFSv3) client provider would been extremely helpful.
Unfortunately, all I could find were a request for code review of a v3
client provider and another request for help developing a v4 provider.
Nothing seems to have come from those initiatives, though.  Any updates?

Thanks.
	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Rainer Orth wrote:
> We recently had a strange NFS performance anomaly between a V880 running
> snv_124 and two NetApp filers.  To investigate, a DTrace NFSv4 (and
> eventually NFSv3) client provider would been extremely helpful.
> Unfortunately, all I could find were a request for code review of a v3
> client provider and another request for help developing a v4 provider.
> Nothing seems to have come from those initiatives, though.  Any updates?
>
> Thanks.
> 	Rainer
>
>
-----------------------------------------------------------------------------
> Rainer Orth, Center for Biotechnology, Bielefeld University
>
> _______________________________________________
> nfs-discuss mailing list
> nfs-discuss at opensolaris.org
>   
Rainer,

This (NFSv4) has been integrated for some time (Jan 2006) and google 
shows that:

http://blogs.sun.com/samf/entry/a_dtrace_provider_for_nfs

Sam also gave a presentation at Connecthathon in 2006:
http://www.connectathon.org/talks06/falkner.pdf

NFSv3 was added later, you can look at: 
http://wikis.sun.com/display/DTrace/nfsv3+Provider

You can use snv_124 as is to get at these probes.

Later,
Tom

Tom,
> This (NFSv4) has been integrated for some time (Jan 2006) and google 
> shows that:
> 
> http://blogs.sun.com/samf/entry/a_dtrace_provider_for_nfs
> 
> Sam also gave a presentation at Connecthathon in 2006:
> http://www.connectathon.org/talks06/falkner.pdf
> 
> NFSv3 was added later, you can look at: 
> http://wikis.sun.com/display/DTrace/nfsv3+Provider
unfortunately, both providers are for the NFSv4 and v3 *server* only, as is
mentioned in the DTrace WiKi.  dtrace -l shows (except for the fbt
provider) mostly module nfssrv.  There are a few probes in the nfs module,
but nothing generic.

This year''s Connectathon talk on Solaris pNFS Client WIP

	http://www.connectathon.org/talks09/cthon09-client-wip.pdf

mentions a NFSv4 client provider on slide 14, but I haven''t been able
to
find it in the nfs41-gate repository.

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Yep my mistake, I skipped the "client" in the title.

:->

Email is sometimes a bad communication medium, i.e., I''m not
trying to be snotty with the following:

You have the source to snv_124, you could add your own client
providers and contribute back to OpenSolaris. The server providers
were written by people who really needed them.

Tom,
> Email is sometimes a bad communication medium, i.e., I''m not
> trying to be snotty with the following:
> 
> You have the source to snv_124, you could add your own client
> providers and contribute back to OpenSolaris. The server providers
> were written by people who really needed them.
understood, but this would be a terrible waste of time if the work has
already been done or at least mostly done, as seems to be the case here.

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Rainer Orth wrote:
> Tom,
>
>   
>> Email is sometimes a bad communication medium, i.e., I''m not
>> trying to be snotty with the following:
>>
>> You have the source to snv_124, you could add your own client
>> providers and contribute back to OpenSolaris. The server providers
>> were written by people who really needed them.
>>     
>
> understood, but this would be a terrible waste of time if the work has
> already been done or at least mostly done, as seems to be the case here.
>
> 	Rainer
>
>
-----------------------------------------------------------------------------
> Rainer Orth, Center for Biotechnology, Bielefeld University
>   
Just to confirm, no one is actively working on client providers. The 
person who
did the v3 server provider was an intern and had a prototype for the client.

It never went anywhere after that.

Robert,
> http://cr.opensolaris.org/~danhua/webrev/onnv-clone.patch
> 
> I haven''t tried to apply the patch (yet, but i will later this  
> evening) to the
> current onnv-gate. but this patch is a good starting place.
thanks for the pointer.  At first glance, the patch looks straightforward
enough: after the general idea is in place, the rest seems mostly
mechanical.

This might be more involved for NFSv4, though, with the move from separate
RPC calls to COMPOUND4 with multiple ops.  One probably wants to instrument
both the whole COMPOUND4 and the individual ops in some way.  Obviously,
there''s considerable design work involved here.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Piyush,
> Just to elaborate on "It never went anywhere" part, in case
someone is
> interested in picking up the patch that Robert Gordon pointed to 
> (http://cr.opensolaris.org/~danhua/webrev/onnv-clone.patch.).
> 
> The prototype for the NFS v3 client provider was done by Danhua Shao. It 
> was working and also went through a code review process on nfs-discuss 
> alias.  The provider and the probes were demonstrated to work as 
> expected. The key issue in the current provider design is the use of 
> tsd_set and tsd_get functions in order to get a handle on RPC xid where 
> the probes get fired. The problem is that we need a reasonable way to 
> harvest the RPC xid in the higher level nfs functions. As of now, the 
> design uses tsd_set and tsd_get routines. I believe that there is high 
> overhead associated with the use of these functions. At the time this 
> work was done, we did not come out with a more appropriate approach to 
> harvest RPC xid values.
at least from inspection of their implementation
(uts/common/disp/thread.c), the overhead doesn''t seem so high except
for
the case when tsd_set() is used for the first time on a thread.  Obviously,
one would have to quantify the overhead and decide if this is deemed
acceptable for the observability gained.
> I am also attaching the written description of NFSv3 client provider 
> specification, which was posted earlier on these email aliases when the 
> work was being  done actively.
Thanks.  I''ll have a look, but doubt that I''ll be able to do
anything soon:
too much on my plate right now.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University