dtrace discuss - Apr 2006 - Physical vs. Logical NFS operations

Howdy,

I am writing some DTrace scripts to trace NFSv3 operations. Since numerous 
operations can be satisified by caches on the client, I started to wonder 
if there was a way to determine if a given NFS operation was satisfied 
from a local cache, or if a network I/O took place. After reading through 
part of the NFSv3 client implementation and reviewing the flowindent 
between nfs3_read:entry and nfs3_read:return, it occurred to me that I 
might be able to set a variable in a probe (e.g., tcp_send_data), and use
that variable to determine if the operation was satisified from the server 
or a local cache. After a bunch of testing, it looks like this approach 
won''t work because of read-ahead, NFS block size vs read size, and
because
the code paths change over time. Does anyone that is bit more familar with 
the kernel / NFS client implementation happen to know if there is a way to 
tell if an NFS read, lookup, etc. generated an network I/O to the server?

Thanks for any insight,
- Ryan
--
UNIX Administrator
http://daemons.net/~matty

Matty wrote:
>
> Howdy,
>
> I am writing some DTrace scripts to trace NFSv3 operations. Since 
> numerous operations can be satisified by caches on the client, I 
> started to wonder if there was a way to determine if a given NFS 
> operation was satisfied from a local cache, or if a network I/O took 
> place. After reading through part of the NFSv3 client implementation 
> and reviewing the flowindent between nfs3_read:entry and 
> nfs3_read:return, it occurred to me that I might be able to set a 
> variable in a probe (e.g., tcp_send_data), and use
> that variable to determine if the operation was satisified from the 
> server or a local cache. After a bunch of testing, it looks like this 
> approach won''t work because of read-ahead, NFS block size vs read 
> size, and because the code paths change over time. Does anyone that is 
> bit more familar with the kernel / NFS client implementation happen to 
> know if there is a way to tell if an NFS read, lookup, etc. generated 
> an network I/O to the server?
For reads I think a simple way would be to see if a page fault was taken 
and if  nfs3_getapage() was called.   For lookups it would be 
nfs3lookup_otw().

Rao.
>
> Thanks for any insight,
> - Ryan
> -- 
> UNIX Administrator
> http://daemons.net/~matty
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On Sat, Matty wrote:
> 
> Howdy,
> 
> I am writing some DTrace scripts to trace NFSv3 operations. Since numerous 
> operations can be satisified by caches on the client, I started to wonder 
> if there was a way to determine if a given NFS operation was satisfied 
> from a local cache, or if a network I/O took place. After reading through 
> part of the NFSv3 client implementation and reviewing the flowindent 
> between nfs3_read:entry and nfs3_read:return, it occurred to me that I 
> might be able to set a variable in a probe (e.g., tcp_send_data), and use
> that variable to determine if the operation was satisified from the server 
> or a local cache. After a bunch of testing, it looks like this approach 
> won''t work because of read-ahead, NFS block size vs read size, and
because
> the code paths change over time. Does anyone that is bit more familar with 
> the kernel / NFS client implementation happen to know if there is a way to 
> tell if an NFS read, lookup, etc. generated an network I/O to the server?
So the best way I can think of to accomplish what you want is to
hook each of the VOP entry points in the NFSv3 client code (which have
likely done already) and then track use of rfs3call.  As you note,
this will capture all over-the-wire activity for the NFSv3 client
VOP in question.  In fact, you will be able to capture if the client
does more than one RPC to satisfy the upper level VOP call.

As you note, this works for everything except asynchronous events the
client does.  This would be things like read, write, and async readdir.
For those, I would suggest tracking use of nfs3read, nfs3write and checking
the vnode in question (by matching with the initial vnode used for the VOP
call); nfs3read/nfs3write will do over-the-wire work so that is an
easy match.

For readdir, you will want to track the nfs3readdir and nfs3readdirplus
with the vnode tracking and that should track that behavior as well.

Spencer

On Mon, 17 Apr 2006, Spencer Shepler wrote:
>
> So the best way I can think of to accomplish what you want is to
> hook each of the VOP entry points in the NFSv3 client code (which have
> likely done already) and then track use of rfs3call.  As you note,
> this will capture all over-the-wire activity for the NFSv3 client
> VOP in question.  In fact, you will be able to capture if the client
> does more than one RPC to satisfy the upper level VOP call.
Cool. I will conduct some tests this week.
> As you note, this works for everything except asynchronous events the
> client does.  This would be things like read, write, and async readdir.
> For those, I would suggest tracking use of nfs3read, nfs3write and checking
> the vnode in question (by matching with the initial vnode used for the VOP
> call); nfs3read/nfs3write will do over-the-wire work so that is an
> easy match.
Which field(s) in the vnode_t would you recommend matching against? Do you 
by any chance have a sample D script that could illustrate this?
> For readdir, you will want to track the nfs3readdir and nfs3readdirplus
> with the vnode tracking and that should track that behavior as well.
Thanks. I will add these routines to the list of stuff to study.

Thanks for the feedback,
- Ryan
--
UNIX Administrator
http://daemons.net/~matty

On Mon, Matty wrote:
> 
> On Mon, 17 Apr 2006, Spencer Shepler wrote:
> 
> >
> >So the best way I can think of to accomplish what you want is to
> >hook each of the VOP entry points in the NFSv3 client code (which have
> >likely done already) and then track use of rfs3call.  As you note,
> >this will capture all over-the-wire activity for the NFSv3 client
> >VOP in question.  In fact, you will be able to capture if the client
> >does more than one RPC to satisfy the upper level VOP call.
> 
> Cool. I will conduct some tests this week.
> 
> >As you note, this works for everything except asynchronous events the
> >client does.  This would be things like read, write, and async readdir.
> >For those, I would suggest tracking use of nfs3read, nfs3write and
checking
> >the vnode in question (by matching with the initial vnode used for the
VOP
> >call); nfs3read/nfs3write will do over-the-wire work so that is an
> >easy match.
> 
> Which field(s) in the vnode_t would you recommend matching against? Do you 
> by any chance have a sample D script that could illustrate this?
I have attached a rudimentary script that captures the essence.

Spencer

-------------- next part --------------
#!/usr/sbin/dtrace -FCs

int activevnodes[struct vnode *];

nfs3_access:entry
{
	self->start = vtimestamp;
	self->v_path = args[0]->v_path;
	self->nfsvop = "nfs3_access";
}

nfs3_access:return
/self->start && self->otw/
{
	printf("otw took %d\n", vtimestamp - self->start);
	self->start = 0;
	self->otw = 0;
}

nfs3_access:return
/self->start/
{
	printf("nfs3_access: no otw\n");
	self->start = 0;
	self->otw = 0;
}

nfs3_read:entry
{
	self->start = vtimestamp;
	self->vp = args[0];
	self->v_path = self->vp->v_path;
	self->nfsvop = "nfs3_read";
	activevnodes[self->vp] += 1;
}

nfs3_read:return
/self->start/
{
	self->start = 0;
	self->otw = 1;
	activevnodes[self->vp] -= 1;
}

rfs3call:entry
/self->start/
{
	self->otw = 1;
	printf("%s %s\n", stringof(self->nfsvop),
stringof(self->v_path));
}

nfs3read:entry
/activevnodes[args[0]] != 0/
{
	printf("read for %s\n", stringof(args[0]->v_path));
}