dtrace discuss - Jun 2010 - Monitoring filessytem access

I haven''t looked into dtrace much yet, because it''s obviously
complex, and
I''m not sure if it''s the right tool for the job.  Can dtrace
help me solve
this?

 

When somebody is hammering on the system, I want to be able to detect
who''s
doing it, and hopefully even what they''re doing.

 

Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me
performance statistics and so forth.  I''m looking for something more
granular.  Either *who* the activity belongs to, or *what* files are active
(which might indirectly let me figure out who) ... etc.

 

If it makes the problem any easier (or harder) all the access is coming
across NFS.  And maybe a little CIFS.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100617/9aa970a1/attachment.html>

Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop should do
what you are asking for on files

Regards
Martin


On 17.06.10 14:57, Edward Ned Harvey wrote:
> I haven''t looked into dtrace much yet, because it''s
obviously complex,
> and I''m not sure if it''s the right tool for the job.  Can
dtrace help me
> solve this?
> 
>  
> 
> When somebody is hammering on the system, I want to be able to detect
> who''s doing it, and hopefully even what they''re doing.
> 
>  
> 
> Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me
> performance statistics and so forth.  I''m looking for something
more
> granular.  Either *who* the activity belongs to, or *what* files are
> active (which might indirectly let me figure out who) ... etc.
> 
>  
> 
> If it makes the problem any easier (or harder) all the access is coming
> across NFS.  And maybe a little CIFS.
> 
>  
> 
> 
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Martin Mueller                        martin.mueller at sun.com
Systems Infrastructure Ambassador     Tel.: +49 2102 45 11740
Sun Microsystems GmbH                 Fax: +49 2102 499516
D-40880 Ratingen                      Mobile: +49 172 8618483

Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551
Kirchheim-Heimstetten
Amtsgericht Muenchen: HRB 161028
Geschaeftsfuehrer: Juergen Kunz

> From: dtrace-discuss-bounces at opensolaris.org 
> [dtrace-discuss-bounces at opensolaris.org] On Behalf Of Martin 
> Mueller - Sun Germany SE [Martin.Mueller at Sun.COM]
> 
> Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop 
> should do what you are asking for on files
This seemed very promising when I first saw it ... But it doesn''t seem
to capture NFS activity.  I sit here and watch it, meanwhile start "dd
if=/dev/zero of=junk.file" on some other machine, writing over NFS, and ...
No result appears on the rwsnoop output.  ;-(

Any other suggestions?

Thank you for this one.

G''Day,

----- solaris2 at nedharvey.com wrote:
> > From: dtrace-discuss-bounces at opensolaris.org
> > [dtrace-discuss-bounces at opensolaris.org] On Behalf Of Martin
> > Mueller - Sun Germany SE [Martin.Mueller at Sun.COM]
> >
> > Have a look at the Dtrace Toolkit from Brendan Gregg. rwsnoop
> > should do what you are asking for on files
>
> This seemed very promising when I first saw it ... But it doesn''t
seem
> to capture NFS activity.  I sit here and watch it, meanwhile start "dd
> if=/dev/zero of=junk.file" on some other machine, writing over NFS,
> and ... No result appears on the rwsnoop output.  ;-(
>
> Any other suggestions?
>
rwsnoop traces syscalls, and would work from the client.  On the server,
NFS is kernel-based and accesses files without syscalls.

Use the NFS providers, if available, since they are a stable interface.  Eg
(NFSv3):

# dtrace -n ''nfsv3:::op-read-start,nfsv3:::op-write-start {
@[probefunc,
# args[1]->noi_curpath] = count(); }''

If they aren''t available, you can try the fsinfo provider - if
available.  Eg:

# dtrace -n ''fsinfo:::read,fsinfo:::write { @[probefunc,
args[0]->fi_pathname]
# = count(); }''

If that isn''t available, you can dig it out of the (unstable) fbt
provider,
and maybe sdt.  You could probe NFS functions, the filesystem, or VFS.  eg,
VFS on Solaris:

# dtrace -n ''fbt::fop_read:entry,fbt::fop_write:entry { @[probefunc,
# stringof(args[0]->v_path)] = count(); }''

Brendan

-- 
Brendan Gregg, Fishworks                       http://blogs.sun.com/brendan

Yessss....

 

man nfslogd

      (and google nfslogd, etc)

totally nailed it.  (Thanks to Cameron.)

 

Just incase anyone stumbles across this by search, here''s the
start-to-end
answer:

man nfslogd

First, edit /etc/default/nfslogd  I am using:

     MIN_PROCESSING_SIZE=1

     IDLE_TIME=1

Assuming you already have svc:/network/nfs/server:default enabled

If you are using /etc/dfs/dfstab instead of the sharenfs property, just add
",log" as specified by man share_nfs

    If you wish, run that share command now.

    By simply adding ,log to the dfstab, nfslogd will automatically start
after reboot

    or start it manually now:  /usr/lib/nfs/nfslogd

If you are using the sharenfs property instead of dfstab, just figure out
how to add the ,log option

tail -f /var/nfs/nfslog

 

Again man nfslogd.  Information in the log won''t be complete until NFS
services are restarted, which is well documented in the man page.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100617/71ea34c5/attachment.html>

[Apologies for the delayed response]

NFS logging kills performance. I have witnessed more than 85% less
performance when NFS logging was enabled. The DTrace providers are 
much more efficient.
 00 rucgard
-- 
This message posted from opensolaris.org