When somebody is hammering on the system, I want to be able to detect who''s doing it, and hopefully even what they''re doing. I can''t seem to find any way to do that. Any suggestions? Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me performance statistics and so forth. I''m looking for something more granular. Either *who* the activity belongs to, or *what* files are active (which might indirectly let me figure out who) ... etc. If it makes the problem any easier (or harder) all the access is coming across NFS. And maybe a little CIFS. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100617/c6fb93b7/attachment.html>
Have you looked at ''lsof'' or the native BSM auditing features? Admittedly audit is not really intended for realtime, but lsof certainly is. fpsm On Thu, Jun 17, 2010 at 8:56 AM, Edward Ned Harvey <solaris2 at nedharvey.com> wrote:> When somebody is hammering on the system, I want to be able to detect who''s > doing it, and hopefully even what they''re doing. > > > > I can''t seem to find any way to do that.? Any suggestions? > > > > Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me > performance statistics and so forth.? I''m looking for something more > granular.? Either *who* the activity belongs to, or *what* files are active > (which might indirectly let me figure out who) ... etc. > > > > If it makes the problem any easier (or harder) all the access is coming > across NFS.? And maybe a little CIFS. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >
> From: Fredrich Maney [mailto:fredrichmaney at gmail.com] > > Have you looked at ''lsof'' or the native BSM auditing features? > Admittedly audit is not really intended for realtime, but lsof > certainly is.I''m not familiar with the BSM auditing, and audit, that you mentioned. Any pointers? lsof and fuser only tell you who has a file open. But there are millions of files in the filesystem, and at any given time, thousands of them will be open, and there''s still no way for me to know which files are seeing heavy IO. I can see via iostat, etc, that there is heavy activity taking place, but I don''t know who or where. Also fuser and lsof don''t work on a network filesystem.
You can turn on NFS auditing with the sharenfs option "rw=host1:host2,log". Audit logs are in /var/nfs. I believe there''s a bit of setup in /etc/nfs/nfslog.conf and /etc/default/nfslogd, but I think I''m running fine off of defaults. This only works for NFSv3. NFSv4 won''t log. -- Cameron Hanover chanover at umich.edu "Giving money and power to the government is like giving whiskey and car keys to teenage boys." --P.J. O''Rourke On Jun 17, 2010, at 8:56 AM, Edward Ned Harvey wrote:> When somebody is hammering on the system, I want to be able to detect who''s doing it, and hopefully even what they''re doing. > > > > I can''t seem to find any way to do that. Any suggestions? > > > > Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me performance statistics and so forth. I''m looking for something more granular. Either *who* the activity belongs to, or *what* files are active (which might indirectly let me figure out who) ... etc. > > > > If it makes the problem any easier (or harder) all the access is coming across NFS. And maybe a little CIFS. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Oh, and start /usr/lib/nfs/nfslogd. If you just set sharenfs, nfslogd won''t start automatically, but it will at reboot. - Cameron Hanover chanover at umich.edu "Our integrity sells for so little, but it is all we really have. It is the very last inch of us, but within that inch, we are free." --Valerie (V for Vendetta) On Jun 17, 2010, at 9:24 AM, Cameron Hanover wrote:> You can turn on NFS auditing with the sharenfs option "rw=host1:host2,log". Audit logs are in /var/nfs. I believe there''s a bit of setup in /etc/nfs/nfslog.conf and /etc/default/nfslogd, but I think I''m running fine off of defaults. This only works for NFSv3. NFSv4 won''t log. > > -- > Cameron Hanover > chanover at umich.edu > > "Giving money and power to the government is like giving whiskey and car keys to teenage boys." > --P.J. O''Rourke > > On Jun 17, 2010, at 8:56 AM, Edward Ned Harvey wrote: > >> When somebody is hammering on the system, I want to be able to detect who''s doing it, and hopefully even what they''re doing. >> >> >> >> I can''t seem to find any way to do that. Any suggestions? >> >> >> >> Everything I can find ... iostat, nfsstat, etc ... AFAIK, just show me performance statistics and so forth. I''m looking for something more granular. Either *who* the activity belongs to, or *what* files are active (which might indirectly let me figure out who) ... etc. >> >> >> >> If it makes the problem any easier (or harder) all the access is coming across NFS. And maybe a little CIFS. >> >> >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >
On 17/06/2010 14:12, Edward Ned Harvey wrote:>> From: Fredrich Maney [mailto:fredrichmaney at gmail.com] >> >> Have you looked at ''lsof'' or the native BSM auditing features? >> Admittedly audit is not really intended for realtime, but lsof >> certainly is. > > I''m not familiar with the BSM auditing, and audit, that you mentioned. Any > pointers?Start here: http://hub.opensolaris.org/bin/view/Project+audit/ -- Darren J Moffat
On Thu, June 17, 2010 09:36, Darren J Moffat wrote:> On 17/06/2010 14:12, Edward Ned Harvey wrote: >>> From: Fredrich Maney [mailto:fredrichmaney at gmail.com] >>> >>> Have you looked at ''lsof'' or the native BSM auditing features? >>> Admittedly audit is not really intended for realtime, but lsof >>> certainly is. >> >> I''m not familiar with the BSM auditing, and audit, that you mentioned. >> Any >> pointers? > > Start here: > > http://hub.opensolaris.org/bin/view/Project+audit/Ben Rockwood had a good posts on it as well: http://www.cuddletech.com/blog/pivot/entry.php?id=647 http://www.cuddletech.com/blog/pivot/entry.php?id=1081 Also audit(1M).
Yessss....
man nfslogd
(and google nfslogd, etc)
totally nailed it. Thank you Cameron.
Just incase anyone stumbles across this by search, here''s the
start-to-end answer:
man nfslogd
First, edit /etc/default/nfslogd I am using:
MIN_PROCESSING_SIZE=1
IDLE_TIME=1
Assuming you already have svc:/network/nfs/server:default enabled
If you are using /etc/dfs/dfstab instead of the sharenfs property, just add
",log" as specified by man share_nfs
If you wish, run that share command now.
By simply adding ,log to the dfstab, nfslogd will automatically start after
reboot
or start it manually now: /usr/lib/nfs/nfslogd
If you are using the sharenfs property instead of dfstab, just figure out how to
add the ,log option
tail -f /var/nfs/nfslog
Again man nfslogd. Information in the log won''t be complete until NFS
services are restarted, which is well documented in the man page.
On Jun 17, 2010, at 12:48 PM, Edward Ned Harvey wrote:> Yessss.... > > man nfslogd > (and google nfslogd, etc) > totally nailed it. Thank you Cameron.Before you get too happy... NFS logging can have a severe negative impact on performance and scaling. Not recommended for anyone with lots of active clients. Meanwhile, you might look into some of the dtrace scripts available now because of the new NFS provider. http://wikis.sun.com/display/DTrace/nfsv3+Provider -- richard> Just incase anyone stumbles across this by search, here''s the start-to-end answer: > man nfslogd > First, edit /etc/default/nfslogd I am using: > MIN_PROCESSING_SIZE=1 > IDLE_TIME=1 > Assuming you already have svc:/network/nfs/server:default enabled > If you are using /etc/dfs/dfstab instead of the sharenfs property, just add ",log" as specified by man share_nfs > If you wish, run that share command now. > By simply adding ,log to the dfstab, nfslogd will automatically start after reboot > or start it manually now: /usr/lib/nfs/nfslogd > If you are using the sharenfs property instead of dfstab, just figure out how to add the ,log option > tail -f /var/nfs/nfslog > > Again man nfslogd. Information in the log won''t be complete until NFS services are restarted, which is well documented in the man page. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Richard Elling richard at nexenta.com +1-760-896-4422 ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
Here is a dtrace script based of one of the examples for the nfs provider.
Especially useful when you use NFS for ESX or other hypervisors.
Andreas
#!/usr/sbin/dtrace -s
#pragma D option quiet
inline int TOP_FILES = 50;
dtrace:::BEGIN
{
printf("Tracing... Hit Ctrl-C to end.\n");
startscript = timestamp;
}
nfsv3:::op-read-start,
nfsv3:::op-write-start
{
start[args[1]->noi_xid] = timestamp;
size[args[1]->noi_xid] = args[2]->count;
}
nfsv3:::op-read-done,
nfsv3:::op-write-done
/start[args[1]->noi_xid] != 0/
{
this->elapsed = timestamp - start[args[1]->noi_xid];
this->size = size[args[1]->noi_xid];
@rw[probename == "op-read-done" ? "read" :
"write"] = quantize(this->elapsed / 1000);
@host[args[0]->ci_remote] = sum(this->elapsed);
@file[args[1]->noi_curpath] = sum(this->elapsed);
@rwsc[probename == "op-read-done" ? "read" :
"write"] = count();
@rws[probename == "op-read-done" ? "read" :
"write"] = quantize(this->size);
/* @rwsl[probename == "op-read-done" ? "read" :
"write"] = lquantize(this->size,4096,8256,64);
*/ @hosts[args[0]->ci_remote] = sum(this->size);
@files[args[1]->noi_curpath] = sum(this->size);
this->size = 0;
size[args[1]->noi_xid] = 0;
start[args[1]->noi_xid] = 0;
}
dtrace:::END
{
this->seconds = (timestamp - startscript)/1000000000;
printf("\nNFSv3 read/write top %d files (total us):\n", TOP_FILES);
normalize(@file, 1000);
trunc(@file, TOP_FILES);
printa(@file);
printf("NFSv3 read/write distributions (us):\n");
printa(@rw);
printf("\nNFSv3 read/write top %d files (total MByte):\n",
TOP_FILES);
normalize(@files, 1024*1024);
trunc(@files, TOP_FILES);
printa(@files);
printf("\nNFSv3 read/write by host (total ns):\n");
printa(@host);
printf("\nNFSv3 read/write by host (total s):\n");
normalize(@host, 1000000000);
printa(@host);
printf("\nNFSv3 read/write by host (total Byte):\n");
printa(@hosts);
printf("\nNFSv3 read/write by host (total kByte):\n");
normalize(@hosts,1024);
printa(@hosts);
denormalize(@hosts);
printf("\nNFSv3 read/write by host (total kByte/s):\n");
normalize(@hosts,this->seconds*1024);
printa(@hosts);
printf("NFSv3 read/write distributions (Byte):\n");
printa(@rws);
/* printf("NFSv3 read/write distributions (Byte):\n");
printa(@rwsl);
*/
printf("NFSv3 read/write counts:\n");
printa(@rwsc);
printf("\nScript running for %20d seconds ",this->seconds);
}
--
This message posted from opensolaris.org