I have several ZFS file systems being served via NFS to lots of NFS clients. I need to figure out who is writing to what files so did what most of us do, searched the Internet for a script that someone else wrote. Found something, tested it out on our Engineer systems and it worked great for what I want with some minor modifications. When I went to use it on a production system, I started seeing lots of errors. This is what I see: DATE R/W/D cr_uid: cr_uid cr_gid: cr_gid cr_ruid: cr_ruid cr_rgid: cr_rgid Path dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 --------- SNIPPED LOTS of similar errors ----------- 2012 Mar 26 12:45:23 D cr_uid: 769 cr_gid: 40 cr_ruid: 769 cr_rgid: 40 2012 Mar 26 12:45:23 D cr_uid: 769 cr_gid: 40 cr_ruid: 769 cr_rgid: 40 dtrace: error on enabled probe ID 3 (ID 26643: fbt:zfs:zfs_read:return): invalid address (0x0) in predicate at DIF offset 28 2012 Mar 26 12:45:31 W cr_uid: 769 cr_gid: 40 cr_ruid: 769 cr_rgid: 40 /clients/abc0101/rep/homes/tsgops/LCPPAGNITE/2012-03-23/nitelog.txt dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 dtrace: error on enabled probe ID 5 (ID 26645: fbt:zfs:zfs_write:return): invalid address (0x0) in predicate at DIF offset 28 The script is below. Both the Engineering and production systems are Sol10 Update10. Zpool versions are different. 15 on the Engineering system and 10 on the production system. The script is invoked as follows: zfs_user.d /clients/abc0101/rep/. The really odd things is, if I remove the ''zfs_write'' tracing from the script and leave the ''zfs_read'' in the script runs fine, which you would expect since there are no read related errors above. But if I then simply search and replace read -> write, I see the errors. The function prototypes for both zfs_read and zfs_write are identical. So I don''t expect that is the problem. My understanding of DTrace, which is somewhat limited at this point, is that the ''self->'' variables are thread specific so I should not be causing problems due to multiple invocations by different threads. It seems to me that I missing something very simple, but I cannot figure it out. Any thoughts? #!/usr/sbin/dtrace -s #pragma D option quiet BEGIN { printf("%20s%20s cr_uid: %5s cr_gid: %5s cr_ruid: %5s cr_rgid: %5s %s\n", "DATE", "R/W/D", "cr_uid", "cr_gid", "cr_ruid", "cr_rgid", "Path" ); } zfs_read:entry { self->filepath = args[0]->v_path; self->cred = (cred_t *)args[3]; } zfs_read:return /strstr(stringof(self->filepath), $1) != NULL/ { printf("%20Y%20s cr_uid: %5d cr_gid: %5d cr_ruid: %5d cr_rgid: %5d %s\n", walltimestamp, "R", self->cred->cr_uid, self->cred->cr_gid, self->cred->cr_ruid, self->cred->cr_rgid, stringof(self->filepath) ); self->filepath = 0; self->cred = 0; } zfs_write:entry { self->filepath = args[0]->v_path; self->cred = (cred_t *)args[3]; } zfs_write:return /strstr(stringof(self->filepath), $1) != NULL/ { printf("%20Y%20s cr_uid: %5d cr_gid: %5d cr_ruid: %5d cr_rgid: %5d %s\n", walltimestamp, "W", self->cred->cr_uid, self->cred->cr_gid, self->cred->cr_ruid, self->cred->cr_rgid, stringof(self->filepath) ); } zfs_remove:entry { self->filepath = strjoin (stringof(args[0]->v_path), "/"); self->filepath = strjoin ( self->filepath, stringof(args[1])); self->cred = (cred_t *)args[2]; } zfs_remove:return /strstr(stringof(self->filepath), $1) != NULL/ { printf("%20Y%20s cr_uid: %5d cr_gid: %5d cr_ruid: %5d cr_rgid: %5d %s\n", walltimestamp, "D", self->cred->cr_uid, self->cred->cr_gid, self->cred->cr_ruid, self->cred->cr_rgid, stringof(self->filepath) ); self->filepath = 0; } -----RTU