I have a script, attached, which is giving me headaches. When I run it on a freshly booted system, it works just fine. That is, if I do a rename (mv a b) I will see the probe fire for fop_rename. However, it only works the first time. The next time I do a rename (or link, or symlink) I don''t see anything. If I modify the script, by deleting all the entry/return probes above rename, then it will work all the time. Is there a limit to the number of probes you can have in a script? Is it a memory problem? Thanks for any help/advice. Jim Wahlig ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop emitting anything at all. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: nfsvops.d URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20051026/04bda00a/attachment.ksh>
oops, should mention not to try to use the script. It uses static probes that I put in common_dispatch() (in nfs_server.c) Also, it doesn''t work for IPv6 (yet). This is just a prototype for the project I''m working on. Jim Wahlig wrote:> I have a script, attached, which is giving me headaches. When I run > it on a freshly > booted system, it works just fine. That is, if I do a rename (mv a b) > I will see the probe > fire for fop_rename. However, it only works the first time. The next > time I do a rename > (or link, or symlink) I don''t see anything. > > If I modify the script, by deleting all the entry/return probes above > rename, then it will > work all the time. Is there a limit to the number of probes you can > have in a script? > Is it a memory problem? > > Thanks for any help/advice. > > Jim Wahlig > > ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop > emitting anything at > all. > > >
Hi Jim, There is an implicit limit to the number of probe descriptions you can specify, but that''s gated by the maximum allowable size of the stuff sent down to the kernel by the DTrace consumer (the DOF). If that limit is crossed, you''ll immediately get an error message so you''re not hitting that problem. There''s also a limit on the amount of memory your script can use, but, again, you''ll see an error or a drop notification if DTrace runs out of memory. In general, in fact, DTrace will let you know if you''ve lost data. It''s tough to debug this script without being able to run it, but make sure that your thread-local variables are being set and cleared as you expect. There seems to be a slight confusion in your use of thread-locals: self->me = curthread; That line seems to be intended to get a unique value into self->me, but keep in mind that self-> denotes a thread-local variable so there''s no need to add another degree of ''uniqueness'' by setting it to the thread pointer value. Adam On Wed, Oct 26, 2005 at 10:05:54AM -0500, Jim Wahlig wrote:> I have a script, attached, which is giving me headaches. When I run it > on a freshly > booted system, it works just fine. That is, if I do a rename (mv a b) I > will see the probe > fire for fop_rename. However, it only works the first time. The next > time I do a rename > (or link, or symlink) I don''t see anything. > > If I modify the script, by deleting all the entry/return probes above > rename, then it will > work all the time. Is there a limit to the number of probes you can > have in a script? > Is it a memory problem? > > Thanks for any help/advice. > > Jim Wahlig > > ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop > emitting anything at > all. > >> #!/usr/sbin/dtrace -Cqs > > #include <sys/uio.h> > > uint_t cmpndID; > > BEGIN > { > printf("# Cmpnd ID Client ID File Op err lat args\t\toff len bytes\n"); > } > > sdt:::nfs-dispatch > { > self->req = (struct nfssrv`svc_req *)arg0; > self->sa = (struct sockaddr *)self->req->rq_xprt->xp_xpc.xpc_rtaddr.buf; > self->af = self->sa->sa_family; > self->cp = (uchar_t *)&((struct sockaddr_in *)self->sa)->sin_addr; > self->cid = cmpndID++; > } > > fop_read:entry, > fop_write:entry, > fop_readdir:entry, > fop_readlink:entry > / self->sa && ((vnode_t *)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->path = stringof(((vnode_t *)arg0)->v_path); > self->uiop = arg1; > self->resid_start = ((uio_t *)arg1)->uio_resid; > self->offset = ((uio_t *)arg1)->uio_offset; > self->me = curthread; > self->start = vtimestamp; > } > > fop_read:return, > fop_write:return, > fop_readdir:return, > fop_readlink:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s %ld %ld %ld\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->path, > (long)self->offset, > self->resid_start, > self->resid_start - ((uio_t *)(self->uiop))->uio_resid); > > self->path = 0; > self->uiop = 0; > self->resid_start = 0; > self->offset = 0; > self->me = 0; > self->start = 0; > } > > > > fop_open:entry > / self->sa && (*(vnode_t **)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->path = stringof((*(vnode_t **)arg0)->v_path); > self->me = curthread; > self->start = vtimestamp; > } > > fop_close:entry, > fop_ioctl:entry, > fop_setfl:entry, > fop_getattr:entry, > fop_setattr:entry, > fop_access:entry, > fop_fsync:entry, > fop_inactive:entry, > fop_fid:entry, > fop_rwlock:entry, > fop_rwunlock:entry, > fop_seek:entry, > fop_cmp:entry, > fop_frlock:entry, > fop_space:entry, > fop_realvp:entry, > fop_getpage:entry, > fop_putpage:entry, > fop_map:entry, > fop_addmap:entry, > fop_delmap:entry, > fop_poll:entry, > fop_dump:entry, > fop_pathconf:entry, > fop_pageio:entry, > fop_dumpctl:entry, > fop_dispose:entry, > fop_setsecattr:entry, > fop_getsecattr:entry, > fop_shrlock:entry, > fop_vnevent:entry > / self->sa && ((vnode_t *)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->path = stringof(((vnode_t *)arg0)->v_path); > self->me = curthread; > self->start = vtimestamp; > } > > fop_open:return, > fop_close:return, > fop_ioctl:return, > fop_setfl:return, > fop_getattr:return, > fop_setattr:return, > fop_access:return, > fop_fsync:return, > fop_fid:return, > fop_seek:return, > fop_cmp:return, > fop_frlock:return, > fop_space:return, > fop_realvp:return, > fop_getpage:return, > fop_putpage:return, > fop_map:return, > fop_addmap:return, > fop_delmap:return, > fop_poll:return, > fop_dump:return, > fop_pathconf:return, > fop_pageio:return, > fop_dumpctl:return, > fop_setsecattr:return, > fop_getsecattr:return, > fop_shrlock:return, > fop_vnevent:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->path); > self->path = 0; > self->me = 0; > self->start = 0; > } > > > fop_inactive:return, > fop_rwlock:return, > fop_rwunlock:return, > fop_dispose:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s - %d %s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > (vtimestamp - self->start)/1000, > self->path); > self->path = 0; > self->me = 0; > self->start = 0; > } > > > > fop_lookup:entry, > fop_create:entry, > fop_remove:entry, > fop_mkdir:entry, > fop_rmdir:entry > / self->sa && ((vnode_t *)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->path = stringof(((vnode_t *)arg0)->v_path); > self->name = stringof(arg1); > self->me = curthread; > self->start = vtimestamp; > } > > fop_lookup:return, > fop_create:return, > fop_remove:return, > fop_mkdir:return, > fop_rmdir:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s/%s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->path, self->name); > self->path = 0; > self->name = 0; > self->me = 0; > self->start = 0; > } > > > > fop_rename:entry > / self->sa && ((vnode_t *)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->dir1 = stringof(((vnode_t *)arg0)->v_path); > self->name1 = stringof(arg1); > self->dir2 = stringof(((vnode_t *)arg2)->v_path); > self->name2 = stringof(arg3); > self->me = curthread; > self->start = vtimestamp; > } > > fop_rename:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s/%s,%s/%s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->dir1, self->name1, > self->dir2, self->name2); > self->dir1 = 0; > self->name1 = 0; > self->dir2 = 0; > self->name2 = 0; > self->me = 0; > self->start = 0; > } > > fop_link:entry > / self->sa / > { > self->tpath = stringof(((vnode_t *)arg0)->v_path); > self->spath = stringof(((vnode_t *)arg1)->v_path); > self->tname = stringof(arg2); > self->me = curthread; > self->start = vtimestamp; > } > > fop_link:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s,%s/%s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->spath, > self->tpath, self->tname); > self->tpath = 0; > self->spath = 0; > self->tname = 0; > self->me = 0; > self->start = 0; > } > > fop_symlink:entry > / self->sa && ((vnode_t *)arg0)->v_path != NULL > && execname != "dtrace" / > { > self->dir = stringof(((vnode_t *)arg0)->v_path); > self->linkname = stringof(arg1); > self->target = stringof(arg3); > self->me = curthread; > self->start = vtimestamp; > } > > fop_symlink:return > / self->me == curthread / > { > printf("%d %u.%u.%u.%u %-15s %d %d %s/%s,%s\n", > self->cid, self->cp[0], self->cp[1], self->cp[2], self->cp[3], > probefunc, > arg1, > (vtimestamp - self->start)/1000, > self->dir, > self->linkname, > self->target); > self->dir = 0; > self->linkname = 0; > self->target = 0; > self->me = 0; > self->start = 0; > } > > sdt:::nfs-dispatch-end > /self->sa/ > { > self->req = 0; > self->sa = 0; > self->af = 0; > self->cp = 0; > self->cid = 0; > }> _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Thanks for looking... Here is a similar script which has the same problem (it was the script I started from). You should be able to run it because it is all fbt. Note, it has the same curthread stuff that you said was unnecessary. jim Adam Leventhal wrote:>Hi Jim, > >There is an implicit limit to the number of probe descriptions you can >specify, but that''s gated by the maximum allowable size of the stuff >sent down to the kernel by the DTrace consumer (the DOF). If that limit >is crossed, you''ll immediately get an error message so you''re not >hitting that problem. > >There''s also a limit on the amount of memory your script can use, but, >again, you''ll see an error or a drop notification if DTrace runs out of >memory. In general, in fact, DTrace will let you know if you''ve lost >data. > >It''s tough to debug this script without being able to run it, but make >sure that your thread-local variables are being set and cleared as you >expect. There seems to be a slight confusion in your use of thread-locals: > > self->me = curthread; > >That line seems to be intended to get a unique value into self->me, but >keep in mind that self-> denotes a thread-local variable so there''s no >need to add another degree of ''uniqueness'' by setting it to the thread >pointer value. > >Adam > >On Wed, Oct 26, 2005 at 10:05:54AM -0500, Jim Wahlig wrote: > > >>I have a script, attached, which is giving me headaches. When I run it >>on a freshly >>booted system, it works just fine. That is, if I do a rename (mv a b) I >>will see the probe >>fire for fop_rename. However, it only works the first time. The next >>time I do a rename >>(or link, or symlink) I don''t see anything. >> >>If I modify the script, by deleting all the entry/return probes above >>rename, then it will >>work all the time. Is there a limit to the number of probes you can >>have in a script? >>Is it a memory problem? >> >>Thanks for any help/advice. >> >>Jim Wahlig >> >>ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop >>emitting anything at >>all. >> >> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20051027/754642ba/attachment.html> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: vops.d URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20051027/754642ba/attachment.ksh>
Didn''t include all in last reply. Thanks for looking at this. I decided to play with the "curthread" stuff, by removing it, and got interesting results. I see more probes firing, but I''m also getting lots of errors now. I will continue to work on this to make it only fire when I want it to so as not to get all the errors that I''m seeing. Thanks again, jim Adam Leventhal wrote:>Hi Jim, > >There is an implicit limit to the number of probe descriptions you can >specify, but that''s gated by the maximum allowable size of the stuff >sent down to the kernel by the DTrace consumer (the DOF). If that limit >is crossed, you''ll immediately get an error message so you''re not >hitting that problem. > >There''s also a limit on the amount of memory your script can use, but, >again, you''ll see an error or a drop notification if DTrace runs out of >memory. In general, in fact, DTrace will let you know if you''ve lost >data. > >It''s tough to debug this script without being able to run it, but make >sure that your thread-local variables are being set and cleared as you >expect. There seems to be a slight confusion in your use of thread-locals: > > self->me = curthread; > >That line seems to be intended to get a unique value into self->me, but >keep in mind that self-> denotes a thread-local variable so there''s no >need to add another degree of ''uniqueness'' by setting it to the thread >pointer value. > >Adam > >On Wed, Oct 26, 2005 at 10:05:54AM -0500, Jim Wahlig wrote: > > >>I have a script, attached, which is giving me headaches. When I run it >>on a freshly >>booted system, it works just fine. That is, if I do a rename (mv a b) I >>will see the probe >>fire for fop_rename. However, it only works the first time. The next >>time I do a rename >>(or link, or symlink) I don''t see anything. >> >>If I modify the script, by deleting all the entry/return probes above >>rename, then it will >>work all the time. Is there a limit to the number of probes you can >>have in a script? >>Is it a memory problem? >> >>Thanks for any help/advice. >> >>Jim Wahlig >> >>ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop >>emitting anything at >>all. >> >> >> >> > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20051027/f1f148e1/attachment.html>
For anyone interested... My problem was nested vops. mkdir, fsync, create, write, remove all call fop_putpage and readdir, read call fop_getpage. When those functions get called, they clear the variables when done and hence prevent the calling functions from working. jim Jim Wahlig wrote:> Didn''t include all in last reply. Thanks for looking at this. > I decided to play with the "curthread" stuff, by removing it, and got > interesting > results. I see more probes firing, but I''m also getting lots of > errors now. > > I will continue to work on this to make it only fire when I want it to > so as not to get > all the errors that I''m seeing. > > Thanks again, > > jim > > Adam Leventhal wrote: > >>Hi Jim, >> >>There is an implicit limit to the number of probe descriptions you can >>specify, but that''s gated by the maximum allowable size of the stuff >>sent down to the kernel by the DTrace consumer (the DOF). If that limit >>is crossed, you''ll immediately get an error message so you''re not >>hitting that problem. >> >>There''s also a limit on the amount of memory your script can use, but, >>again, you''ll see an error or a drop notification if DTrace runs out of >>memory. In general, in fact, DTrace will let you know if you''ve lost >>data. >> >>It''s tough to debug this script without being able to run it, but make >>sure that your thread-local variables are being set and cleared as you >>expect. There seems to be a slight confusion in your use of thread-locals: >> >> self->me = curthread; >> >>That line seems to be intended to get a unique value into self->me, but >>keep in mind that self-> denotes a thread-local variable so there''s no >>need to add another degree of ''uniqueness'' by setting it to the thread >>pointer value. >> >>Adam >> >>On Wed, Oct 26, 2005 at 10:05:54AM -0500, Jim Wahlig wrote: >> >> >>>I have a script, attached, which is giving me headaches. When I run it >>>on a freshly >>>booted system, it works just fine. That is, if I do a rename (mv a b) I >>>will see the probe >>>fire for fop_rename. However, it only works the first time. The next >>>time I do a rename >>>(or link, or symlink) I don''t see anything. >>> >>>If I modify the script, by deleting all the entry/return probes above >>>rename, then it will >>>work all the time. Is there a limit to the number of probes you can >>>have in a script? >>>Is it a memory problem? >>> >>>Thanks for any help/advice. >>> >>>Jim Wahlig >>> >>>ps. actually, I''ve seen it "skip" from compound id 3 to 45 and stop >>>emitting anything at >>>all. >>> >>> >>> >>> >> >> >> >> >------------------------------------------------------------------------ > >_______________________________________________ >dtrace-discuss mailing list >dtrace-discuss at opensolaris.org > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20051028/befa40b2/attachment.html>