I see Intel has released a new tool. Oh, it requires some patches to the kernel to record latency times. Good thing people don''t mind patching their kernels, eh? So who can write the equivalent latencytop.d the fastest? ;-) http://www.latencytop.org/ -- cburgess at qnx.com
On Fri, Jan 18, 2008 at 04:33:17PM -0500, Colin Burgess wrote:> I see Intel has released a new tool. Oh, it requires some patches to > the kernel to record > latency times. Good thing people don''t mind patching their kernels, eh? > > So who can write the equivalent latencytop.d the fastest? ;-)Brendan, I''m sure -- if he hasn''t by the time I''m writing this... (Of course, that said it''s a pretty lame tool. Isn''t obvious that this is providing such paltry information that it''s raising more questions than answers? Fine, better than nothing -- but not much better...) - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Sun Microsystems Fishworks. http://blogs.sun.com/bmc
G''Day Folks, On Fri, Jan 18, 2008 at 04:52:58PM -0500, Bryan Cantrill wrote:> > On Fri, Jan 18, 2008 at 04:33:17PM -0500, Colin Burgess wrote: > > I see Intel has released a new tool. Oh, it requires some patches to > > the kernel to record > > latency times. Good thing people don''t mind patching their kernels, eh? > > > > So who can write the equivalent latencytop.d the fastest? ;-) > > Brendan, I''m sure -- if he hasn''t by the time I''m writing this...Aw, come on - I''ve barely had half an hour since I saw the message - couldn''t you have waited a bit longer before tempting me to reply? :) It has about 40 statically defined trace points and I''ve coded 8 so far. A bunch of these are specific to Linux, so I''d need to think of what was the closest useful match for Solaris. Anyway, here''s where I''m at: # ./latencytop.d Tracing... Hit Ctrl-C to end. Cause Count Maximum Average mutex lock 10 80 usec 30 usec Reading from file 450 22822 usec 1900 usec Cause Count Maximum Average mutex lock 2 11 usec 10 usec Reading from file 474 26716 usec 1836 usec Cause Count Maximum Average mutex lock 2 85 usec 48 usec Reading from file 392 29972 usec 2312 usec Cause Count Maximum Average mutex lock 19 1379 usec 574 usec Writing to file 162 43011 usec 1655 usec Reading from file 302 80847 usec 2945 usec Cause Count Maximum Average mutex lock 1 8 usec 8 usec Reading from file 468 22978 usec 1813 usec [...] Yes, I added a "Count" column since it proved of limited use without one. Of course, bounding this to Count/Maximum/Average is also of limited use - I''d really like to include distribution plots as well. Source so far is (hot off the keyboard, so beware of typos): -------------------------------------------------------------- #!/usr/sbin/dtrace -Cs #pragma D option quiet #define RW_READER 0x1 dtrace:::BEGIN { printf("Tracing... Hit Ctrl-C to end.\n"); } /* * Reading from file * Writing to file */ io:::start { start[args[0]->b_edev, args[0]->b_blkno] = timestamp; } io:::done /start[args[0]->b_edev, args[0]->b_blkno]/ { this->delta = timestamp - start[args[0]->b_edev, args[0]->b_blkno]; this->cause = args[0]->b_flags & B_READ ? "Reading from file" : "Writing to file"; @lcount[this->cause] = count(); @lmax[this->cause] = max(this->delta); @lavg[this->cause] = avg(this->delta); } /* * mutex lock */ lockstat:::adaptive-block { @lcount["mutex lock"] = count(); @lmax["mutex lock"] = max(arg1); @lavg["mutex lock"] = avg(arg1); } /* * rwsem read lock * rwsem write lock */ lockstat:::rw-block { this->cause = arg2 == RW_READER ? "rwsem read lock" : "rwsum write lock"; @lcount[this->cause] = count(); @lmax[this->cause] = max(arg1); @lavg[this->cause] = avg(arg1); } /* * Unlinking file */ syscall::unlink:entry { self->start_unlink = timestamp; } syscall::unlink:return /self->start_unlink/ { this->delta = timestamp - self->start_unlink; @lcount["Unlink file"] = count(); @lmax["Unlink file"] = max(this->delta); @lavg["Unlink file"] = avg(this->delta); self->start_unlink = 0; } /* * sync system call */ syscall::syssync:entry { self->start_syssync = timestamp; } syscall::syssync:return /self->start_syssync/ { this->delta = timestamp - self->start_syssync; @lcount["sync system call"] = count(); @lmax["sync system call"] = max(this->delta); @lavg["sync system call"] = avg(this->delta); self->start_syssync = 0; } /* * fsync system call */ syscall::fdsync:entry { self->start_fdsync = timestamp; } syscall::fdsync:return /self->start_fdsync/ { this->delta = timestamp - self->start_fdsync; @lcount["fsync system call"] = count(); @lmax["fsync system call"] = max(this->delta); @lavg["fsync system call"] = avg(this->delta); self->start_fdsync = 0; } /* * Print output summary */ profile:::tick-1s, dtrace:::END { normalize(@lmax, 1000); normalize(@lavg, 1000); printf(" %-32s %12s %14s %14s\n", "Cause", "Count", "Maximum", "Average"); printa(" %-32s %@12d %@9d usec %@9d usec\n", @lcount, @lmax, @lavg); printf("\n"); trunc(@lcount); trunc(@lmax); trunc(@lavg); } -------------------------------------------------------------- I''ll add some more later - I have other work to be doing right now. :) If anyone would like to work on some of the other metrics, please do and post the bits here.> (Of course, that said it''s a pretty lame tool. Isn''t obvious that this > is providing such paltry information that it''s raising more questions than > answers? Fine, better than nothing -- but not much better...)Yes, not that impressive. Although getting customers to look and think about latency is certainly worthwhile, and the start of the road to DTrace. Brendan -- Brendan [CA, USA]
Well I see that Brendan did reply to the OSNews link to this. He basically shot them down at hardcoding the instrumentation - as he should have! :-) Shame on Intel - they should know better! Colin Bryan Cantrill wrote: On Fri, Jan 18, 2008 at 04:33:17PM -0500, Colin Burgess wrote:> I see Intel has released a new tool. Oh, it requires some patches to > the kernel to record > latency times. Good thing people don''t mind patching their kernels, eh? > > So who can write the equivalent latencytop.d the fastest? ;-)Brendan, I''m sure -- if he hasn''t by the time I''m writing this... (Of course, that said it''s a pretty lame tool. Isn''t obvious that this is providing such paltry information that it''s raising more questions than answers? Fine, better than nothing -- but not much better...) - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Sun Microsystems Fishworks. http://blogs.sun.com/bmc <http://blogs.sun.com/bmc> -- cburgess at qnx.com <mailto:cburgess at qnx.com>
LOL - you rock, Brendan. You know, at the VERY least they could have used systemtap, I guess. Colin Brendan Gregg - Sun Microsystems wrote: G''Day Folks, On Fri, Jan 18, 2008 at 04:52:58PM -0500, Bryan Cantrill wrote:> > On Fri, Jan 18, 2008 at 04:33:17PM -0500, Colin Burgess wrote: > > I see Intel has released a new tool. Oh, it requires some patches to > > the kernel to record > > latency times. Good thing people don''t mind patching their kernels, eh?> > > > So who can write the equivalent latencytop.d the fastest? ;-) > > Brendan, I''m sure -- if he hasn''t by the time I''m writing this...Aw, come on - I''ve barely had half an hour since I saw the message - couldn''t you have waited a bit longer before tempting me to reply? :) It has about 40 statically defined trace points and I''ve coded 8 so far. A bunch of these are specific to Linux, so I''d need to think of what was the closest useful match for Solaris. Anyway, here''s where I''m at: # ./latencytop.d Tracing... Hit Ctrl-C to end. Cause Count Maximum Average mutex lock 10 80 usec 30 usec Reading from file 450 22822 usec 1900 usec Cause Count Maximum Average mutex lock 2 11 usec 10 usec Reading from file 474 26716 usec 1836 usec Cause Count Maximum Average mutex lock 2 85 usec 48 usec Reading from file 392 29972 usec 2312 usec Cause Count Maximum Average mutex lock 19 1379 usec 574 usec Writing to file 162 43011 usec 1655 usec Reading from file 302 80847 usec 2945 usec Cause Count Maximum Average mutex lock 1 8 usec 8 usec Reading from file 468 22978 usec 1813 usec [...] Yes, I added a "Count" column since it proved of limited use without one. Of course, bounding this to Count/Maximum/Average is also of limited use - I''d really like to include distribution plots as well. Source so far is (hot off the keyboard, so beware of typos): -------------------------------------------------------------- #!/usr/sbin/dtrace -Cs #pragma D option quiet #define RW_READER 0x1 dtrace:::BEGIN { printf("Tracing... Hit Ctrl-C to end.\n"); } /* * Reading from file * Writing to file */ io:::start { start[args[0]->b_edev, args[0]->b_blkno] = timestamp; } io:::done /start[args[0]->b_edev, args[0]->b_blkno]/ { this->delta = timestamp - start[args[0]->b_edev, args[0]->b_blkno]; this->cause = args[0]->b_flags & B_READ ? "Reading from file" : "Writing to file"; @lcount[this->cause] = count(); @lmax[this->cause] = max(this->delta); @lavg[this->cause] = avg(this->delta); } /* * mutex lock */ lockstat:::adaptive-block { @lcount["mutex lock"] = count(); @lmax["mutex lock"] = max(arg1); @lavg["mutex lock"] = avg(arg1); } /* * rwsem read lock * rwsem write lock */ lockstat:::rw-block { this->cause = arg2 == RW_READER ? "rwsem read lock" : "rwsum write lock"; @lcount[this->cause] = count(); @lmax[this->cause] = max(arg1); @lavg[this->cause] = avg(arg1); } /* * Unlinking file */ syscall::unlink:entry { self->start_unlink = timestamp; } syscall::unlink:return /self->start_unlink/ { this->delta = timestamp - self->start_unlink; @lcount["Unlink file"] = count(); @lmax["Unlink file"] = max(this->delta); @lavg["Unlink file"] = avg(this->delta); self->start_unlink = 0; } /* * sync system call */ syscall::syssync:entry { self->start_syssync = timestamp; } syscall::syssync:return /self->start_syssync/ { this->delta = timestamp - self->start_syssync; @lcount["sync system call"] = count(); @lmax["sync system call"] = max(this->delta); @lavg["sync system call"] = avg(this->delta); self->start_syssync = 0; } /* * fsync system call */ syscall::fdsync:entry { self->start_fdsync = timestamp; } syscall::fdsync:return /self->start_fdsync/ { this->delta = timestamp - self->start_fdsync; @lcount["fsync system call"] = count(); @lmax["fsync system call"] = max(this->delta); @lavg["fsync system call"] = avg(this->delta); self->start_fdsync = 0; } /* * Print output summary */ profile:::tick-1s, dtrace:::END { normalize(@lmax, 1000); normalize(@lavg, 1000); printf(" %-32s %12s %14s %14s\n", "Cause", "Count", "Maximum", "Average"); printa(" %-32s %@12d %@9d usec %@9d usec\n", @lcount, @lmax, @lavg); printf("\n"); trunc(@lcount); trunc(@lmax); trunc(@lavg); } -------------------------------------------------------------- I''ll add some more later - I have other work to be doing right now. :) If anyone would like to work on some of the other metrics, please do and post the bits here.> (Of course, that said it''s a pretty lame tool. Isn''t obvious that this > is providing such paltry information that it''s raising more questions than> answers? Fine, better than nothing -- but not much better...)Yes, not that impressive. Although getting customers to look and think about latency is certainly worthwhile, and the start of the road to DTrace. Brendan -- Brendan [CA, USA] -- cburgess at qnx.com <mailto:cburgess at qnx.com>
On Fri, 2008-01-18 at 16:33 -0500, Colin Burgess wrote:> I see Intel has released a new tool. Oh, it requires some patches to > the kernel to record > latency times. Good thing people don''t mind patching their kernels, eh? > > So who can write the equivalent latencytop.d the fastest? ;-) > > http://www.latencytop.org/What I find interesting about these projects that Intel spawns for Linux (PowerTOP, LatencyTOP and couple of others) is that regardless of internal implementation they are very useful end user tools. Here at Sun we seem to be missing interest in creating such things. Which is a bit of a shame. They are ideal vehicles for disseminating DTrace knowledge and exposing neophytes to the raw power of DTrace. To be fair Greg''s DTrace toolkit helps in that respect, but still it sets the bar pretty high for anybody who would like to use it. It is easy to poke fun at LatencyTOP, but asking the right question could sometimes be even more important than being able to deliver the answer. Just my 2c. Thanks, Roman. P.S. I was able to extend the battery time of my Linux laptop 1.5x using PowerTOP. Can the same thing be done with DTrace? Perhaps it can, but I don''t think I can code it up.
On Jan 19, 2008 10:30 AM, Roman Shaposhnik <rvs at sun.com> wrote:> On Fri, 2008-01-18 at 16:33 -0500, Colin Burgess wrote: > > I see Intel has released a new tool. Oh, it requires some patches to > > the kernel to record > > latency times. Good thing people don''t mind patching their kernels, eh? > > > > So who can write the equivalent latencytop.d the fastest? ;-) > > > > http://www.latencytop.org/ > > What I find interesting about these projects that Intel > spawns for Linux (PowerTOP, LatencyTOP and couple of others) > is that regardless of internal implementation they are > very useful end user tools. Here at Sun we seem to be > missing interest in creating such things. Which is a bit of a > shame. They are ideal vehicles for disseminating DTrace > knowledge and exposing neophytes to the raw power of DTrace. > > To be fair Greg''s DTrace toolkit helps in that respect, but > still it sets the bar pretty high for anybody who would > like to use it. > > It is easy to poke fun at LatencyTOP, but asking the right > question could sometimes be even more important than > being able to deliver the answer. > > Just my 2c. > > Thanks, > Roman. > > P.S. I was able to extend the battery time of my Linux laptop > 1.5x using PowerTOP. Can the same thing be done with DTrace? > Perhaps it can, but I don''t think I can code it up.Solaris PowerTOP is almost done. http://www.opensolaris.org/os/project/tesla/Work/Powertop/ -Aubrey Intel OpenSolaris Team
I don''t think anyone disagrees that measuring latency is a bad idea. Note Brendan''s comments in http://osnews.com/permalink?296801 <http://osnews.com/permalink?296801> "Encouraging customers to look at latencies for performance analysis is really important." ... "If this tool does get customers to think more carefully about latency metrics, then that will certainly be valuable. All roads lead to DTrace." However forcing people to patch their kernel to do it is not going to attract neophytes, no matter how simply the user space tool is. That''s the beauty of dynamic tracing systems. You can answer questions that the original designers of the OS didn''t anticipate (or had time to think about) that you would ask. Cheers, Colin Aubrey Li wrote: On Jan 19, 2008 10:30 AM, Roman Shaposhnik <mailto:rvs at sun.com> <rvs at sun.com> wrote:> On Fri, 2008-01-18 at 16:33 -0500, Colin Burgess wrote: > > I see Intel has released a new tool. Oh, it requires some patches to > > the kernel to record > > latency times. Good thing people don''t mind patching their kernels, eh?> > > > So who can write the equivalent latencytop.d the fastest? ;-) > > > > http://www.latencytop.org/ <http://www.latencytop.org/> > > What I find interesting about these projects that Intel > spawns for Linux (PowerTOP, LatencyTOP and couple of others) > is that regardless of internal implementation they are > very useful end user tools. Here at Sun we seem to be > missing interest in creating such things. Which is a bit of a > shame. They are ideal vehicles for disseminating DTrace > knowledge and exposing neophytes to the raw power of DTrace. > > To be fair Greg''s DTrace toolkit helps in that respect, but > still it sets the bar pretty high for anybody who would > like to use it. > > It is easy to poke fun at LatencyTOP, but asking the right > question could sometimes be even more important than > being able to deliver the answer. > > Just my 2c. > > Thanks, > Roman. > > P.S. I was able to extend the battery time of my Linux laptop > 1.5x using PowerTOP. Can the same thing be done with DTrace? > Perhaps it can, but I don''t think I can code it up.Solaris PowerTOP is almost done. http://www.opensolaris.org/os/project/tesla/Work/Powertop/ <http://www.opensolaris.org/os/project/tesla/Work/Powertop/> -Aubrey Intel OpenSolaris Team -- cburgess at qnx.com <mailto:cburgess at qnx.com>
On Jan 19, 2008 9:04 AM, Colin Burgess <cburgess at qnx.com> wrote:> Well I see that Brendan did reply to the OSNews link to this. He basically > shot them down at hardcoding the instrumentation - as he should have! :-) > > Shame on Intel - they should know better! > > Colin >I''m not a member of the linux LatencyTOP team. I haven''t gotten a chance to see how is this tool implemented. But I totally disagree with you about it''s a shame. Regardless of internal implementation, It''s interesting and be very useful. At least, for end users, it helps to visualize system latencies. -Aubrey Intel OpenSolaris Team
Perhaps the wording was a bit strong, and I apologize for that. I perhaps should have prefaced it with some kudos to the intel team for encouraging people to look at their systems as a whole, and measure such things. However I do stand by my assessment that it is way less than ideal to include new static hooks to measure this information. Cheers, Colin Aubrey Li wrote: On Jan 19, 2008 9:04 AM, Colin Burgess <mailto:cburgess at qnx.com> <cburgess at qnx.com> wrote:> Well I see that Brendan did reply to the OSNews link to this. Hebasically> shot them down at hardcoding the instrumentation - as he should have! :-) > > Shame on Intel - they should know better! > > Colin >I''m not a member of the linux LatencyTOP team. I haven''t gotten a chance to see how is this tool implemented. But I totally disagree with you about it''s a shame. Regardless of internal implementation, It''s interesting and be very useful. At least, for end users, it helps to visualize system latencies. -Aubrey Intel OpenSolaris Team -- cburgess at qnx.com <mailto:cburgess at qnx.com>
Fletcher Cocquyt
2008-Jan-21 21:39 UTC
[dtrace-discuss] tcptop error: failed to resolve SS_TCP_FAST_ACCEPT: Unknown variable name
Hi, I am trying to debug the bottle neck(s) in a Solaris 10 Mailman/Spamassassin/Sendmail VMWare VM and get the following error from tcptop: god at smtp:~ 1:35pm 103 # ./tcptop dtrace: failed to compile script /dev/fd/11: line 40: failed to resolve SS_TCP_FAST_ACCEPT: Unknown variable name thanks for any insight, Fletcher. -----Original Message----- From: dtrace-discuss-bounces at opensolaris.org [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Colin Burgess Sent: Friday, January 18, 2008 1:33 PM To: dtrace-discuss at opensolaris.org Subject: [dtrace-discuss] LatencyTop I see Intel has released a new tool. Oh, it requires some patches to the kernel to record latency times. Good thing people don''t mind patching their kernels, eh? So who can write the equivalent latencytop.d the fastest? ;-) http://www.latencytop.org/ -- cburgess at qnx.com _______________________________________________ dtrace-discuss mailing list dtrace-discuss at opensolaris.org
Chris Yoder (kernel)
2008-Jan-21 21:45 UTC
[dtrace-discuss] tcptop error: failed to resolve SS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 01:39:03PM -0800, Fletcher Cocquyt wrote:> Hi, I am trying to debug the bottle neck(s) in a Solaris 10 > Mailman/Spamassassin/Sendmail VMWare VM and get the following error from > tcptop: > > god at smtp:~ 1:35pm 103 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line 40: failed to resolve > SS_TCP_FAST_ACCEPT: Unknown variable nameFrom http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-January/001021.html> I think SS_TCP_FAST_ACCEPT has been changed to SS_DIRECT > by a recent putback, although I was unable to isolate it. (no time!) > Changing my local copy of tcpsnoop.d to use SS_DIRECT fixed > the problem for me.
Fletcher Cocquyt
2008-Jan-21 21:55 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
Followup - this system has a lot of kernel activity and I/O - (top typically
shows CPU > 50% kernel) - but the hotkernel blorked with this (eventhough
load avg was only ~2 and command line is responsive):
god at smtp:~ 1:41pm 114 # ./hotkernel
Sampling... Hit Ctrl-C to end.
dtrace: processing aborted: Abort due to systemic unresponsiveness
FUNCTION COUNT PCNT
I''m working my way down the toolkit list - any help on pinpointing the
bottlenecks with the appropriate 1st pass tools appreciated.
Here is some iotop output - nothing surprising here - sendmail, spamd and
mailman (python) are generating I/O:
2008 Jan 21 13:49:54, load: 1.35, disk_r: 32 KB, disk_w: 2424 KB
UID PID PPID CMD DEVICE MAJ MIN D BYTES
0 13413 13412 sendmail sd0 61 0 W 2048
0 13411 13406 sendmail sd0 61 0 W 4096
0 13409 13370 sendmail sd0 61 0 W 5120
0 3 0 fsflush sd0 61 0 W 8192
0 13420 1 sendmail sd0 61 0 W 22528
555 3809 3140 spamd sd0 61 0 R 32768
0 13419 496 sendmail sd0 61 0 W 41984
0 13412 496 sendmail sd0 61 0 W 44032
0 13370 496 sendmail sd0 61 0 W 50688
0 13413 1 sendmail sd0 61 0 W 51712
0 13406 496 sendmail sd0 61 0 W 71680
0 13414 496 sendmail sd0 61 0 W 96256
35 24406 24400 python2.4 sd0 61 0 W 172032
0 0 0 sched sd0 61 0 W 318464
555 3809 3140 spamd sd0 61 0 W 405504
35 24409 24400 python2.4 sd0 61 0 W 1006592
Ideally I''d like to know what the fixable (tunable) bottlenecks are on
a
system that otherwise has plenty of CPU and memory available
Thanks
-----Original Message-----
From: dtrace-discuss-bounces at opensolaris.org
[mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Fletcher
Cocquyt
Sent: Monday, January 21, 2008 1:39 PM
To: dtrace-discuss at opensolaris.org
Subject: [dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT:
Unknown variable name
Hi, I am trying to debug the bottle neck(s) in a Solaris 10
Mailman/Spamassassin/Sendmail VMWare VM and get the following error from
tcptop:
god at smtp:~ 1:35pm 103 # ./tcptop
dtrace: failed to compile script /dev/fd/11: line 40: failed to resolve
SS_TCP_FAST_ACCEPT: Unknown variable name
thanks for any insight,
Fletcher.
Brendan Gregg - Sun Microsystems
2008-Jan-21 22:00 UTC
[dtrace-discuss] tcptop error: failed to resolve SS_TCP_FAST_ACCEPT: Unknown variable name
G''Day, On Mon, Jan 21, 2008 at 01:39:03PM -0800, Fletcher Cocquyt wrote:> Hi, I am trying to debug the bottle neck(s) in a Solaris 10 > Mailman/Spamassassin/Sendmail VMWare VM and get the following error from > tcptop: > > god at smtp:~ 1:35pm 103 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line 40: failed to resolve > SS_TCP_FAST_ACCEPT: Unknown variable nameTry a newer version of tcptop. The latest has: # grep -n SS_TCP_FAST_ACCEPT tcptop 66:# 20-Apr-2006 " " Fixed SS_TCP_FAST_ACCEPT bug in build 31+. 157: * 0x00200000 has been hardcoded. It was SS_TCP_FAST_ACCEPT, but was Latest version of the toolkit (0.99) here: http://www.brendangregg.com/dtrace.html#DTraceToolkit Brendan -- Brendan [CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-21 22:16 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 01:55:36PM -0800, Fletcher Cocquyt wrote:> Followup - this system has a lot of kernel activity and I/O - (top typically > shows CPU > 50% kernel) - but the hotkernel blorked with this (eventhough > load avg was only ~2 and command line is responsive): > > god at smtp:~ 1:41pm 114 # ./hotkernel > Sampling... Hit Ctrl-C to end. > dtrace: processing aborted: Abort due to systemic unresponsivenessThe system is so busy DTrace has decided to play it safe and abort... Based on a few hunches, try these: - interstat 1 look for a network driver burning CPU - pidpersec.d from the DTraceToolkit (or "sar -c 1 100" if DTrace won''t behave) look for lots of short lived processes - "procsystime -coT" from the DTraceToolkit look for frequent syscalls burning CPU time - dtrace -n ''profile-101 { @[stack(5)] = count(); }'' (this has a slower profile rate than hotuser) look for hot kernel stacks Brendan -- Brendan [CA, USA]
Fletcher Cocquyt
2008-Jan-21 22:17 UTC
[dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable name
Replaced SS_TCP_FAST_ACCEPT with SS_DIRECT in tcptop per the thread you cited - now I get a new error: god at irt-smtp-02:~ 2:14pm 133 # ./tcptop dtrace: failed to compile script /dev/fd/11: line 163: failed to resolve `tcp_g_q: Unknown symbol name I got it from here: http://www.brendangregg.com/DTrace/tcptop is that not up to date? Thanks -----Original Message----- From: dtrace-discuss-bounces at opensolaris.org [mailto:dtrace-discuss-bounces at opensolaris.org] On Behalf Of Chris Yoder (kernel) Sent: Monday, January 21, 2008 1:45 PM To: dtrace-discuss at opensolaris.org Subject: Re: [dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable name On Mon, Jan 21, 2008 at 01:39:03PM -0800, Fletcher Cocquyt wrote:> Hi, I am trying to debug the bottle neck(s) in a Solaris 10 > Mailman/Spamassassin/Sendmail VMWare VM and get the following error from > tcptop: > > god at smtp:~ 1:35pm 103 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line 40: failed to resolve > SS_TCP_FAST_ACCEPT: Unknown variable nameFrom http://mail.opensolaris.org/pipermail/dtrace-discuss/2006-January/001021.htm l> I think SS_TCP_FAST_ACCEPT has been changed to SS_DIRECT > by a recent putback, although I was unable to isolate it. (no time!) > Changing my local copy of tcpsnoop.d to use SS_DIRECT fixed > the problem for me._______________________________________________ dtrace-discuss mailing list dtrace-discuss at opensolaris.org
Brendan Gregg - Sun Microsystems
2008-Jan-21 22:25 UTC
[dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 02:17:46PM -0800, Fletcher Cocquyt wrote:> Replaced SS_TCP_FAST_ACCEPT with SS_DIRECT in tcptop per the thread you > cited - now I get a new error: > > god at irt-smtp-02:~ 2:14pm 133 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line 163: failed to resolve > `tcp_g_q: Unknown symbol name > > I got it from here: > http://www.brendangregg.com/DTrace/tcptop > is that not up to date?Sorry about that - I''ve kept the DTraceToolkit bundle up to date, but not individual copies of those scripts in other locations. I''ll either update that copy, or link it to the DTraceToolkit bundle when I get a chance. Stefan Parvu has an up to date HTML browsable version of the toolkit here: http://www.nbl.fi/~nbl97/solaris/dtrace/dtt_testing.html Click on "0.99". Brendan -- Brendan [CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-21 22:27 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 02:16:02PM -0800, Brendan Gregg - Sun Microsystems wrote:> On Mon, Jan 21, 2008 at 01:55:36PM -0800, Fletcher Cocquyt wrote: > > Followup - this system has a lot of kernel activity and I/O - (top typically > > shows CPU > 50% kernel) - but the hotkernel blorked with this (eventhough > > load avg was only ~2 and command line is responsive): > > > > god at smtp:~ 1:41pm 114 # ./hotkernel > > Sampling... Hit Ctrl-C to end. > > dtrace: processing aborted: Abort due to systemic unresponsiveness > > The system is so busy DTrace has decided to play it safe and abort... > > Based on a few hunches, try these: > > - interstat 1 > look for a network driver burning CPUSorry (typo) - intrstat Brendan -- Brendan [CA, USA]
Fletcher Cocquyt
2008-Jan-21 22:48 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
Forgive me, where do I find ''interstat'' ?
Also, where can I get Sun::Solaris::Kstat for prustat?
god at smtp-02:~ 2:32pm 146 # perl -MCPAN -e ''install
Sun::Solaris::Kstat''
[snip]
Fetching with LWP:
ftp://ftp.perl.org/pub/CPAN/modules/03modlist.data.gz
Going to read /export/home/fcocquyt/.cpan/sources/modules/03modlist.data.gz
Going to write /export/home/fcocquyt/.cpan/Metadata
Warning: Cannot install Sun::Solaris::Kstat, don''t know what it is.
Try the command
i /Sun::Solaris::Kstat/
to find objects with matching identifiers.
god at smtp-02:~ 2:33pm 147 # perl -MCPAN -e ''install
Solaris::Kstat''
[snip]
CPAN.pm: Going to build A/AB/ABURLISON/Solaris-0.05a.tar.gz
Solaris:: is only supported on Solaris 2.5.1, 2.6 & 2.7
Running make test
Make had some problems, maybe interrupted? Won''t test
Running make install
Make had some problems, maybe interrupted? Won''t install
thanks
-----Original Message-----
From: Brendan Gregg - Sun Microsystems [mailto:brendan at sun.com]
Sent: Monday, January 21, 2008 2:16 PM
To: Fletcher Cocquyt
Cc: dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] tcptop error: failed to
resolveSS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 01:55:36PM -0800, Fletcher Cocquyt
wrote:> Followup - this system has a lot of kernel activity and I/O - (top
typically> shows CPU > 50% kernel) - but the hotkernel blorked with this
(eventhough
> load avg was only ~2 and command line is responsive):
>
> god at smtp:~ 1:41pm 114 # ./hotkernel
> Sampling... Hit Ctrl-C to end.
> dtrace: processing aborted: Abort due to systemic unresponsiveness
The system is so busy DTrace has decided to play it safe and abort...
Based on a few hunches, try these:
- interstat 1
look for a network driver burning CPU
- pidpersec.d from the DTraceToolkit
(or "sar -c 1 100" if DTrace won''t behave)
look for lots of short lived processes
- "procsystime -coT" from the DTraceToolkit
look for frequent syscalls burning CPU time
- dtrace -n ''profile-101 { @[stack(5)] = count(); }''
(this has a slower profile rate than hotuser)
look for hot kernel stacks
Brendan
--
Brendan
[CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-21 23:37 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
On Mon, Jan 21, 2008 at 02:48:47PM -0800, Fletcher Cocquyt wrote:> Forgive me, where do I find ''interstat'' ? > > Also, where can I get Sun::Solaris::Kstat for prustat?It''s probably already under /usr/perl5/5.8.4/lib - it''s a vendor (Sun) supplied package. prustat was written as a demo tool - it might be useful, but it will probably fail due to a kernel change. I wrote it when I was a customer to make a point to Sun that this is the sort of tool that customers would like. It turns out that supporting this tool for customers would require stable network providers for DTrace, a project that is still in progress. I never put prustat into the DTraceToolkit because it wasn''t stable enough, despite it providing key resource utilisations by process (which is wonderful, and made possible by DTrace). If anyone hasn''t seen it, it looks like this: # prustat -ct 20 5 PID %CPU %Mem %Disk %Net COMM 22301 78.84 3.16 0.00 0.00 setiathome 22635 4.09 0.20 69.11 0.00 tar 440 2.76 45.39 0.00 0.00 Xsun 2618 0.31 14.34 0.00 0.00 mozilla-bin 22640 3.87 1.49 0.12 0.00 dtrace 582 2.04 2.16 0.00 0.00 gnome-terminal 576 0.02 2.80 0.00 0.00 nautilus 2299 0.33 1.99 0.00 0.00 acroread 22641 0.00 0.00 1.84 0.00 upsmonitor 578 0.37 1.46 0.00 0.00 gnome-panel 574 0.41 1.31 0.00 0.00 metacity 6504 0.00 1.23 0.00 0.00 nautilus-throbb 593 0.04 1.05 0.00 0.00 mixer_applet2 556 0.00 1.05 0.00 0.00 gconfd-2 549 0.00 0.94 0.00 0.00 gnome-session 6510 0.00 0.93 0.00 0.00 nautilus-text-v 591 0.02 0.83 0.00 0.00 galf-server 21551 0.00 0.56 0.00 0.00 dtterm 4789 0.10 0.45 0.00 0.00 vncviewer 553 0.00 0.43 0.00 0.00 gnome-volcheck the screen updates like William LeFebvre''s top. Let me stress again - prustat was written to demonstrate an idea, but is currently unstable as a tool. Brendan -- Brendan [CA, USA]
Fletcher Cocquyt
2008-Jan-22 00:21 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
Thanks - prustat works great once I point it at Sun''s perl (I was using a newer install) I''m going to record some snapshots when the contention is happening... What if I wanted to quantify the latency (wait times) due to DNS lookups (I suspect I could benefit from a local caching install - but I want a "before") picture so I can show how much better it is using a local DNS cache... Thanks, Fletcher -----Original Message----- From: Brendan Gregg - Sun Microsystems [mailto:brendan at sun.com] Sent: Monday, January 21, 2008 3:38 PM To: Fletcher Cocquyt Cc: dtrace-discuss at opensolaris.org Subject: Re: [dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name On Mon, Jan 21, 2008 at 02:48:47PM -0800, Fletcher Cocquyt wrote:> Forgive me, where do I find ''interstat'' ? > > Also, where can I get Sun::Solaris::Kstat for prustat?It''s probably already under /usr/perl5/5.8.4/lib - it''s a vendor (Sun) supplied package. prustat was written as a demo tool - it might be useful, but it will probably fail due to a kernel change. I wrote it when I was a customer to make a point to Sun that this is the sort of tool that customers would like. It turns out that supporting this tool for customers would require stable network providers for DTrace, a project that is still in progress. I never put prustat into the DTraceToolkit because it wasn''t stable enough, despite it providing key resource utilisations by process (which is wonderful, and made possible by DTrace). If anyone hasn''t seen it, it looks like this: # prustat -ct 20 5 PID %CPU %Mem %Disk %Net COMM 22301 78.84 3.16 0.00 0.00 setiathome 22635 4.09 0.20 69.11 0.00 tar 440 2.76 45.39 0.00 0.00 Xsun 2618 0.31 14.34 0.00 0.00 mozilla-bin 22640 3.87 1.49 0.12 0.00 dtrace 582 2.04 2.16 0.00 0.00 gnome-terminal 576 0.02 2.80 0.00 0.00 nautilus 2299 0.33 1.99 0.00 0.00 acroread 22641 0.00 0.00 1.84 0.00 upsmonitor 578 0.37 1.46 0.00 0.00 gnome-panel 574 0.41 1.31 0.00 0.00 metacity 6504 0.00 1.23 0.00 0.00 nautilus-throbb 593 0.04 1.05 0.00 0.00 mixer_applet2 556 0.00 1.05 0.00 0.00 gconfd-2 549 0.00 0.94 0.00 0.00 gnome-session 6510 0.00 0.93 0.00 0.00 nautilus-text-v 591 0.02 0.83 0.00 0.00 galf-server 21551 0.00 0.56 0.00 0.00 dtterm 4789 0.10 0.45 0.00 0.00 vncviewer 553 0.00 0.43 0.00 0.00 gnome-volcheck the screen updates like William LeFebvre''s top. Let me stress again - prustat was written to demonstrate an idea, but is currently unstable as a tool. Brendan -- Brendan [CA, USA]
Hi, I tried both: http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop_snv.html and http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop.html they both give this error: ./tcptop_nevada ./tcptop_nevada[80]: syntax error at line 86 : `<<'' unmatched Is there a central dtrace repository under SVN revision control? Thanks -----Original Message----- From: Brendan Gregg - Sun Microsystems [mailto:brendan at sun.com] Sent: Monday, January 21, 2008 2:25 PM To: Fletcher Cocquyt Cc: Chris.Yoder at sun.com; dtrace-discuss at opensolaris.org Subject: Re: [dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable name On Mon, Jan 21, 2008 at 02:17:46PM -0800, Fletcher Cocquyt wrote:> Replaced SS_TCP_FAST_ACCEPT with SS_DIRECT in tcptop per the thread you > cited - now I get a new error: > > god at irt-smtp-02:~ 2:14pm 133 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line 163: failed to resolve > `tcp_g_q: Unknown symbol name > > I got it from here: > http://www.brendangregg.com/DTrace/tcptop > is that not up to date?Sorry about that - I''ve kept the DTraceToolkit bundle up to date, but not individual copies of those scripts in other locations. I''ll either update that copy, or link it to the DTraceToolkit bundle when I get a chance. Stefan Parvu has an up to date HTML browsable version of the toolkit here: http://www.nbl.fi/~nbl97/solaris/dtrace/dtt_testing.html Click on "0.99". Brendan -- Brendan [CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-24 19:37 UTC
[dtrace-discuss] where is a working tcptop?
G''Day Fletcher, On Thu, Jan 24, 2008 at 10:40:43AM -0800, Fletcher Cocquyt wrote:> Hi, I tried both: > http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop_snv.html and > http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop.html > > they both give this error: > > ./tcptop_nevada > ./tcptop_nevada[80]: syntax error at line 86 : `<<'' unmatchedHmm, sounds like a HTML-izing bug. The latest version should always be here: http://www.brendangregg.com/DTraceToolkit-latest.tar.gz Brendan -- Brendan [CA, USA]
Same error with the ''-latest'': god at irt-smtp-02:~ 1:58pm 55 # DTraceToolkit-0.99/Net/tcptop dtrace: failed to compile script /dev/fd/11: line 166: failed to resolve `tcp_g_q: Unknown symbol name I''d like to get this working as network captures are showing retransmits... Thanks -----Original Message----- From: Brendan Gregg - Sun Microsystems [mailto:brendan at sun.com] Sent: Thursday, January 24, 2008 11:37 AM To: Fletcher Cocquyt Cc: Chris.Yoder at sun.com; dtrace-discuss at opensolaris.org Subject: Re: where is a working tcptop? G''Day Fletcher, On Thu, Jan 24, 2008 at 10:40:43AM -0800, Fletcher Cocquyt wrote:> Hi, I tried both: > http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop_snv.html and > http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Net/tcptop.html > > they both give this error: > > ./tcptop_nevada > ./tcptop_nevada[80]: syntax error at line 86 : `<<'' unmatchedHmm, sounds like a HTML-izing bug. The latest version should always be here: http://www.brendangregg.com/DTraceToolkit-latest.tar.gz Brendan -- Brendan [CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-24 22:47 UTC
[dtrace-discuss] where is a working tcptop?
G''Day Fletcher, On Thu, Jan 24, 2008 at 02:39:19PM -0800, Fletcher Cocquyt wrote:> Same error with the ''-latest'': > > god at irt-smtp-02:~ 1:58pm 55 # DTraceToolkit-0.99/Net/tcptop > dtrace: failed to compile script /dev/fd/11: line 166: failed to resolve > `tcp_g_q: Unknown symbol nameThat''s not quite the latest: DTraceToolkit-0.99> MANPATH=Man man tcptop ... OS Solaris 10 3/05 DTraceToolkit-0.99> MANPATH=Man man tcptop_snv ... OS Solaris Nevada / OpenSolaris, circa late 2007 Try tcptop_snv. I put the OS field in the man pages for this rev, not only to point out Solaris version support, but also for MacOS X and other OSes with DTrace.> I''d like to get this working as network captures are showing retransmits... > ThanksThis won''t help directly with retransmits. What is your retransmit ratio? Brendan -- Brendan [CA, USA]
Different error with that one: god at irt-smtp-02:~ 2:49pm 53 # DTraceToolkit-0.99/Net/tcptop_snv dtrace: failed to compile script /dev/fd/11: line 198: probe description fbt:ip:tcp_xchg:entry does not match any probes Retransmit rate is low (9/30000) - but the fact I''m seeing any warrants further analysis Thanks -----Original Message----- From: Brendan Gregg - Sun Microsystems [mailto:brendan at sun.com] Sent: Thursday, January 24, 2008 2:48 PM To: Fletcher Cocquyt Cc: Chris.Yoder at sun.com; dtrace-discuss at opensolaris.org Subject: Re: where is a working tcptop? G''Day Fletcher, On Thu, Jan 24, 2008 at 02:39:19PM -0800, Fletcher Cocquyt wrote:> Same error with the ''-latest'': > > god at irt-smtp-02:~ 1:58pm 55 # DTraceToolkit-0.99/Net/tcptop > dtrace: failed to compile script /dev/fd/11: line 166: failed to resolve > `tcp_g_q: Unknown symbol nameThat''s not quite the latest: DTraceToolkit-0.99> MANPATH=Man man tcptop ... OS Solaris 10 3/05 DTraceToolkit-0.99> MANPATH=Man man tcptop_snv ... OS Solaris Nevada / OpenSolaris, circa late 2007 Try tcptop_snv. I put the OS field in the man pages for this rev, not only to point out Solaris version support, but also for MacOS X and other OSes with DTrace.> I''d like to get this working as network captures are showingretransmits...> ThanksThis won''t help directly with retransmits. What is your retransmit ratio? Brendan -- Brendan [CA, USA]
Brendan Gregg - Sun Microsystems
2008-Jan-25 00:08 UTC
[dtrace-discuss] where is a working tcptop?
G''Day Fletcher, On Thu, Jan 24, 2008 at 03:26:23PM -0800, Fletcher Cocquyt wrote:> Different error with that one: > > god at irt-smtp-02:~ 2:49pm 53 # DTraceToolkit-0.99/Net/tcptop_snv > dtrace: failed to compile script /dev/fd/11: line 198: probe description > fbt:ip:tcp_xchg:entry does not match any probesWow, this sounds like the history of tcpsnoop bugs. Sorry about this; it is inevitable until there are stable network providers. The Notes/ALLfbt_notes.txt file in the DTraceToolkit-0.99 will make a lot of sense to you right now, if you haven''t already read it (the man page for the tcptop scripts refer to it); it''s also online here: http://www.nbl.fi/~nbl97/solaris/dtrace/099html/Notes/ALLfbt_notes.txt.html Did you say what version of Solaris are you on? I don''t have access to a Solaris server that does this; however these changes in tcp are usually easy to fix (so far). So you, Fletcher, or someone else who has access to the Solaris version, can help out by posting a fix. Or wait until Solaris has more stable support for these probes (which is in progress).> Retransmit rate is low (9/30000) - but the fact I''m seeing any warrants > further analysisMaybe. My workstation is at 63649/186843238, which is 0.034%. You are at 0.03%. Whatever you have, I have too. :) Brendan -- Brendan [CA, USA]
Fletcher Cocquyt
2008-Jan-25 18:48 UTC
[dtrace-discuss] where is a working tcptop for Solaris 10 8/07 s10x_u4wos_12b X86?
This is my version
10:46am 61 > more /etc/release
Solaris 10 8/07 s10x_u4wos_12b X86
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007
>Did you say what version of Solaris are you on?
Thanks
Brendan Gregg - Sun Microsystems
2008-Jan-25 19:14 UTC
[dtrace-discuss] where is a working tcptop for Solaris 10 8/07 s10x_u4wos_12b X86?
G''Day Fletcher, On Fri, Jan 25, 2008 at 10:48:03AM -0800, Fletcher Cocquyt wrote:> This is my version > 10:46am 61 > more /etc/release > Solaris 10 8/07 s10x_u4wos_12b X86 > Copyright 2007 Sun Microsystems, Inc. All Rights Reserved. > Use is subject to license terms. > Assembled 16 August 2007Thanks. If a fellow dtrace-discusser is running this version and can write and test a fix, please post it here. I would like to fix this myself, but maintaining the DTraceToolkit is not part of my current role at Sun - it''s also not a Sun product. I can''t spend a couple of hours here to setup both x86 and sparc test machines running this version, write a fix, test the fix through a variety of network loads, etc. What is part of my job at Sun is to help create stable network providers, after which these tcp tools should always work. Brendan -- Brendan [CA, USA]
Fletcher Cocquyt
2008-Feb-26 19:29 UTC
[dtrace-discuss] tcptop error: failed to resolveSS_TCP_FAST_ACCEPT: Unknown variable name
I wanted to post a closing message for this thread. I resolved the system contention on this Solaris VM - although it was not by way of Dtrace. Turns out the VMWare settings in the vmx file for this Solaris VM were not optimal:> memsize = "2048" (old file) > sched.mem.max = "256" (old file) - (If sched.mem.max is smaller than memsize, > the balloon driver can start consuming memory (especially if the Guest > Operating system application has peaky memory usage). However, this setting > can cause the balloon driver to retain it''s hold on memory continuously, even > if the Guest Operating System requires it again. This causes Guest Operating > System to start swapping and will slow down considerably.)Now I recognize the vmware-memctld process consuming so much CPU was a red flag for this. Once the two settings were brought into line (by using VC and checking Memory resources "unlimited") the VM functioned 100x better (responsiveness, workload throughput etc_ Thanks On 1/21/08 2:16 PM, "Brendan Gregg - Sun Microsystems" <brendan at sun.com> wrote:> On Mon, Jan 21, 2008 at 01:55:36PM -0800, Fletcher Cocquyt wrote: >> Followup - this system has a lot of kernel activity and I/O - (top typically >> shows CPU > 50% kernel) - but the hotkernel blorked with this (eventhough >> load avg was only ~2 and command line is responsive): >> >> god at smtp:~ 1:41pm 114 # ./hotkernel >> Sampling... Hit Ctrl-C to end. >> dtrace: processing aborted: Abort due to systemic unresponsiveness > > The system is so busy DTrace has decided to play it safe and abort... > > Based on a few hunches, try these: > > - interstat 1 > look for a network driver burning CPU > > - pidpersec.d from the DTraceToolkit > (or "sar -c 1 100" if DTrace won''t behave) > look for lots of short lived processes > > - "procsystime -coT" from the DTraceToolkit > look for frequent syscalls burning CPU time > > - dtrace -n ''profile-101 { @[stack(5)] = count(); }'' > (this has a slower profile rate than hotuser) > look for hot kernel stacks > > Brendan
Hi Brendan, I have been trying to get the tcptop script working on one of our servers. Its givin me the same issue whcih has been reported earlier . I am using Solaris 5.10 /i86pc. The error is the "tcp_q_g Unable to resolve symbol error" . I have verified it for the latest version based on your comments for the scripts and find them to be latest. Any help would be great. I get errors on tcptop_snv as :probe description fbt:ip:tcp_xchg:entry does not match any probes. Thanks in Advance, Satya. -- This message posted from opensolaris.org
Samuel Bercovici
2009-Mar-19 20:03 UTC
[dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable n
Please check the following: replace the following line in the tcptop: this->queuep = (queue_t *)`tcp_g_q; /* ` */ with this line: this->queuep = (queue_t *)((tcp_stack_t *)arg8)->tcps_g_q; -Sam.> Replaced SS_TCP_FAST_ACCEPT with SS_DIRECT in tcptop > per the thread you > cited - now I get a new error: > > god at irt-smtp-02:~ 2:14pm 133 # ./tcptop > dtrace: failed to compile script /dev/fd/11: line > 163: failed to resolve > `tcp_g_q: Unknown symbol name > > I got it from here: > http://www.brendangregg.com/DTrace/tcptop > is that not up to date? > > > Thanks > > > -----Original Message----- > From: dtrace-discuss-bounces at opensolaris.org > [mailto:dtrace-discuss-bounces at opensolaris.org] On > Behalf Of Chris Yoder > (kernel) > Sent: Monday, January 21, 2008 1:45 PM > To: dtrace-discuss at opensolaris.org > Subject: Re: [dtrace-discuss] tcptop error: failed > toresolve > SS_TCP_FAST_ACCEPT: Unknown variable name > > On Mon, Jan 21, 2008 at 01:39:03PM -0800, Fletcher > Cocquyt wrote: > > Hi, I am trying to debug the bottle neck(s) in a > Solaris 10 > > Mailman/Spamassassin/Sendmail VMWare VM and get the > following error from > > tcptop: > > > > god at smtp:~ 1:35pm 103 # ./tcptop > > dtrace: failed to compile script /dev/fd/11: line > 40: failed to resolve > > SS_TCP_FAST_ACCEPT: Unknown variable name > > From > http://mail.opensolaris.org/pipermail/dtrace-discuss/2 > 006-January/001021.htm > l > > > I think SS_TCP_FAST_ACCEPT has been changed to > SS_DIRECT > > by a recent putback, although I was unable to > isolate it. (no time!) > > Changing my local copy of tcpsnoop.d to use > SS_DIRECT fixed > > the problem for me. > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- This message posted from opensolaris.org
Paul Anderson
2009-Mar-19 20:12 UTC
[dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable n
Following this suggestion I get the following error: dtrace: failed to compile script /dev/fd/11: line 198: probe description fbt:ip: tcp_xchg:entry does not match any probes Paul> Date: Thu, 19 Mar 2009 13:03:59 -0700 > From: sam_bercovici at yahoo.com > To: dtrace-discuss at opensolaris.org > Subject: Re: [dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Unknown variable n > > Please check the following: > > replace the following line in the tcptop: > this->queuep = (queue_t *)`tcp_g_q; /* ` */ > > with this line: > this->queuep = (queue_t *)((tcp_stack_t *)arg8)->tcps_g_q; > > -Sam. > > > > Replaced SS_TCP_FAST_ACCEPT with SS_DIRECT in tcptop > > per the thread you > > cited - now I get a new error: > > > > god at irt-smtp-02:~ 2:14pm 133 # ./tcptop > > dtrace: failed to compile script /dev/fd/11: line > > 163: failed to resolve > > `tcp_g_q: Unknown symbol name > > > > I got it from here: > > http://www.brendangregg.com/DTrace/tcptop > > is that not up to date? > > > > > > Thanks > > > > > > -----Original Message----- > > From: dtrace-discuss-bounces at opensolaris.org > > [mailto:dtrace-discuss-bounces at opensolaris.org] On > > Behalf Of Chris Yoder > > (kernel) > > Sent: Monday, January 21, 2008 1:45 PM > > To: dtrace-discuss at opensolaris.org > > Subject: Re: [dtrace-discuss] tcptop error: failed > > toresolve > > SS_TCP_FAST_ACCEPT: Unknown variable name > > > > On Mon, Jan 21, 2008 at 01:39:03PM -0800, Fletcher > > Cocquyt wrote: > > > Hi, I am trying to debug the bottle neck(s) in a > > Solaris 10 > > > Mailman/Spamassassin/Sendmail VMWare VM and get the > > following error from > > > tcptop: > > > > > > god at smtp:~ 1:35pm 103 # ./tcptop > > > dtrace: failed to compile script /dev/fd/11: line > > 40: failed to resolve > > > SS_TCP_FAST_ACCEPT: Unknown variable name > > > > From > > http://mail.opensolaris.org/pipermail/dtrace-discuss/2 > > 006-January/001021.htm > > l > > > > > I think SS_TCP_FAST_ACCEPT has been changed to > > SS_DIRECT > > > by a recent putback, although I was unable to > > isolate it. (no time!) > > > Changing my local copy of tcpsnoop.d to use > > SS_DIRECT fixed > > > the problem for me. > > _______________________________________________ > > dtrace-discuss mailing list > > dtrace-discuss at opensolaris.org > > > > > > _______________________________________________ > > dtrace-discuss mailing list > > dtrace-discuss at opensolaris.org > -- > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org_________________________________________________________________ Windows Live? Contacts: Organize your contact list. http://windowslive.com/connect/post/marcusatmicrosoft.spaces.live.com-Blog-cns!503D1D86EBB2B53C!2285.entry?ocid=TXT_TAGLM_WL_UGC_Contacts_032009 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20090319/8ff11864/attachment.html>
Samuel Bercovici
2009-Mar-19 23:14 UTC
[dtrace-discuss] tcptop error: failed toresolve SS_TCP_FAST_ACCEPT: Un
oops, sorry but this is whta seams to be needed: this->queuep = (queue_t *)((tcp_stack_t *)args[7)->tcps_g_q;> <style type="text/css"> > > .hmmessage P > { > margin:0px; > padding:0px > } > body.hmmessage > { > font-size: 10pt; > font-family:Verdana > } > > </style> > <div id="jive-html-wrapper-div"> > > Following this suggestion I get the following > error:<br><br>dtrace: failed to compile script > /dev/fd/11: line 198: probe description fbt:ip: > tcp_xchg:entry does not match any > probes<br><br><br>Paul<br><br>> Date: Thu, 19 Mar > 2009 13:03:59 -0700<br>> From: > sam_bercovici at yahoo.com<br>> To: > dtrace-discuss at opensolaris.org<br>> Subject: Re: > [dtrace-discuss] tcptop error: failed toresolve > SS_TCP_FAST_ACCEPT: Unknown variable n<br>> > <br>> Please check the following:<br>> <br>> > replace the following line in the tcptop:<br>> > this->queuep = (queue_t *)`tcp_g_q; /* ` > */<br>> <br>> with this line:<br>> > this->queuep = (queue_t *)((tcp_stack_t > *)arg8)->tcps_g_q;<br>> <br>> -Sam.<br>> > <br>> <br>> > Replaced SS_TCP_FAST_ACCEPT > with SS_DIRECT in tcptop<br>> > per the thread > you<br>> > cited - now I get a new > error:<br>> > <br>> > god at irt-smtp-02:~ > 2:14pm 133 # ./tcptop<br>> > dtrace: failed to > compile script /dev/fd/11: line<br>> > 163: > failed to resolve<br>> > `tcp_g_q: Unknown > symbol name<br>> > <br>> > I got it from > here:<br>> > > http://www.brendangregg.com/DTrace/tcptop<br>> > > is that not up to date?<br>> > <br>> > > <br>> > Thanks<br>> > <br>> > > <br>> > -----Original Message-----<br>> > > From: dtrace-discuss-bounces at opensolaris.org<br>> > > [mailto:dtrace-discuss-bounces at opensolaris.org] > On<br>> > Behalf Of Chris Yoder<br>> > > (kernel)<br>> > Sent: Monday, January 21, 2008 > 1:45 PM<br>> > To: > dtrace-discuss at opensolaris.org<br>> > Subject: > Re: [dtrace-discuss] tcptop error: failed<br>> > > toresolve<br>> > SS_TCP_FAST_ACCEPT: > Unknown variable name<br>> > <br>> > On > Mon, Jan 21, 2008 at 01:39:03PM -0800, > Fletcher<br>> > Cocquyt wrote:<br>> > > > Hi, I am trying to debug the bottle neck(s) in > a<br>> > Solaris 10<br>> > > > Mailman/Spamassassin/Sendmail VMWare VM and get > the<br>> > following error from<br>> > > > tcptop:<br>> > > <br>> > > > god at smtp:~ 1:35pm 103 # ./tcptop<br>> > > > dtrace: failed to compile script /dev/fd/11: > line<br>> > 40: failed to resolve<br>> > > > SS_TCP_FAST_ACCEPT: Unknown variable > name<br>> > <br>> > From<br>> > > http://mail.opensolaris.org/pipermail/dtrace-discuss/2 > <br>> > 006-January/001021.htm<br>> > > l<br>> > <br>> > > I think > SS_TCP_FAST_ACCEPT has been changed to<br>> > > SS_DIRECT<br>> > > by a recent putback, > although I was unable to<br>> > isolate it. (no > time!)<br>> > > Changing my local copy of > tcpsnoop.d to use<br>> > SS_DIRECT > fixed<br>> > > the problem for me.<br>> > > > _______________________________________________<br>> > ; > dtrace-discuss mailing list<br>> > > dtrace-discuss at opensolaris.org<br>> > <br>> > > <br>> > > _______________________________________________<br>> > ; > dtrace-discuss mailing list<br>> > > dtrace-discuss at opensolaris.org<br>> -- <br>> > This message posted from opensolaris.org<br>> > _______________________________________________<br>> > ; dtrace-discuss mailing list<br>> > dtrace-discuss at opensolaris.org<br><br /><hr />Windows > Live? Contacts: Organize your contact list. <a > href=''http://windowslive.com/connect/post/marcusatmicr > osoft.spaces.live.com-Blog-cns!503D1D86EBB2B53C!2285.e > ntry?ocid=TXT_TAGLM_WL_UGC_Contacts_032009'' > target=''_new''>Check it out.</a> > </div>_______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- This message posted from opensolaris.org