Adding link and flow stats to netstat A couple of months ago, Matty had a question about tools to monitor (V)NIC utilization and specifically mentioned Brendan Gregg''s nicstat utility. Since then, we have added per flow kstats and I have also been working on extending netstat to display per-link and per-flow statistics. The column format is similar but not identical to nicstat. My original proposal was to use netstat -u. While the Solaris version didn''t use this option, some BSD variants did. As a result, I''ve changed the proposed option to -k because it is one of the few options not in use by Solaris/Linux/BSD and we are using kstats as the basis for our data. netstat -k will provide per-link statistics and netstat -K will provide per-flow statistics. Both use libcurses and will be continuously updating in a manner similar to prstat(1M) and ipfstat -t. #netstat -k Link iKB/s oKB/s iPk/s oPk/s %Util e1000g0 3.00 7.00 48.00 77.00 0.01 e1000g1 0.00 0.00 0.00 0.00 0.00 e1000g2 0.00 0.00 0.00 0.00 0.00 e1000g3 0.00 0.00 0.00 0.00 0.00 vnic3 0.00 0.00 0.00 0.00 0.00 Totals 3.0 7.00 48.00 77.00 #netstat -K Flow Link iKB/s oKB/s iPk/s oPk/s tcp1 e1000g1 0.00 0.00 0.00 0.00 udp2 e1000g2 0.00 0.00 0.00 0.00 vnic3 e1000g3 0.00 0.00 0.00 0.00 e1000g3 e1000g3 0.00 0.00 0.00 0.00 Totals 0.00 0.00 0.00 0.00 <the final output will be column aligned> Comments welcome. -Mike This message posted from opensolaris.org
Sean McGrath - Sun Microsystems Ireland
2007-Jan-05 20:23 UTC
[crossbow-discuss] crossbow stats with netstat
Michael Lim stated:
< Adding link and flow stats to netstat
<
< A couple of months ago, Matty had a question about tools to monitor (V)NIC
utilization and specifically mentioned Brendan Gregg''s nicstat utility.
<
< Since then, we have added per flow kstats and I have also been working on
extending netstat to display per-link and per-flow statistics. The column
format is similar but not identical to nicstat.
<
< My original proposal was to use netstat -u. While the Solaris version
didn''t use this option, some BSD variants did. As a result,
I''ve changed the proposed option to -k because it is one of the few
options not in use by Solaris/Linux/BSD and we are using kstats as the basis for
our data.
<
< netstat -k will provide per-link statistics and netstat -K will provide
per-flow statistics. Both use libcurses and will be continuously updating in a
manner similar to prstat(1M) and ipfstat -t.
Not a complaint, just a question. Why with libcurses ?
Long time netstat user, I''ve typically used it like mpstat or vmstat,
where
you''d still see the past stats on the previous lines. One can then
more
easily discern any patterm emerging, sudden bouts of cpu idleness, sudden
instances of collisions every N seconds.
Watching mpstat on a T2000 is hard enough :) If mpstat was libcursed then
things would be even harder :) Say 30 zones each having their own vnic and
now watch that netstat -k...
Just more of a user interface query.
Having said that, this is nice. Having the throughput in K/s etc, long time
wanted. No more estimating (num packets by mtu size :)
Perhaps an addition ? I''ve me own stat tool which does the below and
also
has in/out errors and collisions. Can we add them ?
Oh and loopback ?-) I''ll stop now.
For example;
Network Statistics
Net Ipkts Ierrs Opkts Oerrs Colls Obytes Rbytes Tbytes
lo0 5 0 5 0 0 0 0 0
ce0 4 0 9 0 0 753 255 1008
ce1 1 0 1 0 0 303 161 464
ce2 13 0 7 0 0 725 1352 2077
ce3 0 0 1 0 0 273 0 273
ce7 5 0 1 0 0 273 312 585
ce4 314 0 600 0 0 891464 22039 913503
ce5 1 0 18 0 0 4201 60 4261
xge1 1 0 1 0 0 212 58 270
Regards,
Sean.
.
<
< #netstat -k
<
< Link iKB/s oKB/s iPk/s oPk/s %Util
<
< e1000g0 3.00 7.00 48.00 77.00 0.01
< e1000g1 0.00 0.00 0.00 0.00 0.00
< e1000g2 0.00 0.00 0.00 0.00 0.00
< e1000g3 0.00 0.00 0.00 0.00 0.00
< vnic3 0.00 0.00 0.00 0.00 0.00
<
< Totals 3.0 7.00 48.00 77.00
<
< #netstat -K
<
< Flow Link iKB/s oKB/s iPk/s oPk/s
<
< tcp1 e1000g1 0.00 0.00 0.00 0.00
< udp2 e1000g2 0.00 0.00 0.00 0.00
< vnic3 e1000g3 0.00 0.00 0.00 0.00
< e1000g3 e1000g3 0.00 0.00 0.00 0.00
<
< Totals 0.00 0.00 0.00 0.00
<
< <the final output will be column aligned>
<
< Comments welcome.
<
< -Mike
<
<
< This message posted from opensolaris.org
< _______________________________________________
< crossbow-discuss mailing list
< crossbow-discuss at opensolaris.org
< http://opensolaris.org/mailman/listinfo/crossbow-discuss
--
Sean.
.
Sean McGrath - Sun Microsystems Ireland wrote:> Michael Lim stated: > < Adding link and flow stats to netstat > < > < A couple of months ago, Matty had a question about tools to monitor (V)NIC utilization and specifically mentioned Brendan Gregg''s nicstat utility. > < > < Since then, we have added per flow kstats and I have also been working on extending netstat to display per-link and per-flow statistics. The column format is similar but not identical to nicstat. > < > < My original proposal was to use netstat -u. While the Solaris version didn''t use this option, some BSD variants did. As a result, I''ve changed the proposed option to -k because it is one of the few options not in use by Solaris/Linux/BSD and we are using kstats as the basis for our data. > < > < netstat -k will provide per-link statistics and netstat -K will provide per-flow statistics. Both use libcurses and will be continuously updating in a manner similar to prstat(1M) and ipfstat -t. > > Not a complaint, just a question. Why with libcurses ?With large numbers of items (like processes, or IP Filter rules), I think it is easier to see the top 20 items instead of a repeating list. As we are introducing vnics and network flows, there is the potential for having quite a few.> Long time netstat user, I''ve typically used it like mpstat or vmstat, where > you''d still see the past stats on the previous lines. One can then more > easily discern any patterm emerging, sudden bouts of cpu idleness, sudden > instances of collisions every N seconds.If you want to monitor the stats of a single link, vnic or flow over time.. #dladm show-link -s -i 1 e1000g0 #dladm show-vnic -s -i 1 1 (for vnic1) #netrcm show-flow -s -i 1 -d e1000g1 tcp1> Watching mpstat on a T2000 is hard enough :) If mpstat was libcursed then > things would be even harder :) Say 30 zones each having their own vnic and > now watch that netstat -k...We plan to have more involved interface using extended accounting (acctadm) to view statistics over time.> Just more of a user interface query. > > Having said that, this is nice. Having the throughput in K/s etc, long time > wanted. No more estimating (num packets by mtu size :) > > Perhaps an addition ? I''ve me own stat tool which does the below and also > has in/out errors and collisions. Can we add them ? > Oh and loopback ?-) I''ll stop now.Collisions are tracked for devices but not for vnics or flows. I''ll look into loopback. Thanks for the comments. -Mike> For example; > Network Statistics > Net Ipkts Ierrs Opkts Oerrs Colls Obytes Rbytes Tbytes > lo0 5 0 5 0 0 0 0 0 > ce0 4 0 9 0 0 753 255 1008 > ce1 1 0 1 0 0 303 161 464 > ce2 13 0 7 0 0 725 1352 2077 > ce3 0 0 1 0 0 273 0 273 > ce7 5 0 1 0 0 273 312 585 > ce4 314 0 600 0 0 891464 22039 913503 > ce5 1 0 18 0 0 4201 60 4261 > xge1 1 0 1 0 0 212 58 270 > > > Regards, > Sean. > . > > < > < #netstat -k > < > < Link iKB/s oKB/s iPk/s oPk/s %Util > < > < e1000g0 3.00 7.00 48.00 77.00 0.01 > < e1000g1 0.00 0.00 0.00 0.00 0.00 > < e1000g2 0.00 0.00 0.00 0.00 0.00 > < e1000g3 0.00 0.00 0.00 0.00 0.00 > < vnic3 0.00 0.00 0.00 0.00 0.00 > < > < Totals 3.0 7.00 48.00 77.00 > < > < #netstat -K > < > < Flow Link iKB/s oKB/s iPk/s oPk/s > < > < tcp1 e1000g1 0.00 0.00 0.00 0.00 > < udp2 e1000g2 0.00 0.00 0.00 0.00 > < vnic3 e1000g3 0.00 0.00 0.00 0.00 > < e1000g3 e1000g3 0.00 0.00 0.00 0.00 > < > < Totals 0.00 0.00 0.00 0.00 > < > < <the final output will be column aligned> > < > < Comments welcome. > < > < -Mike > < > < > < This message posted from opensolaris.org > < _______________________________________________ > < crossbow-discuss mailing list > < crossbow-discuss at opensolaris.org > < http://opensolaris.org/mailman/listinfo/crossbow-discuss >
> My original proposal was to use netstat -u. While the Solaris version> didn''t use this option, some BSD variants did. As a result, I''ve > changed the proposed option to -k because it is one of the few options > not in use by Solaris/Linux/BSD and we are using kstats as the basis > for our data. FWIW, netstat -k has a long history on Solaris and was an undocumented but well-known way to dump network kstats up until (and including) Solaris 9. As there are a lot of customers still using Solaris 9, I''m not sure about recycling this option just yet. Unless the BSD -u functionality is likely to be implemented in Solaris at some point, I''m not sure why it should be a factor. -- meem
David.Comay at Sun.COM
2007-Jan-19 17:10 UTC
[crossbow-discuss] crossbow stats with netstat
> > My original proposal was to use netstat -u. While the Solaris version > > didn''t use this option, some BSD variants did. As a result, I''ve > > changed the proposed option to -k because it is one of the few options > > not in use by Solaris/Linux/BSD and we are using kstats as the basis > > for our data. > > FWIW, netstat -k has a long history on Solaris and was an undocumented but > well-known way to dump network kstats up until (and including) Solaris 9. > As there are a lot of customers still using Solaris 9, I''m not sure about > recycling this option just yet.I also am somewhat uneasy about resurrecting -k for this (different) purpose. Are there really no other letters? dsc
david.comay at sun.com wrote:>> > My original proposal was to use netstat -u. While the Solaris version >> > didn''t use this option, some BSD variants did. As a result, I''ve >> > changed the proposed option to -k because it is one of the few options >> > not in use by Solaris/Linux/BSD and we are using kstats as the basis >> > for our data. >> >> .... > > > I also am somewhat uneasy about resurrecting -k for this (different) > purpose. Are there really no other letters?What about -k/-K as modifiers to netstat''s -i. So the usage would be "netstat -i -k"...in days of yore, "netstat -i -k" would dump "netstat -i" followed by "netstat -k". While I recognise that "netstat -k" was used, how often did anyone do it like this? Anyway, when I see the output presented, to me it is "interface output" and looks vaguely like what "netstat -i" does today, so I think it (somehow) fits with that option. Also, this should mean we automatically get "netstat -i -I e1000g3 -k" or "netstat -i -I vnic3 -K"...while the default mode of output is top-style, is there a "dump it all" option too? And include matching, so we could do "-I e1000g" (for all e1000g''s), etc? Darren
> What about -k/-K as modifiers to netstat''s -i.I don''t see why that would be preferable to simply picking a letter that has not otherwise been used on Solaris. While we should not be recklessly incompatible with other Unix variants, I don''t see why we need to tiptoe around every option letter that might''ve been used on every other variant of Unix. -- meem
David.Comay at sun.com wrote:>> > My original proposal was to use netstat -u. While the Solaris version >> > didn''t use this option, some BSD variants did. As a result, I''ve >> > changed the proposed option to -k because it is one of the few options >> > not in use by Solaris/Linux/BSD and we are using kstats as the basis >> > for our data. >> >> FWIW, netstat -k has a long history on Solaris and was an undocumented >> but >> well-known way to dump network kstats up until (and including) Solaris 9. >> As there are a lot of customers still using Solaris 9, I''m not sure about >> recycling this option just yet. > > I also am somewhat uneasy about resurrecting -k for this (different) > purpose. Are there really no other letters?I actually went through the various manpages and -k/-K were the most appropriate for this. However, in light of the history of -k/-c as unsupported and subsequently removed options, reintroduction would be the wrong thing. -Mike
Peter Memishian wrote:> > What about -k/-K as modifiers to netstat''s -i. > > I don''t see why that would be preferable to simply picking a letter that > has not otherwise been used on Solaris. While we should not be recklessly > incompatible with other Unix variants, I don''t see why we need to tiptoe > around every option letter that might''ve been used on every other variant > of Unix.I would really prefer to not use an option that is currently being used by another variant of Unix. I just recall how annoying when typing ''ps -aux'' when ''ps -ef'' was the right choice. ''netstat -u'' may not be as commonplace but I have no idea how widespread it''s use has become for FreeBSD. As for the -i + modifier idea, I think this has the most merit as it will be easiear to come up with a meaningful new sub-option. I also consider the new feature to be an extension of the existing ''netstat -i'' functionality. -Mike