i''m using the following probe to calculate how many bytes are being
written by tcp write calls, by process and total:
fbt:ip:tcp_output:entry
{
this->tcpout_size = msgdsize(args[1]);
@tcpout_size[execname] = sum(this->tcpout_size);
@tcpout_size["TOTAL_TCP_OUT"] = sum(this->tcpout_size);
}
I run this probe for N seconds.
I suppose that if i get the tcpout_size["TOTAL_TCP_OUT"] value and i
divide it by N (seconds), i get the number of bytes per second written by
tcp_output calls, is this right?
But the value i get is much smaller than the one expected (i tried a check by
running an http file transfer while the probe is collecting data).
This message posted from opensolaris.org
alessioc wrote:> i''m using the following probe to calculate how many bytes are being written by tcp write calls, by process and total: > > fbt:ip:tcp_output:entry > { > this->tcpout_size = msgdsize(args[1]); > @tcpout_size[execname] = sum(this->tcpout_size); > @tcpout_size["TOTAL_TCP_OUT"] = sum(this->tcpout_size); > } > > I run this probe for N seconds. > I suppose that if i get the tcpout_size["TOTAL_TCP_OUT"] value and i divide it by N (seconds), i get the number of bytes per second written by tcp_output calls, is this right? > But the value i get is much smaller than the one expected (i tried a check by running an http file transfer while the probe is collecting data).I''m not sure that tcp_output is guaranteed to run in the context of the user process which did the write. Try tracing a routine closer to the user/kernel boundary. - Jeremy
alessioc,???
How about change "this" to "self"?
fbt:ip:tcp_output:entry
{
self->tcpout_size = msgdsize(args[1]);
@tcpout_size[execname] = sum(self->tcpout_size);
@tcpout_size["TOTAL_TCP_OUT"] = sum(self->tcpout_size);
}
======= 2006-05-08 06:13:07 ????????======>
>i''m using the following probe to calculate how many bytes are being
written by tcp write calls, by process and total:
>
>fbt:ip:tcp_output:entry
>{
> this->tcpout_size = msgdsize(args[1]);
> @tcpout_size[execname] = sum(this->tcpout_size);
> @tcpout_size["TOTAL_TCP_OUT"] = sum(this->tcpout_size);
>}
>
>I run this probe for N seconds.
>I suppose that if i get the tcpout_size["TOTAL_TCP_OUT"] value and
i divide it by N (seconds), i get the number of bytes per second written by
tcp_output calls, is this right?
>But the value i get is much smaller than the one expected (i tried a check
by running an http file transfer while the probe is collecting data).
>
>
>This message posted from opensolaris.org
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>
= = = = = = = = = = = = = = = = = = =
Cheers
????????zheh
????????zheh at 163.com
??????????2006-05-09
G''Day, On Mon, 8 May 2006, Jeremy Harris wrote:> alessioc wrote: > > i''m using the following probe to calculate how many bytes are being written by tcp write calls, by process and total: > > > > fbt:ip:tcp_output:entry > > { > > this->tcpout_size = msgdsize(args[1]); > > @tcpout_size[execname] = sum(this->tcpout_size); > > @tcpout_size["TOTAL_TCP_OUT"] = sum(this->tcpout_size); > > } > > > > I run this probe for N seconds. > > I suppose that if i get the tcpout_size["TOTAL_TCP_OUT"] value and i divide it by N (seconds), i get the number of bytes per second written by tcp_output calls, is this right? > > But the value i get is much smaller than the one expected (i tried a check by running an http file transfer while the probe is collecting data). > > I''m not sure that tcp_output is guaranteed to > run in the context of the user process which did the write. > > Try tracing a routine closer to the user/kernel boundary.TCP tracing is certainly difficult as most isn''t tied to the context of the process. tcp_output however, is much more likely to be synchronous than other tcp probes (although I don''t use it in my more serious tcp scripts, such as tcpsnoop). Just trying to replicate your test, I get, # dtrace -qn '' fbt:ip:tcp_output:entry { this->size = msgdsize(args[1]); @size[pid, execname] = sum(this->size); } dtrace:::END { printf("%6s %-16s %12s\n", "PID", "CMD", "BYTES"); printa("%6d %-16s %@12d\n", @size); } '' ^C PID CMD BYTES 599 sshd 52 3132 Xvnc 1540 5056 httpd 1049325 I transferred the following file, # ls -l /var/apache/htdocs/1mb -rw-r--r-- 1 root root 1048576 May 10 00:06 /var/apache/htdocs/1mb which is very close in size to what DTrace measured. Minus HTTP headers, I imagine. So I''m not seeing what you are seeing at all. You are measuring this on the Web server, and with a clear client cache? ... I''ll create a DTrace FAQ on the OpenSolaris site. It would start with the following, Q1. Will DTrace be released for Solaris 9? A1. No. Q2. How do you accurately measure <something TCP/IP>, <by PID/...>? A2. a) The easy way - wait for the Net provider. b) The hard way - write your test cases first, read tcp.c, ip.c, write a lot of DTrace, stay on top of kernel changes in minor updates, and test all protocols on all platforms. Although, that would make it a DTrace Frequently Actually Asked Questions, rather than the usual Frequenly Wished We Were Asked Questions. :-) cheers, Brendan
Brendan Gregg wrote:> Q2. How do you accurately measure <something TCP/IP>, <by PID/...>? > A2. a) The easy way - wait for the Net provider.If there is going to be a net (or ip, tcp, ...) provider, what kind of info do people like to gather? -- K. Poon. kacheong.poon at sun.com
> Just trying to replicate your test, I get, > > # dtrace -qn '' > fbt:ip:tcp_output:entry > { > this->size = msgdsize(args[1]); > @size[pid, execname] = sum(this->size); > } > dtrace:::END > { > printf("%6s %-16s %12s\n", "PID", > "CMD", "BYTES"); > printa("%6d %-16s %@12d\n", @size); > > PID CMD BYTES > 599 sshd 52 > 3132 Xvnc 1540 > 5056 httpd 1049325 > > I transferred the following file, > > # ls -l /var/apache/htdocs/1mb > -rw-r--r-- 1 root root 1048576 May 10 00:06 > /var/apache/htdocs/1mb > > which is very close in size to what DTrace measured. > Minus HTTP headers, > I imagine. > > So I''m not seeing what you are seeing at all. You are > measuring this on > the Web server, and with a clear client cache?I run your same probe (plus fbt:ip:tcp_input:entry). While my bowser is downloading, let''s say, OpenOffice from one of the mirrors, the Mozilla download manager says i''m downloading it at about 150k/sec. In the meanwhile i run the probe with a sample time of 10 secs (or greater), an this is the output: PID CMD BYTES 2202 seamonkey-bin 1393 So dtrace is reporting my tcp_output traffic is 1.3k/sec, and the tcp_in traffic is 0. Shouldn''t i get somewhere a number similar to the 150k/sec reported by the browser''s download manager? This message posted from opensolaris.org
G''Day, On Wed, 10 May 2006, alessioc wrote:> > Just trying to replicate your test, I get, > >[...]> > I run your same probe (plus fbt:ip:tcp_input:entry).Ahh - you didn''t ask about tcp_input! tcp_input almost definatly won''t run in the correct context, so it cannot be used in this way.> While my bowser is downloading, let''s say, OpenOffice from one of the > mirrors, the Mozilla download manager says i''m downloading it at about > 150k/sec. > In the meanwhile i run the probe with a sample time of 10 secs (or > greater), an this is the output: > PID CMD BYTES > 2202 seamonkey-bin 1393 > > So dtrace is reporting my tcp_output traffic is 1.3k/sec, and the > tcp_in traffic is 0.tcp_output''s 1.3 Kbytes/sec is correct - as you are downloading not uploading.> Shouldn''t i get somewhere a number similar to the 150k/sec reported by > the browser''s download manager?Only if tcp_input could be used in this way - which it can''t. no worries, Brendan
> tcp_output''s 1.3 Kbytes/sec is correct - as you are downloading > not > uploading. > > > Shouldn''t i get somewhere a number similar to the 150k/sec > reported by > > the browser''s download manager? > > Only if tcp_input could be used in this way - which it can''t.i see, i ingenuously thought that tcp_input could be used in a similar context of tcp_ouput. so is there any other simple way to get incoming tcp traffic like what''s being done for outgoing traffic with the probe tcp_output? (with "simple" way i mean anything which does not involve a mix of hundreds of lines of shell and dtrace scripting ;)
Roch Bourbonnais - Performance Engineering
2006-May-10 10:29 UTC
[dtrace-discuss] monitoring tcp writes
Kacheong Poon writes: > Brendan Gregg wrote: > > > Q2. How do you accurately measure <something TCP/IP>, <by PID/...>? > > A2. a) The easy way - wait for the Net provider. > > > If there is going to be a net (or ip, tcp, ...) provider, > what kind of info do people like to gather? Just to get started, how about: something that would trigger on new connections. Arguments needs to have enough info that allows predication based on : execname, pid/fd, local and remote host/ip/port, active/passive open Then some probe that triggers when data arrives/leaves IP to/from network when data arrives/leaves STREAMHEAD to/from application The args should contain the # bytes and of course the connection ID. -r > > > -- > > K. Poon. > kacheong.poon at sun.com > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
Roch Bourbonnais - Performance Engineering wrote:> Kacheong Poon writes: > > Brendan Gregg wrote: > > > > > Q2. How do you accurately measure <something TCP/IP>, <by PID/...>? > > > A2. a) The easy way - wait for the Net provider. > > > > > > If there is going to be a net (or ip, tcp, ...) provider, > > what kind of info do people like to gather? > > Just to get started, how about: > > something that would trigger on new connections. Arguments > needs to have enough info that allows predication based on : > > execname, pid/fd, local and remote host/ip/port, active/passive open > > > Then some probe that triggers > > when data arrives/leaves IP to/from network > when data arrives/leaves STREAMHEAD to/from application > > The args should contain the # bytes and of course the connection ID.Something I have used successfully in the past is based on the fact that most of the tcp_* functions take a tcp_t pointer as their first argument. The tricky part is finding the value of that pointer for the connection you are interested in. I have found the tcp_mss_set function pretty useful for this. Since this function is used at the beginning of both active and passive tcp connections, it is ideal. Here is what I use: BEGIN { mytcp = 0; } fbt::tcp_mss_set:entry /mytcp == 0/ { mytcp = arg0 } With this, I get mytcp set to the value of the tcp_t pointer of the next new TCP connection. Of course, I have to arrange for that connection to be the one I want, which could be tricky in a production environment, but isn''t hard on a relatively idle system. Of course, it might be possible to use some of the values in the tcp block to be more choosy. After that, I just use a clause like this: fbt::tcp_*:entry /mytcp==arg0/ to trace my tcp functions. Alas, this doesn''t work for tcp_input, because it doesn''t take a tcp_t pointer for its first argument. I haven''t specifically tried tracing tcp_input, but I have had the same effect by tracing from eri_intr, that is, starting at the interface rather than at tcp_input. But that was just due to what I was after at the time. You can use the same technique I did starting at tcp_input. What I did was to use speculations. You start a new speculation at tcp_input entry, and set a flag on any tcp_* call that matches mytcp. Then when you get to tcp_input return, if the flag has been set, commit the speculation, otherwise discard it. Simple. -- blu Rose are #FF0000, Violets are #0000FF. All my base are belong to you. ---------------------------------------------------------------------- Brian Utterback - OP/N1 RPE, Sun Microsystems, Inc. Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom
> > Just trying to replicate your test, I get, > > > > # dtrace -qn '' > > fbt:ip:tcp_output:entry > > { > > this->size = msgdsize(args[1]); > > @size[pid, execname] = sum(this->size); > > } > > dtrace:::END > > { > > printf("%6s %-16s %12s\n", "PID", > > "CMD", "BYTES"); > > printa("%6d %-16s %@12d\n", @size); > > > > PID CMD BYTES > > 599 sshd 52 > > 3132 Xvnc 1540 > > 5056 httpd 1049325Why not just use tcp_wput()? It is most likely to be called in the caller''s context. You need to filter M_DATA messages. - Alexander Kolbasov
Roch Bourbonnais - Performance Engineering wrote:>> Kacheong Poon writes:> > Brendan Gregg wrote: > > > > Q2. How do you accurately measure <something TCP/IP>, <by > PID/...>? > > > A2. a) The easy way - wait for the Net provider. > > > > If there is going to be a net (or ip, tcp, ...) provider, > > what kind of info do people like to gather? > > Just to get started, how about: > > something that would trigger on new connections. Arguments > needs to have enough info that allows predication based on : > > execname, pid/fd, local and remote host/ip/port, active/passive open > > > Then some probe that triggers > when data arrives/leaves IP to/from network when > data arrives/leaves STREAMHEAD to/from application > > The args should contain the # bytes and of course the connection ID.The problems with the proposed Net provider have always been a) deciding what might be needed :-) b) the context changes which the stack uses Since wanting to look at the inbound packets for a particular process is so common, how about building it into the provider to make it as easy-to-use as possible? Hide the speculation under the hood, perhaps. What about tagging every inbound packet with a system-unique ID? Would that facilitate the above? Would the overhead be too much? Better or worse than using speculation? We might end up maintaining lists of IDs on each object as it traversed the stack, due to splits and combines..... Likewise, on output, tagging {writes, segments, fragments, packets} by enough info to correlate to the owning process. As for probe locations, I''d suggest the boundaries between protocol layers (specifically, ignoring any implementation merging of layers). - Jeremy
G''Day, On Sat, 13 May 2006, Jeremy Harris wrote:> > Roch Bourbonnais - Performance Engineering wrote:>> Kacheong Poon writes: > > > Brendan Gregg wrote: > > > > > Q2. How do you accurately measure <something TCP/IP>, <by > > PID/...>? > > > > A2. a) The easy way - wait for the Net provider. > > > > > If there is going to be a net (or ip, tcp, ...) provider, > > > what kind of info do people like to gather? > > > > Just to get started, how about: > > > > something that would trigger on new connections. Arguments > > needs to have enough info that allows predication based on : > > > > execname, pid/fd, local and remote host/ip/port, active/passive open > > > > > > Then some probe that triggers > > when data arrives/leaves IP to/from network when > > data arrives/leaves STREAMHEAD to/from application > > > > The args should contain the # bytes and of course the connection ID. > > The problems with the proposed Net provider have always been > a) deciding what might be needed :-) > b) the context changes which the stack uses > > Since wanting to look at the inbound packets for a particular process > is so common, how about building it into the provider to make it as > easy-to-use as possible? Hide the speculation under the hood, perhaps. > > What about tagging every inbound packet with a system-unique ID? > Would that facilitate the above? Would the overhead be too much? > Better or worse than using speculation? > We might end up maintaining lists of IDs on each object as it traversed > the stack, due to splits and combines.....Sounds like a lot of overhead. Remember that TCP/IP events are fast. Although, I''d have to benchmark various solutions to be sure. I use the conn_t address as a unique ID, which you can access from most of the TCP/IP functions. One of the solutions I''ve thought of involves adding a procp to conn_t (or tcp_t), which would be set during functions such as tcp_connect() and sotpi_accept(). It wouldn''t be unusual to do so, as tcp_t already contains something similar, $ grep pid /usr/include/inet/tcp.h pid_t tcp_cpid; /* Process id when this was opened */ which is accessible from conn_t. Anyhow, coding in a conn_procp would be hardwiring what my TCP DTrace scripts from the toolkit do. It would make writing fbt based network scripts much easier, and would be a step in the right direction for a provider.. ... BTW - tcp_cpid doesn''t solve the TCP-by-process question, # dtrace -n ''fbt:ip:tcp_send_data:entry { @[args[0]->tcp_cpid] = count(); }'' dtrace: description ''fbt:ip:tcp_send_data:entry '' matched 1 probe ^C -1 3 6264 13 0 51 as it tracks the on-CPU PID when the session was opened, tcp_open(). Brendan
Brendan Gregg wrote:>> What about tagging every inbound packet with a system-unique ID? >> Would that facilitate the above? Would the overhead be too much? >> Better or worse than using speculation? >> We might end up maintaining lists of IDs on each object as it traversed >> the stack, due to splits and combines..... > > Sounds like a lot of overhead. Remember that TCP/IP events are fast.It can''t possibly be more than running a speculated action, later discarded, for every packet? Roughly similar (*) to an enabled probe with a predicate evaluating false, yes. I''d agree doing the latter would be preferable to packet-tagging, being more in keeping with the dtrace spirit. I''m not sure it''s always possible while providing the facilities wanted. *) I''m assuming an atomic-update counter per cpu, with permitted range set at cpu-online time. I''m ignoring the wrap issue. Cheers, Jeremy