I want to measure incoming/outgoing HTTP traffic in bytes.
Seems simple using the examples in Brendan''s DTrace book talks.
HTTP is discussed starting on p609.
There is a data structure that looks great:
typedef struct {
string hri_uri; /* uri requested */
string hri_user; /* authenticated user */
string hri_method; /* method name (GET, POST, ...) */
string hri_useragent; /* "User-agent" header (browser) */
uint64_t hri_request; /* request id, unique at a given time */
u*int64_t hri_bytesread; /* bytes SENT to the client */*
*uint64_t hri_byteswritten; /* bytes RECEIVED from the client */*
uint32_t hri_respcode; /* response code */
} http_reqinfo_t;
but I don''t see it on illumos:
http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist
Nor do I see any http probes as mentioned int he examples
$ sudo dtrace -ln ''http*:::''
ID PROVIDER MODULE FUNCTION NAME
dtrace: failed to match http*:::: No probe matches description
Looking for any pointers on measuring http traffic.
--
- Kyle
O: +1.415.341.3430
F: +1.650.494.1676
275 Middlefield Road, Suite 50
Menlo Park, CA 94025
http://www.delphix.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120120/eb5cfa40/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 821 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120120/eb5cfa40/attachment.png>
On 1/20/12 11:20 AM, Kyle Hailey wrote:> I want to measure incoming/outgoing HTTP traffic in bytes. > Seems simple using the examples in Brendan''s DTrace book talks. > HTTP is discussed starting on p609. > > There is a data structure that looks great: > > typedef struct { > string hri_uri; /* uri requested */ > string hri_user; /* authenticated user */ > string hri_method; /* method name (GET, POST, ...) */ > string hri_useragent; /* "User-agent" header (browser) */ > uint64_t hri_request; /* request id, unique at a given time */ > u*int64_t hri_bytesread; /* bytes SENT to the client */* > *uint64_t hri_byteswritten; /* bytes RECEIVED from the client */* > uint32_t hri_respcode; /* response code */ > } http_reqinfo_t; > > > but I don''t see it on illumos: > > http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist> > > Nor do I see any http probes as mentioned int he examples > > $ sudo dtrace -ln ''http*:::'' > ID PROVIDER MODULE FUNCTION NAME > dtrace: failed to match http*:::: No probe matches description > > > Looking for any pointers on measuring http traffic.Hi Kyle, You''re right, there is nothing in illumos related to this. This snippet happens to be from a version of USDT probes added to apache. For an easy way to do this with apache you should see what Dave Pacheco did with mod-usdt here: https://github.com/davepacheco/mod_usdt. If you want this for something that isn''t apache, you''re going to have to implement USDT probes for your webserver. More info on that and the history of where Brendan''s example came from are here: http://dtrace.org/blogs/dap/2011/12/13/usdt-providers-redux/. Robert
hi Kyle On 20/01/2012 19:20, Kyle Hailey wrote:> > I want to measure incoming/outgoing HTTP traffic in bytes. > Seems simple using the examples in Brendan''s DTrace book talks. > HTTP is discussed starting on p609. >What about using the TCP provider and using port 80 as the predicate for the destination port on tcp:::receive and 80 as the source port on tcp:::send and counting payload bytes? In other words /usr/demo/dtrace/tcpbytes.d with a slight modification to count for port 80 only: tcp:::receive / args[4]->tcp_dport == 80/ { @bytes[args[2]->ip_saddr, args[4]->tcp_dport] sum(args[2]->ip_plength - args[4]->tcp_offset); } tcp:::send / args[4]->tcp_sport == 80/ { @bytes[args[2]->ip_daddr, args[4]->tcp_sport] sum(args[2]->ip_plength - args[4]->tcp_offset); } You could split sent and received bytes into separate aggregations of course. Hope this helps, Alan> There is a data structure that looks great: > > typedef struct { > string hri_uri; /* uri requested */ > string hri_user; /* authenticated user */ > string hri_method; /* method name (GET, POST, ...) */ > string hri_useragent; /* "User-agent" header (browser) */ > uint64_t hri_request; /* request id, unique at a given time */ > u*int64_t hri_bytesread; /* bytes SENT to the client */* > *uint64_t hri_byteswritten; /* bytes RECEIVED from the client */* > uint32_t hri_respcode; /* response code */ > } http_reqinfo_t; > > > but I don''t see it on illumos: > > http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist> <http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist=> > > > Nor do I see any http probes as mentioned int he examples > > $ sudo dtrace -ln ''http*:::'' > ID PROVIDER MODULE > FUNCTION NAME > dtrace: failed to match http*:::: No probe matches description > > > Looking for any pointers on measuring http traffic. > > > -- > - Kyle > > O: +1.415.341.3430 > F: +1.650.494.1676 > 275 Middlefield Road, Suite 50 > Menlo Park, CA 94025 > http://www.delphix.com <http://www.delphix.com/> > > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
That is a cool idea to filter by port 80. Unfortunately this will be too much overhead. Some of the connections over 10GbE can do 200+MB/s thus into the 10s of thousand probes a second. I think it will have to be higher up the stack (or a different tool). Thanks - Kyle On Fri, Jan 20, 2012 at 3:42 PM, Alan Maguire <alan.maguire at oracle.com>wrote:> hi Kyle > > > On 20/01/2012 19:20, Kyle Hailey wrote: > >> >> I want to measure incoming/outgoing HTTP traffic in bytes. >> Seems simple using the examples in Brendan''s DTrace book talks. >> HTTP is discussed starting on p609. >> >> What about using the TCP provider and using port 80 as the predicate > for the destination port on tcp:::receive and 80 as the source port on > tcp:::send and counting payload bytes? In other words > /usr/demo/dtrace/tcpbytes.d with a slight modification to count for > port 80 only: > > tcp:::receive > / args[4]->tcp_dport == 80/ > { > @bytes[args[2]->ip_saddr, args[4]->tcp_dport] > sum(args[2]->ip_plength - args[4]->tcp_offset); > } > > tcp:::send > / args[4]->tcp_sport == 80/ > { > @bytes[args[2]->ip_daddr, args[4]->tcp_sport] > sum(args[2]->ip_plength - args[4]->tcp_offset); > } > > You could split sent and received bytes into separate > aggregations of course. Hope this helps, > > Alan > >> There is a data structure that looks great: >> >> typedef struct { >> string hri_uri; /* uri requested */ >> string hri_user; /* authenticated user */ >> string hri_method; /* method name (GET, POST, ...) */ >> string hri_useragent; /* "User-agent" header (browser) */ >> uint64_t hri_request; /* request id, unique at a given time */ >> u*int64_t hri_bytesread; /* bytes SENT to the client */* >> *uint64_t hri_byteswritten; /* bytes RECEIVED from the client */* >> >> uint32_t hri_respcode; /* response code */ >> } http_reqinfo_t; >> >> >> but I don''t see it on illumos: >> >> http://src.illumos.org/source/**search?q=&project=illumos-** >> gate&defs=http_reqinfo_t&refs=**&path=&hist=<http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist=> >> <http://src.illumos.org/**source/search?q=&project=** >> illumos-gate&defs=http_**reqinfo_t&refs=&path=&hist=<http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist=> >> > >> >> >> Nor do I see any http probes as mentioned int he examples >> >> $ sudo dtrace -ln ''http*:::'' >> ID PROVIDER MODULE >> FUNCTION NAME >> dtrace: failed to match http*:::: No probe matches description >> >> >> Looking for any pointers on measuring http traffic. >> >> >> -- >> - Kyle >> >> O: +1.415.341.3430 >> F: +1.650.494.1676 >> 275 Middlefield Road, Suite 50 >> Menlo Park, CA 94025 >> http://www.delphix.com <http://www.delphix.com/> >> >> >> >> ______________________________**_________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >> > >-- - Kyle O: +1.415.341.3430 F: +1.650.494.1676 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120120/d2e4f3be/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 821 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120120/d2e4f3be/attachment.png>
http://labs.omniti.com/labs/project-dtrace/wiki/Applications We use this in production. It works well. I *believe* that it will be incorporated into the 2.4.x release of Apache, but I''ve honestly not tracked it that closely. As we do our own rolls, we''re happy to maintain the patcheset. The apr-util patches are very useful as well as every module becomes instrumented due to the use of "HOOKS" On Fri, Jan 20, 2012 at 7:05 PM, Kyle Hailey <kyle.hailey at delphix.com>wrote:> > That is a cool idea to filter by port 80. > Unfortunately this will be too much overhead. Some of the connections over > 10GbE can do 200+MB/s thus into the 10s of thousand probes a second. > I think it will have to be higher up the stack (or a different tool). > > Thanks > > - Kyle > > > > > On Fri, Jan 20, 2012 at 3:42 PM, Alan Maguire <alan.maguire at oracle.com>wrote: > >> hi Kyle >> >> >> On 20/01/2012 19:20, Kyle Hailey wrote: >> >>> >>> I want to measure incoming/outgoing HTTP traffic in bytes. >>> Seems simple using the examples in Brendan''s DTrace book talks. >>> HTTP is discussed starting on p609. >>> >>> What about using the TCP provider and using port 80 as the predicate >> for the destination port on tcp:::receive and 80 as the source port on >> tcp:::send and counting payload bytes? In other words >> /usr/demo/dtrace/tcpbytes.d with a slight modification to count for >> port 80 only: >> >> tcp:::receive >> / args[4]->tcp_dport == 80/ >> { >> @bytes[args[2]->ip_saddr, args[4]->tcp_dport] >> sum(args[2]->ip_plength - args[4]->tcp_offset); >> } >> >> tcp:::send >> / args[4]->tcp_sport == 80/ >> { >> @bytes[args[2]->ip_daddr, args[4]->tcp_sport] >> sum(args[2]->ip_plength - args[4]->tcp_offset); >> } >> >> You could split sent and received bytes into separate >> aggregations of course. Hope this helps, >> >> Alan >> >>> There is a data structure that looks great: >>> >>> typedef struct { >>> string hri_uri; /* uri requested */ >>> string hri_user; /* authenticated user */ >>> string hri_method; /* method name (GET, POST, ...) */ >>> string hri_useragent; /* "User-agent" header (browser) */ >>> uint64_t hri_request; /* request id, unique at a given time */ >>> u*int64_t hri_bytesread; /* bytes SENT to the client */* >>> *uint64_t hri_byteswritten; /* bytes RECEIVED from the client */* >>> >>> uint32_t hri_respcode; /* response code */ >>> } http_reqinfo_t; >>> >>> >>> but I don''t see it on illumos: >>> >>> http://src.illumos.org/source/**search?q=&project=illumos-** >>> gate&defs=http_reqinfo_t&refs=**&path=&hist=<http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist=> >>> <http://src.illumos.org/**source/search?q=&project=** >>> illumos-gate&defs=http_**reqinfo_t&refs=&path=&hist=<http://src.illumos.org/source/search?q=&project=illumos-gate&defs=http_reqinfo_t&refs=&path=&hist=> >>> > >>> >>> >>> Nor do I see any http probes as mentioned int he examples >>> >>> $ sudo dtrace -ln ''http*:::'' >>> ID PROVIDER MODULE >>> FUNCTION NAME >>> dtrace: failed to match http*:::: No probe matches description >>> >>> >>> Looking for any pointers on measuring http traffic. >>> >>> >>> -- >>> - Kyle >>> >>> O: +1.415.341.3430 >>> F: +1.650.494.1676 >>> 275 Middlefield Road, Suite 50 >>> Menlo Park, CA 94025 >>> http://www.delphix.com <http://www.delphix.com/> >>> >>> >>> >>> ______________________________**_________________ >>> dtrace-discuss mailing list >>> dtrace-discuss at opensolaris.org >>> >> >> > > > -- > - Kyle > > O: +1.415.341.3430 > F: +1.650.494.1676 > 275 Middlefield Road, Suite 50 > Menlo Park, CA 94025 > http://www.delphix.com > > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >-- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120121/df520018/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 821 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20120121/df520018/attachment.png>