dtrace discuss - Sep 2006 - [networking-discuss] Re: Re: DTrace Network Provider

Darren.reed at sun.com wrote:
> Can you comment on what the provider is being designed to do,
> if it won''t allow for following a packet through the kernel?
> 
> If I look at the design presented on the web page, it tells us
> what probes will be implemented and some ways in which they can
> be used.  What appears to be missing is a statement of "this is
> what we''re trying to achieve", from a more loftly level.
> 
> Are you just trying to provide stable names for interesting
> places to observe information inside IP or something else?
> 
> Whatever the goal, it might be would worth mentioning it.
Seconded!

for example, there is a class of problems that falls under the
following description (from Nicolas Williams)
> The thing that makes it easy to thread function calls/returns is that
> they all happen in the same thread of execution and in sequence.  The
> same is not necessarily true of "packets."  There''s
multicast, and
> SOCK_RAW, fragmentation, etc...  Also, do you want to trace flows all
> the way from the link layer through applications (e.g., NFS, Apache)?
> That seems even more daunting.
> 
> How do you thread receipt of fragmented packets into a nice call flow-
> style of report?
but I think the previous discussions have determined that this
is not currently in the scope of the current provider, and dtrace
scripts that need to solve the above must use associative arrays
or other existing means. 

This is a reasonable scope-definition to make, but then  the
network-savviness of an audience that seeks to do the following is unclear:
> ip:::send
> /args[2]->ip_ver == 4/
> {
> 	args[2] contains valid IPv4 members, straight from the IPv4 header
> }
> 
> ip:::send
> /args[2]->ip_ver == 6/
> {
> 	args[2] contains valid IPv6 members, straight from the IPv6 header
> }
System Administrators are probably more accustomed to well-known
tools like ethereal/snoop/tcpdump that allows them to filter above
info out from the wire. Are we trying to find out if/why the packet
got all the way to ip''s send routine, but didn''t make it out
to the wire?
IOW info like the following:
> It''s not the kstat value that''s important, it''s
being able to see
> what the packet looked like that when the kernel decided to ldrop it.
i.e., we want to get ancillary information from "well-known"
points in the stack (where "well-known" is some commonly accepted
rfc/state-machine/whatever), then perhaps it makes more sense to  first
enhance the existing mib provider to provide the pertinent 
ancillary information (SNMP MIBs already have the charter of
portability, interoperability etc. that are needed for stability)?

--Sowmini

On Fri, Sep 29, 2006 at 09:50:37AM -0400, Sowmini.Varadhan at Sun.COM
wrote:
> System Administrators are probably more accustomed to well-known
> tools like ethereal/snoop/tcpdump that allows them to filter above
> info out from the wire. Are we trying to find out if/why the packet
> got all the way to ip''s send routine, but didn''t make it
out to the wire?
> IOW info like the following:
I think the "threading the packet processing" idea is something that
someone interested in seeing _how_ Solaris handles TCP/IP will be
interested in.  But when you''re trying to solve actual problems I
suspect it''ll be something else that one needs.

BTW, is there some key that one can use as a key to an associative array
to track a single tcp "output" down to the network layer?  Some sort
of
packet ID or STREAMS message ID, or something of the sort?

I really think that threading packet processing is just not a well-
thought out feature and not necessarily easy to implement with the
current infrastructure.  I don''t think the project team should be
burdened with adding it.

Folks who need such tracing can rely on higher-cost associative arrays
and/or unstable fbt probes.

Nicolas Williams writes:
> BTW, is there some key that one can use as a key to an associative array
> to track a single tcp "output" down to the network layer?  Some
sort of
> packet ID or STREAMS message ID, or something of the sort?
No.

It''s a bit of an ugly problem.  Messages are transformed in the stack
(allocate new dblk, copy data over with some changes; think IPsec
ESP), split into separate messages (TCP segmentation and IP
fragmentation), joined together (reassembly), and even spawn new
messages (ack-clocked transmission) that are logically related.

When I trace messages in the stack, I usually use an associative array
with either the mblk_t or dblk_t address as a key.  It''s functional
for some trivial debugging cases, but when things get complicated or
involve long-duration problems or post-mortem analysis, it''s lame.
> I really think that threading packet processing is just not a well-
> thought out feature and not necessarily easy to implement with the
> current infrastructure.  I don''t think the project team should be
> burdened with adding it.
Agreed.  More importantly, it''s not related to the current proposal.
(It''s an data-oriented view rather than a function- or action-oriented
view.)

But I do think that it''d be a useful feature for known customer
problems ("I see packet X on the wire, but Solaris seems to ignore it
... why?"), if someone could pull it off.

-- 
James Carlson, KISS Network                    <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677