dtrace discuss - Apr 2007 - Using dtrace to snoop messages between two Streams modules

I''m working on a case where customer has a 3rd party streams 
driver/module, called uplink, which sits over Sun''s ce driver.  This
3rd
party module is used by the telco to perform telco grade NIC failover.

The customer was given an IDR ce driver, to avoid a panic they were 
given.  The IDR driver was successful in avoiding the panic, but now the 
customer is getting many "WARNING: IP: Hardware address 
''xx:xx:xx:xx:xx:xx'' trying to be our address
010.100.001.091!", where
the hardware address and IP are the receiving host (itself).  When they 
put original ce driver back on, they don''t get these messages.  We 
understand that these messages aren''t harmful, but they want to know
why
new ce driver gets them, when original driver doesn''t.

One of the things that would be helpful is to see the messages getting 
passed between the ce driver and 3rd party driver.  The system has 
Solaris 10 installed, so wondering if it would be possible to do this 
with dtrace.

Thanks,
Jim

Jim,

I suppose printing a stack when you hit this condition might provide you 
some insight.  Only question is how to know you''ve hit the condition. 
My understanding is that ARP reports this to IP.  So you will probably 
need to set the condition to print the stack in ARP.

Looking through the arp source, AR_BOGON looks promising which would be 
a return value.

http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/inet/arp/arp.c

Dave

Jim Nissen wrote:
> I''m working on a case where customer has a 3rd party streams 
> driver/module, called uplink, which sits over Sun''s ce driver. 
This 3rd
> party module is used by the telco to perform telco grade NIC failover.
> 
> The customer was given an IDR ce driver, to avoid a panic they were 
> given.  The IDR driver was successful in avoiding the panic, but now the 
> customer is getting many "WARNING: IP: Hardware address 
> ''xx:xx:xx:xx:xx:xx'' trying to be our address
010.100.001.091!", where
> the hardware address and IP are the receiving host (itself).  When they 
> put original ce driver back on, they don''t get these messages.  We
> understand that these messages aren''t harmful, but they want to
know why
> new ce driver gets them, when original driver doesn''t.
> 
> One of the things that would be helpful is to see the messages getting 
> passed between the ce driver and 3rd party driver.  The system has 
> Solaris 10 installed, so wondering if it would be possible to do this 
> with dtrace.
> 
> Thanks,
> Jim
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

--

Dave,
Yes, we already have the stack, from IP''s point-of-view.  I also see 
that these messages only get produced is when AR_CN_BOGON gets set (and 
see arp code where this happens), so experimenting with trying to 
capture when this is set.

However, one of the confusing things is that the orig ce driver never 
gets these messages, and therefore AR_CN_BOGON is never getting set, and 
ip_arp_news is not getting called.  So, somehow, the new driver has 
packets coming through this codepath, and orig driver doesn''t.

Therefore, I thought it would be handy to be able to see packet content 
summary, as they get passed upward, from ce.  Then, we could compare 
between new/old driver.

Jim
> Jim,
> 
> I suppose printing a stack when you hit this condition might provide you 
> some insight.  Only question is how to know you''ve hit the
condition. My
> understanding is that ARP reports this to IP.  So you will probably need 
> to set the condition to print the stack in ARP.
> 
> Looking through the arp source, AR_BOGON looks promising which would be 
> a return value.
> 
>
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/inet/arp/arp.c
> 
> 
> Dave
> 
> Jim Nissen wrote:
>> I''m working on a case where customer has a 3rd party streams 
>> driver/module, called uplink, which sits over Sun''s ce driver.
This
>> 3rd party module is used by the telco to perform telco grade NIC 
>> failover.
>>
>> The customer was given an IDR ce driver, to avoid a panic they were 
>> given.  The IDR driver was successful in avoiding the panic, but now 
>> the customer is getting many "WARNING: IP: Hardware address 
>> ''xx:xx:xx:xx:xx:xx'' trying to be our address
010.100.001.091!", where
>> the hardware address and IP are the receiving host (itself).  When 
>> they put original ce driver back on, they don''t get these
messages.
>> We understand that these messages aren''t harmful, but they
want to
>> know why new ce driver gets them, when original driver
doesn''t.
>>
>> One of the things that would be helpful is to see the messages getting 
>> passed between the ce driver and 3rd party driver.  The system has 
>> Solaris 10 installed, so wondering if it would be possible to do this 
>> with dtrace.
>>
>> Thanks,
>> Jim
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
> 
>

Hi,

I have some occasionally hanging IO read/write issue to diagnose. 
IOs which normally take milliseconds, sometimes take 20-30 seconds.

I''m putting a DTrace script together which uses io provider and syscall
provider to get more complete picture of where the time is spent - I
don''t
want to reinvent the wheel though.

It looks like in the DTrace toolkit there are separate scripts like iosnoop
and dtruss etc, but does anyone have a script which snoops all possible
layers on an IO operation''s path? (it''s fine to trace the IO
for a single
PID only)

We are using thread-emulated asynch IO on vxfs, which in turn is using EMC
powerpath, on solaris 10/sparc.

Thanks,
Tanel