dtrace discuss - Jan 2007 - fbt:ip:tcp_xchg:entry does not exists

Hello,

I am trying to use one of scripts from Dtrace toolkit and get:
bash-3.00# dtrace -l|wc -l
   47351
bash-3.00# cat /etc/release 
                        Solaris 10 6/06 s10x_u2wos_09a X86
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 09 June 2006
bash-3.00# uname -a
SunOS e2 5.10 Generic_118855-19 i86pc i386 i86pc
bash-3.00# ./tcpsnoop.d
dtrace: failed to compile script ./tcpsnoop.d: line 233: probe description
fbt:ip:tcp_xchg:entry does not match any probes

Indeed, there is no such probe:
bash-3.00# dtrace -l|wc -l
   47351
bash-3.00# dtrace -l|grep tcp_xchg

On the other x86 system I get it:
# cat /etc/release
                        Solaris 10 6/06 s10x_u2wos_09a X86
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 09 June 2006
# uname -a
SunOS bolek 5.10 Generic_118855-19 i86pc i386 i86pc
# dtrace -l|grep tcp_xchg
25995        fbt                ip                          tcp_xchg entry
25996        fbt                ip                          tcp_xchg return
36350        fbt               nca                          tcp_xchg entry
36351        fbt               nca                          tcp_xchg return

Can anyone explain me what''s going on ?

przemol

----------------------------------------------------------------------
Nawet 50MB w jednym liscie>>> http://link.interia.pl/f19e7

Anyone know what happened to fbt:ip:tcp_xchg:entry probe ?

przemol

On Thu, Jan 11, 2007 at 12:09:55PM +0100, przemolicc at poczta.fm
wrote:
> Hello,
> 
> I am trying to use one of scripts from Dtrace toolkit and get:
> bash-3.00# dtrace -l|wc -l
>    47351
> bash-3.00# cat /etc/release 
>                         Solaris 10 6/06 s10x_u2wos_09a X86
>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>                         Use is subject to license terms.
>                              Assembled 09 June 2006
> bash-3.00# uname -a
> SunOS e2 5.10 Generic_118855-19 i86pc i386 i86pc
> bash-3.00# ./tcpsnoop.d
> dtrace: failed to compile script ./tcpsnoop.d: line 233: probe description
fbt:ip:tcp_xchg:entry does not match any probes
> 
> Indeed, there is no such probe:
> bash-3.00# dtrace -l|wc -l
>    47351
> bash-3.00# dtrace -l|grep tcp_xchg
> 
> On the other x86 system I get it:
> # cat /etc/release
>                         Solaris 10 6/06 s10x_u2wos_09a X86
>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>                         Use is subject to license terms.
>                              Assembled 09 June 2006
> # uname -a
> SunOS bolek 5.10 Generic_118855-19 i86pc i386 i86pc
> # dtrace -l|grep tcp_xchg
> 25995        fbt                ip                          tcp_xchg entry
> 25996        fbt                ip                          tcp_xchg return
> 36350        fbt               nca                          tcp_xchg entry
> 36351        fbt               nca                          tcp_xchg return
> 
> Can anyone explain me what''s going on ?
> 
> przemol
> 
> ----------------------------------------------------------------------
> Nawet 50MB w jednym liscie>>> http://link.interia.pl/f19e7
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
----------------------------------------------------------------------
Bryka za cwajhundert ojro! Zobacz >> http://link.interia.pl/f19df

Being an fbt probe, and there has to be function of that name in the 
kernel, what do you get from each system for:

     /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg

You also might compare the outputs of modinfo on each system.  Could 
there have been a patch that eliminated that function name?  (yea, I 
know: that''s unlikely.)

Chip

przemolicc at poczta.fm wrote:
> Anyone know what happened to fbt:ip:tcp_xchg:entry probe ?
>
> przemol
>
> On Thu, Jan 11, 2007 at 12:09:55PM +0100, przemolicc at poczta.fm wrote:
>   
>> Hello,
>>
>> I am trying to use one of scripts from Dtrace toolkit and get:
>> bash-3.00# dtrace -l|wc -l
>>    47351
>> bash-3.00# cat /etc/release 
>>                         Solaris 10 6/06 s10x_u2wos_09a X86
>>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>>                         Use is subject to license terms.
>>                              Assembled 09 June 2006
>> bash-3.00# uname -a
>> SunOS e2 5.10 Generic_118855-19 i86pc i386 i86pc
>> bash-3.00# ./tcpsnoop.d
>> dtrace: failed to compile script ./tcpsnoop.d: line 233: probe
description fbt:ip:tcp_xchg:entry does not match any probes
>>
>> Indeed, there is no such probe:
>> bash-3.00# dtrace -l|wc -l
>>    47351
>> bash-3.00# dtrace -l|grep tcp_xchg
>>
>> On the other x86 system I get it:
>> # cat /etc/release
>>                         Solaris 10 6/06 s10x_u2wos_09a X86
>>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>>                         Use is subject to license terms.
>>                              Assembled 09 June 2006
>> # uname -a
>> SunOS bolek 5.10 Generic_118855-19 i86pc i386 i86pc
>> # dtrace -l|grep tcp_xchg
>> 25995        fbt                ip                          tcp_xchg
entry
>> 25996        fbt                ip                          tcp_xchg
return
>> 36350        fbt               nca                          tcp_xchg
entry
>> 36351        fbt               nca                          tcp_xchg
return
>>
>> Can anyone explain me what''s going on ?
>>
>> przemol
>>
>> ----------------------------------------------------------------------
>> Nawet 50MB w jednym liscie>>> http://link.interia.pl/f19e7
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>     
>
> ----------------------------------------------------------------------
> Bryka za cwajhundert ojro! Zobacz >> http://link.interia.pl/f19df
>
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Chip Bennett writes:
> Being an fbt probe, and there has to be function of that name in the 
> kernel, what do you get from each system for:
> 
>      /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
The symbol in question is still in Solaris and has been there for more
than five years.  It hasn''t changed.

At a guess, I think this would have to be either a build problem
(we''re still talking about Open Solaris, right?) with the compiler in
use, or perhaps something amiss with the way dtrace finds this symbol.

However, you do bring up a good point.  Undocumented kernel symbols
(like this one) are private interfaces, and may disappear at any time
without notice.

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

On Mon, Jan 15, 2007 at 09:54:04AM -0600, Chip Bennett
wrote:
> Being an fbt probe, and there has to be function of that name in the 
> kernel, what do you get from each system for:
> 
>     /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
Both servers are Solaris 10 6/06 s10x_u2wos:

server _without_ tcp_xchg (servername: e2 - 64bit AMD Opteron):

bash-3.00# /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
[14872]	|18446744073441173472|          34|FUNC |LOCL |0    |ABS |tcp_xchg
[8206]	|18446744073437902160|          34|FUNC |LOCL |0    |ABS |tcp_xchg
bash-3.00# dtrace -l|grep tcp_xchg
bash-3.00# 


server with tcp_xchg (servername: bolek - 32bit):
bash-3.00# /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
[10128] |4273276023|      52|FUNC |LOCL |0    |ABS    |tcp_xchg
[14903] |4192498854|      52|FUNC |LOCL |0    |ABS    |tcp_xchg
bash-3.00# dtrace -l|grep tcp_xchg
25995        fbt                ip                          tcp_xchg entry
25996        fbt                ip                          tcp_xchg return
36350        fbt               nca                          tcp_xchg entry
36351        fbt               nca                          tcp_xchg return
bash-3.00# 
> You also might compare the outputs of modinfo on each system.  Could 
> there have been a patch that eliminated that function name?  (yea, I 
> know: that''s unlikely.)
The diff between the two (left=e2, right=32bit bolek):
1,2d0
< FX (Fixed priority sched class)
< FX_DPTBL (Fixed priority dispatch table)
10,11d7
< aes (AES Kernel SW Provider 1.8)
< aggr (Link Aggregation MAC)
16,19d11
< autofs (AUTOFS syscall (32-bit))
< autofs (AUTOFS syscall)
< autofs (filesystem for autofs)
< bge (BCM579x driver v0.48)
21,22d12
< blowfish (Blowfish Kernel SW Provider 1.9)
< bmc (BMC 1.0)
24a15
> c2audit (C2 system call)
33d23
< cpc (32-bit cpc sampling system call)
36d25
< cpu.AuthenticAMD.15 (AMD Athlon64/Opteron C
44,45d32
< des (DES Kernel SW Provider 1.22)
< des (des encryption)
50d36
< doorfs (32-bit door syscalls)
55d40
< elfexec (32-bit exec module for elf)
56a42
> elxl (3Com 90XX LAN driver 1.34)
60,61d45
< fcp (SunFC FCP v20060831-1.109)
< fcsm (Sun FC SAN Management v20060831)
65d48
< fp (SunFC Port v20060831-1.71)
71,72d53
< hid (USB HID Client Driver 1.36)
< hidparser (HID PARSER 1.13)
82,83d62
< inst_sync (32-bit instance binding syscall)
< inst_sync (instance binding syscall)
88d66
< ipc (common ipc code)
90d67
< ippctl (IP Policy Configuration Driver )
103d79
< klmmod (lock mgr common module)
119d94
< mc-amd (Memory Controller for AMD proce)
123c98
< md_hotspares (Solaris Volume Manager hot sp
---
> md_hotspares (Solaris Volume Manager hot spar)
132,133c107
< msgsys (32-bit System V message facilit)
< msgsys (System V message facility)
---
> mouse8042 (PS/2 Mouse 1.43, 05/06/01)
138d111
< nfs (NFS syscall, client, and common)
143,144d115
< nge (nVidia 1Gb Ethernet 1.3)
< npe (Host to PCIe nexus driver 1.3)
150c121,122
< pcbe.AuthenticAMD.15 (AMD Performance Count
---
> pcbe.GenuineIntel.15 (Pentium 4 Performance Counter
> pci (host to PCI nexus driver 1.65)
153d124
< pcie_pci (PCIe to PCI nexus driver 1.3)
159d129
< pipe (32-bit pipe(2) syscall)
164d133
< portfs (32-bit event ports syscalls)
169d137
< pset (32-bit pset(2) syscall)
177d144
< qlc (SunFC Qlogic FCA v20060630-2.16)
181d147
< rpcmod (32-bit RPC syscall)
196,198d161
< semsys (32-bit System V semaphore facil)
< semsys (System V semaphore facility)
< ses (SCSI Enclosure Services Driver )
201,202d163
< shmsys (32-bit System V shared memory)
< shmsys (System V shared memory)
210d170
< st (SCSI tape Driver 1.243)
234d193
< usb_mid (USB Multi Interface Driver 1.38)
236,237d194
< usbkbm (USB keyboard streams 1.34)
< usbms (USB mouse streams 1.18)
241a199
> vuid3ps2 (mouse events to vuid events)

przemol

----------------------------------------------------------------------
Slodkie jest zycie bandziora. Zobacz >> http://link.interia.pl/f19e5

On Mon, Jan 15, 2007 at 11:27:33AM -0500, James Carlson
wrote:
> However, you do bring up a good point.  Undocumented kernel symbols
> (like this one) are private interfaces, and may disappear at any time
> without notice.
To continue the point, this naturally leads to the question of how one would
instrument anything in networking without groveling around in fbt. We know
that a networking provider is desperately needed and one is in the works.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

This is sounding like a bug. What does this command yeild on the two
systems:

# echo tcp_xchg::nm | mdb -k

Thanks.

Adam

On Tue, Jan 16, 2007 at 10:10:55AM +0100, przemolicc at poczta.fm
wrote:
> On Mon, Jan 15, 2007 at 09:54:04AM -0600, Chip Bennett wrote:
> > Being an fbt probe, and there has to be function of that name in the 
> > kernel, what do you get from each system for:
> > 
> >     /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
> 
> Both servers are Solaris 10 6/06 s10x_u2wos:
> 
> server _without_ tcp_xchg (servername: e2 - 64bit AMD Opteron):
> 
> bash-3.00# /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
> [14872]	|18446744073441173472|          34|FUNC |LOCL |0    |ABS |tcp_xchg
> [8206]	|18446744073437902160|          34|FUNC |LOCL |0    |ABS |tcp_xchg
> bash-3.00# dtrace -l|grep tcp_xchg
> bash-3.00# 
> 
> 
> server with tcp_xchg (servername: bolek - 32bit):
> bash-3.00# /usr/ccs/bin/nm /dev/ksyms | grep tcp_xchg
> [10128] |4273276023|      52|FUNC |LOCL |0    |ABS    |tcp_xchg
> [14903] |4192498854|      52|FUNC |LOCL |0    |ABS    |tcp_xchg
> bash-3.00# dtrace -l|grep tcp_xchg
> 25995        fbt                ip                          tcp_xchg entry
> 25996        fbt                ip                          tcp_xchg return
> 36350        fbt               nca                          tcp_xchg entry
> 36351        fbt               nca                          tcp_xchg return
> bash-3.00# 
> 
> > You also might compare the outputs of modinfo on each system.  Could 
> > there have been a patch that eliminated that function name?  (yea, I 
> > know: that''s unlikely.)
> 
> The diff between the two (left=e2, right=32bit bolek):
> 1,2d0
> < FX (Fixed priority sched class)
> < FX_DPTBL (Fixed priority dispatch table)
> 10,11d7
> < aes (AES Kernel SW Provider 1.8)
> < aggr (Link Aggregation MAC)
> 16,19d11
> < autofs (AUTOFS syscall (32-bit))
> < autofs (AUTOFS syscall)
> < autofs (filesystem for autofs)
> < bge (BCM579x driver v0.48)
> 21,22d12
> < blowfish (Blowfish Kernel SW Provider 1.9)
> < bmc (BMC 1.0)
> 24a15
> > c2audit (C2 system call)
> 33d23
> < cpc (32-bit cpc sampling system call)
> 36d25
> < cpu.AuthenticAMD.15 (AMD Athlon64/Opteron C
> 44,45d32
> < des (DES Kernel SW Provider 1.22)
> < des (des encryption)
> 50d36
> < doorfs (32-bit door syscalls)
> 55d40
> < elfexec (32-bit exec module for elf)
> 56a42
> > elxl (3Com 90XX LAN driver 1.34)
> 60,61d45
> < fcp (SunFC FCP v20060831-1.109)
> < fcsm (Sun FC SAN Management v20060831)
> 65d48
> < fp (SunFC Port v20060831-1.71)
> 71,72d53
> < hid (USB HID Client Driver 1.36)
> < hidparser (HID PARSER 1.13)
> 82,83d62
> < inst_sync (32-bit instance binding syscall)
> < inst_sync (instance binding syscall)
> 88d66
> < ipc (common ipc code)
> 90d67
> < ippctl (IP Policy Configuration Driver )
> 103d79
> < klmmod (lock mgr common module)
> 119d94
> < mc-amd (Memory Controller for AMD proce)
> 123c98
> < md_hotspares (Solaris Volume Manager hot sp
> ---
> > md_hotspares (Solaris Volume Manager hot spar)
> 132,133c107
> < msgsys (32-bit System V message facilit)
> < msgsys (System V message facility)
> ---
> > mouse8042 (PS/2 Mouse 1.43, 05/06/01)
> 138d111
> < nfs (NFS syscall, client, and common)
> 143,144d115
> < nge (nVidia 1Gb Ethernet 1.3)
> < npe (Host to PCIe nexus driver 1.3)
> 150c121,122
> < pcbe.AuthenticAMD.15 (AMD Performance Count
> ---
> > pcbe.GenuineIntel.15 (Pentium 4 Performance Counter
> > pci (host to PCI nexus driver 1.65)
> 153d124
> < pcie_pci (PCIe to PCI nexus driver 1.3)
> 159d129
> < pipe (32-bit pipe(2) syscall)
> 164d133
> < portfs (32-bit event ports syscalls)
> 169d137
> < pset (32-bit pset(2) syscall)
> 177d144
> < qlc (SunFC Qlogic FCA v20060630-2.16)
> 181d147
> < rpcmod (32-bit RPC syscall)
> 196,198d161
> < semsys (32-bit System V semaphore facil)
> < semsys (System V semaphore facility)
> < ses (SCSI Enclosure Services Driver )
> 201,202d163
> < shmsys (32-bit System V shared memory)
> < shmsys (System V shared memory)
> 210d170
> < st (SCSI tape Driver 1.243)
> 234d193
> < usb_mid (USB Multi Interface Driver 1.38)
> 236,237d194
> < usbkbm (USB keyboard streams 1.34)
> < usbms (USB mouse streams 1.18)
> 241a199
> > vuid3ps2 (mouse events to vuid events)
> 
> 
> przemol
> 
> ----------------------------------------------------------------------
> Slodkie jest zycie bandziora. Zobacz >> http://link.interia.pl/f19e5
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Tue, Jan 16, 2007 at 02:43:35AM -0800, Adam Leventhal
wrote:
> This is sounding like a bug. What does this command yeild on the two
> systems:
> 
> # echo tcp_xchg::nm | mdb -k
On e2 (the server without the probe):
bash-3.00# echo tcp_xchg::nm | mdb -k
Value              Size               Type  Bind  Other Shndx    Name
0xfffffffff000dfe0|0x0000000000000022|FUNC |LOCL |0x0  |2       |tcp_xchg

On bolek (the server with the probe):
bash-3.00# echo tcp_xchg::nm | mdb -k
Value      Size       Type  Bind  Other Shndx    Name
0xf9e474a6|0x00000034|FUNC |LOCL |0x0  |1       |tcp_xchg

Regards
przemol

----------------------------------------------------------------------
Slodkie jest zycie bandziora. Zobacz >> http://link.interia.pl/f19e5

G''Day Przemol,

On Thu, Jan 11, 2007 at 12:09:55PM +0100, przemolicc at poczta.fm
wrote:
> Hello,
> 
> I am trying to use one of scripts from Dtrace toolkit and get:
> bash-3.00# dtrace -l|wc -l
>    47351
> bash-3.00# cat /etc/release 
>                         Solaris 10 6/06 s10x_u2wos_09a X86
>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>                         Use is subject to license terms.
>                              Assembled 09 June 2006
> bash-3.00# uname -a
> SunOS e2 5.10 Generic_118855-19 i86pc i386 i86pc
> bash-3.00# ./tcpsnoop.d
> dtrace: failed to compile script ./tcpsnoop.d: line 233: probe description
fbt:ip:tcp_xchg:entry does not match any probes
> 
> Indeed, there is no such probe:
> bash-3.00# dtrace -l|wc -l
>    47351
> bash-3.00# dtrace -l|grep tcp_xchg
Sorry about this; While I hope that the tcpsnoop/tcptop scripts don''t
break for Solaris updates, they are indeed the most fragile as they are
based on private unstable kernel implementation (fbt), as Jim said.
Ideally they would be based on a stable network provider, as Adam said,
but in the meantime it is either fbt or nothing.

This is also the reason I only wrote tcpsnoop/tcptop; I wanted (and still
want) to write numerous networking based scripts - but after writing
tcpsnoop.d it was obvious how fragile TCP tracing was. There is going
to be some real fun when we have a stable TCP provider.

I still don''t know why tcp_xchg is missing - I can fix it, but first
I need to find out why it broke...

Brendan

-- 
Brendan
[CA, USA]

I had Przemol give me the disassembly for tcp_xchg:

tcp_xchg:                       movl   %edx,%r8d
tcp_xchg+3:                     jmp    +0x13    <tcp_xchg+0x16>
tcp_xchg+5:                     movslq %r8d,%rax
tcp_xchg+8:                     movzbl (%rax,%rdi),%ecx
tcp_xchg+0xc:                   movzbl (%rax,%rsi),%edx
tcp_xchg+0x10:                  movb   %dl,(%rax,%rdi)
tcp_xchg+0x13:                  movb   %cl,(%rax,%rsi)
tcp_xchg+0x16:                  movl   %r8d,%eax
tcp_xchg+0x19:                  decl   %r8d
tcp_xchg+0x1c:                  testl  %eax,%eax
tcp_xchg+0x1e:                  jg     -0x19    <tcp_xchg+5>
tcp_xchg+0x20:                  repz ret

The problem is that DTrace won''t instrument a leaf routine (one that
doesn''t
explicitly push a stack frame) on x86 and amd64. Leaf routines can often
indicate that we''re operating in a delicate context whereas functions
that
push stack frames are known to be safe. On SPARC, we use the trick of checking
for CTF data to see if a function was compiled from C code (and therefore
able to execute in unconstrained contexts); perhaps it''s time to move
this
trick to amd64 since the compiler seems to be increasingly aggressive about
optimizing out those stack frames.

I''ve filed this bug to track the issue:

    6513936 fbt can''t instrumeent C-derived leaf routines on amd64

- ahl

On Tue, Jan 16, 2007 at 12:31:30PM +0100, przemolicc at poczta.fm
wrote:
> On Tue, Jan 16, 2007 at 02:43:35AM -0800, Adam Leventhal wrote:
> > This is sounding like a bug. What does this command yeild on the two
> > systems:
> > 
> > # echo tcp_xchg::nm | mdb -k
> 
> On e2 (the server without the probe):
> bash-3.00# echo tcp_xchg::nm | mdb -k
> Value              Size               Type  Bind  Other Shndx    Name
> 0xfffffffff000dfe0|0x0000000000000022|FUNC |LOCL |0x0  |2       |tcp_xchg
> 
> On bolek (the server with the probe):
> bash-3.00# echo tcp_xchg::nm | mdb -k
> Value      Size       Type  Bind  Other Shndx    Name
> 0xf9e474a6|0x00000034|FUNC |LOCL |0x0  |1       |tcp_xchg
> 
> Regards
> przemol
> 
> ----------------------------------------------------------------------
> Slodkie jest zycie bandziora. Zobacz >> http://link.interia.pl/f19e5
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl