dtrace discuss - Feb 2010 - how to know whether the network is working in interrupt mode or polling mode?

In the book of "Solaris 10 and OpenSolaris Kernel Architecture", it
said:

the squeue is a common FIFO for both inbound and outbound packets, and only
one thread is allowed to process it at any given time. As such, the per-CPU
backlog is easy to figure out. If packets are queued, the squeue can switch
the NIC associated with it from interrupt to polling mode.

How can I know which mode the network card is currently using?

Suppose the network card is bound to cpu 7, when working in interrupt mode,
only cpu 7 will process the network data, so if cpu 7 is 100% used, and all
other cpus are idle, the sever still has to problem to get/send data to
client, right? And if work in polling mode, is it possible to use all cpu to
get/send data?

And is it possible to use dtrace to measure the latency because of the cpu
is busy to handle network data? e.g,  data were send from client at t1, and
the data was first queued in t2 and then it processed by cpu at t3. Can we
measure t2-t1 and t3-t1 using dtrace?

# echo ::interrupts -d | mdb -k
IRQ  Vector IPL Bus   Type  CPU Share APIC/INT# Driver Name(s)
4    0xb0   12  ISA   Fixed 4   1     0x0/0x4   asy#0
....
21   0x42   5   PCI   Fixed 6   2     0x0/0x15  nv_sata#1, ohci#0
*22   0x60   6   PCI   Fixed 7   2     0x0/0x16  nv_sata#2, nge#0*
....
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100202/29945468/attachment.html>

On 02/02/10 17:30, Qihua Wu wrote:
>
> In the book of "Solaris 10 and OpenSolaris Kernel Architecture",
it said:
>
> the squeue is a common FIFO for both inbound and outbound packets, and 
> only one thread is allowed to process it at any given time. As such, 
> the per-CPU backlog is easy to figure out. If packets are queued, the 
> squeue can switch the NIC associated with it from interrupt to polling 
> mode.
>
> How can I know which mode the network card is currently using?
When in interrupt mode, the network rx stack starts from _interrupt() on 
x86, while in polling mode, the network rx stack starts from a polling 
thread, that is, "unix''thread_start".

The interrupt mode and the polling mode are switched on the fly, you can 
just do dtrace aggregations to see the portion of each mode. like,
::tcp_rput_data:entry
{
    @[stack()]=count();
}
>
> Suppose the network card is bound to cpu 7, when working in interrupt 
> mode, only cpu 7 will process the network data, so if cpu 7 is 100% 
> used, and all other cpus are idle, the sever still has to problem to 
> get/send data to client, right?
Yes.

In fact, since solaris interrupt thread has higher priority than any 
other user and kernel threads, so when cpu7 is saturated, almost all of 
the CPU cycles should be spent handling the interrupt bound to that cpu.
Since the interrupt thread may block also, if so, other threads with 
lower priority may be dispatched on this cpu, you can fence the cpu by 
putting it in a processor set(see psrset(1M), so other thread would not 
be dispatched on this cpu. Anyway, solaris makes sure the cpu handle the 
interrupt promptly. If there are more interrupts occurring , or the ISR 
does too much work in it, then the system would not handle the following 
interrupt in time after it gets saturated, so that the whole throughput 
would be limited. A worker kernel thread is useful in this case, with 
it, some of the works which had done by interrupt thread now can be done 
in the kernel threads, and these threads can be dispatched on all of the 
cpus. This feature is named interrupt fanout.
> And if work in polling mode, is it possible to use all cpu to get/send 
> data?
Yes.
>
> And is it possible to use dtrace to measure the latency because of the 
> cpu is busy to handle network data? e.g,  data were send from client 
> at t1, and the data was first queued in t2 and then it processed by 
> cpu at t3. Can we measure t2-t1 and t3-t1 using dtrace?
I am not quite sure what the latency you mention refers to. In general, 
the variable ?timestamp? in dtrace can not measure an accurate latency 
since the dtrace itself introduces latency when adding probes to the 
system. so if you are trying to measure small latency, a kernel 
profiling(sampling the PC in a profile provider) is required, instead of 
using timestamp in fbt provider.

-Brian
>
> # echo ::interrupts -d | mdb -k
> IRQ  Vector IPL Bus   Type  CPU Share APIC/INT# Driver Name(s)
> 4    0xb0   12  ISA   Fixed 4   1     0x0/0x4   asy#0
> ....
> 21   0x42   5   PCI   Fixed 6   2     0x0/0x15  nv_sata#1, ohci#0
> *22   0x60   6   PCI   Fixed 7   2     0x0/0x16  nv_sata#2, nge#0*
> ....
> ------------------------------------------------------------------------
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100204/2878a052/attachment.html>