Why sched is often interrupt on cpu 0 instead of distribution evenly on all
cpu?
dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] = count();
}''
dtrace: description ''sdt:::interrupt-start '' matched 1 probe
^C
sched 4
355
sched 18
358
sched 8
369
sched 10
370
sched 11
370
sched 12
408
sched 13
408
sched 15
408
sched 19
408
sched 20
408
sched 21
408
sched 22
408
sched 23
408
sched 14
409
sched 9
410
sched 2
411
sched 16
411
sched 6
422
sched 17
456
sched 3
469
sched 5
510
sched 7
564
sched 1
3079
sched 0
535071
Thanks,
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100121/ff6fdf18/attachment.html>
Qihua Wu wrote:> Why sched is often interrupt on cpu 0 instead of distribution evenly on > all cpu? > > dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] = count(); }'' > dtrace: description ''sdt:::interrupt-start '' matched 1 probethe output of intrstat may help. Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
"sched" is the execname of the PID 0 process (run "ps -e"). The string "sched" gets plugged into the DTrace execname variable if the CPU is in an interrupt handler when a probe fires. CPU 0 is very likely taking the clock interrupts, which by default occur every 10 milliseconds. HTH, /jim Qihua Wu wrote:> Why sched is often interrupt on cpu 0 instead of distribution evenly > on all cpu? > > dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] = count(); }'' > dtrace: description ''sdt:::interrupt-start '' matched 1 probe > ^C > > sched > 4 355 > sched > 18 358 > sched > 8 369 > sched > 10 370 > sched > 11 370 > sched > 12 408 > sched > 13 408 > sched > 15 408 > sched > 19 408 > sched > 20 408 > sched > 21 408 > sched > 22 408 > sched > 23 408 > sched > 14 409 > sched > 9 410 > sched > 2 411 > sched > 16 411 > sched > 6 422 > sched > 17 456 > sched > 3 469 > sched > 5 510 > sched > 7 564 > sched > 1 3079 > sched > 0 535071 > > Thanks, > Daniel > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
Now cpu7 has most of the interrupt (delete other cpu stats as they are 0).
so why the execname is sched when interrupt happens?
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3
%tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
------------+--------------------------------------------------------------------------------------------------------------------------
ehci#0 | 0 0.0 0 0.0 0 0.0 0
0.0 0 0.0 2 0.0 0 0.0 0 0.0
nge#0 | 0 0.0 0 0.0 0 0.0 0
0.0 0 0.0 0 0.0 0 0.0 6260 2.9
nge#1 | 0 0.0 0 0.0 0 0.0 0
0.0 0 0.0 0 0.0 0 0.0 2 0.0
nv_sata#0 | 0 0.0 0 0.0 0 0.0 0
0.0 0 0.0 2 0.0 0 0.0 0 0.0
nv_sata#2 | 0 0.0 0 0.0 0 0.0 0
0.0 0 0.0 0 0.0 0 0.0 6260 0.5
On Thu, Jan 21, 2010 at 6:54 PM, Michael Schuster
<Michael.Schuster at sun.com>wrote:
> Qihua Wu wrote:
>
>> Why sched is often interrupt on cpu 0 instead of distribution evenly on
>> all cpu?
>>
>> dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] =
count(); }''
>> dtrace: description ''sdt:::interrupt-start '' matched
1 probe
>>
>
> the output of intrstat may help.
>
> Michael
> --
> Michael Schuster http://blogs.sun.com/recursion
> Recursion, n.: see ''Recursion''
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100122/c8884a66/attachment.html>
So do we have a way to know what causes the interrupt (this may be got by intrstat) and which pid/execname is interrupted? Thanks Daniel On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro <James.Mauro at sun.com> wrote:> "sched" is the execname of the PID 0 process (run "ps -e"). > > The string "sched" gets plugged into the DTrace execname variable > if the CPU is in an interrupt handler when a probe fires. > > CPU 0 is very likely taking the clock interrupts, which by default > occur every 10 milliseconds. > > HTH, > /jim > > > > > Qihua Wu wrote: > >> Why sched is often interrupt on cpu 0 instead of distribution evenly on >> all cpu? >> >> dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] = count(); }'' >> dtrace: description ''sdt:::interrupt-start '' matched 1 probe >> ^C >> >> sched 4 >> 355 >> sched 18 >> 358 >> sched 8 >> 369 >> sched 10 >> 370 >> sched 11 >> 370 >> sched 12 >> 408 >> sched 13 >> 408 >> sched 15 >> 408 >> sched 19 >> 408 >> sched 20 >> 408 >> sched 21 >> 408 >> sched 22 >> 408 >> sched 23 >> 408 >> sched 14 >> 409 >> sched 9 >> 410 >> sched 2 >> 411 >> sched 16 >> 411 >> sched 6 >> 422 >> sched 17 >> 456 >> sched 3 >> 469 >> sched 5 >> 510 >> sched 7 >> 564 >> sched 1 >> 3079 >> sched 0 >> 535071 >> >> Thanks, >> Daniel >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100122/3a20cf86/attachment.html>
Hi Qihua Wu, Qihua Wu wrote:> Now cpu7 has most of the interrupt (delete other cpu stats as they are 0). > so why the execname is sched when interrupt happens? >Most interrupts run in a kernel thread set aside for handling interrupts. There are 10 threads per cpu for interrupt levels 1 through 10. All of these threads will show up as being part of the "sched" process. (In fact, on an idle system, you''ll probably have over 300 threads, and about 200 of them will say they belong to "sched". These are various taskq(9f) threads, garbage collectors, interrupt threads, and others.). I think what you probably want to know is who was interrupted. max> device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 > %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim > ------------+-------------------------------------------------------------------------------------------------------------------------- > ehci#0 | 0 0.0 0 0.0 0 0.0 0 > 0.0 0 0.0 2 0.0 0 0.0 0 0.0 > nge#0 | 0 0.0 0 0.0 0 0.0 0 > 0.0 0 0.0 0 0.0 0 0.0 6260 2.9 > nge#1 | 0 0.0 0 0.0 0 0.0 0 > 0.0 0 0.0 0 0.0 0 0.0 2 0.0 > nv_sata#0 | 0 0.0 0 0.0 0 0.0 0 > 0.0 0 0.0 2 0.0 0 0.0 0 0.0 > nv_sata#2 | 0 0.0 0 0.0 0 0.0 0 > 0.0 0 0.0 0 0.0 0 0.0 6260 0.5 > > On Thu, Jan 21, 2010 at 6:54 PM, Michael Schuster > <Michael.Schuster at sun.com>wrote: > > >> Qihua Wu wrote: >> >> >>> Why sched is often interrupt on cpu 0 instead of distribution evenly on >>> all cpu? >>> >>> dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] = count(); }'' >>> dtrace: description ''sdt:::interrupt-start '' matched 1 probe >>> >>> >> the output of intrstat may help. >> >> Michael >> -- >> Michael Schuster http://blogs.sun.com/recursion >> Recursion, n.: see ''Recursion'' >> >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
Hi Qihua Wu,
Here is a script that shows process args for process that was
interrupted, and cpu. It is easily modified
to add pid.
#!/usr/sbin/dtrace -s
interrupt-start
{
self->intr = curthread->t_intr;
}
interrupt-start
/self->intr/
{
@[stringof(self->intr->t_procp->p_user.u_psargs), cpu] = count();
}
interrupt-complete
/self->intr/
{
self->intr = 0;
}
max
Qihua Wu wrote:> So do we have a way to know what causes the interrupt (this may be got by
> intrstat) and which pid/execname is interrupted?
>
> Thanks
> Daniel
>
> On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro <James.Mauro at sun.com>
wrote:
>
>
>> "sched" is the execname of the PID 0 process (run "ps
-e").
>>
>> The string "sched" gets plugged into the DTrace execname
variable
>> if the CPU is in an interrupt handler when a probe fires.
>>
>> CPU 0 is very likely taking the clock interrupts, which by default
>> occur every 10 milliseconds.
>>
>> HTH,
>> /jim
>>
>>
>>
>>
>> Qihua Wu wrote:
>>
>>
>>> Why sched is often interrupt on cpu 0 instead of distribution
evenly on
>>> all cpu?
>>>
>>> dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] =
count(); }''
>>> dtrace: description ''sdt:::interrupt-start ''
matched 1 probe
>>> ^C
>>>
>>> sched 4
>>> 355
>>> sched 18
>>> 358
>>> sched 8
>>> 369
>>> sched 10
>>> 370
>>> sched 11
>>> 370
>>> sched 12
>>> 408
>>> sched 13
>>> 408
>>> sched 15
>>> 408
>>> sched 19
>>> 408
>>> sched 20
>>> 408
>>> sched 21
>>> 408
>>> sched 22
>>> 408
>>> sched 23
>>> 408
>>> sched 14
>>> 409
>>> sched 9
>>> 410
>>> sched 2
>>> 411
>>> sched 16
>>> 411
>>> sched 6
>>> 422
>>> sched 17
>>> 456
>>> sched 3
>>> 469
>>> sched 5
>>> 510
>>> sched 7
>>> 564
>>> sched 1
>>> 3079
>>> sched 0
>>> 535071
>>>
>>> Thanks,
>>> Daniel
>>>
------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> dtrace-discuss mailing list
>>> dtrace-discuss at opensolaris.org
>>>
>>>
>>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
According to your intrstat data, it looks like a nge network device and disk sata controller both have interrupts bound to cpu 7. Max provided a DTrace script to determine which process is having their thread(s) pinned when an interrupt comes in. If you''re concerned about the possibility of a CPU handling a high rate of interrupts, and running application threads, causing a performance issue, just fence the CPU off. It''s easy, and dynamic. For example, if cpu7 is handling most of the interrupts, just put it in a processor set; psrset -c 7 Once you''ve done that, the kernel will not schedule user threads on cpu7 unless you explicitly bind processes/threads to the processor set. By NOT doing any binding, you''ve effectively fenced off cpu7 - all it will do is handle interrupts. If it doesn''t help, or makes things worse, just "psradm -d 1" to delete the set. /jim Qihua Wu wrote:> So do we have a way to know what causes the interrupt (this may be got > by intrstat) and which pid/execname is interrupted? > > Thanks > Daniel > > On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro <James.Mauro at sun.com > <mailto:James.Mauro at sun.com>> wrote: > > "sched" is the execname of the PID 0 process (run "ps -e"). > > The string "sched" gets plugged into the DTrace execname variable > if the CPU is in an interrupt handler when a probe fires. > > CPU 0 is very likely taking the clock interrupts, which by default > occur every 10 milliseconds. > > HTH, > /jim > > > > > Qihua Wu wrote: > > Why sched is often interrupt on cpu 0 instead of distribution > evenly on all cpu? > > dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] > count(); }'' > dtrace: description ''sdt:::interrupt-start '' matched 1 probe > ^C > > sched 4 > 355 > sched 18 > 358 > sched 8 > 369 > sched 10 > 370 > sched 11 > 370 > sched 12 > 408 > sched 13 > 408 > sched 15 > 408 > sched 19 > 408 > sched 20 > 408 > sched 21 > 408 > sched 22 > 408 > sched 23 > 408 > sched 14 > 409 > sched 9 > 410 > sched 2 > 411 > sched 16 > 411 > sched 6 > 422 > sched 17 > 456 > sched 3 > 469 > sched 5 > 510 > sched 7 > 564 > sched 1 > 3079 > sched 0 > 535071 > > Thanks, > Daniel > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > <mailto:dtrace-discuss at opensolaris.org> > > >
Hi Max/Jim,
Based on your script, I created the following script to print which pid is
interrupted how many times every 2 second. The result shows pid 0 is
interrupted most: about 50k every 2 second, that''s 25k per second. How
could
it be interrupted so frequently? For the clock interrupt, it only interrupts
100 times per second, which is much less than 25k per second. Where could it
come from?
The box has lots of network traffic from mysql, I think the network
interrupt will cause the mysqld (pid 10490) to be interrupted instead of pid
0.
#!/usr/sbin/dtrace -s
interrupt-start
{
self->intr = curthread->t_intr;
}
interrupt-start
/self->intr/
{
@inter[self->intr->t_procp->p_pidp->pid_id, cpu] = count();
}
interrupt-complete
/self->intr/
{
self->intr = 0;
}
profile:::tick-2sec
{
printf("report completed at time:%Y\n",walltimestamp);
printf("*******************************************\n");
printa("%d-%d: %@d\n", @inter);
trunc(@inter,0);
}
0 89152 :tick-2sec report completed at time:2010 Jan
24 02:48:56
*******************************************
0-0: 1
0-2: 1
0-8: 1
0-12: 1
0-14: 1
0-16: 1
0-20: 1
0-21: 1
0-22: 1
1095-7: 4
0-6: 5
0-5: 8
264-7: 8
10490-7: 4962
0-7: 49131
0 89152 :tick-2sec report completed at time:2010
Jan 24 02:48:58
*******************************************
0-0: 1
0-5: 4
1095-7: 4
10490-7: 4200
0-7: 50305
0 89152 :tick-2sec report completed at time:2010
Jan 24 02:49:00
*******************************************
0-3: 4
1095-7: 4
0-5: 8
25388-7: 818
10490-7: 1582
0-7: 51507
On Fri, Jan 22, 2010 at 4:21 PM, max at bruningsystems.com <
max at bruningsystems.com> wrote:
> Hi Qihua Wu,
>
> Here is a script that shows process args for process that was interrupted,
> and cpu. It is easily modified
> to add pid.
>
> #!/usr/sbin/dtrace -s
>
> interrupt-start
> {
> self->intr = curthread->t_intr;
> }
>
> interrupt-start
> /self->intr/
> {
> @[stringof(self->intr->t_procp->p_user.u_psargs), cpu] =
count();
> }
>
> interrupt-complete
> /self->intr/
> {
> self->intr = 0;
> }
>
>
> max
>
>
> Qihua Wu wrote:
>
>> So do we have a way to know what causes the interrupt (this may be got
by
>> intrstat) and which pid/execname is interrupted?
>>
>> Thanks
>> Daniel
>>
>> On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro <James.Mauro at
sun.com> wrote:
>>
>>
>>
>>> "sched" is the execname of the PID 0 process (run
"ps -e").
>>>
>>> The string "sched" gets plugged into the DTrace execname
variable
>>> if the CPU is in an interrupt handler when a probe fires.
>>>
>>> CPU 0 is very likely taking the clock interrupts, which by default
>>> occur every 10 milliseconds.
>>>
>>> HTH,
>>> /jim
>>>
>>>
>>>
>>>
>>> Qihua Wu wrote:
>>>
>>>
>>>
>>>> Why sched is often interrupt on cpu 0 instead of distribution
evenly on
>>>> all cpu?
>>>>
>>>> dtrace -n ''sdt:::interrupt-start { @num[execname,cpu]
= count(); }''
>>>> dtrace: description ''sdt:::interrupt-start ''
matched 1 probe
>>>> ^C
>>>>
>>>> sched 4
>>>> 355
>>>> sched 18
>>>> 358
>>>> sched 8
>>>> 369
>>>> sched 10
>>>> 370
>>>> sched 11
>>>> 370
>>>> sched 12
>>>> 408
>>>> sched 13
>>>> 408
>>>> sched 15
>>>> 408
>>>> sched 19
>>>> 408
>>>> sched 20
>>>> 408
>>>> sched 21
>>>> 408
>>>> sched 22
>>>> 408
>>>> sched 23
>>>> 408
>>>> sched 14
>>>> 409
>>>> sched 9
>>>> 410
>>>> sched 2
>>>> 411
>>>> sched 16
>>>> 411
>>>> sched 6
>>>> 422
>>>> sched 17
>>>> 456
>>>> sched 3
>>>> 469
>>>> sched 5
>>>> 510
>>>> sched 7
>>>> 564
>>>> sched 1
>>>> 3079
>>>> sched 0
>>>> 535071
>>>>
>>>> Thanks,
>>>> Daniel
>>>>
------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> dtrace-discuss mailing list
>>>> dtrace-discuss at opensolaris.org
>>>>
>>>>
>>>>
>>>>
>>>
>>
------------------------------------------------------------------------
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100125/79df52bb/attachment.html>
PID 0 isn''t really a process. It''s the kernel - that''s something of an oversimplification, but for the purposes of this discussion, just know that pid 0 is not a process. Interrupts do not target processes. Devices bind interrupts to CPU at initialization time. Use ''echo "::interrupts" | mdb -k" to determine which devices have interrupts bound to which CPUs, and/or intrstat. mpstat will show you interrupt rates per CPU as well. As I said, if you think interrupts are getting in the way, create a processor set with the CPU(s) handling high rates of interrupts, thereby fencing it off from running application code. Thanks, /jim Qihua Wu wrote:> Hi Max/Jim, > > Based on your script, I created the following script to print which > pid is interrupted how many times every 2 second. The result shows pid > 0 is interrupted most: about 50k every 2 second, that''s 25k per > second. How could it be interrupted so frequently? For the clock > interrupt, it only interrupts 100 times per second, which is much less > than 25k per second. Where could it come from? > > The box has lots of network traffic from mysql, I think the network > interrupt will cause the mysqld (pid 10490) to be interrupted instead > of pid 0. > > #!/usr/sbin/dtrace -s > > interrupt-start > { > self->intr = curthread->t_intr; > } > > interrupt-start > /self->intr/ > { > @inter[self->intr->t_procp->p_pidp->pid_id, cpu] = count(); > } > > interrupt-complete > /self->intr/ > { > self->intr = 0; > } > > profile:::tick-2sec > { > printf("report completed at time:%Y\n",walltimestamp); > printf("*******************************************\n"); > printa("%d-%d: %@d\n", @inter); > trunc(@inter,0); > } > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:48:56 > ******************************************* > 0-0: 1 > 0-2: 1 > 0-8: 1 > 0-12: 1 > 0-14: 1 > 0-16: 1 > 0-20: 1 > 0-21: 1 > 0-22: 1 > 1095-7: 4 > 0-6: 5 > 0-5: 8 > 264-7: 8 > 10490-7: 4962 > 0-7: 49131 > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:48:58 > ******************************************* > 0-0: 1 > 0-5: 4 > 1095-7: 4 > 10490-7: 4200 > 0-7: 50305 > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:49:00 > ******************************************* > 0-3: 4 > 1095-7: 4 > 0-5: 8 > 25388-7: 818 > 10490-7: 1582 > 0-7: 51507 > > > On Fri, Jan 22, 2010 at 4:21 PM, max at bruningsystems.com > <mailto:max at bruningsystems.com> <max at bruningsystems.com > <mailto:max at bruningsystems.com>> wrote: > > Hi Qihua Wu, > > Here is a script that shows process args for process that was > interrupted, and cpu. It is easily modified > to add pid. > > #!/usr/sbin/dtrace -s > > interrupt-start > { > self->intr = curthread->t_intr; > } > > interrupt-start > /self->intr/ > { > @[stringof(self->intr->t_procp->p_user.u_psargs), cpu] = count(); > } > > interrupt-complete > /self->intr/ > { > self->intr = 0; > } > > > max > > > Qihua Wu wrote: > > So do we have a way to know what causes the interrupt (this > may be got by > intrstat) and which pid/execname is interrupted? > > Thanks > Daniel > > On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro > <James.Mauro at sun.com <mailto:James.Mauro at sun.com>> wrote: > > > > "sched" is the execname of the PID 0 process (run "ps -e"). > > The string "sched" gets plugged into the DTrace execname > variable > if the CPU is in an interrupt handler when a probe fires. > > CPU 0 is very likely taking the clock interrupts, which by > default > occur every 10 milliseconds. > > HTH, > /jim > > > > > Qihua Wu wrote: > > > > Why sched is often interrupt on cpu 0 instead of > distribution evenly on > all cpu? > > dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] > = count(); }'' > dtrace: description ''sdt:::interrupt-start '' matched 1 > probe > ^C > > sched > 4 > 355 > sched > 18 > 358 > sched > 8 > 369 > sched > 10 > 370 > sched > 11 > 370 > sched > 12 > 408 > sched > 13 > 408 > sched > 15 > 408 > sched > 19 > 408 > sched > 20 > 408 > sched > 21 > 408 > sched > 22 > 408 > sched > 23 > 408 > sched > 14 > 409 > sched > 9 > 410 > sched > 2 > 411 > sched > 16 > 411 > sched > 6 > 422 > sched > 17 > 456 > sched > 3 > 469 > sched > 5 > 510 > sched > 7 > 564 > sched > 1 > 3079 > sched > 0 > 535071 > > Thanks, > Daniel > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > <mailto:dtrace-discuss at opensolaris.org> > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > <mailto:dtrace-discuss at opensolaris.org> > > >
Brian Xu - Sun Microsystems - Beijing China
2010-Jan-26 03:02 UTC
[dtrace-discuss] about sched interrupt
On 01/25/10 11:07, Qihua Wu wrote:> Hi Max/Jim, > > Based on your script, I created the following script to print which > pid is interrupted how many times every 2 second. The result shows pid > 0 is interrupted most: about 50k every 2 second, that''s 25k per > second. How could it be interrupted so frequently? For the clock > interrupt, it only interrupts 100 times per second, which is much less > than 25k per second. Where could it come from?In solaris, the basic schedule unit is ''thread'', so when a user process is forked, at lease one lwp/thread pair will be created in the kernel for that process.(if the process is a multithread process, the kernel would create multiple those pairs). Besides those user created process, which would be scheduled on the CPU through its underlying thread, there are also kernel threads running on the CPU, which have no corresponding user process(PID). For simplicity, a PID 0 is defined for all of the kernel threads. so all kernel thread( a kthread_t structure in the kernel) has its t_procp as &p0(and pid_id as 0 of course). For interrupt, in solaris, all of the level interrupt(NIC, HBA etc) as handled as kernel thread. so for your dtrace script, when the ''interrult-start'' probe fires, the curthread is the interrupt thread, and curthread->t_intr is the thread which is interrupted by the hardware interrupt. This hardware interrupt can occur at any time, when the CPU may run a kernel thread, or a user thread, saying mysqld, or a special kernel thread named ''idle''.(this thread has also t_procp as &p0, and has a priority as 0, and can be preempted by any other thread). Now let me explain your dtrace output, 10490-7: 4962 means the interrupt occurs when the CPU is running the mysqld process. 0-7: 49131 means the interrupt occurs when the CPU is idle or running a kernel thread. You can know which CPU the interrupt is on with mpstat(1M), and know which CPU is assigned to a hardware interupt by ''::interrupts'' in mdb. -Brian> > The box has lots of network traffic from mysql, I think the network > interrupt will cause the mysqld (pid 10490) to be interrupted instead > of pid 0. > > #!/usr/sbin/dtrace -s > > interrupt-start > { > self->intr = curthread->t_intr; > } > > interrupt-start > /self->intr/ > { > @inter[self->intr->t_procp->p_pidp->pid_id, cpu] = count(); > } > > interrupt-complete > /self->intr/ > { > self->intr = 0; > } > > profile:::tick-2sec > { > printf("report completed at time:%Y\n",walltimestamp); > printf("*******************************************\n"); > printa("%d-%d: %@d\n", @inter); > trunc(@inter,0); > } > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:48:56 > ******************************************* > 0-0: 1 > 0-2: 1 > 0-8: 1 > 0-12: 1 > 0-14: 1 > 0-16: 1 > 0-20: 1 > 0-21: 1 > 0-22: 1 > 1095-7: 4 > 0-6: 5 > 0-5: 8 > 264-7: 8 > 10490-7: 4962 > 0-7: 49131 > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:48:58 > ******************************************* > 0-0: 1 > 0-5: 4 > 1095-7: 4 > 10490-7: 4200 > 0-7: 50305 > > 0 89152 :tick-2sec report completed at > time:2010 Jan 24 02:49:00 > ******************************************* > 0-3: 4 > 1095-7: 4 > 0-5: 8 > 25388-7: 818 > 10490-7: 1582 > 0-7: 51507 > > > On Fri, Jan 22, 2010 at 4:21 PM, max at bruningsystems.com > <mailto:max at bruningsystems.com> <max at bruningsystems.com > <mailto:max at bruningsystems.com>> wrote: > > Hi Qihua Wu, > > Here is a script that shows process args for process that was > interrupted, and cpu. It is easily modified > to add pid. > > #!/usr/sbin/dtrace -s > > interrupt-start > { > self->intr = curthread->t_intr; > } > > interrupt-start > /self->intr/ > { > @[stringof(self->intr->t_procp->p_user.u_psargs), cpu] = count(); > } > > interrupt-complete > /self->intr/ > { > self->intr = 0; > } > > > max > > > Qihua Wu wrote: > > So do we have a way to know what causes the interrupt (this > may be got by > intrstat) and which pid/execname is interrupted? > > Thanks > Daniel > > On Thu, Jan 21, 2010 at 10:43 PM, Jim Mauro > <James.Mauro at sun.com <mailto:James.Mauro at sun.com>> wrote: > > > > "sched" is the execname of the PID 0 process (run "ps -e"). > > The string "sched" gets plugged into the DTrace execname > variable > if the CPU is in an interrupt handler when a probe fires. > > CPU 0 is very likely taking the clock interrupts, which by > default > occur every 10 milliseconds. > > HTH, > /jim > > > > > Qihua Wu wrote: > > > > Why sched is often interrupt on cpu 0 instead of > distribution evenly on > all cpu? > > dtrace -n ''sdt:::interrupt-start { @num[execname,cpu] > = count(); }'' > dtrace: description ''sdt:::interrupt-start '' matched 1 > probe > ^C > > sched > 4 > 355 > sched > 18 > 358 > sched > 8 > 369 > sched > 10 > 370 > sched > 11 > 370 > sched > 12 > 408 > sched > 13 > 408 > sched > 15 > 408 > sched > 19 > 408 > sched > 20 > 408 > sched > 21 > 408 > sched > 22 > 408 > sched > 23 > 408 > sched > 14 > 409 > sched > 9 > 410 > sched > 2 > 411 > sched > 16 > 411 > sched > 6 > 422 > sched > 17 > 456 > sched > 3 > 469 > sched > 5 > 510 > sched > 7 > 564 > sched > 1 > 3079 > sched > 0 > 535071 > > Thanks, > Daniel > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > <mailto:dtrace-discuss at opensolaris.org> > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org > <mailto:dtrace-discuss at opensolaris.org> > > > > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20100126/0c292dd2/attachment.html>