dtrace discuss - Feb 2010 - __lwp_park() showing up as most time-consuming when dtrace an application

Hi,  when doing dtrace on an appliction, __lwp_park() seems to be taking a lot
of time.  What does it really do?   is it waiting for threads?

Thanks,
-- 
This message posted from opensolaris.org

It''s used to put threads to sleep that are blocking on user locks
(at least that''s my recollection).

Run "prstat -Lmp <PID_OF_APP) - what does  the "LCK" column
look like?

Try running plockstat -A -p <PID_OF_APP>.

Thanks,
/jim


Dtrace Email wrote:
> Hi,  when doing dtrace on an appliction, __lwp_park() seems to be taking a
lot of time.  What does it really do?   is it waiting for threads?
>
> Thanks,
>

Yep.  The system call is lwp_park().  Given that it''s a blocking
system call, the time a thread spends here is effectively unbounded.

Chad

On Wed, Feb 3, 2010 at 3:47 PM, Jim Mauro <James.Mauro at sun.com>
wrote:
> It''s used to put threads to sleep that are blocking on user locks
> (at least that''s my recollection).
>
> Run "prstat -Lmp <PID_OF_APP) - what does ?the "LCK"
column look like?
>
> Try running plockstat -A -p <PID_OF_APP>.
>
> Thanks,
> /jim
>
>
> Dtrace Email wrote:
>>
>> Hi, ?when doing dtrace on an appliction, __lwp_park() seems to be
taking a
>> lot of time. ?What does it really do? ? is it waiting for threads?
>>
>> Thanks,
>>
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Yes.  Lots of info in 
http://www.sun.com/software/whitepapers/solaris9/multithread.pdf

Jim Mauro wrote:
> It''s used to put threads to sleep that are blocking on user locks
> (at least that''s my recollection).
>
> Run "prstat -Lmp <PID_OF_APP) - what does  the "LCK"
column look like?
>
> Try running plockstat -A -p <PID_OF_APP>.
>
> Thanks,
> /jim
>
>
> Dtrace Email wrote:
>> Hi,  when doing dtrace on an appliction, __lwp_park() seems to be 
>> taking a lot of time.  What does it really do?   is it waiting for 
>> threads?
>>
>> Thanks,
>>   
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

First, thanks to Andrew for referencing my paper. Although it was 
written for the great Solaris 9 thread model U-turn, a lot of what it 
says still applies today (except that the thread library has now been 
folded into libc).

If you''re using the DTrace syscall provider you will only see lwp_park.
However, if you''re looking at stack traces, truss(1) output or the 
OpenSolaris source you may see that the one syscall entry point actually 
implements five private syscalls: lwp_park, lwp_unpark, lwp_unpark_all, 
lwp_unpark_cancel and lwp_park with schedctl support. Of these, only the 
first three are of real interest.

The lwp_park syscalls are designed for the simplest and quickest thread 
sleep/wakeup mechanism with intra-process scope. They are used to 
implement PTHREAD_PROCESS_PRIVATE (aka USYNC_THREAD) mutexes, condvars, 
semaphores and rwlocks.

For inter-process synchronisation (i.e. PTHREAD_PROCESS_SHARED (aka 
USYNC_PROCESS) mutexes, condvars, semaphores and rwlocks) you will see 
object specific syscalls (e.g. lwp_mutex_lock, lwp_cond_broadcast, 
lwp_sema_post, lwp_rwlock_unlock).

PTHREAD_PROCESS_PRIVATE synchronisation uses user-level sleep queues 
(implemented in libc). When a thread needs to sleep on a queue, it calls 
lwp_park(). When another thread wants to wake just one thread from a 
user-level sleep queue is uses lwp_unpark(). To wake all sleepers it 
uses lwp_unpark_all(). For PTHREAD_PROCESS_PRIVATE synchronisation the 
kernel has no notion of the type of synchronisation object being waited 
for, it just provides the simple park/unpark mechanism

PTHREAD_PROCESS_SHARED synchronisation uses kernel sleep queues (known 
as LWP wait channels) because only the kernel has visibility across LWPs 
in multiple processes. For PTHREAD_PROCESS_SHARED synchronisation the 
kernel has to deal with object specific semantics, so hence requires 
many more syscalls.

By analysing lwp_* syscalls, together with their associated stack 
traces, one can understand a lot about an application''s scalability and
the quality of its implementation. If you see a lot of lwp_park activity 
around mutexes, you know that you have potentially limiting mutex 
contention (i.e. that a mutex couldn''t be acquired with a simple 
instruction or after a short spin). If you see excessive lwp_park 
activity around condvars, you probably have poor workflow between the 
application''s various threads.

Of course, it can a lot of experience and a trained eye to really 
understand what is going on and to take steps to improve application 
scalability. For such situations you might consider hiring in some 
expert help :)

Phil Harman
Harman Holistix - focusing on the detail and the big picture
Our holistic services include: performance health checks, system tuning, 
DTrace training, coding advice, developer assassinations

http://blogs.sun.com/pgdh (mothballed)
http://harmanholistix.com/mt (current)
http://linkedin.com/in/philharman


On 03/02/2010 20:55, Andrew Gabriel wrote:
> Yes.  Lots of info in 
> http://www.sun.com/software/whitepapers/solaris9/multithread.pdf
>
> Jim Mauro wrote:
>> It''s used to put threads to sleep that are blocking on user
locks
>> (at least that''s my recollection).
>>
>> Run "prstat -Lmp <PID_OF_APP) - what does  the "LCK"
column look like?
>>
>> Try running plockstat -A -p <PID_OF_APP>.
>>
>> Thanks,
>> /jim
>>
>>
>> Dtrace Email wrote:
>>> Hi,  when doing dtrace on an appliction, __lwp_park() seems to be 
>>> taking a lot of time.  What does it really do?   is it waiting for 
>>> threads?
>>>
>>> Thanks,
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org