Some stack sections in the application disappeared, Both pstack/dbx can not find the call stack and only knows they stopped in lwp_park. below pmap resoults showed that the stack of thread 9 disappeared FAA74000 32 32 32 - 8K rw--R [ stack tid=12 ] FAB78000 8 8 - - - rw--R [ anon ] FAB7A000 8 8 8 - 8K rw--R [ stack tid=11 ] FAC7A000 8 8 - - - rw--R [ stack tid=10 ] FAD78000 8 8 8 - 8K rw--R [ anon ] FAE78000 16 16 - - - rw--R [ stack tid=8 ] FAF74000 8 8 - - - rw--R [ anon ] FAF76000 24 24 24 - 8K rw--R [ stack tid=7 ] FB078000 16 16 - - - rw--R [ stack tid=6 ] So I wonder if it is possible for some other threads to unmap those stack sections mistakenly. I tried to use dtrace to check who called munmap but found nothing when even more threads lost their stack space. Later, some more stranger thing happened, some stack losted previously come back! Things like below: Previously, I can''t find thread#4 this morning by pstack ----------------- lwp# 4 / thread# 4 -------------------- fbac8a08 lwp_park (0, 0, 0) however, now I can find it by pstack . why ? ----------------- lwp# 4 / thread# 4 -------------------- fbac8a08 lwp_park (0, 0, 0) fbac2a1c cond_wait_queue (fb27b960, 188900, 0, 0, 1c00, 0) + 4c fbac2f64 cond_wait (fb27b960, 188900, ffffffff, fffffff8, ffffffe0, fb27b979) + 10 fbac2fa0 pthread_cond_wait (fb27b960, 188900, 0, fb27bbc8, 1, 0) + 8 fc6b76b4 int ACE_OS::cond_timedwait(_pthread_cond*,_pthread_mutex*,ACE_Time_Value*) (fb27b960, 188900, 0, 0, b, 0) + 8c fc6b7d9c int ACE_Condition_Thread_Mutex::wait(ACE_Thread_Mutex&,const ACE_Time_Value*) (fb27b960, 188900, 0, 0, fb27bc68, 1) + 24 fc6b7de8 int ACE_Condition_Thread_Mutex::wait(const ACE_Time_Value*) (fb27b960, 0, 188930, 0, 0, 2e) + 20 fc778658 int ACE_Token::ACE_Token_Queue_Entry::wait(ACE_Time_Value*,ACE_Thread_Mutex&) (fb27b958, 0, 188900, 188930, fb3e1200, 8b2e8) + 20 fc777b8c int ACE_Token::shared_acquire(void(*)(void*),void*,ACE_Time_Value*,ACE_Token::ACE_Token_Op_Type) (1888e8, fc77ab40, 0, 0, 1, fc69e7e4) + 2f4 fc77ab14 int ACE_Token::acquire_read(void(*)(void*),void*,ACE_Time_Value*) (1888e8, fc77ab40, 0, 0, 75, fda008) + 34 fc7787a0 int ACE_TP_Token_Guard::acquire_read_token(ACE_Time_Value*) (fb27baf4, 0, 0, 4, b, 1) + b0 fc778bdc int ACE_TP_Reactor::handle_events(ACE_Time_Value*) (188248, 0, 4, fb27bbc8, 1, 0) + 4c fc742588 int ACE_Reactor::handle_events(ACE_Time_Value*) (1b5d50, 0, fcb03154, 0, b, fbb303a8) + 40 fca573a4 int TAO_ORB_Core::run(ACE_Time_Value*,int) (117488, 0, 0, 12, fb27bc68, 1) + 364 fca47544 void CORBA::ORB::run(ACE_Time_Value*) (13e788, 0, 0, 10, 1cf4, fbb36000) + 2c fca474c0 void CORBA::ORB::run() (13e788, 13e788, 0, 0, fb3e1200, 10) + 10 fdcd3794 void*FMK_CorbaThreadPool::poolThread(void*) (1e3a08, fb27c000, 0, 0, fdcd35a0, 1) + 1f4 fbac8968 _lwp_start (0, 0, 0, 0, 0, 0) I have used dtrace to track the calling to munmap but can not find any meaningful thing. Some more stacks lost their stack without any "munmap" probes has been triggered. Another more stranger thing is that some thread stack come back! So seems the application used some other methods to deal with the mapping, Or, I wonder if it is possible for the kernel to move those stacks underground, for those threads have never been switched on (Or else, it will core dump) pmap.out.2:FAC7A000 8K rw--R [ stack tid=10 ] pmap.out.2:FAF74000 32K rw--R [ stack tid=7 ] pmap.out.2:FB07A000 8K rw--R [ stack tid=6 ] pmap.out.2:FB274000 32K rw--R [ stack tid=4 ] pmap.out.2:FB374000 32K rw--R [ stack tid=3 ] pmap.out.2:FFBF4000 48K rw--- [ stack ] in above thread 2,5,8,9 lost stack mapping pmap.out.dec.16:FAB6C000 64K rw--R [ stack tid=11 ] pmap.out.dec.16:FAE6C000 64K rw--R [ stack tid=8 ] pmap.out.dec.16:FAF6C000 64K rw--R [ stack tid=7 ] pmap.out.dec.16:FB16C000 64K rw--R [ stack tid=5 ] pmap.out.dec.16:FB26C000 64K rw--R [ stack tid=4 ] pmap.out.dec.16:FB36C000 64K rw--R [ stack tid=3 ] pmap.out.dec.16:FFBF4000 48K rw--- [ stack ] here thread 5,8 get their stack back. So, Are there any good idea to find out who is doing these kind of jobs behind? Thanks and Rds. Lamb.