similar to: find thread that is causing high cpu usage

[Just starting out with DTrace and was hoping to get some guidance.] I have a "benchmark" program that I monitored with both prstat (prstat -mL -P <PID>) and pfilestat (from the DTrace toolkit). Prstat reports LAT values in the 0.1-0.2% range, but pfilestat reports "waitcpu" values in the 6-10%. Since those two numbers supposedly represent time waiting for the CPU,

I have a SunRay server that I am looking at to determine some sizing requirements in my department. The machine has 16G of ram and 10G of swap. Currently, I have about 4G of swap used. I am wondering if dtrace/mdb can be used to find out what lwp/processes have been swapped out? Any hints? This message posted from opensolaris.org

I''m looking for a suggestion on a good way to hunt down the source of high context switching and mutex contentions... Is dtrace the way to go now, or should I stick with something like lockstat? Russ This is a 5 second interval for mpstat: CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 16 0 0 1115 1241 206 9095 912 2420 7393 0 12105 68 25

Hi there. I'm running an AD DC in a KVM VM, a centos7 server. named is showing a very hight cpu usage, close to 100%. I just increased the number of CPUs to 2 to cope. This is a small company, with perhaps 20 workstations and another 20 linux servers. There shouldn't be that much happening. top - 17:27:48 up 2:29, 1 user, load average: 1.08, 1.13, 1.09 Tasks: 139 total, 1 running,

>From time to time a jboss process would end eating all, the available CPU and the load avg would skyrocket. Once the operators restarted jboss, the system''d be normal again (sometimes for weeks) until the next incident Since we moved the app from a v440 running Solaris 10 8/07 to a t2000 running Solaris 10 5/08 the problem started to happen more frequently (2-3 times a week). The

Hi world, I have a 10-disk RAID-Z2 system with 4 GB of DDR2 RAM and a 3 GHz Core 2 Duo. It''s exporting ~280 filesystems over NFS to about half a dozen machines. Under some loads (in particular, any attempts to rsync between another machine and this one over SSH), the machine''s load average sometimes goes insane (27+), and it appears to all be in kernel-land (as nothing in

Folks, Perplexing question about load average display with prstat -Z Solaris 10 OS U4 (08/07) We have 4 zones with very different processes and workloads.. The prstat -Z command issued within each of the zones, correctly displays the number of processes and lwps, but the load average value looks exactly the same on all non-global zones..I mean all 3 values (1,5,15 load averages) are the same

Just to let people know what my big picture is, I''m trying to write a script that will let me run a program, and name a progeny of that program that I want to debug. My script should find the first occurrence of that progeny, and run it until it finishes initializing the runtime linker, but stop it before it runs any shared library startup routines. (Failing that, I''d be okay

when check java process lock statistics, plockstat failed, please see below: # prstat -mLp 21162 PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 21162 7677 0.9 0.1 0.0 0.0 0.0 99 0.0 0.3 83 89 215 0 java/81 21162 7677 0.3 0.1 0.0 0.0 0.0 0.0 99 0.2 106 33 305 0 java/35 21162 7677 0.1 0.0 0.0 0.0 0.0 100 0.0 0.1 79 6 85 0 java/59

On Mon, 4 Sep 2017 at 20:04, Serkan ?oban <cobanserkan at gmail.com> wrote: > I have been using a 60 server 1560 brick 3.7.11 cluster without > problems for 1 years. I did not see this problem with it. > Note that this problem does not happen when I install packages & start > glusterd & peer probe and create the volumes. But after glusterd > restart. > > Also

On Fri, Sep 1, 2017 at 8:47 AM, Milind Changire <mchangir at redhat.com> wrote: > Serkan, > I have gone through other mails in the mail thread as well but responding > to this one specifically. > > Is this a source install or an RPM install ? > If this is an RPM install, could you please install the > glusterfs-debuginfo RPM and retry to capture the gdb backtrace. >

On Mon, Sep 4, 2017 at 5:28 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > >1. On 80 nodes cluster, did you reboot only one node or multiple ones? > Tried both, result is same, but the logs/stacks are from stopping and > starting glusterd only on one server while others are running. > > >2. Are you sure that pstack output was always constantly pointing on >

hello! First of all thank you for all your information you give us on this sites. It is a great help to learn for example dtrace. i''ve a little script in work, which shows the cpu utilization per process in percent. this output is refreshed every 2 seconds for example. now i want to extend this script with the output of the allocated bytes of memory per process. how could i display this

Hi, You can find pstack sampes here: https://www.dropbox.com/s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 Here is the first one: Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000310fe37d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 #3

Serkan, I have gone through other mails in the mail thread as well but responding to this one specifically. Is this a source install or an RPM install ? If this is an RPM install, could you please install the glusterfs-debuginfo RPM and retry to capture the gdb backtrace. If this is a source install, then you'll need to configure the build with --enable-debug and reinstall and retry

Some corrections about the previous mails. Problem does not happen when no volumes created. Problem happens volumes created but in stopped state. Problem also happens when volumes started state. Below is the 5 stack traces taken by 10 min intervals and volumes stopped state. --1-- Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1

Here you can find 10 stack trace samples from glusterd. I wait 10 seconds between each trace. https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 Content of the first stack trace is here: Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 #2

>1. On 80 nodes cluster, did you reboot only one node or multiple ones? Tried both, result is same, but the logs/stacks are from stopping and starting glusterd only on one server while others are running. >2. Are you sure that pstack output was always constantly pointing on strcmp being stuck? It stays 70-80 minutes in %100 cpu consuming state, the stacks I send is from first 5-10 minutes.

No worries Serkan, You can continue to use your 40 node clusters. The backtrace has resolved the function names and it *should* be sufficient to debug the issue. Thanks for letting us know. We'll post on this thread again to notify you about the findings. On Sat, Sep 2, 2017 at 2:42 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Milind, > > Anything new about the

G''Day Folks, I''ve finally typed up my classic DTrace demo, which I use to introduce people to DTrace (I delivered this at SOSUG#1), http://www.brendangregg.com/DTrace/dtracevstruss.html Here I create a fault and show the difficulty in analysing it using previous tools. Then I compare the impact of analysing the problem using both DTrace and truss. DTrace wins (a lot!).