search for: gluster_strac

...e data to cluster from 100 clients for 12 hours again no problem. But when I try to reboot a node, glusterd process hangs on %100 CPU usage and seems to do nothing, no brick processes come online. You can find strace of glusterd process for 1 minutes here: https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 Here is the glusterd logs: https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 By the way, reboot of one server completes without problem if I reboot the servers before creating any volumes.

...00 clients for 12 hours again > no problem. But when I try to reboot a node, glusterd process hangs on > %100 CPU usage and seems to do nothing, no brick processes come > online. You can find strace of glusterd process for 1 minutes here: > > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 > > Here is the glusterd logs: > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 > > > By the way, reboot of one server completes without problem if I reboot > the servers before creating any volumes.

...gain > > no problem. But when I try to reboot a node, glusterd process hangs on > > %100 CPU usage and seems to do nothing, no brick processes come > > online. You can find strace of glusterd process for 1 minutes here: > > > > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 > > > > Here is the glusterd logs: > > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 > > > > > > By the way, reboot of one server completes without problem if I reboot > > the servers before creating any volumes. > _______________...

...o problem. But when I try to reboot a node, glusterd process hangs on >> > %100 CPU usage and seems to do nothing, no brick processes come >> > online. You can find strace of glusterd process for 1 minutes here: >> > >> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >> > >> > Here is the glusterd logs: >> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >> > >> > >> > By the way, reboot of one server completes without problem if I reboot >> > the servers before creating any v...

...hen I try to reboot a node, glusterd process hangs on >>> > %100 CPU usage and seems to do nothing, no brick processes come >>> > online. You can find strace of glusterd process for 1 minutes here: >>> > >>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >>> > >>> > Here is the glusterd logs: >>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >>> > >>> > >>> > By the way, reboot of one server completes without problem if I reboot >>> > the s...

...oot a node, glusterd process hangs on >>>> > %100 CPU usage and seems to do nothing, no brick processes come >>>> > online. You can find strace of glusterd process for 1 minutes here: >>>> > >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >>>> > >>>> > Here is the glusterd logs: >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >>>> > >>>> > >>>> > By the way, reboot of one server completes without problem if I reboot...

...ess hangs > on > >>>> > %100 CPU usage and seems to do nothing, no brick processes come > >>>> > online. You can find strace of glusterd process for 1 minutes here: > >>>> > > >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 > >>>> > > >>>> > Here is the glusterd logs: > >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 > >>>> > > >>>> > > >>>> > By the way, reboot of one server complet...

...n >> >>>> > %100 CPU usage and seems to do nothing, no brick processes come >> >>>> > online. You can find strace of glusterd process for 1 minutes here: >> >>>> > >> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >> >>>> > >> >>>> > Here is the glusterd logs: >> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >> >>>> > >> >>>> > >> >>>> > By the way, rebo...

...for 1 >> minutes >> >> >>>> >>>> > here: >> >> >>>> >>>> > >> >> >>>> >>>> > >> >> >>>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace. >> out?dl=0 >> >> >>>> >>>> > >> >> >>>> >>>> > Here is the glusterd logs: >> >> >>>> >>>> > https://www.dropbox.com/s/hkst >> b3mdeil9a5u/glusterd.log?dl=0 >> &g...

...>> >>>> >>>> > >> >>> >> >>>> >>>> > >> >>> >> >>>> >>>> > >> >>> >> >>>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >> >>> >> >>>> >>>> > >> >>> >> >>>> >>>> > Here is the glusterd logs: >> >>> >> >>>> >>>> > >> >>> >> >>>> >...

...;>>> >>>> > >>> >>> >> >>>> >>>> > >>> >>> >> >>>> >>>> > >>> >>> >> >>>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >>> >>> >> >>>> >>>> > >>> >>> >> >>>> >>>> > Here is the glusterd logs: >>> >>> >> >>>> >>>> > >>> >>> >> >&...

...; >>>> > here: >> >>> >>> >>>> > >> >>> >>> >>>> > >> >>> >>> >>>> > >> >>> >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >> >>> >>> >>>> > >> >>> >>> >>>> > Here is the glusterd logs: >> >>> >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >> >>> >>&...

...> > minutes > > >>> >>> >>>> > here: > > >>> >>> >>>> > > > >>> >>> >>>> > > > >>> >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 > > >>> >>> >>>> > > > >>> >>> >>>> > Here is the glusterd logs: > > >>> >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 > > >>> >...

...ace of glusterd process for 1 minutes > >>> >>> >>>> > here: > >>> >>> >>>> > > >>> >>> >>>> > > >>> >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 > >>> >>> >>>> > > >>> >>> >>>> > Here is the glusterd logs: > >>> >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 > >>> >>> >>&gt...

...tes >> > >>> >>> >>>> > here: >> > >>> >>> >>>> > >> > >>> >>> >>>> > >> > >>> >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >> > >>> >>> >>>> > >> > >>> >>> >>>> > Here is the glusterd logs: >> > >>> >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >> > &g...

Hi Gaurav, Any progress about the problem? On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: > Thank you Gaurav, > Here is more findings: > Problem does not happen using only 20 servers each has 68 bricks. > (peer probe only 20 servers) > If we use 40 servers with single volume, glusterd cpu %100 state > continues for 5 minutes and it goes to

...t; > here: >>> >> >>>> >>>> > >>> >> >>>> >>>> > >>> >> >>>> >>>> > >>> >> >>>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >>> >> >>>> >>>> > >>> >> >>>> >>>> > Here is the glusterd logs: >>> >> >>>> >>>> > >>> >> >>>> >>>> > https://www.dropbo...

Serkan, I have gone through other mails in the mail thread as well but responding to this one specifically. Is this a source install or an RPM install ? If this is an RPM install, could you please install the glusterfs-debuginfo RPM and retry to capture the gdb backtrace. If this is a source install, then you'll need to configure the build with --enable-debug and reinstall and retry

...and seems to do nothing, no brick processes come >>> >>>> > online. You can find strace of glusterd process for 1 minutes >>> >>>> > here: >>> >>>> > >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0 >>> >>>> > >>> >>>> > Here is the glusterd logs: >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >>> >>>> > >>> >>>> > >>> >>>&g...

...rocesses come >> >>> >>>> > online. You can find strace of glusterd process for 1 minutes >> >>> >>>> > here: >> >>> >>>> > >> >>> >>>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace. >> out?dl=0 >> >>> >>>> > >> >>> >>>> > Here is the glusterd logs: >> >>> >>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0 >> >>> >>>> > >> >&gt...