thr3ads.net - Gluster users - [Gluster-users] Glusterd proccess hangs on reboot [Aug 2017]

If this information is useful, please help other people find it:
Share via:

Atin Mukherjee

2017-Aug-22 15:47 UTC

[Gluster-users] Glusterd proccess hangs on reboot

My guess is there is a corruption in vol list or peer list which has lead
glusterd to get into a infinite loop of traversing a peer/volume list and
CPU to hog up. Again this is a guess and I've not got a chance to take a
detail look at the logs and the strace output.

I believe if you get to reboot the node again the problem will disappear.

On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at gmail.com>
wrote:
> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42 during
> glusterd %100 cpu usage
> Hope this helps...
>
> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban <cobanserkan at
gmail.com>
> wrote:
> > Hi there,
> >
> > I have a strange problem.
> > Gluster version in 3.10.5, I am testing new servers. Gluster
> > configuration is 16+4 EC, I have three volumes, each have 1600 bricks.
> > I can successfully create the cluster and volumes without any
> > problems. I write data to cluster from 100 clients for 12 hours again
> > no problem. But when I try to reboot a node, glusterd process hangs on
> > %100 CPU usage and seems to do nothing, no brick processes come
> > online. You can find strace of glusterd process for 1 minutes here:
> >
> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
> >
> > Here is the glusterd logs:
> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
> >
> >
> > By the way, reboot of one server completes without problem if I reboot
> > the servers before creating any volumes.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170822/5a95e3d2/attachment.html>

Serkan Çoban

2017-Aug-22 16:00 UTC

head link

[Gluster-users] Glusterd proccess hangs on reboot

I reboot multiple times, also I destroyed the gluster configuration
and recreate multiple times. The behavior is same.

On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at redhat.com>
wrote:> My guess is there is a corruption in vol list or peer list which has lead
> glusterd to get into a infinite loop of traversing a peer/volume list and
> CPU to hog up. Again this is a guess and I've not got a chance to take
a
> detail look at the logs and the strace output.
>
> I believe if you get to reboot the node again the problem will disappear.
>
> On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at gmail.com>
wrote:
>>
>> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42 during
>> glusterd %100 cpu usage
>> Hope this helps...
>>
>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban <cobanserkan at
gmail.com>
>> wrote:
>> > Hi there,
>> >
>> > I have a strange problem.
>> > Gluster version in 3.10.5, I am testing new servers. Gluster
>> > configuration is 16+4 EC, I have three volumes, each have 1600
bricks.
>> > I can successfully create the cluster and volumes without any
>> > problems. I write data to cluster from 100 clients for 12 hours
again
>> > no problem. But when I try to reboot a node, glusterd process
hangs on
>> > %100 CPU usage and seems to do nothing, no brick processes come
>> > online. You can find strace of glusterd process for 1 minutes
here:
>> >
>> > https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>> >
>> > Here is the glusterd logs:
>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>> >
>> >
>> > By the way, reboot of one server completes without problem if I
reboot
>> > the servers before creating any volumes.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> --
> - Atin (atinm)

Serkan Çoban

2017-Aug-23 07:02 UTC

head link

[Gluster-users] Glusterd proccess hangs on reboot

Same thing happens with 3.12.rc0. This time perf top shows hanging in
libglusterfs.so and below is the glusterd logs, which are different
from 3.10.
With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see
brick processes come online and system starts to answer commands like
"gluster peer status"..

[2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.152287] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153503] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153647] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153866] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153948] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154018] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154108] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154162] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154250] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154322] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154425] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154494] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154575] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154649] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154705] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154774] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154852] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154903] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154995] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.155052] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.155141] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:27.074052] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:27.077034] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]

On Tue, Aug 22, 2017 at 7:00 PM, Serkan ?oban <cobanserkan at gmail.com>
wrote:> I reboot multiple times, also I destroyed the gluster configuration
> and recreate multiple times. The behavior is same.
>
> On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at
redhat.com> wrote:
>> My guess is there is a corruption in vol list or peer list which has
lead
>> glusterd to get into a infinite loop of traversing a peer/volume list
and
>> CPU to hog up. Again this is a guess and I've not got a chance to
take a
>> detail look at the logs and the strace output.
>>
>> I believe if you get to reboot the node again the problem will
disappear.
>>
>> On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>>>
>>> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42
during
>>> glusterd %100 cpu usage
>>> Hope this helps...
>>>
>>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban <cobanserkan at
gmail.com>
>>> wrote:
>>> > Hi there,
>>> >
>>> > I have a strange problem.
>>> > Gluster version in 3.10.5, I am testing new servers. Gluster
>>> > configuration is 16+4 EC, I have three volumes, each have 1600
bricks.
>>> > I can successfully create the cluster and volumes without any
>>> > problems. I write data to cluster from 100 clients for 12
hours again
>>> > no problem. But when I try to reboot a node, glusterd process
hangs on
>>> > %100 CPU usage and seems to do nothing, no brick processes
come
>>> > online. You can find strace of glusterd process for 1 minutes
here:
>>> >
>>> >
https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>>> >
>>> > Here is the glusterd logs:
>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>>> >
>>> >
>>> > By the way, reboot of one server completes without problem if
I reboot
>>> > the servers before creating any volumes.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> --
>> - Atin (atinm)

Possibly Parallel Threads

Search for more apparently analagous threads

Gluster users - Aug 2017 - Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

Possibly Parallel Threads