thr3ads.net - Gluster users - [Gluster-users] Glusterd proccess hangs on reboot [Aug 2017]

If this information is useful, please help other people find it:
Share via:

Serkan Çoban

2017-Aug-23 07:02 UTC

[Gluster-users] Glusterd proccess hangs on reboot

Same thing happens with 3.12.rc0. This time perf top shows hanging in
libglusterfs.so and below is the glusterd logs, which are different
from 3.10.
With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see
brick processes come online and system starts to answer commands like
"gluster peer status"..

[2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.152287] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153503] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153647] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153866] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.153948] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154018] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154108] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154162] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154250] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154322] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154425] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154494] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154575] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154649] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154705] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154774] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154852] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154903] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.154995] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.155052] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:02.155141] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:27.074052] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
[2017-08-23 06:46:27.077034] E [client_t.c:324:gf_client_ref]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
[0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
[0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
[0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]

On Tue, Aug 22, 2017 at 7:00 PM, Serkan ?oban <cobanserkan at gmail.com>
wrote:> I reboot multiple times, also I destroyed the gluster configuration
> and recreate multiple times. The behavior is same.
>
> On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at
redhat.com> wrote:
>> My guess is there is a corruption in vol list or peer list which has
lead
>> glusterd to get into a infinite loop of traversing a peer/volume list
and
>> CPU to hog up. Again this is a guess and I've not got a chance to
take a
>> detail look at the logs and the strace output.
>>
>> I believe if you get to reboot the node again the problem will
disappear.
>>
>> On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>>>
>>> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42
during
>>> glusterd %100 cpu usage
>>> Hope this helps...
>>>
>>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban <cobanserkan at
gmail.com>
>>> wrote:
>>> > Hi there,
>>> >
>>> > I have a strange problem.
>>> > Gluster version in 3.10.5, I am testing new servers. Gluster
>>> > configuration is 16+4 EC, I have three volumes, each have 1600
bricks.
>>> > I can successfully create the cluster and volumes without any
>>> > problems. I write data to cluster from 100 clients for 12
hours again
>>> > no problem. But when I try to reboot a node, glusterd process
hangs on
>>> > %100 CPU usage and seems to do nothing, no brick processes
come
>>> > online. You can find strace of glusterd process for 1 minutes
here:
>>> >
>>> >
https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>>> >
>>> > Here is the glusterd logs:
>>> > https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>>> >
>>> >
>>> > By the way, reboot of one server completes without problem if
I reboot
>>> > the servers before creating any volumes.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> --
>> - Atin (atinm)

Serkan Çoban

2017-Aug-23 14:44 UTC

head link

[Gluster-users] Glusterd proccess hangs on reboot

Hi Atin,

Do you have time to check the logs?

On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com>
wrote:> Same thing happens with 3.12.rc0. This time perf top shows hanging in
> libglusterfs.so and below is the glusterd logs, which are different
> from 3.10.
> With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see
> brick processes come online and system starts to answer commands like
> "gluster peer status"..
>
> [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.152287] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.153503] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.153647] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.153866] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.153948] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154018] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154108] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154162] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154250] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154322] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154425] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154494] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154575] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154649] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154705] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154774] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154852] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154903] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.154995] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.155052] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:02.155141] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:27.074052] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> [2017-08-23 06:46:27.077034] E [client_t.c:324:gf_client_ref]
> (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
>
> On Tue, Aug 22, 2017 at 7:00 PM, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>> I reboot multiple times, also I destroyed the gluster configuration
>> and recreate multiple times. The behavior is same.
>>
>> On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at
redhat.com> wrote:
>>> My guess is there is a corruption in vol list or peer list which
has lead
>>> glusterd to get into a infinite loop of traversing a peer/volume
list and
>>> CPU to hog up. Again this is a guess and I've not got a chance
to take a
>>> detail look at the logs and the strace output.
>>>
>>> I believe if you get to reboot the node again the problem will
disappear.
>>>
>>> On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at
gmail.com> wrote:
>>>>
>>>> As an addition perf top shows %80 libc-2.12.so __strcmp_sse42
during
>>>> glusterd %100 cpu usage
>>>> Hope this helps...
>>>>
>>>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban <cobanserkan
at gmail.com>
>>>> wrote:
>>>> > Hi there,
>>>> >
>>>> > I have a strange problem.
>>>> > Gluster version in 3.10.5, I am testing new servers.
Gluster
>>>> > configuration is 16+4 EC, I have three volumes, each have
1600 bricks.
>>>> > I can successfully create the cluster and volumes without
any
>>>> > problems. I write data to cluster from 100 clients for 12
hours again
>>>> > no problem. But when I try to reboot a node, glusterd
process hangs on
>>>> > %100 CPU usage and seems to do nothing, no brick processes
come
>>>> > online. You can find strace of glusterd process for 1
minutes here:
>>>> >
>>>> >
https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
>>>> >
>>>> > Here is the glusterd logs:
>>>> >
https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
>>>> >
>>>> >
>>>> > By the way, reboot of one server completes without problem
if I reboot
>>>> > the servers before creating any volumes.
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>> --
>>> - Atin (atinm)

Atin Mukherjee

2017-Aug-23 14:52 UTC

head link

[Gluster-users] Glusterd proccess hangs on reboot

Not yet. Gaurav will be taking a look at it tomorrow.

On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com>
wrote:
> Hi Atin,
>
> Do you have time to check the logs?
>
> On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at
gmail.com>
> wrote:
> > Same thing happens with 3.12.rc0. This time perf top shows hanging in
> > libglusterfs.so and below is the glusterd logs, which are different
> > from 3.10.
> > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see
> > brick processes come online and system starts to answer commands like
> > "gluster peer status"..
> >
> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.152287] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.153503] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.153647] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.153866] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.153948] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154018] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154108] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154162] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154250] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154322] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154425] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154494] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154575] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154649] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154705] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154774] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154852] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154903] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.154995] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.155052] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:02.155141] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:27.074052] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> > [2017-08-23 06:46:27.077034] E [client_t.c:324:gf_client_ref]
> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1)
> > [0x7f5ae2c091b1]
-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c)
> > [0x7f5ae2c0851c]
-->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9)
> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument]
> >
> > On Tue, Aug 22, 2017 at 7:00 PM, Serkan ?oban <cobanserkan at
gmail.com>
> wrote:
> >> I reboot multiple times, also I destroyed the gluster
configuration
> >> and recreate multiple times. The behavior is same.
> >>
> >> On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at
redhat.com>
> wrote:
> >>> My guess is there is a corruption in vol list or peer list
which has
> lead
> >>> glusterd to get into a infinite loop of traversing a
peer/volume list
> and
> >>> CPU to hog up. Again this is a guess and I've not got a
chance to take
> a
> >>> detail look at the logs and the strace output.
> >>>
> >>> I believe if you get to reboot the node again the problem will
> disappear.
> >>>
> >>> On Tue, 22 Aug 2017 at 20:07, Serkan ?oban <cobanserkan at
gmail.com>
> wrote:
> >>>>
> >>>> As an addition perf top shows %80 libc-2.12.so
__strcmp_sse42 during
> >>>> glusterd %100 cpu usage
> >>>> Hope this helps...
> >>>>
> >>>> On Tue, Aug 22, 2017 at 2:41 PM, Serkan ?oban
<cobanserkan at gmail.com>
> >>>> wrote:
> >>>> > Hi there,
> >>>> >
> >>>> > I have a strange problem.
> >>>> > Gluster version in 3.10.5, I am testing new servers.
Gluster
> >>>> > configuration is 16+4 EC, I have three volumes, each
have 1600
> bricks.
> >>>> > I can successfully create the cluster and volumes
without any
> >>>> > problems. I write data to cluster from 100 clients
for 12 hours
> again
> >>>> > no problem. But when I try to reboot a node, glusterd
process hangs
> on
> >>>> > %100 CPU usage and seems to do nothing, no brick
processes come
> >>>> > online. You can find strace of glusterd process for 1
minutes here:
> >>>> >
> >>>> >
https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0
> >>>> >
> >>>> > Here is the glusterd logs:
> >>>> >
https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0
> >>>> >
> >>>> >
> >>>> > By the way, reboot of one server completes without
problem if I
> reboot
> >>>> > the servers before creating any volumes.
> >>>> _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users at gluster.org
> >>>> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>> --
> >>> - Atin (atinm)
>-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170823/307c1031/attachment.html>

Maybe Matching Threads

Search for more maybe matching threads

Gluster users - Aug 2017 - Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

[Gluster-users] Glusterd proccess hangs on reboot

Maybe Matching Threads