search for: rpcsvc_request_init

...ecomes normal and we see brick processes come online and system starts to answer commands like "gluster peer status".. [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) [0x7f5ae2c091b1] -->/usr/lib64/li...

...brick processes come online and system starts to answer commands like > "gluster peer status".. > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > [0x7f5ae2c0...

...ine and system starts to answer commands like > > "gluster peer status".. > > > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] > > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x...

...s to answer commands like >> > "gluster peer status".. >> > >> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] >> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_r...

...status".. > >>>> > > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0( > rpcsvc_request_init+0x9c) > >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. > 0(gf_client_ref+0x1a9) > >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > >>>> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] > >...

...t; > "gluster peer status".. >>>> > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>>> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] >>>> > (-->/...

...>>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >> >>>> > [0x7f5ae2c091b1] > >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >> >>>> > [0x7f5ae2c0851c] > >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > >> >>>> > [2017-08-23 06...

I reboot multiple times, also I destroyed the gluster configuration and recreate multiple times. The behavior is same. On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at redhat.com> wrote: > My guess is there is a corruption in vol list or peer list which has lead > glusterd to get into a infinite loop of traversing a peer/volume list and > CPU to hog up. Again this is a

...017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >> >>>> > [0x7f5ae2c091b1] >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >> >>>> > [0x7f5ae2c0851c] >>> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>> >> >...

...> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>>> > [0x7f5ae2c091b1] >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>>> > [0x7f5ae2c0851c] >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> >>>> > [2017-08-23 06:46:02.152181] E [cl...

...like >>> > "gluster peer status".. >>> > >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] >>> > (-->/usr/lib64/libgfr...

...> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >> >>>> > [0x7f5ae2c091b1] >> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >> >>>> > [0x7f5ae2c0851c] >> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> >> >>>> &gt...

...; [client_t.c:324:gf_client_ref] >> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> >> >>>> > [0x7f5ae2c091b1] >> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>> >> >>>> > [0x7f5ae2c0851c] >> >>> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >>> >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid >>...

...:02.150472] E [client_t.c:324:gf_client_ref] > >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >> >>>> > [0x7f5ae2c091b1] > >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >>> >> >>>> > [0x7f5ae2c0851c] > >>> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > >>> >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > >&g...

...gt; >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >>> > [0x7f5ae2c091b1] > >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >>> >>> > [0x7f5ae2c0851c] > >>> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > >>> >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > >>> >>> > [2017-08-23 06...

...c:324:gf_client_ref] >>> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> >> >>>> > [0x7f5ae2c091b1] >>> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >>> >> >>>> > [0x7f5ae2c0851c] >>> >>> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> >>> >> >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Inval...

...> >>> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> > [0x7f5ae2c091b1] >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >>> > [0x7f5ae2c0851c] >>> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>> >>> > [2017-08-23 06:46:02.152181] E [cl...

My guess is there is a corruption in vol list or peer list which has lead glusterd to get into a infinite loop of traversing a peer/volume list and CPU to hog up. Again this is a guess and I've not got a chance to take a detail look at the logs and the strace output. I believe if you get to reboot the node again the problem will disappear. On Tue, 22 Aug 2017 at 20:07, Serkan ?oban

...gt; >>> > >> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> > [0x7f5ae2c091b1] >> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>> > [0x7f5ae2c0851c] >> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> >>> > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_...

...> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> >>> > [0x7f5ae2c091b1] >> >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>> >>> > [0x7f5ae2c0851c] >> >>> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >>> >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> >>> >>> &gt...