search for: rpcsvc_request_create

...which are different from 3.10. With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see brick processes come online and system starts to answer commands like "gluster peer status".. [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] (-->/usr/lib64/libg...

...0. > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > brick processes come online and system starts to answer commands like > "gluster peer status".. > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > [2017-08-23 06:46:02.152181] E [client_t.c:324:gf_client_ref] &g...

...r 60-70 minutes CPU usage becomes normal and we see > > brick processes come online and system starts to answer commands like > > "gluster peer status".. > > > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] > > [2017-08-23 06:46:02.152181] E [client_t.c:3...

...usage becomes normal and we see >> > brick processes come online and system starts to answer commands like >> > "gluster peer status".. >> > >> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >> > [2017-08-23 06:46:02.152181]...

...and we see >>> > brick processes come online and system starts to answer commands like >>> > "gluster peer status".. >>> > >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>> > [2017-08-23...

...sses come online and system starts to answer commands > like > >>>> > "gluster peer status".. > >>>> > > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0( > rpcsvc_request_init+0x9c) > >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. > 0(gf_client_ref+0x1a9) > >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid a...

...>> > brick processes come online and system starts to answer commands like >>>> > "gluster peer status".. >>>> > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] >>>> &...

...>>>> > like > >> >>>> > "gluster peer status".. > >> >>>> > > >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >> >>>> > [0x7f5ae2c091b1] > >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >> >>>> > [0x7f5ae2c0851c] > >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) &g...

I reboot multiple times, also I destroyed the gluster configuration and recreate multiple times. The behavior is same. On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at redhat.com> wrote: > My guess is there is a corruption in vol list or peer list which has lead > glusterd to get into a infinite loop of traversing a peer/volume list and > CPU to hog up. Again this is a

...>> >> >>>> > "gluster peer status".. >>> >> >>>> > >>> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >> >>>> > [0x7f5ae2c091b1] >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >> >>>> > [0x7f5ae2c0851c] >>> >> >>>> > -->/usr/lib64/libgluste...

...er commands >> >>>> > like >> >>>> > "gluster peer status".. >> >>>> > >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>>> > [0x7f5ae2c091b1] >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>>> > [0x7f5ae2c0851c] >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> >>>&...

...> > brick processes come online and system starts to answer commands like > >>> > "gluster peer status".. > >>> > > >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0( > rpcsvc_request_init+0x9c) > >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. > 0(gf_client_ref+0x1a9) > >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] &gt...

...come online and system starts to answer commands >> like >> >>> > "gluster peer status".. >> >>> > >> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rp >> csvc_request_init+0x9c) >> >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. >> 0(gf_client_ref+0x1a9) >> >>> > [0x7f5ae2ea3949] ) 0-client_t: null client [I...

...; > like >> >> >>>> > "gluster peer status".. >> >> >>>> > >> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >> >>>> > [0x7f5ae2c091b1] >> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >> >>>> > [0x7f5ae2c0851c] >> >> >>>> > -->/usr/lib64/libglusterfs.so.0(gf_clie...

...t; >>> >> >>>> > >> >>> >> >>>> > [2017-08-23 06:46:02.150472] E >> >>> >> >>>> > [client_t.c:324:gf_client_ref] >> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> >> >>>> > [0x7f5ae2c091b1] >> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>> >> >>>> > [0x7f5ae2c0851c] >> >>> >> >>>...

...t; >>>> > "gluster peer status".. > >>> >> >>>> > > >>> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >> >>>> > [0x7f5ae2c091b1] > >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >>> >> >>>> > [0x7f5ae2c0851c] > >>> >> >>>> > -->...

...> >>> > like > >>> >>> > "gluster peer status".. > >>> >>> > > >>> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >>> > [0x7f5ae2c091b1] > >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >>> >>> > [0x7f5ae2c0851c] > >>> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) &g...

...gt; >> >>>> > >>> >>> >> >>>> > [2017-08-23 06:46:02.150472] E >>> >>> >> >>>> > [client_t.c:324:gf_client_ref] >>> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> >> >>>> > [0x7f5ae2c091b1] >>> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >>> >> >>>> > [0x7f5ae2c0851c] >>> >>> >&...

...er commands >>> >>> > like >>> >>> > "gluster peer status".. >>> >>> > >>> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> > [0x7f5ae2c091b1] >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >>> > [0x7f5ae2c0851c] >>> >>> > -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> >>&...

My guess is there is a corruption in vol list or peer list which has lead glusterd to get into a infinite loop of traversing a peer/volume list and CPU to hog up. Again this is a guess and I've not got a chance to take a detail look at the logs and the strace output. I believe if you get to reboot the node again the problem will disappear. On Tue, 22 Aug 2017 at 20:07, Serkan ?oban