search for: client_t

...ows hanging in libglusterfs.so and below is the glusterd logs, which are different from 3.10. With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see brick processes come online and system starts to answer commands like "gluster peer status".. [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid argument] [2017-08-...

...o and below is the glusterd logs, which are different > from 3.10. > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > brick processes come online and system starts to answer commands like > "gluster peer status".. > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > [0x7f5ae2ea3949] ) 0-client_t: null client [Invalid...

...gs, which are different > > from 3.10. > > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > > brick processes come online and system starts to answer commands like > > "gluster peer status".. > > > > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) > > [0x7f5ae2ea3949] ) 0-client_t:...

...>> > from 3.10. >> > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see >> > brick processes come online and system starts to answer commands like >> > "gluster peer status".. >> > >> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >> > [0x7f5ae2ea3949...

....10. >>> > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see >>> > brick processes come online and system starts to answer commands like >>> > "gluster peer status".. >>> > >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >>> >...

...60-70 minutes CPU usage becomes normal and we see > >>>> > brick processes come online and system starts to answer commands > like > >>>> > "gluster peer status".. > >>>> > > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0( > rpcsvc_request_init+0x9c) > >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. > 0...

...; > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see >>>> > brick processes come online and system starts to answer commands like >>>> > "gluster peer status".. >>>> > >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so.0(gf_client_ref+0x1a9) >&...

...t;>>> > brick processes come online and system starts to answer commands > >> >>>> > like > >> >>>> > "gluster peer status".. > >> >>>> > > >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >> >>>> > [0x7f5ae2c091b1] > >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >> >>>> >...

I reboot multiple times, also I destroyed the gluster configuration and recreate multiple times. The behavior is same. On Tue, Aug 22, 2017 at 6:47 PM, Atin Mukherjee <amukherj at redhat.com> wrote: > My guess is there is a corruption in vol list or peer list which has lead > glusterd to get into a infinite loop of traversing a peer/volume list and > CPU to hog up. Again this is a

...>>> >> >>>> > commands >>> >> >>>> > like >>> >> >>>> > "gluster peer status".. >>> >> >>>> > >>> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >> >>>> > [0x7f5ae2c091b1] >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>>...

...> see >> >>>> > brick processes come online and system starts to answer commands >> >>>> > like >> >>>> > "gluster peer status".. >> >>>> > >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>>> > [0x7f5ae2c091b1] >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >>>> > [0x7f5ae2c0851c] &gt...

...t; With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > >>> > brick processes come online and system starts to answer commands like > >>> > "gluster peer status".. > >>> > > >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0( > rpcsvc_request_init+0x9c) > >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. > 0(gf_client_r...

...70 minutes CPU usage becomes normal and we see >> >>> > brick processes come online and system starts to answer commands >> like >> >>> > "gluster peer status".. >> >>> > >> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> > [0x7f5ae2c091b1] -->/usr/lib64/libgfrpc.so.0(rp >> csvc_request_init+0x9c) >> >>> > [0x7f5ae2c0851c] -->/usr/lib64/libglusterfs.so. &g...

...gt; brick processes come online and system starts to answer commands >> >> >>>> > like >> >> >>>> > "gluster peer status".. >> >> >>>> > >> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >> >>>> > [0x7f5ae2c091b1] >> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >> >> >&g...

...gt;>> > like >> >>> >> >>>> > "gluster peer status".. >> >>> >> >>>> > >> >>> >> >>>> > [2017-08-23 06:46:02.150472] E >> >>> >> >>>> > [client_t.c:324:gf_client_ref] >> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >> >>> >> >>>> > [0x7f5ae2c091b1] >> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_requ...

...; >>>> > commands > >>> >> >>>> > like > >>> >> >>>> > "gluster peer status".. > >>> >> >>>> > > >>> >> >>>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >> >>>> > [0x7f5ae2c091b1] > >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9...

...; >>> > brick processes come online and system starts to answer commands > >>> >>> > like > >>> >>> > "gluster peer status".. > >>> >>> > > >>> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] > >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) > >>> >>> > [0x7f5ae2c091b1] > >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) > >>> >>> >...

...like >>> >>> >> >>>> > "gluster peer status".. >>> >>> >> >>>> > >>> >>> >> >>>> > [2017-08-23 06:46:02.150472] E >>> >>> >> >>>> > [client_t.c:324:gf_client_ref] >>> >>> >> >>>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> >> >>>> > [0x7f5ae2c091b1] >>> >>> >> >>>> > -->/usr/lib64/libgfrpc.so.0...

...> see >>> >>> > brick processes come online and system starts to answer commands >>> >>> > like >>> >>> > "gluster peer status".. >>> >>> > >>> >>> > [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_ref] >>> >>> > (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf1) >>> >>> > [0x7f5ae2c091b1] >>> >>> > -->/usr/lib64/libgfrpc.so.0(rpcsvc_request_init+0x9c) >>> >>> > [0x7f5ae2c0851c] &gt...

My guess is there is a corruption in vol list or peer list which has lead glusterd to get into a infinite loop of traversing a peer/volume list and CPU to hog up. Again this is a guess and I've not got a chance to take a detail look at the logs and the strace output. I believe if you get to reboot the node again the problem will disappear. On Tue, 22 Aug 2017 at 20:07, Serkan ?oban