Gluster users - Sep 2017 - Glusterd proccess hangs on reboot

Hi Serkan,

I was wondering if you resolved your issue with the high CPU usage and hang
after starting gluster?

I'm setting up a 3 server (replica 3, arbiter 1), 300 volume, Gluster 3.12
cluster on CentOS 7 and am having what looks to be exactly the same issue as
you.

With no volumes created CPU usage / load is normal, but after creating all the
volumes even with no data CPU and RAM usage keeps creeping up and the logs are
filling up with:

[2017-09-14 05:47:45.447772] E [client_t.c:324:gf_client_ref]
(-->/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf8) [0x7fe3f2a1b7e8]
-->/lib64/libgfrpc.so.0(rpcsvc_request_init+0x7f) [0x7fe3f2a1893f]
-->/lib64/libglusterfs.so.0(gf_client_ref+0x179) [0x7fe3f2cb2e59] )
0-client_t: null client [Invalid argument]
[2017-09-14 05:47:45.486593] E [client_t.c:324:gf_client_ref]
(-->/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf8) [0x7fe3f2a1b7e8]
-->/lib64/libgfrpc.so.0(rpcsvc_request_init+0x7f) [0x7fe3f2a1893f] --

etc...

It's not an overly helpful error message as although it says a null client
gave an invalid argument, it doesn't state which client and what the
argument was.

I've tried strace and valgrind on glusterd as well as starting glusterd with
--debug to no avail.

--
Sam McLeod 
@s_mcleod
https://smcleod.net

I have the %100 CPU usage issue when I restart a glusterd instance and
I do not have null client errors in log.
The issue was related to number of bricks/servers so I decreased the
brick count in volume. It resolved the problem.


On Thu, Sep 14, 2017 at 9:02 AM, Sam McLeod <mailinglists at smcleod.net>
wrote:
> Hi Serkan,
>
> I was wondering if you resolved your issue with the high CPU usage and hang
after starting gluster?
>
> I'm setting up a 3 server (replica 3, arbiter 1), 300 volume, Gluster
3.12 cluster on CentOS 7 and am having what looks to be exactly the same issue
as you.
>
> With no volumes created CPU usage / load is normal, but after creating all
the volumes even with no data CPU and RAM usage keeps creeping up and the logs
are filling up with:
>
> [2017-09-14 05:47:45.447772] E [client_t.c:324:gf_client_ref]
(-->/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf8) [0x7fe3f2a1b7e8]
-->/lib64/libgfrpc.so.0(rpcsvc_request_init+0x7f) [0x7fe3f2a1893f]
-->/lib64/libglusterfs.so.0(gf_client_ref+0x179) [0x7fe3f2cb2e59] )
0-client_t: null client [Invalid argument]
> [2017-09-14 05:47:45.486593] E [client_t.c:324:gf_client_ref]
(-->/lib64/libgfrpc.so.0(rpcsvc_request_create+0xf8) [0x7fe3f2a1b7e8]
-->/lib64/libgfrpc.so.0(rpcsvc_request_init+0x7f) [0x7fe3f2a1893f] --
>
> etc...
>
> It's not an overly helpful error message as although it says a null
client gave an invalid argument, it doesn't state which client and what the
argument was.
>
> I've tried strace and valgrind on glusterd as well as starting glusterd
with --debug to no avail.
>
> --
> Sam McLeod
> @s_mcleod
> https://smcleod.net
>