search for: rpc_clnt_ping_timer_expir

...scrubber_scale_down] 0-glustervol-bit-rot-0: Scaling down scr ubbers [36 => 0] [2017-09-06 08:33:41.884031] I [MSGID: 118051] [bit-rot-ssm.c:80:br_scrub_ssm_state_stall] 0-glustervol-bit-rot-0: Volume is under ac tive scrubbing. Pausing scrub.. [2017-09-06 08:34:26.477106] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-glustervol-client-970: server 192.168.0.21:49177 has not responded in the last 42 seconds, disconnecting. [2017-09-06 08:34:29.477438] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-glustervol-client-980: server 192.168.0.21:49178 has not responded in the last 42 seconds, disconnecting....

On Fri, Sep 8, 2017 at 5:56 AM, Pavel Szalbot <pavel.szalbot at gmail.com> wrote: > This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > > 1 second is not an ideal value for ping timeout. Can you please set it to 30 seconds or so and simulate the problem? I would be interested in observing your logs with a...

...sley. logs are silent after resync - 3 gluster fuse client nodes: working flawlessley. logs are silent after resync - 1 gluster fuse client node: spitting errors every 2 minutes since the maintenane Client log beginning at the maintenance time: [2017-06-17 09:08:26.032957] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-1: server 100.64.252.1:49154 has not responded in the last 42 seconds, disconnecting. [2017-06-17 09:08:26.033302] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-2: server 100.64.252.1:49155 has not responded in the last 42 seconds, disconnecting. [2017-0...

This is the qemu log of instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwin...

Oh, you really don't want to go below 30s, I was told. I'm using 30 seconds for the timeout, and indeed when a node goes down the VM freez for 30 seconds, but I've never seen them go read only for that. I _only_ use virtio though, maybe it's that. What are you using ? On Fri, Sep 08, 2017 at 11:41:13AM +0200, Pavel Szalbot wrote: > Back to replica 3 w/o arbiter. Two fio jobs

...gt; changing network ping timeout. We discussed this via IRC recently. > > Diego > > On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at gmail.com> wrote: > > This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrp...

...gt; changing network ping timeout. We discussed this via IRC recently. > > Diego > > On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at gmail.com> wrote: > > This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrp...

...rfsd processes. JoeJulian recommends against changing network ping timeout. We discussed this via IRC recently. Diego On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at gmail.com> wrote: This is the qemu log of instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwin...

...gt; changing network ping timeout. We discussed this via IRC recently. > > Diego > > On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at gmail.com> wrote: > > This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrp...

...gt; changing network ping timeout. We discussed this via IRC recently. > > Diego > > On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at gmail.com> wrote: > > This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrp...