search for: _gf_log_callingfn

...instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (-- > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0...

...:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] > (-- >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfr...

...:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] > (-- >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfr...

...:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] > (-- > > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] > (--> /lib64/libgfrpc.so.0...

...instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (-- > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0...

...:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] > (-- >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfr...

Oh, you really don't want to go below 30s, I was told. I'm using 30 seconds for the timeout, and indeed when a node goes down the VM freez for 30 seconds, but I've never seen them go read only for that. I _only_ use virtio though, maybe it's that. What are you using ? On Fri, Sep 08, 2017 at 11:41:13AM +0200, Pavel Szalbot wrote: > Back to replica 3 w/o arbiter. Two fio jobs

...I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:45.221576] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x...

...sterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-04-03 22:20:24.705256] T [rpc-clnt.c:1261:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 [2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu /libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x8 6_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x...

...isconnecting. [2017-06-17 09:08:26.033302] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-2: server 100.64.252.1:49155 has not responded in the last 42 seconds, disconnecting. [2017-06-17 09:08:26.033751] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] )))))))))) 0-gluster1-client-1: forced unwinding frame...

...p idle -1 on socket 20, Invalid argument > [2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect] > 0-management: Failed to set keep-alive: Invalid argument > [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] > (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] > (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] > (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_ > connection_cleanu...

...2] E [socket.c:2966:socket_connect] >>>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>> [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind] >>>>>>> (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f767c2384ce] >>>>>>> (-->...

...g-rpc-limit : 16 performance.io-thread-count : 16 /var/log/glusterfs/nfs.log [2017-06-14 10:02:03.964405] I [MSGID: 108006] [afr-common.c:4941:afr_local_init] 0-gvol01-replicate-0: no subvolumes up [2017-06-14 10:02:04.026299] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f2729b3ae8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f27299018ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f27299019fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7f2729903170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x...

...rfs/3.12.5/xlator/mgmt/glusterd.so(+0x2d198) [0x7f46020f3198] -->/usr/lib64/glusterfs/3.12.5/xlator/mgmt/glusterd.so(+0xe4755) [0x7f46021aa755] ) 0-management: Lock for vol ovirtprod_vol not held [2018-02-26 14:31:47.331066] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f460d64dedb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f460d412e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f460d412f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7f460d414710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x...

...gt;>>>>> 0-management: Failed to set keep-alive: Invalid argument >>>>>>>> [2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_unwind] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f767c46d483] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f767c2383af] >>>>>>>> (--> >>>>>>>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved...

...a ppc64 client and an x86 client. Weirdly the client logs were almost identical. Here's the ppc64 gluster client log of attempting to create a folder... ------------- [2017-09-20 13:34:23.344321] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xfdf60)[0x3fff9ec56fe0] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked-0x26060)[0x3fff9ebd9e20] (--> /usr/lib64/libgfrpc.so.0(+0x1a614)[0x3fff9ebda614] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit-0x29300)[0x3fff9ebd69b0] (--> /usr/lib64/glusterfs/3.10.5/xlator/p...

...92469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(clien...

...92469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(clien...

...+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) > /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib > 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(...

Looks like it is an issue with architecture compatibility in RPC layer (ie, with XDRs and how it is used). Just glance the logs of the client process where you saw the errors, which could give some hints. If you don't understand the logs, share them, so we will try to look into it. -Amar On Wed, Sep 20, 2017 at 2:40 AM, Walter Deignan <WDeignan at uline.com> wrote: > I recently