search for: rpc_clnt_remove_ping_timer_lock

....c:1261:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 [2017-04-03 22:20:24.705662] T [socket.c:2879:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f012fd21953] (--> /usr/lib/x86_64-linux-gnu /libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f012f69add4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x55)[0x7f012f697af5] (--> /usr/lib/x8 6_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f012f698338] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f012f6945b3]...

I put the share into debug mode and then repeated the process from a ppc64 client and an x86 client. Weirdly the client logs were almost identical. Here's the ppc64 gluster client log of attempting to create a folder... ------------- [2017-09-20 13:34:23.344321] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xfdf60)[0x3fff9ec56fe0] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked-0x26060)[0x3fff9ebd9e20] (--> /usr/lib64/libgfrpc.so.0(+0x1a614)[0x3fff9ebda614] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit-0x29300)[0x...

...OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/...

...OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/...

...re no frames (0 || 0) > /DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) > /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db] (--> /lib > 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->...

Looks like it is an issue with architecture compatibility in RPC layer (ie, with XDRs and how it is used). Just glance the logs of the client process where you saw the errors, which could give some hints. If you don't understand the logs, share them, so we will try to look into it. -Amar On Wed, Sep 20, 2017 at 2:40 AM, Walter Deignan <WDeignan at uline.com> wrote: > I recently

I recently compiled the 3.10-5 client from source on a few PPC64 systems running RHEL 7.3. They are mounting a Gluster volume which is hosted on more traditional x86 servers. Everything seems to be working properly except for creating new directories from the PPC64 clients. The mkdir command gives a "Input/output error" and for the first few minutes the new directory is

...erfsd-mgmt.c:2171:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network [2017-08-20 20:30:40.466764] I [glusterfsd-mgmt.c:2171:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network [2017-08-20 20:30:40.658255] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3f0f82abbb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f3f0f5fd074] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xd4)[0x7f3f0f5f9774] (...

...mt_rpc_notify] > 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network > [2017-08-20 20:30:40.466764] I [glusterfsd-mgmt.c:2171:mgmt_rpc_notify] > 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network > [2017-08-20 20:30:40.658255] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] > (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3f0f82abbb] > (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ > ping_timer_locked+0x84)[0x7f3f0f5fd074] (--> /usr/lib/x86_64-linux-gnu/ > libgfrpc.so.0(rpc_clnt_connection_cleanup+0...

...gt;> 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network >> [2017-08-20 20:30:40.466764] I [glusterfsd-mgmt.c:2171:mgmt_rpc_notify] >> 0-glusterfsd-mgmt: disconnected from remote-host: web1.dasilva.network >> [2017-08-20 20:30:40.658255] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] >> (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3f0f82abbb] >> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping >> _timer_locked+0x84)[0x7f3f0f5fd074] (--> /usr/lib/x86_64-linux-gnu/libg >> frpc.so.0(rpc_clnt_conn...