Riccardo Murri
2018-Jul-19 13:46 UTC
[Gluster-users] Cannot list `.snaps/<snapname>` directory
Re-sending the log files as attachments, to avoid MUAs corrupting them... Ciao, R -------------- next part -------------- ### client logs: [2018-07-19 13:30:10.206657] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-07-19 13:30:10.316710] I [MSGID: 114007] [client.c:2402:client_check_remote_host] 0-glusterfs-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument] [2018-07-19 13:30:10.316910] D [io-stats.c:3870:reconfigure] 0-glusterfs: reconfigure returning 0 [2018-07-19 13:30:10.316969] D [glusterfsd-mgmt.c:1920:mgmt_getspec_cbk] 0-glusterfsd-mgmt: No need to re-load volfile, reconfigure done [2018-07-19 13:30:31.823912] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:31.837598] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: FLUSH scheduled as normal fop [2018-07-19 13:30:33.174222] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:30:33.175564] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt returned with op_ret 0 [2018-07-19 13:30:35.546485] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop" repeated 3 times between [2018-07-19 13:30:31.823912] and [2018-07-19 13:30:35.540106] [2018-07-19 13:30:35.546479] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.548707] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.562717] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2018-07-19 13:30:35.562298] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.562713] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:35.562919] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.564746] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.574831] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2018-07-19 13:30:35.570561] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.574827] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:35.575078] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.576864] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:37.546586] D [logging.c:1855:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: FLUSH scheduled as normal fop" repeated 2 times between [2018-07-19 13:30:31.837598] and [2018-07-19 13:30:31.838077] The message "D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt returned with op_ret 0" repeated 4 times between [2018-07-19 13:30:33.175564] and [2018-07-19 13:30:33.176783] [2018-07-19 13:30:35.582728] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:37.546747] D [logging.c:1817:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2018-07-19 13:30:38.647775] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:38.650530] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:38.653807] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:38.678216] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:30:38.680700] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.200:49153: ping timer event already removed [2018-07-19 13:30:38.681239] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-snapd-client: Ping latency is 0ms [2018-07-19 13:30:44.547902] D [rpc-clnt-ping.c:336:rpc_clnt_start_ping] 0-glusterfs-client-10: returning as transport is already disconnected OR there are no frames (0 || 0) [2018-07-19 13:30:44.548027] D [rpc-clnt-ping.c:336:rpc_clnt_start_ping] 0-glusterfs-client-12: returning as transport is already disconnected OR there are no frames (0 || 0) [2018-07-19 13:30:44.548042] D [rpc-clnt-ping.c:336:rpc_clnt_start_ping] 0-glusterfs-client-13: returning as transport is already disconnected OR there are no frames (0 || 0) [2018-07-19 13:30:44.548053] D [rpc-clnt-ping.c:336:rpc_clnt_start_ping] 0-glusterfs-client-8: returning as transport is already disconnected OR there are no frames (0 || 0) [2018-07-19 13:30:44.548065] D [rpc-clnt-ping.c:336:rpc_clnt_start_ping] 0-glusterfs-client-9: returning as transport is already disconnected OR there are no frames (0 || 0) [2018-07-19 13:31:01.189801] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.196:49152: ping timer event already removed [2018-07-19 13:31:01.190172] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.200:49152: ping timer event already removed [2018-07-19 13:31:01.190441] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.197:49152: ping timer event already removed [2018-07-19 13:31:01.190697] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.199:49152: ping timer event already removed [2018-07-19 13:31:01.190973] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f396011b4bb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7f395feec524] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x12c4d)[0x7f395feecc4d] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x416)[0x7f395fee9ad6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/client.so(+0xf45e)[0x7f3959ff345e] ))))) 0-: 172.23.62.198:49152: ping timer event already removed [2018-07-19 13:31:01.191210] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-client-8: Ping latency is 1ms [2018-07-19 13:31:01.191336] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-client-13: Ping latency is 0ms [2018-07-19 13:31:01.191558] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-client-12: Ping latency is 1ms [2018-07-19 13:31:01.191626] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-client-9: Ping latency is 0ms [2018-07-19 13:31:01.191657] D [rpc-clnt-ping.c:211:rpc_clnt_ping_cbk] 0-glusterfs-client-10: Ping latency is 0ms [2018-07-19 13:31:01.193908] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt returned with op_ret 0 [2018-07-19 13:31:01.197302] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop" repeated 2 times between [2018-07-19 13:30:38.678216] and [2018-07-19 13:31:01.196236] [2018-07-19 13:31:01.197299] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod returned with op_ret 0 [2018-07-19 13:31:01.201816] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod returned with op_ret 0" repeated 4 times between [2018-07-19 13:31:01.197299] and [2018-07-19 13:31:01.198446] [2018-07-19 13:31:01.201812] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:31:01.202137] D [MSGID: 0] [dht-common.c:2694:dht_lookup] 0-glusterfs-dht: calling revalidate lookup for /opt/lmod/lmod at glusterfs-client-9 [2018-07-19 13:31:01.203034] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/lmod returned with op_ret 0 [2018-07-19 13:31:01.203243] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READLINK scheduled as fast fop [2018-07-19 13:31:01.204160] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:31:01.204997] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0 returned with op_ret 0 [2018-07-19 13:31:01.206434] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0 returned with op_ret 0" repeated 4 times between [2018-07-19 13:31:01.204997] and [2018-07-19 13:31:01.206213] [2018-07-19 13:31:01.206431] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:31:01.207314] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0/init returned with op_ret 0 [2018-07-19 13:31:01.210478] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0/init returned with op_ret 0" repeated 4 times between [2018-07-19 13:31:01.207314] and [2018-07-19 13:31:01.210221] [2018-07-19 13:31:01.210475] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:31:01.210647] D [MSGID: 0] [dht-common.c:2694:dht_lookup] 0-glusterfs-dht: calling revalidate lookup for /opt/lmod/7.0/init/bash at glusterfs-client-9 [2018-07-19 13:31:01.211394] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0/init/bash returned with op_ret 0 [2018-07-19 13:31:01.211590] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPEN scheduled as fast fop [2018-07-19 13:31:01.211961] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: FLUSH scheduled as normal fop [2018-07-19 13:31:01.217981] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READLINK scheduled as fast fop [2018-07-19 13:31:01.219091] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:31:01.219269] D [MSGID: 0] [dht-common.c:2694:dht_lookup] 0-glusterfs-dht: calling revalidate lookup for /opt/lmod/7.0/init/lmod_bash_completions at glusterfs-client-9 [2018-07-19 13:31:01.220009] D [MSGID: 0] [dht-common.c:1056:dht_revalidate_cbk] 0-glusterfs-dht: revalidate lookup of /opt/lmod/7.0/init/lmod_bash_completions returned with op_ret 0 [2018-07-19 13:31:01.913330] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: FLUSH scheduled as normal fop ### brick 1 logs: [2018-07-19 13:29:46.290627] D [MSGID: 0] [options.c:1141:xlator_reconfigure_rec] 0-glusterfs-io-stats: reconfigured [2018-07-19 13:29:46.290642] D [MSGID: 0] [options.c:1160:xlator_reconfigure_rec] 0-/srv/glusterfs: No reconfigure() found [2018-07-19 13:29:46.290685] D [glusterfsd-mgmt.c:1920:mgmt_getspec_cbk] 0-glusterfsd-mgmt: No need to re-load volfile, reconfigure done [2018-07-19 13:30:01.770678] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/li b/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:01.770776] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:01.771062] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/ lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8 c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:10.062875] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-07-19 13:30:10.062935] D [rpc-clnt-ping.c:317:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2018-07-19 13:30:10.197421] I [glusterfsd-mgmt.c:1888:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2018-07-19 13:30:31.824458] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/li b/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:31.824855] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:31.846677] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:31.847060] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:33.172175] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:33.172544] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:33.174656] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:33.174787] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:30:33.175224] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.540359] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.540619] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.546862] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x25b67) [0x7fe1eeaffb67] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.546923] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop" repeated 4 times between [2018-07-19 13:30:01.770776] and [2018-07-19 13:30:35.540423] [2018-07-19 13:30:35.546920] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.547607] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1baed) [0x7fe1eeaf5aed] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.553814] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x27fe2) [0x7fe1eeb01fe2] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.553856] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.554407] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x205d1) [0x7fe1eeafa5d1] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.562703] D [MSGID: 0] [posix-helpers.c:1466:posix_janitor_thread_proc] 0-glusterfs-posix: janitor: closing dir fd=0x7fe1b465e7e0 [2018-07-19 13:30:35.563193] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x25b67) [0x7fe1eeaffb67] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.563242] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.563528] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1baed) [0x7fe1eeaf5aed] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.567429] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x27fe2) [0x7fe1eeb01fe2] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.567477] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.567894] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x205d1) [0x7fe1eeafa5d1] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.570973] D [MSGID: 0] [posix-helpers.c:1466:posix_janitor_thread_proc] 0-glusterfs-posix: janitor: closing dir fd=0x7fe1b465e7e0 [2018-07-19 13:30:35.575311] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x25b67) [0x7fe1eeaffb67] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.575385] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:35.575610] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1baed) [0x7fe1eeaf5aed] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.579637] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x27fe2) [0x7fe1eeb01fe2] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:35.579677] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:35.580059] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x205d1) [0x7fe1eeafa5d1] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:35.583122] D [MSGID: 0] [posix-helpers.c:1466:posix_janitor_thread_proc] 0-glusterfs-posix: janitor: closing dir fd=0x7fe1b465e7e0 [2018-07-19 13:30:38.648701] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.648855] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:38.649119] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:38.650779] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x25b67) [0x7fe1eeaffb67] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.651968] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:38.652317] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1baed) [0x7fe1eeaf5aed] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:38.656497] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x27fe2) [0x7fe1eeb01fe2] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.656601] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:38.657507] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x205d1) [0x7fe1eeafa5d1] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:38.660985] D [MSGID: 0] [posix-helpers.c:1466:posix_janitor_thread_proc] 0-glusterfs-posix: janitor: closing dir fd=0x7fe1bc8c1df0 [2018-07-19 13:30:38.662589] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x25b67) [0x7fe1eeaffb67] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.662661] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: OPENDIR scheduled as fast fop [2018-07-19 13:30:38.662920] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1baed) [0x7fe1eeaf5aed] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:38.666150] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x27fe2) [0x7fe1eeb01fe2] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.666231] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: READDIRP scheduled as fast fop [2018-07-19 13:30:38.666714] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x205d1) [0x7fe1eeafa5d1] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:30:38.669800] D [MSGID: 0] [posix-helpers.c:1466:posix_janitor_thread_proc] 0-glusterfs-posix: janitor: closing dir fd=0x7fe1bc8c1df0 [2018-07-19 13:30:38.677105] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:30:38.677185] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:30:38.677445] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.190565] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.191249] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.192975] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.193545] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.196731] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.197537] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.205377] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.205873] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.209143] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.209748] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:01.926270] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:01.926603] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:31:18.408919] D [logging.c:1855:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop" repeated 4 times between [2018-07-19 13:30:33.174787] and [2018-07-19 13:31:01.209416] The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop" repeated 2 times between [2018-07-19 13:30:38.677185] and [2018-07-19 13:31:01.926360] [2018-07-19 13:31:18.409423] D [logging.c:1817:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2018-07-19 13:31:32.006449] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:31:32.006595] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:31:32.006948] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:32:02.079972] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:32:02.080792] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:32:32.169915] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:32:32.171022] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:02.242813] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:02.243529] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:11.329101] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:11.329482] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:11.331254] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:11.331508] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop [2018-07-19 13:33:11.333006] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:11.334574] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:11.335693] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:11.341771] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:11.342322] D [MSGID: 0] [posix.c:355:posix_lookup] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-posix returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342574] D [MSGID: 0] [changetimerecorder.c:309:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-changetimerecorder returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342624] D [MSGID: 0] [bit-rot-stub.c:2930:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-bitrot-stub returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342694] D [MSGID: 0] [posix-acl.c:1017:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-access-control returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342853] D [MSGID: 0] [posix.c:2639:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-locks returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342884] D [MSGID: 0] [upcall.c:796:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-upcall returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342932] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-threads returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.342949] D [MSGID: 0] [marker.c:2948:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-marker returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.343041] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-index returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.343062] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-stats returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:11.343094] D [MSGID: 115050] [server-rpc-fops.c:175:server_lookup_cbk] 0-glusterfs-server: 6700: LOOKUP /opt/bin/ruptime (65ce8807-e978-430c-821d-d83c8fb04d6c/ruptime), client: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0, error-xlator: glusterfs-posix [No such file or directory] [2018-07-19 13:33:11.343271] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:12.887244] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:12.887634] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:12.889028] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:12.889443] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:12.890704] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:12.891445] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:12.897167] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:12.897529] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2018-07-19 13:33:12.897522] D [MSGID: 0] [changetimerecorder.c:309:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-changetimerecorder returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897528] D [MSGID: 0] [bit-rot-stub.c:2930:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-bitrot-stub returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897644] D [MSGID: 0] [posix-acl.c:1017:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-access-control returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897673] D [MSGID: 0] [posix.c:2639:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-locks returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897703] D [MSGID: 0] [upcall.c:796:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-upcall returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897730] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-threads returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897763] D [MSGID: 0] [marker.c:2948:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-marker returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897787] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-index returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897814] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-stats returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:12.897844] D [MSGID: 115050] [server-rpc-fops.c:175:server_lookup_cbk] 0-glusterfs-server: 6704: LOOKUP /opt/bin/ruptime (65ce8807-e978-430c-821d-d83c8fb04d6c/ruptime), client: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0, error-xlator: glusterfs-posix [No such file or directory] 4d6c/ruptime), client: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0, error-xlator: glusterfs-posix [No such file or directory] [2018-07-19 13:33:12.897973] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:14.489599] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:14.489988] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:14.491647] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:14.492095] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:14.494026] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:14.494421] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:14.499289] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2e103) [0x7fe1eeb08103] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:14.499587] D [logging.c:1979:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [posix.c:355:posix_lookup] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-posix returned -1 error: No such file or directory [No such file or directory]" repeated 2 times between [2018-07-19 13:33:11.342322] and [2018-07-19 13:33:14.499568] [2018-07-19 13:33:14.499586] D [MSGID: 0] [changetimerecorder.c:309:ctr_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-changetimerecorder returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499694] D [MSGID: 0] [bit-rot-stub.c:2930:br_stub_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-bitrot-stub returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499722] D [MSGID: 0] [posix-acl.c:1017:posix_acl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-access-control returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499756] D [MSGID: 0] [posix.c:2639:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-locks returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499779] D [MSGID: 0] [upcall.c:796:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-upcall returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499834] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-threads returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499874] D [MSGID: 0] [marker.c:2948:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-marker returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499903] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-index returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499928] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fe1d0dac0b0, glusterfs-io-stats returned -1 error: No such file or directory [No such file or directory] [2018-07-19 13:33:14.499965] D [MSGID: 115050] [server-rpc-fops.c:175:server_lookup_cbk] 0-glusterfs-server: 6708: LOOKUP /opt/bin/ruptime (65ce8807-e978-430c-821d-d83c8fb04d6c/ruptime), client: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0, error-xlator: glusterfs-posix [No such file or directory] [2018-07-19 13:33:14.500134] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x2d814) [0x7fe1eeb07814] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1 [2018-07-19 13:33:18.426934] D [logging.c:1855:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop" repeated 6 times between [2018-07-19 13:31:32.006595] and [2018-07-19 13:33:14.489706] The message "D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: LOOKUP scheduled as fast fop" repeated 8 times between [2018-07-19 13:33:11.331508] and [2018-07-19 13:33:14.499450] [2018-07-19 13:33:18.427036] D [logging.c:1817:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2018-07-19 13:33:32.315885] D [client_t.c:331:gf_client_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x28287) [0x7fe1eeb02287] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0xd37d) [0x7fe1eeae737d] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_ref+0xbf) [0x7fe1fbd4b6ff] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 2 [2018-07-19 13:33:32.316005] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-glusterfs-io-threads: STAT scheduled as fast fop [2018-07-19 13:33:32.316627] D [client_t.c:440:gf_client_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x1f955) [0x7fe1eeaf9955] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.9/xlator/protocol/server.so(+0x89f3) [0x7fe1eeae29f3] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_client_unref+0x96) [0x7fe1fbd4b8c6] ) 0-client_t: slurm-master-001-19740-2018/07/18-09:48:19:570945-glusterfs-client-12-0-0: ref-count 1
Riccardo Murri
2018-Jul-24 13:12 UTC
[Gluster-users] Cannot list `.snaps/<snapname>` directory
Hello, I have set up a test cluster with GlusterFS 4.1 and Ubuntu 16.04.5 and I get the same behavior: `ls .snaps/test/` hangs indefinitely in a getdents() system call. I can mount and list the snapshot just fine with `mount -t glusterfs`, it's just the USS feature that is not working. Is this a known bug? Any hints on how to work it around? Thanks, Riccardo