Hi all, I have one particular replicated volume keep getting these errors in the log. glustershd.log: [2013-04-08 17:18:30.178495] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.179174] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.180011] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.180735] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.181520] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.182282] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.183042] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.183800] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.184699] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.185371] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.186137] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.186867] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory [2013-04-08 17:18:30.187638] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote operation failed: No such file or directory glusterfs/bricks/EXPORT-CSEVOL.log: [2013-04-08 17:18:30.197321] E [posix.c:224:posix_stat] 0-CSEVOL-posix: lstat on /EXPORT/CSEVOL/.glusterfs/35/14/351416f3-e0c2-4a4e-952e-83 4c98ba6093 failed: No such file or directory [2013-04-08 17:18:30.197351] I [server3_1-fops.c:1085:server_unlink_cbk] 0-CSEVOL-server: 145425303: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6 4e4a5580495>/8d46ac2f-0b48-4080-a539-6c4ced346567 (351416f3-e0c2-4a4e-952e-834c98ba6093) ==> -1 (No such file or directory) [2013-04-08 17:18:30.197922] W [marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot add a new contribution node [2013-04-08 17:18:30.198055] E [posix.c:224:posix_stat] 0-CSEVOL-posix: lstat on /EXPORT/CSEVOL/.glusterfs/b1/ab/b1abf4ad-b7a6-4171-af0e-47 e9809b9d7e failed: No such file or directory [2013-04-08 17:18:30.198091] I [server3_1-fops.c:1085:server_unlink_cbk] 0-CSEVOL-server: 145425305: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6 4e4a5580495>/135cf78e-d4ad-4c2a-b7e3-66a79322e982 (b1abf4ad-b7a6-4171-af0e-47e9809b9d7e) ==> -1 (No such file or directory) [2013-04-08 17:18:30.198681] W [marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot add a new contribution node [2013-04-08 17:18:30.198810] E [posix.c:224:posix_stat] 0-CSEVOL-posix: lstat on /EXPORT/CSEVOL/.glusterfs/36/fe/36fe047c-dcd9-4c94-b9a1-50 fc6b4d4d24 failed: No such file or directory [2013-04-08 17:18:30.198839] I [server3_1-fops.c:1085:server_unlink_cbk] 0-CSEVOL-server: 145425307: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6 4e4a5580495>/34fe4e2b-3731-45f0-916c-87852c4af4b4 (36fe047c-dcd9-4c94-b9a1-50fc6b4d4d24) ==> -1 (No such file or directory) [2013-04-08 17:18:30.199496] W [marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot add a new contribution node [2013-04-08 17:18:30.199627] E [posix.c:224:posix_stat] 0-CSEVOL-posix: lstat on /EXPORT/CSEVOL/.glusterfs/88/3e/883e86df-432c-4631-8977-9e a5b3a46c7d failed: No such file or directory [2013-04-08 17:18:30.199662] I [server3_1-fops.c:1085:server_unlink_cbk] 0-CSEVOL-server: 145425309: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6 4e4a5580495>/412da580-0b2c-42f6-9dc5-b8baf18c1f81 (883e86df-432c-4631-8977-9ea5b3a46c7d) ==> -1 (No such file or directory) [2013-04-08 17:18:30.200213] W [marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot add a new contribution node [2013-04-08 17:18:30.200345] E [posix.c:224:posix_stat] 0-CSEVOL-posix: lstat on /EXPORT/CSEVOL/.glusterfs/bb/af/bbaf760f-86ea-4b74-b817-c2 818823df3d failed: No such file or directory [2013-04-08 17:18:30.200381] I [server3_1-fops.c:1085:server_unlink_cbk] 0-CSEVOL-server: 145425311: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6 4e4a5580495>/a7a313e9-3b0d-4118-9575-af50a1d9fccf (bbaf760f-86ea-4b74-b817-c2818823df3d) ==> -1 (No such file or directory) [2013-04-08 17:18:38.445324] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory [2013-04-08 17:18:38.462209] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory [2013-04-08 17:18:50.625541] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory [2013-04-08 17:19:02.784643] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory [2013-04-08 17:19:02.790217] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory [2013-04-08 17:19:14.968659] E [posix-helpers.c:701:posix_handle_pair] 0-CSEVOL-posix: /EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime error:No such file or directory My volume config: Volume Name: CSEVOL Type: Replicate Volume ID: 0ced464e-554f-4840-82e3-1bfb50d6df36 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: t4-01.store:/EXPORT/CSEVOL Brick2: t4-03.store:/EXPORT/CSEVOL Options Reconfigured: network.ping-timeout: 5 geo-replication.indexing: on features.quota: on features.limit-usage: ( i have some quotas set) Anyone has any idea how these errors are generated? This 2 logs are growing too fast that weekly logrotate is insufficient. Appreciate if anybody can give some advise. Thanks. Keith