hi All, I try to remove some directories from our cluster, but ther eis an error: $ rm -rf _torolni rm: cannot remove `_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001': Directory not empty rm: cannot remove `_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800': Directory not empty From the log of the client: [2014-07-23 17:25:58.532161] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: gfid different on w-vol-client-1 [2014-07-23 17:25:58.532247] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: gfid different on w-vol-client-3 [2014-07-23 17:25:58.532277] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: gfid different on w-vol-client-4 [2014-07-23 17:25:58.532458] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: gfid different on w-vol-client-0 [2014-07-23 17:25:58.533283] W [client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-2: remote operation failed: No such file or directory. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 (63cf6edf-f1ba-4657-acd5-c28d225bddf1) [2014-07-23 17:25:58.543416] W [client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-2: remote operation failed: No such file or directory. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 (63cf6edf-f1ba-4657-acd5-c28d225bddf1) [2014-07-23 17:25:58.557966] W [client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-2: remote operation failed: File exists. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 (63cf6edf-f1ba-4657-acd5-c28d225bddf1) [2014-07-23 17:25:58.653867] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800: gfid different on w-vol-client-1 [2014-07-23 17:25:58.669396] W [client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 (6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c) [2014-07-23 17:25:58.676365] W [client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 (6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c) [2014-07-23 17:25:58.677956] W [client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-1: remote operation failed: File exists. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 (6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c) [2014-07-23 17:25:58.678121] W [client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-3: remote operation failed: File exists. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 (6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c) I found this tool: https://github.com/vbellur/gfid Probably it will help. Why did that happen? Will it happen in the future as well, is this a persistent bug? Thanks, tamas