I am seeing these messsages after I delete large amounts of data using gluster 3.6.2. cannot delete non-empty directory: old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final>From the FUSE mount (as root), the directory shows up as empty:# pwd /backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final # ls -al total 5 d--------- 2 root root 4106 Feb 6 13:55 . drwxrws--- 3 601 dmiller 72 Feb 6 13:55 .. However, when you look at the bricks, the files are still there (none on brick01bkp, all files are on brick02bkp). All of the files are 0-length and have ------T permissions. Any suggestions on how to fix this and how to prevent it from happening? # ls -al /data/brick*/homegfs_bkp/backup.0/old_shelf4/Aegis/\!\!\!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final /data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final: total 4 d---------+ 2 root root 10 Feb 6 13:55 . drwxrws---+ 3 601 raven 36 Feb 6 13:55 .. /data/brick02bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final: total 8 d---------+ 3 root root 4096 Dec 31 1969 . drwxrws---+ 3 601 raven 36 Feb 6 13:55 .. ---------T 5 601 raven 0 Nov 20 00:08 read_inset.f.gz ---------T 5 601 raven 0 Nov 20 00:08 readbc.f.gz ---------T 5 601 raven 0 Nov 20 00:08 readcn.f.gz ---------T 5 601 raven 0 Nov 20 00:08 readinp.f.gz ---------T 5 601 raven 0 Nov 20 00:08 readinp_v1_2.f.gz ---------T 5 601 raven 0 Nov 20 00:08 readinp_v1_3.f.gz ---------T 5 601 raven 0 Nov 20 00:08 rotatept.f.gz d---------+ 2 root root 118 Feb 6 13:54 save1 ---------T 5 601 raven 0 Nov 20 00:08 sepvec.f.gz ---------T 5 601 raven 0 Nov 20 00:08 shadow.f.gz ---------T 5 601 raven 0 Nov 20 00:08 snksrc.f.gz ---------T 5 601 raven 0 Nov 20 00:08 source.f.gz ---------T 5 601 raven 0 Nov 20 00:08 step.f.gz ---------T 5 601 raven 0 Nov 20 00:08 stoprog.f.gz ---------T 5 601 raven 0 Nov 20 00:08 summer6.f.gz ---------T 5 601 raven 0 Nov 20 00:08 totforc.f.gz ---------T 5 601 raven 0 Nov 20 00:08 tritet.f.gz ---------T 5 601 raven 0 Nov 20 00:08 wallrsd.f.gz ---------T 5 601 raven 0 Nov 20 00:08 wheat.f.gz ---------T 5 601 raven 0 Nov 20 00:08 write_inset.f.gz This is using gluster 3.6.2 on a distributed gluster volume that resides on a single machine. Both of the bricks are on one machine consisting of 2x RAID-6 arrays. df -h | grep brick /dev/mapper/vg01-lvol1 88T 22T 66T 25% /data/brick01bkp /dev/mapper/vg02-lvol1 88T 22T 66T 26% /data/brick02bkp # gluster volume info homegfs_bkp Volume Name: homegfs_bkp Type: Distribute Volume ID: 96de8872-d957-4205-bf5a-076e3f35b294 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: gfsib01bkp.corvidtec.com:/data/brick01bkp/homegfs_bkp Brick2: gfsib01bkp.corvidtec.com:/data/brick02bkp/homegfs_bkp Options Reconfigured: storage.owner-gid: 100 performance.io-thread-count: 32 server.allow-insecure: on network.ping-timeout: 10 performance.cache-size: 128MB performance.write-behind-window-size: 128MB server.manage-gids: on changelog.rollover-time: 15 changelog.fsync-interval: 3 ==============================David F. Robinson, Ph.D. President - Corvid Technologies 704.799.6944 x101 [office] 704.252.1310 [cell] 704.799.7974 [fax] David.Robinson at corvidtec.com http://www.corvidtechnologies.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150208/90359099/attachment.html>
Shyam
2015-Feb-09 16:11 UTC
[Gluster-users] [Gluster-devel] cannot delete non-empty directory
On 02/08/2015 12:19 PM, David F. Robinson wrote:> I am seeing these messsages after I delete large amounts of data using > gluster 3.6.2. > cannot delete non-empty directory: > old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final > *_From the FUSE mount (as root), the directory shows up as empty:_* > # pwd > /backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final > > # ls -al > total 5 > d--------- 2 root root 4106 Feb 6 13:55 . > drwxrws--- 3 601 dmiller 72 Feb 6 13:55 .. > However, when you look at the bricks, the files are still there (none on > brick01bkp, all files are on brick02bkp). All of the files are 0-length > and have ------T permissions.These files are linkto files that are created by DHT, which basically mean the files were either renamed, or the brick layout changed (I suspect the former to be the cause). These files should have been deleted when the files that they point to were deleted, looks like this did not happen. Can I get the following information for some of the files here, - getfattr -d -m . -e text <path to file on brick> - The output of trusted.glusterfs.dht.linkto xattr should state where the real file belongs, in this case as there are only 2 bricks, it should be brick01bkp subvol - As the second brick is empty, we should be able to safely delete these files from the brick and proceed to do an rmdir on the mount point of the volume as the directory is now empty. - Please check, the one sub-directory that is showing up in this case as well, "save1"> Any suggestions on how to fix this and how to prevent it from happening?I believe there are renames happening here, possibly by the archive creator, one way to prevent the rename from creating a linkto file is to use the DHT set parameter to set a pattern so that file name hash considers only the static part of the name. The set parameter is, cluster.extra-hash-regex. A link on a similar problem and how to use this set parameter (there a few in the gluster forums) would be, http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html Additionally, there is a bug here, the unlink of the file should have cleaned up the linkto as well, so that all of the above is not required, we have noticed this with NFS and FUSE mounts (ref bugs, 1117923, 1139992), and investigation is in progress on the same. We will step up the priority on this so that we have a clean fix that can be used to prevent this in the future. Shyam