thr3ads.net - Gluster users - [Gluster-users] [Gluster-devel] cannot delete non-empty directory [Feb 2015]

If this information is useful, please help other people find it:
Share via:

Shyam

2015-Feb-09 16:11 UTC

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

On 02/08/2015 12:19 PM, David F. Robinson wrote:> I am seeing these messsages after I delete large amounts of data using
> gluster 3.6.2.
> cannot delete non-empty directory:
> old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
> *_From the FUSE mount (as root), the directory shows up as empty:_*
> # pwd
>
/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>
> # ls -al
> total 5
> d--------- 2 root root    4106 Feb  6 13:55 .
> drwxrws--- 3  601 dmiller   72 Feb  6 13:55 ..
> However, when you look at the bricks, the files are still there (none on
> brick01bkp, all files are on brick02bkp).  All of the files are 0-length
> and have ------T permissions.
These files are linkto files that are created by DHT, which basically 
mean the files were either renamed, or the brick layout changed (I 
suspect the former to be the cause).

These files should have been deleted when the files that they point to 
were deleted, looks like this did not happen.

Can I get the following information for some of the files here,
- getfattr -d -m . -e text <path to file on brick>
   - The output of trusted.glusterfs.dht.linkto xattr should state where 
the real file belongs, in this case as there are only 2 bricks, it 
should be brick01bkp subvol
- As the second brick is empty, we should be able to safely delete these 
files from the brick and proceed to do an rmdir on the mount point of 
the volume as the directory is now empty.
- Please check, the one sub-directory that is showing up in this case as 
well, "save1"
> Any suggestions on how to fix this and how to prevent it from happening?
I believe there are renames happening here, possibly by the archive 
creator, one way to prevent the rename from creating a linkto file is to 
use the DHT set parameter to set a pattern so that file name hash 
considers only the static part of the name.

The set parameter is, cluster.extra-hash-regex.

A link on a similar problem and how to use this set parameter (there a 
few in the gluster forums) would be, 
http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html

Additionally, there is a bug here, the unlink of the file should have 
cleaned up the linkto as well, so that all of the above is not required, 
we have noticed this with NFS and FUSE mounts (ref bugs, 1117923, 
1139992), and investigation is in progress on the same. We will step up 
the priority on this so that we have a clean fix that can be used to 
prevent this in the future.

Shyam

David F. Robinson

2015-Feb-09 17:17 UTC

head link

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24
[root at gfs01bkp C24]# ls -al
total 1
drwx------ 2 jcowan users 39 Feb  6 12:41 .
drwxrw-rw- 4 jcowan users 62 Feb  6 19:19 ..

[root at gfs01bkp C24]# ls -al 
/data/brick*/homegfs_bkp/backup.0/old_shelf4/Aegis/\!\!\!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/
total 4
drwxrw-rw-+ 2 jcowan users 4096 Feb  6 12:41 .
drwxrw-rw-+ 3 jcowan users   29 Feb  6 12:41 ..
---------T  5 jcowan users    0 Nov 19 23:30 
c24-airbox_vr_z=25_zoom.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=26.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=27.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=28.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 
c24-airbox_vr_z=29.5_zoom.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=30.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=31.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=32.5.jpeg
[root at gfs01bkp C24]# getfattr -d -m . -e text 
/data/brick*bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/\!\!\!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/*
getfattr: Removing leading '/' from absolute path names
# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=25_zoom.jpeg
trusted.gfid="?r'V*N???F?"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=26.jpeg
trusted.gfid="L??}??Ldza?U"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=27.jpeg
trusted.gfid="?.???2@???d???%_"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=28.jpeg
trusted.gfid="0?
                 /D??x???"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=29.5_zoom.jpeg
trusted.gfid="?9T'$?C??E?x!1"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=30.jpeg
trusted.gfid="t?
8r?"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=31.jpeg
trusted.gfid="x??
                  E???Zm?W?"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=32.5.jpeg
trusted.gfid="d+0?x?M?Gx?@>?"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"




------ Original Message ------
From: "Shyam" <srangana at redhat.com>
To: "David F. Robinson" <david.robinson at corvidtec.com>;
"Gluster Devel"
<gluster-devel at gluster.org>; "gluster-users at gluster.org" 
<gluster-users at gluster.org>; "Susant Palai" <spalai at
redhat.com>
Sent: 2/9/2015 11:11:20 AM
Subject: Re: [Gluster-devel] cannot delete non-empty directory
>On 02/08/2015 12:19 PM, David F. Robinson wrote:
>>I am seeing these messsages after I delete large amounts of data using
>>gluster 3.6.2.
>>cannot delete non-empty directory:
>>old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>*_From the FUSE mount (as root), the directory shows up as empty:_*
>># pwd
>>/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>
>># ls -al
>>total 5
>>d--------- 2 root root 4106 Feb 6 13:55 .
>>drwxrws--- 3 601 dmiller 72 Feb 6 13:55 ..
>>However, when you look at the bricks, the files are still there (none 
>>on
>>brick01bkp, all files are on brick02bkp). All of the files are 
>>0-length
>>and have ------T permissions.
>
>These files are linkto files that are created by DHT, which basically 
>mean the files were either renamed, or the brick layout changed (I 
>suspect the former to be the cause).
>
>These files should have been deleted when the files that they point to 
>were deleted, looks like this did not happen.
>
>Can I get the following information for some of the files here,
>- getfattr -d -m . -e text <path to file on brick>
>   - The output of trusted.glusterfs.dht.linkto xattr should state where 
>the real file belongs, in this case as there are only 2 bricks, it 
>should be brick01bkp subvol
>- As the second brick is empty, we should be able to safely delete 
>these files from the brick and proceed to do an rmdir on the mount 
>point of the volume as the directory is now empty.
>- Please check, the one sub-directory that is showing up in this case 
>as well, "save1"
>
>>Any suggestions on how to fix this and how to prevent it from 
>>happening?
>
>I believe there are renames happening here, possibly by the archive 
>creator, one way to prevent the rename from creating a linkto file is 
>to use the DHT set parameter to set a pattern so that file name hash 
>considers only the static part of the name.
>
>The set parameter is, cluster.extra-hash-regex.
>
>A link on a similar problem and how to use this set parameter (there a 
>few in the gluster forums) would be, 
>http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html
>
>Additionally, there is a bug here, the unlink of the file should have 
>cleaned up the linkto as well, so that all of the above is not 
>required, we have noticed this with NFS and FUSE mounts (ref bugs, 
>1117923, 1139992), and investigation is in progress on the same. We 
>will step up the priority on this so that we have a clean fix that can 
>be used to prevent this in the future.
>
>Shyam

David F. Robinson

2015-Feb-09 17:25 UTC

head link

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

So, just to be sure before I do this, it is okay to do the following if 
I want to get rid of everything in the /old_shelf4/Aegis directory and 
below?

  rm -rf /data/brick*/homegfs_bkp/backup.0/old_shelf4/Aegis

What happens to all of the files in the .glusterfs directory?  Does this 
get rebuilt or do the links stay there for files that now no longer 
exist?

And, is this same issue what causes all of the broken links in 
.glusterfs.  See attached image for example.  There appears to be a lot 
of broken links the .glusterfs directories.  Is this normal or does it 
indicate another problem.

Finally, if I search through the /data/brick* directories, should I find 
no entries of "-------T" permission files with zero length files?  Do
I
need to clean all of these up somehow?  A quick look at 
/data/brick01bkp/homegfs_bkp/.glusterfs/2f/54 shows many of these files. 
  They look like
---------T   3 rbhinge      pme_ics           0 Jan  9 16:45 
2f54d7d6-968b-442f-8cfe-eff01d6cefe7
---------T   2 rbhinge      pme_ics           0 Jan  9 21:40 
2f54d7e7-b198-4fd4-aec7-f5d0ff020f72

How do I find out what file these entries were pointing to?

David




------ Original Message ------
From: "Shyam" <srangana at redhat.com>
To: "David F. Robinson" <david.robinson at corvidtec.com>;
"Gluster Devel"
<gluster-devel at gluster.org>; "gluster-users at gluster.org" 
<gluster-users at gluster.org>; "Susant Palai" <spalai at
redhat.com>
Sent: 2/9/2015 11:11:20 AM
Subject: Re: [Gluster-devel] cannot delete non-empty directory
>On 02/08/2015 12:19 PM, David F. Robinson wrote:
>>I am seeing these messsages after I delete large amounts of data using
>>gluster 3.6.2.
>>cannot delete non-empty directory:
>>old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>*_From the FUSE mount (as root), the directory shows up as empty:_*
>># pwd
>>/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>
>># ls -al
>>total 5
>>d--------- 2 root root 4106 Feb 6 13:55 .
>>drwxrws--- 3 601 dmiller 72 Feb 6 13:55 ..
>>However, when you look at the bricks, the files are still there (none 
>>on
>>brick01bkp, all files are on brick02bkp). All of the files are 
>>0-length
>>and have ------T permissions.
>
>These files are linkto files that are created by DHT, which basically 
>mean the files were either renamed, or the brick layout changed (I 
>suspect the former to be the cause).
>
>These files should have been deleted when the files that they point to 
>were deleted, looks like this did not happen.
>
>Can I get the following information for some of the files here,
>- getfattr -d -m . -e text <path to file on brick>
>   - The output of trusted.glusterfs.dht.linkto xattr should state where 
>the real file belongs, in this case as there are only 2 bricks, it 
>should be brick01bkp subvol
>- As the second brick is empty, we should be able to safely delete 
>these files from the brick and proceed to do an rmdir on the mount 
>point of the volume as the directory is now empty.
>- Please check, the one sub-directory that is showing up in this case 
>as well, "save1"
>
>>Any suggestions on how to fix this and how to prevent it from 
>>happening?
>
>I believe there are renames happening here, possibly by the archive 
>creator, one way to prevent the rename from creating a linkto file is 
>to use the DHT set parameter to set a pattern so that file name hash 
>considers only the static part of the name.
>
>The set parameter is, cluster.extra-hash-regex.
>
>A link on a similar problem and how to use this set parameter (there a 
>few in the gluster forums) would be, 
>http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html
>
>Additionally, there is a bug here, the unlink of the file should have 
>cleaned up the linkto as well, so that all of the above is not 
>required, we have noticed this with NFS and FUSE mounts (ref bugs, 
>1117923, 1139992), and investigation is in progress on the same. We 
>will step up the priority on this so that we have a clean fix that can 
>be used to prevent this in the future.
>
>Shyam-------------- next part --------------
A non-text attachment was scrubbed...
Name: n4yv5cww.png
Type: image/png
Size: 29006 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150209/72fc863e/attachment.png>

David Robinson

2015-Feb-10 05:05 UTC

head link

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

Shyam,
> These files are linkto files that are created by DHT, which basically mean
the files were either renamed, or the brick layout changed (I suspect the former
to be the cause).
> 
> These files should have been deleted when the files that they point to were
deleted, looks like this did not happen.
Could these linkto files be causing my performance issues? I will forward you a
couple of emails that I sent Ben and justin related to severe performance issues
I am having with gluster.


> 
>> On 02/08/2015 12:19 PM, David F. Robinson wrote:
>> I am seeing these messsages after I delete large amounts of data using
>> gluster 3.6.2.
>> cannot delete non-empty directory:
>> old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>> *_From the FUSE mount (as root), the directory shows up as empty:_*
>> # pwd
>>
/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>> 
>> # ls -al
>> total 5
>> d--------- 2 root root    4106 Feb  6 13:55 .
>> drwxrws--- 3  601 dmiller   72 Feb  6 13:55 ..
>> However, when you look at the bricks, the files are still there (none
on
>> brick01bkp, all files are on brick02bkp).  All of the files are
0-length
>> and have ------T permissions.
> 
> These files are linkto files that are created by DHT, which basically mean
the files were either renamed, or the brick layout changed (I suspect the former
to be the cause).
> 
> These files should have been deleted when the files that they point to were
deleted, looks like this did not happen.
> 
> Can I get the following information for some of the files here,
> - getfattr -d -m . -e text <path to file on brick>
>  - The output of trusted.glusterfs.dht.linkto xattr should state where the
real file belongs, in this case as there are only 2 bricks, it should be
brick01bkp subvol
> - As the second brick is empty, we should be able to safely delete these
files from the brick and proceed to do an rmdir on the mount point of the volume
as the directory is now empty.
> - Please check, the one sub-directory that is showing up in this case as
well, "save1"
> 
>> Any suggestions on how to fix this and how to prevent it from
happening?
> 
> I believe there are renames happening here, possibly by the archive
creator, one way to prevent the rename from creating a linkto file is to use the
DHT set parameter to set a pattern so that file name hash considers only the
static part of the name.
> 
> The set parameter is, cluster.extra-hash-regex.
> 
> A link on a similar problem and how to use this set parameter (there a few
in the gluster forums) would be,
http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html
> 
> Additionally, there is a bug here, the unlink of the file should have
cleaned up the linkto as well, so that all of the above is not required, we have
noticed this with NFS and FUSE mounts (ref bugs, 1117923, 1139992), and
investigation is in progress on the same. We will step up the priority on this
so that we have a clean fix that can be used to prevent this in the future.
> 
> Shyam-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150210/d3a1b586/attachment.html>

Gluster users - Feb 2015 - [Gluster-devel] cannot delete non-empty directory

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

[Gluster-users] [Gluster-devel] cannot delete non-empty directory

[Gluster-users] [Gluster-devel] cannot delete non-empty directory