SINCOCK John
2014-Jun-15 13:46 UTC
[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs
Hi Everybody,
We have recently setup a 2-node gluster, and are having some problems
deleting some files!
We haven't accessed the gluster bricks directly, all access has been
through the glusterfs mount
The 1st node in our cluster also handles sharing the glusterfs volume
via samba, to windows clients.
Environment is:
Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 6.4 (Final)
glusterfs-libs-3.4.1-3.el6.x86_64
glusterfs-cli-3.4.1-3.el6.x86_64
glusterfs-fuse-3.4.1-3.el6.x86_64
glusterfs-server-3.4.1-3.el6.x86_64
glusterfs-3.4.1-3.el6.x86_64
4 bricks on each node
Gluster is setup without replication or striping
Bricks on one of the two nodes (the other node is setup
the same):
/dev/sda1 9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1
/dev/sda2 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2
/dev/sda3 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3
/dev/sda4 9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4
Gluster volume mounted on 1st node:
g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00
Mount options:
g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs
(rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072
)
So the problem is this:
[root at g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final
rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not
empty
rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not
empty
[root at g-unit-1 ~]#
And the delete leaves behind 2000 or so files:
[root at g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/
-type f | wc -l
2185
Almost all the left behind files are on the 2nd node, g-unit-2, but
there are a few on g-unit-1
>From the 2nd node, g-unit-2:
[root at g-unit-2 ~]# getfattr -m . -d -e hex
/mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.e
fi
getfattr: Removing leading '/' from absolute path names
# file:
mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.ef
i
trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f
So the file is there, is tagged with some gluster info, and has
definitely been copied onto the volume properly, via the glusterfs
mount. It hasn't been copied directly onto the brick.
But from g-unit-1, accessing via the glusterfs mount, and trying to
delete this file:
[root at g-unit-1 ~]# rm
/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
rm: remove regular file
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y
rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': No
such file or directory
What the hell!?!!!!!
Can someone please suggest the best course of action to resolve this and
prevent it happening in the future?
Thanks for any assistance.
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140615/3de40562/attachment.html>
Pranith Kumar Karampuri
2014-Jun-18 02:18 UTC
[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs
I think Susant knows about this issue CCed him Pranith On 06/15/2014 07:16 PM, SINCOCK John wrote:> > Hi Everybody, > > We have recently setup a 2-node gluster, and are having some problems > deleting some files! > > We haven't accessed the gluster bricks directly, all access has been > through the glusterfs mount > > The 1^st node in our cluster also handles sharing the glusterfs volume > via samba, to windows clients. > > Environment is: > > Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC > 2013 x86_64 x86_64 x86_64 GNU/Linux > > CentOS release 6.4 (Final) > > glusterfs-libs-3.4.1-3.el6.x86_64 > > glusterfs-cli-3.4.1-3.el6.x86_64 > > glusterfs-fuse-3.4.1-3.el6.x86_64 > > glusterfs-server-3.4.1-3.el6.x86_64 > > glusterfs-3.4.1-3.el6.x86_64 > > 4 bricks on each node > > Gluster is setup without replication or striping > > Bricks on one of the two nodes (the other node is setup > the same): > > /dev/sda1 9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1 > > /dev/sda2 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2 > > /dev/sda3 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3 > > /dev/sda4 9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4 > > Gluster volume mounted on 1st node: > > g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00 > > Mount options: > > g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs > (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072) > > *So the problem is this:* > > [root at g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not > empty > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not > empty > > [root at g-unit-1 ~]# > > And the delete leaves behind 2000 or so files: > > [root at g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ > -type f | wc -l > > 2185 > > Almost all the left behind files are on the 2^nd node, g-unit-2, but > there are a few on g-unit-1 > > From the 2^nd node, g-unit-2: > > [root at g-unit-2 ~]# getfattr -m . -d -e hex > /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > getfattr: Removing leading '/' from absolute path names > > # file: > mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f > > So the file is there, is tagged with some gluster info, and has > definitely been copied onto the volume properly, via the glusterfs > mount. It hasn't been copied directly onto the brick. > > But from g-unit-1, accessing via the glusterfs mount, and trying to > delete this file: > > [root at g-unit-1 ~]# rm > /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > rm: remove regular file > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': > No such file or directory > > What the hell!?!!!!! > > Can someone please suggest the best course of action to resolve this > and prevent it happening in the future? > > Thanks for any assistance. > > John > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140618/dc041c25/attachment.html>