Alessandro Ipe
2015-Feb-23 15:49 UTC
[Gluster-users] rm -rf some_dir results in "Directory not empty"
On 4 of the 6 bricks, it is empty. However, on tsunami 3-4, ls -lsa gives
total 16
d--------- 2 root root 61440 Feb 23 15:42 .
drwxrwxrwx 3 gerb users 61 Feb 22 21:10 ..
---------T 2 gerb users 0 Apr 16 2014
akonadi-googledata-1.2.0-2.5.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
bluedevil-debugsource-1.2.2-1.8.3.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 bovo-4.7.4-3.12.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
digikam-debugsource-2.2.0-3.12.9.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
dolphin-debuginfo-4.7.4-4.22.6.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 freetds-doc-0.91-2.5.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
kanagram-debuginfo-4.7.4-2.10.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
kdebase4-runtime-4.7.4-3.17.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 kdebindings-smokegen-
debuginfo-4.7.4-2.9.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 kdesdk4-strigi-
debuginfo-4.7.4-3.12.5.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 kradio-4.0.2-9.9.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 kremotecontrol-4.7.4-2.12.9.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
kreversi-debuginfo-4.7.4-3.12.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 krfb-4.7.4-2.13.6.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 krusader-doc-2.0.0-23.9.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libalkimia-devel-4.3.1-2.5.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libdmtx0-0.7.4-2.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libdmtx0-debuginfo-0.7.4-2.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libkdegames4-
debuginfo-4.7.4-3.12.7.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libksane0-4.7.4-2.10.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
libkvkontakte-debugsource-1.0.0-2.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libmediawiki-
debugsource-2.5.0-4.6.1.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 libsmokeqt-4.7.4-2.10.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 NetworkManager-vpnc-
kde4-0.9.1git20111027-1.11.5.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 qtcurve-kde4-1.8.8-3.6.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014
QtZeitgeist-devel-0.7.0-7.4.2.i586.rpm
---------T 2 gerb users 0 Apr 16 2014 umbrello-4.7.4-3.12.5.i586.rpm
so that might be the reason of error. How can I fix this ?
Thanks,
A.
On Monday 23 February 2015 21:06:58 Ravishankar N wrote:
Just noticed that your `gluster volume status` shows that rebalance was
triggered. Maybe DHT developers can help out. I see a similar bug[1] has been
fixed
some time back. FWIW, can you check if "
/linux/suse/12.1/KDE4.7.4/i586" on all 6
bricks is indeed empty?
On 02/23/2015 08:15 PM, Alessandro Ipe wrote:
Hi,
Gluster version is 3.5.3-1.
/var/log/gluster.log (client log) gives during the rm -rf the following
logs:
[2015-02-23 14:42:50.180091] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-2: remote operation failed: Directory not empty
[2015-02-23 14:42:50.180134] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-3: remote operation failed: Directory not empty
[2015-02-23 14:42:50.180740] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-5: remote operation failed: File exists. Path:
/linux/suse/12.1/KDE4.7.4/i586
[2015-02-23 14:42:50.180772] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-4: remote operation failed: File exists. Path:
/linux/suse/12.1/KDE4.7.4/i586
[2015-02-23 14:42:50.181129] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-3: remote operation failed: File exists. Path:
/linux/suse/12.1/KDE4.7.4/i586
[2015-02-23 14:42:50.181160] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-2: remote operation failed: File exists. Path:
/linux/suse/12.1/KDE4.7.4/i586
[2015-02-23 14:42:50.319213] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-3: remote operation failed: Directory not empty
[2015-02-23 14:42:50.319762] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-2: remote operation failed: Directory not empty
[2015-02-23 14:42:50.320501] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-0: remote operation failed: File exists. Path:
/linux/suse/12.1/src-
oss/suse/src
[2015-02-23 14:42:50.320552] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-1: remote operation failed: File exists. Path:
/linux/suse/12.1/src-
oss/suse/src
[2015-02-23 14:42:50.320842] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-2: remote operation failed: File exists. Path:
/linux/suse/12.1/src-
oss/suse/src
[2015-02-23 14:42:50.320884] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-3: remote operation failed: File exists. Path:
/linux/suse/12.1/src-
oss/suse/src
[2015-02-23 14:42:50.438982] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-3: remote operation failed: Directory not empty
[2015-02-23 14:42:50.439347] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-2: remote operation failed: Directory not empty
[2015-02-23 14:42:50.440235] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-0: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/noarch
[2015-02-23 14:42:50.440344] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-1: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/noarch
[2015-02-23 14:42:50.440603] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-2: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/noarch
[2015-02-23 14:42:50.440665] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-3: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/noarch
[2015-02-23 14:42:50.680827] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-2: remote operation failed: Directory not empty
[2015-02-23 14:42:50.681721] W
[client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-
md1-client-3: remote operation failed: Directory not empty
[2015-02-23 14:42:50.682482] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-3: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/i586
[2015-02-23 14:42:50.682517] W
[client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-
md1-client-2: remote operation failed: File exists. Path:
/linux/suse/12.1/oss/suse/i586
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150223/39da7714/attachment.html>
Ravishankar N
2015-Feb-23 16:10 UTC
[Gluster-users] rm -rf some_dir results in "Directory not empty"
On 02/23/2015 09:19 PM, Alessandro Ipe wrote:> > On 4 of the 6 bricks, it is empty. However, on tsunami 3-4, ls -lsa gives > > total 16 > > d--------- 2 root root 61440 Feb 23 15:42 . > > drwxrwxrwx 3 gerb users 61 Feb 22 21:10 .. > > ---------T 2 gerb users 0 Apr 16 2014 > akonadi-googledata-1.2.0-2.5.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > bluedevil-debugsource-1.2.2-1.8.3.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 bovo-4.7.4-3.12.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > digikam-debugsource-2.2.0-3.12.9.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > dolphin-debuginfo-4.7.4-4.22.6.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 freetds-doc-0.91-2.5.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > kanagram-debuginfo-4.7.4-2.10.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > kdebase4-runtime-4.7.4-3.17.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > kdebindings-smokegen-debuginfo-4.7.4-2.9.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > kdesdk4-strigi-debuginfo-4.7.4-3.12.5.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 kradio-4.0.2-9.9.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 kremotecontrol-4.7.4-2.12.9.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > kreversi-debuginfo-4.7.4-3.12.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 krfb-4.7.4-2.13.6.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 krusader-doc-2.0.0-23.9.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > libalkimia-devel-4.3.1-2.5.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 libdmtx0-0.7.4-2.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > libdmtx0-debuginfo-0.7.4-2.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > libkdegames4-debuginfo-4.7.4-3.12.7.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 libksane0-4.7.4-2.10.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > libkvkontakte-debugsource-1.0.0-2.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > libmediawiki-debugsource-2.5.0-4.6.1.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 libsmokeqt-4.7.4-2.10.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > NetworkManager-vpnc-kde4-0.9.1git20111027-1.11.5.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 qtcurve-kde4-1.8.8-3.6.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 > QtZeitgeist-devel-0.7.0-7.4.2.i586.rpm > > ---------T 2 gerb users 0 Apr 16 2014 umbrello-4.7.4-3.12.5.i586.rpm > > so that might be the reason of error. How can I fix this ? >The '----T' files are DHT link-to files. The actual files must be present on the other distribute subolumes (tsunami 1-2 or tsunami 5-6) in the same path. But since that doesn't seem to be the case, the something went wrong with the re-balance process. You could run `gluster volume rebalance <volname> start+status` again and see if they disappear.> Thanks, > > A. > > On Monday 23 February 2015 21:06:58 Ravishankar N wrote: > > Just noticed that your `gluster volume status` shows that rebalance > was triggered. Maybe DHT developers can help out. I see a similar bug > <https://bugzilla.redhat.com/show_bug.cgi?id=1125824> has been fixed > some time back. > FWIW, can you check if " /linux/suse/12.1/KDE4.7.4/i586" on all 6 > bricks is indeed empty? > > > On 02/23/2015 08:15 PM, Alessandro Ipe wrote: > > Hi, > > Gluster version is 3.5.3-1. > > /var/log/gluster.log (client log) gives during the rm -rf the > following logs: > > [2015-02-23 14:42:50.180091] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-2: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.180134] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-3: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.180740] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-5: remote > operation failed: File exists. Path: /linux/suse/12.1/KDE4.7.4/i586 > > [2015-02-23 14:42:50.180772] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-4: remote > operation failed: File exists. Path: /linux/suse/12.1/KDE4.7.4/i586 > > [2015-02-23 14:42:50.181129] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-3: remote > operation failed: File exists. Path: /linux/suse/12.1/KDE4.7.4/i586 > > [2015-02-23 14:42:50.181160] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-2: remote > operation failed: File exists. Path: /linux/suse/12.1/KDE4.7.4/i586 > > [2015-02-23 14:42:50.319213] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-3: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.319762] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-2: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.320501] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-0: remote > operation failed: File exists. Path: /linux/suse/12.1/src-oss/suse/src > > [2015-02-23 14:42:50.320552] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-1: remote > operation failed: File exists. Path: /linux/suse/12.1/src-oss/suse/src > > [2015-02-23 14:42:50.320842] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-2: remote > operation failed: File exists. Path: /linux/suse/12.1/src-oss/suse/src > > [2015-02-23 14:42:50.320884] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-3: remote > operation failed: File exists. Path: /linux/suse/12.1/src-oss/suse/src > > [2015-02-23 14:42:50.438982] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-3: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.439347] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-2: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.440235] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-0: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/noarch > > [2015-02-23 14:42:50.440344] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-1: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/noarch > > [2015-02-23 14:42:50.440603] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-2: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/noarch > > [2015-02-23 14:42:50.440665] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-3: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/noarch > > [2015-02-23 14:42:50.680827] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-2: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.681721] W > [client-rpc-fops.c:696:client3_3_rmdir_cbk] 0-md1-client-3: remote > operation failed: Directory not empty > > [2015-02-23 14:42:50.682482] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-3: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/i586 > > [2015-02-23 14:42:50.682517] W > [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-md1-client-2: remote > operation failed: File exists. Path: /linux/suse/12.1/oss/suse/i586 > > Thanks, > > A. > > On Monday 23 February 2015 20:06:17 Ravishankar N wrote: > > > On 02/23/2015 07:04 PM, Alessandro Ipe wrote: > > Hi Ravi, > > gluster volume status md1 returns > > Status of volume: md1 > > Gluster process Port Online Pid > > ------------------------------------------------------------------------------ > > > Brick tsunami1:/data/glusterfs/md1/brick1 49157 Y 2260 > > Brick tsunami2:/data/glusterfs/md1/brick1 49152 Y 2320 > > Brick tsunami3:/data/glusterfs/md1/brick1 49156 Y 20715 > > Brick tsunami4:/data/glusterfs/md1/brick1 49156 Y 10544 > > Brick tsunami5:/data/glusterfs/md1/brick1 49152 Y 12588 > > Brick tsunami6:/data/glusterfs/md1/brick1 49152 Y 12242 > > Self-heal Daemon on localhost N/A Y 2336 > > Self-heal Daemon on tsunami2 N/A Y 2359 > > Self-heal Daemon on tsunami5 N/A Y 27619 > > Self-heal Daemon on tsunami4 N/A Y 12318 > > Self-heal Daemon on tsunami3 N/A Y 19118 > > Self-heal Daemon on tsunami6 N/A Y 27650 > > Task Status of Volume md1 > > ------------------------------------------------------------------------------ > > > Task : Rebalance > > ID : 9dfee1a2-49ac-4766-bdb6-00de5e5883f6 > > Status : completed > > so it seems that all brick server are up. > > gluster volume heal md1 info returns > > Brick tsunami1.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Brick tsunami2.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Brick tsunami3.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Brick tsunami4.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Brick tsunami5.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Brick tsunami6.oma.be:/data/glusterfs/md1/brick1/ > > Number of entries: 0 > > Should I run "gluster volume heal md1 full" ? > > Hi Alessandro, > > Looks like there is no pending-self heals, so no need to run the heal > command. Can you share the output of the client (mount) log when you > get the ENOTEMPTY during the rm -rf? > > What version of gluster are you using? > Thanks, > Ravi > > Thanks, > > A. > > On Monday 23 February 2015 18:12:43 Ravishankar N wrote: > > > On 02/23/2015 05:42 PM, Alessandro Ipe wrote: > > Hi, > > We have a "md1" volume under gluster 3.5.3 over 6 servers configured > as distributed and replicated. When trying on a client, thourgh fuse > mount (which turns out to be also a brick server) to delete (as root) > recursively a directory with "rm -rf /home/.md1/linux/suse/12.1", I > get the error messages > > rm: cannot remove ?/home/.md1/linux/suse/12.1/KDE4.7.4/i586?: > Directory not empty > > rm: cannot remove ?/home/.md1/linux/suse/12.1/src-oss/suse/src?: > Directory not empty > > rm: cannot remove ?/home/.md1/linux/suse/12.1/oss/suse/noarch?: > Directory not empty > > rm: cannot remove ?/home/.md1/linux/suse/12.1/oss/suse/i586?: > Directory not empty > > (the same occurs as unprivileged user but with "Permission denied".) > > while a "ls -Ral /home/.md1/linux/suse/12.1" gives me > > /home/.md1/linux/suse/12.1: > > total 0 > > drwxrwxrwx 5 gerb users 151 Feb 20 16:22 . > > drwxr-xr-x 6 gerb users 245 Feb 23 12:55 .. > > drwxrwxrwx 3 gerb users 95 Feb 23 13:03 KDE4.7.4 > > drwxrwxrwx 3 gerb users 311 Feb 20 16:57 oss > > drwxrwxrwx 3 gerb users 86 Feb 20 16:20 src-oss > > /home/.md1/linux/suse/12.1/KDE4.7.4: > > total 28 > > drwxrwxrwx 3 gerb users 95 Feb 23 13:03 . > > drwxrwxrwx 5 gerb users 151 Feb 20 16:22 .. > > d--------- 2 root root 61452 Feb 23 13:03 i586 > > /home/.md1/linux/suse/12.1/KDE4.7.4/i586: > > total 28 > > d--------- 2 root root 61452 Feb 23 13:03 . > > drwxrwxrwx 3 gerb users 95 Feb 23 13:03 .. > > /home/.md1/linux/suse/12.1/oss: > > total 0 > > drwxrwxrwx 3 gerb users 311 Feb 20 16:57 . > > drwxrwxrwx 5 gerb users 151 Feb 20 16:22 .. > > drwxrwxrwx 4 gerb users 90 Feb 23 13:03 suse > > /home/.md1/linux/suse/12.1/oss/suse: > > total 536 > > drwxrwxrwx 4 gerb users 90 Feb 23 13:03 . > > drwxrwxrwx 3 gerb users 311 Feb 20 16:57 .. > > d--------- 2 root root 368652 Feb 23 13:03 i586 > > d--------- 2 root root 196620 Feb 23 13:03 noarch > > /home/.md1/linux/suse/12.1/oss/suse/i586: > > total 360 > > d--------- 2 root root 368652 Feb 23 13:03 . > > drwxrwxrwx 4 gerb users 90 Feb 23 13:03 .. > > /home/.md1/linux/suse/12.1/oss/suse/noarch: > > total 176 > > d--------- 2 root root 196620 Feb 23 13:03 . > > drwxrwxrwx 4 gerb users 90 Feb 23 13:03 .. > > /home/.md1/linux/suse/12.1/src-oss: > > total 0 > > drwxrwxrwx 3 gerb users 86 Feb 20 16:20 . > > drwxrwxrwx 5 gerb users 151 Feb 20 16:22 .. > > drwxrwxrwx 3 gerb users 48 Feb 23 13:03 suse > > /home/.md1/linux/suse/12.1/src-oss/suse: > > total 220 > > drwxrwxrwx 3 gerb users 48 Feb 23 13:03 . > > drwxrwxrwx 3 gerb users 86 Feb 20 16:20 .. > > d--------- 2 root root 225292 Feb 23 13:03 src > > /home/.md1/linux/suse/12.1/src-oss/suse/src: > > total 220 > > d--------- 2 root root 225292 Feb 23 13:03 . > > drwxrwxrwx 3 gerb users 48 Feb 23 13:03 .. > > Is there a cure such as manually forcing a healing on that directory ? > > > Are all bricks up? Are there any pending self-heals ? Does `gluster > volume heal md1` info show any output? If it does, run 'gluster volume > heal md1' to manually trigger heal. > -Ravi > > Many thanks, > > Alessandro. > > gluster volume info md1 outputs: > > Volume Name: md1 > > Type: Distributed-Replicate > > Volume ID: 6da4b915-1def-4df4-a41c-2f3300ebf16b > > Status: Started > > Number of Bricks: 3 x 2 = 6 > > Transport-type: tcp > > Bricks: > > Brick1: tsunami1:/data/glusterfs/md1/brick1 > > Brick2: tsunami2:/data/glusterfs/md1/brick1 > > Brick3: tsunami3:/data/glusterfs/md1/brick1 > > Brick4: tsunami4:/data/glusterfs/md1/brick1 > > Brick5: tsunami5:/data/glusterfs/md1/brick1 > > Brick6: tsunami6:/data/glusterfs/md1/brick1 > > Options Reconfigured: > > performance.write-behind: on > > performance.write-behind-window-size: 4MB > > performance.flush-behind: off > > performance.io-thread-count: 64 > > performance.cache-size: 512MB > > nfs.disable: on > > features.quota: off > > cluster.read-hash-mode: 2 > > server.allow-insecure: on > > cluster.lookup-unhashed: off > > > > _______________________________________________Gluster-users mailing > listGluster-users at gluster.org > <mailto:Gluster-users at gluster.org>http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > -- >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150223/09d1c454/attachment.html>
Hi, I launched a couple a days ago a rebalance on my gluster distribute-replicate volume (see below) through its CLI, while allowing my users to continue using the volume. Yesterday, they managed to fill completely the volume. It now results in unavailable files on the client (using fuse) with the message "Transport endpoint is not connected". Investigating to associated files on the bricks, I noticed that these are displayed with ls -l as ---------T 2 user group 0 Jan 15 22:00 file Performing a ls -lR /data/glusterfs/home/brick1/* | grep -F -- "---------T" on a single brick gave me a LOT of files in that above-mentioned state. Why are the files in that state ? Did I lose all these files or can they still be recovered from the replicate copy of another brick ? Regards, Alessandro. gluster volume info home output: Volume Name: home Type: Distributed-Replicate Volume ID: 501741ed-4146-4022-af0b-41f5b1297766 Status: Started Number of Bricks: 12 x 2 = 24 Transport-type: tcp Bricks: Brick1: tsunami1:/data/glusterfs/home/brick1 Brick2: tsunami2:/data/glusterfs/home/brick1 Brick3: tsunami1:/data/glusterfs/home/brick2 Brick4: tsunami2:/data/glusterfs/home/brick2 Brick5: tsunami1:/data/glusterfs/home/brick3 Brick6: tsunami2:/data/glusterfs/home/brick3 Brick7: tsunami1:/data/glusterfs/home/brick4 Brick8: tsunami2:/data/glusterfs/home/brick4 Brick9: tsunami3:/data/glusterfs/home/brick1 Brick10: tsunami4:/data/glusterfs/home/brick1 Brick11: tsunami3:/data/glusterfs/home/brick2 Brick12: tsunami4:/data/glusterfs/home/brick2 Brick13: tsunami3:/data/glusterfs/home/brick3 Brick14: tsunami4:/data/glusterfs/home/brick3 Brick15: tsunami3:/data/glusterfs/home/brick4 Brick16: tsunami4:/data/glusterfs/home/brick4 Brick17: tsunami5:/data/glusterfs/home/brick1 Brick18: tsunami6:/data/glusterfs/home/brick1 Brick19: tsunami5:/data/glusterfs/home/brick2 Brick20: tsunami6:/data/glusterfs/home/brick2 Brick21: tsunami5:/data/glusterfs/home/brick3 Brick22: tsunami6:/data/glusterfs/home/brick3 Brick23: tsunami5:/data/glusterfs/home/brick4 Brick24: tsunami6:/data/glusterfs/home/brick4 Options Reconfigured: features.default-soft-limit: 95% cluster.ensure-durability: off performance.cache-size: 512MB performance.io-thread-count: 64 performance.flush-behind: off performance.write-behind-window-size: 4MB performance.write-behind: on nfs.disable: on features.quota: on cluster.read-hash-mode: 2 diagnostics.brick-log-level: CRITICAL cluster.lookup-unhashed: off server.allow-insecure: on -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150310/b75d33d1/attachment.html>
Alessandro Ipe
2015-Mar-11 11:05 UTC
[Gluster-users] Input/output error when trying to access a file on client
Hi, When trying to access a file on a gluster client (through fuse), I get an "Input/output error" message. Getting the attributes for the file gives me for the first brick # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2 trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA trusted.afr.md1-client-3=0sAAABdAAAAAAAAAAA trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ= while for the second (replicate) brick # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2 trusted.afr.md1-client-2=0sAAABJAAAAAAAAAAA trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ= It seems that I have a split-brain. How can I solve this issue by resetting the attributes, please ? Thanks, Alessandro. =================gluster volume info md1 Volume Name: md1 Type: Distributed-Replicate Volume ID: 6da4b915-1def-4df4-a41c-2f3300ebf16b Status: Started Number of Bricks: 3 x 2 = 6 Transport-type: tcp Bricks: Brick1: tsunami1:/data/glusterfs/md1/brick1 Brick2: tsunami2:/data/glusterfs/md1/brick1 Brick3: tsunami3:/data/glusterfs/md1/brick1 Brick4: tsunami4:/data/glusterfs/md1/brick1 Brick5: tsunami5:/data/glusterfs/md1/brick1 Brick6: tsunami6:/data/glusterfs/md1/brick1 Options Reconfigured: server.allow-insecure: on cluster.read-hash-mode: 2 features.quota: off performance.write-behind: on performance.write-behind-window-size: 4MB performance.flush-behind: off performance.io-thread-count: 64 performance.cache-size: 512MB nfs.disable: on cluster.lookup-unhashed: off