Lana Deere
2010-Dec-20 23:27 UTC
[Gluster-users] permission denied and no such file for real files
I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport, fuse clients. The servers are running CentOS 5.5, the clients CentOS 5.4. The symptom I'm having which reproduces the most frequently goes like this. There are a bunch of jobs (several hundred) running on various client nodes (~50), but for no obvious reason one or several of them will get a failure to open a file (errno 13, permission denied). Subsequently it will not be possible to delete that directory because it is not empty. Attempts to delete the files which make it non-empty fail, "no such file or directory". I attached the program and driver script which I use to reproduce this, but not some log files (I ran them with trace and even compressed they are 150M). It is not perfect, it only reproduces maybe one-third of the runs. Here's a typical output: $ ATTACK-QRSH Quantity of jobs: 1000 Mon Dec 20 14:59:52 EST 2010: rm ..... Mon Dec 20 14:59:52 EST 2010: create data ..... Mon Dec 20 14:59:54 EST 2010: wait for creates..... Couldn't create data/051/307-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Couldn't create data/051/563-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Mon Dec 20 15:04:45 EST 2010: check data ..... Mon Dec 20 15:04:47 EST 2010: wait for checks..... Unable to reopen for read data/051/563-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Unable to reopen for read data/051/307-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Mon Dec 20 15:08:53 EST 2010: rm again ..... rm: cannot remove directory `data/051': Directory not empty Mon Dec 20 15:09:16 EST 2010: done $ cd data/051 $ ls 179-this!"":is:!""the""filename""and""it""is""very""strange 435-this!"":is:!""the""filename""and""it""is""very""strange $ rm * rm: cannot remove `179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory $ cd .. $ rm -r 051 rm: cannot remove `051/179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `051/435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory Note that the files which won't delete are in the same directory as had the permission error earlier. .. Lana (lana.deere at gmail.com)
Thai. Ngo Bao
2011-Jan-07 06:03 UTC
[Gluster-users] permission denied and no such file for real files
Dear All, I am having a problem, and I think it is similar to Lana's case. The problem is glusterfs prevents clients to create/delete files and folders. This problem appears so regularly. I attached the log file of our client. Please also keep informed that I'm running glusterfs 3.1.1 with 4 bricks, volumes distributed, tcp transport, fuse clients. The servers and clients are running CentOS 5.4. Would you have any suggestions, please let me know. Thanks. ~Thai -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Lana Deere Sent: Tuesday, December 21, 2010 6:27 AM To: gluster-users at gluster.org Subject: [Gluster-users] permission denied and no such file for real files I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport, fuse clients. The servers are running CentOS 5.5, the clients CentOS 5.4. The symptom I'm having which reproduces the most frequently goes like this. There are a bunch of jobs (several hundred) running on various client nodes (~50), but for no obvious reason one or several of them will get a failure to open a file (errno 13, permission denied). Subsequently it will not be possible to delete that directory because it is not empty. Attempts to delete the files which make it non-empty fail, "no such file or directory". I attached the program and driver script which I use to reproduce this, but not some log files (I ran them with trace and even compressed they are 150M). It is not perfect, it only reproduces maybe one-third of the runs. Here's a typical output: $ ATTACK-QRSH Quantity of jobs: 1000 Mon Dec 20 14:59:52 EST 2010: rm ..... Mon Dec 20 14:59:52 EST 2010: create data ..... Mon Dec 20 14:59:54 EST 2010: wait for creates..... Couldn't create data/051/307-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Couldn't create data/051/563-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Mon Dec 20 15:04:45 EST 2010: check data ..... Mon Dec 20 15:04:47 EST 2010: wait for checks..... Unable to reopen for read data/051/563-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Unable to reopen for read data/051/307-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Mon Dec 20 15:08:53 EST 2010: rm again ..... rm: cannot remove directory `data/051': Directory not empty Mon Dec 20 15:09:16 EST 2010: done $ cd data/051 $ ls 179-this!"":is:!""the""filename""and""it""is""very""strange 435-this!"":is:!""the""filename""and""it""is""very""strange $ rm * rm: cannot remove `179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory $ cd .. $ rm -r 051 rm: cannot remove `051/179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `051/435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory Note that the files which won't delete are in the same directory as had the permission error earlier. .. Lana (lana.deere at gmail.com) _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users