Andrus, Brian Contractor
2016-Jan-28 19:20 UTC
[Gluster-users] Moved files from one directory to another, now gone
All, I have a glusterfs setup with a disperse volume over 3 zfs bricks, one on each node. I just did a 'mv' of some log files from one directory to another and when I look in the directory, they are not there at all! Neither is any of the data I used to have. It is completely empty. I try doing a 'touch' of a file and nothing shows up. I do see the files on the zpool mount, but they seem to be partially there. They are all just text files, but they look incomplete. There seems to be many errors in the log: [2016-01-28 18:57:41.353265] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-DATA-disperse-0: Heal failed [Transport endpoint is not connected] Any ideas on what this is? Other directories seem ok. Thanks in advance, Brian Andrus -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160128/9e925fd8/attachment.html>
Pranith Kumar Karampuri
2016-Jan-29 06:36 UTC
[Gluster-users] Moved files from one directory to another, now gone
On 01/29/2016 12:50 AM, Andrus, Brian Contractor wrote:> > All, > > I have a glusterfs setup with a disperse volume over 3 zfs bricks, one > on each node. > > I just did a ?mv? of some log files from one directory to another and > when I look in the directory, they are not there at all! > > Neither is any of the data I used to have. It is completely empty. I > try doing a ?touch? of a file and nothing shows up. > > I do see the files on the zpool mount, but they seem to be partially > there. They are all just text files, but they look incomplete. > > There seems to be many errors in the log: > > [2016-01-28 18:57:41.353265] W [MSGID: 122002] > [ec-common.c:71:ec_heal_report] 0-DATA-disperse-0: Heal failed > [Transport endpoint is not connected] > > Any ideas on what this is? Other directories seem ok. >Could you give me the following information: 1) Output of: gluster volume info 2) Log files of the setup? Both client and brick logs, they should be present in /var/log/glusterfs 3) What is the version of glusterfs you are using? Pranith> > Thanks in advance, > > Brian Andrus > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160129/b4a83fa7/attachment.html>
Andrus, Brian Contractor
2016-Mar-03 19:50 UTC
[Gluster-users] Moved files from one directory to another, now gone
Pranith, Sorry for taking so long. Work required my time. So I can look at this a little more right now. I am using glusterfs-3.7.6-1 Here is some pertinent info: ]# gluster volume info DATA Volume Name: DATA Type: Disperse Volume ID: 8374314b-796a-48d0-acf7-dea51fdd83ae Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server45:/zpool1/DATABRICK Brick2: server46:/zpool1/DATABRICK Brick3: server47:/zpool1/DATABRICK Options Reconfigured: performance.io-cache: off performance.quick-read: off performance.readdir-ahead: disable # df -h /zpool1 /DATA Filesystem Size Used Avail Use% Mounted on zpool1 7.2T 1.2T 6.0T 17% /zpool1 server45:/DATA 15T 2.5T 12T 18% /DATA # du -hsc /DATA/ 232G /DATA/ 232G total I have been cleaning up some of the stuff, but there are 2 directories I cannot remove: # ls /DATA/ FINISHED_LOGS home JOBS LOGS m4p2 OLDLOGS # ls /zpool1/DATABRICK/ FINISHED_LOGS/ .glusterfs/ home/ JOBS/ LOGS/ m4p2/ OLDLOGS/ .trashcan/ # ls /zpool1/DATABRICK/ FINISHED_LOGS home JOBS LOGS m4p2 OLDLOGS # rm -rf /DATA/OLDLOGS/ # rm -rf /DATA/FINISHED_LOGS/ # ls /DATA/ FINISHED_LOGS home JOBS LOGS m4p2 OLDLOGS Those two directories are empty on the glusterfs mount, but still have files when I look on the brick. Strange bug. I will be just reformatting the thing when some of the work completes, but I can run some tests and provide info if you want to try and track it down. Brian Andrus From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] Sent: Thursday, January 28, 2016 10:36 PM To: Andrus, Brian Contractor <bdandrus at nps.edu>; gluster-users at gluster.org Subject: Re: [Gluster-users] Moved files from one directory to another, now gone On 01/29/2016 12:50 AM, Andrus, Brian Contractor wrote: All, I have a glusterfs setup with a disperse volume over 3 zfs bricks, one on each node. I just did a 'mv' of some log files from one directory to another and when I look in the directory, they are not there at all! Neither is any of the data I used to have. It is completely empty. I try doing a 'touch' of a file and nothing shows up. I do see the files on the zpool mount, but they seem to be partially there. They are all just text files, but they look incomplete. There seems to be many errors in the log: [2016-01-28 18:57:41.353265] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-DATA-disperse-0: Heal failed [Transport endpoint is not connected] Any ideas on what this is? Other directories seem ok. Could you give me the following information: 1) Output of: gluster volume info 2) Log files of the setup? Both client and brick logs, they should be present in /var/log/glusterfs 3) What is the version of glusterfs you are using? Pranith Thanks in advance, Brian Andrus _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160303/1d7b91ee/attachment.html>