Krutika Dhananjay
2015-Mar-12 11:51 UTC
[Gluster-users] Input/output error when trying to access a file on client
Hi, Could you provide the xattrs in hex format? You can execute `getfattr -d -m . -e hex <path-to-the-directory/file-on-the-brick(s)>` -Krutika ----- Original Message -----> From: "Alessandro Ipe" <Alessandro.Ipe at meteo.be> > To: "Krutika Dhananjay" <kdhananj at redhat.com> > Cc: gluster-users at gluster.org > Sent: Thursday, March 12, 2015 5:15:08 PM > Subject: Re: [Gluster-users] Input/output error when trying to access a file > on client> Hi,> Actually, my gluster volume is distribute-replicate so I should provide the > attributes on all the bricks. Here they are:> 1. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= > 2. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= > 3. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-3=0sAAAAAAAAAAEAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA= > 4. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-2=0sAAAAAAAAAAEAAAAA> trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA= > 5. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-4=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-5=0sAAAAAAAAAAEAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ= > 6. # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-4=0sAAAAAAAAAAEAAAAA> trusted.afr.md1-client-5=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ= > so it seems in fact that there are discrepancies between 3-4 and 5-6 > (replicate pairs).> A.> On Thursday 12 March 2015 11:33:00 Alessandro Ipe wrote:> Hi,> "gluster volume heal md1 info split-brain" returns approximatively 2000 files > (already divided by 2> due to replicate volume). So manually repairing each split-brain is > unfeasable. Before scripting some> procedure, I need to be sure that I will not harm further the gluster system.> Moreover, I noticed that the messages printed in the logs are all about > directories,> e.g.> [2015-03-12 10:06:53.423856] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-1: > Unable to self-heal contents of '/root' (possible split-brain). Please > delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 > 1 ] [ 1 0 ] ]> [2015-03-12 10:06:53.424005] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2: > Unable to self-heal contents of '/root' (possible split-brain). Please > delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 > 1 ] [ 1 0 ] ]> [2015-03-12 10:06:53.424110] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-md1-replicate-1: metadata self heal failed, on /root> [2015-03-12 10:06:53.424290] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-md1-replicate-2: metadata self heal failed, on /root> Getting the attributes of that directory on each brick gives me for the first> # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= > and for the second> # file: data/glusterfs/md1/brick1/root> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw= > trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= > so it seems that there are both rigorously identical. However, according to > your split -brain tutorial,> none of them has 0x000000000000000000000000. What 0sAAAAAAAAAAAAAAAA means> in fact ?> Should I change both attributes on each directory to > 0x000000000000000000000000 ?> Many thanks,> A.> On Wednesday 11 March 2015 08:02:56 Krutika Dhananjay wrote:> Hi,> Have you gone through > https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md > ?> If not, could you go through that once and try the steps given there? Do let > us know if something is not clear in the doc.> -Krutika> From: "Alessandro Ipe" <Alessandro.Ipe at meteo.be> > To: gluster-users at gluster.org > Sent: Wednesday, March 11, 2015 4:54:09 PM > Subject: Re: [Gluster-users] Input/output error when trying to access a file > on client> Well, it is even worse. Now when doing a "ls -R" on the volume results in a > lot of> [2015-03-11 11:18:31.957505] E > [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2: > Unable to self-heal contents of '/library' (possible split-brain). Please > delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 > 2 ] [ 1 0 ] ] > [2015-03-11 11:18:31.957692] E > [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] > 0-md1-replicate-2: metadata self heal failed, on /library> I am desperate...> A.> On Wednesday 11 March 2015 12:05:33 you wrote: > > Hi, > > > > > > When trying to access a file on a gluster client (through fuse), I get an > > "Input/output error" message. > > > > Getting the attributes for the file gives me for the first brick > > # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2 > > trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA > > trusted.afr.md1-client-3=0sAAABdAAAAAAAAAAA > > trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ=> > > > while for the second (replicate) brick > > # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2 > > trusted.afr.md1-client-2=0sAAABJAAAAAAAAAAA > > trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA > > trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ=> > > > It seems that I have a split-brain. How can I solve this issue by resetting > > the attributes, please ? > > > > > > Thanks, > > > > > > Alessandro. > > > > =================> > gluster volume info md1 > > > > Volume Name: md1 > > Type: Distributed-Replicate > > Volume ID: 6da4b915-1def-4df4-a41c-2f3300ebf16b > > Status: Started > > Number of Bricks: 3 x 2 = 6 > > Transport-type: tcp > > Bricks: > > Brick1: tsunami1:/data/glusterfs/md1/brick1 > > Brick2: tsunami2:/data/glusterfs/md1/brick1 > > Brick3: tsunami3:/data/glusterfs/md1/brick1 > > Brick4: tsunami4:/data/glusterfs/md1/brick1 > > Brick5: tsunami5:/data/glusterfs/md1/brick1 > > Brick6: tsunami6:/data/glusterfs/md1/brick1 > > Options Reconfigured: > > server.allow-insecure: on > > cluster.read-hash-mode: 2 > > features.quota: off > > performance.write-behind: on > > performance.write-behind-window-size: 4MB > > performance.flush-behind: off > > performance.io-thread-count: 64 > > performance.cache-size: 512MB > > nfs.disable: on > > cluster.lookup-unhashed: off> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users> --> Dr. Ir. Alessandro Ipe> Department of Observations Tel. +32 2 373 06 31> Remote Sensing from Space Fax. +32 2 374 67 88> Royal Meteorological Institute> Avenue Circulaire 3 Email:> B-1180 Brussels Belgium Alessandro.Ipe at meteo.be> Web: http://gerb.oma.be-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150312/5624c62c/attachment.html>
Alessandro Ipe
2015-Mar-12 11:59 UTC
[Gluster-users] Input/output error when trying to access a file on client
Hi, Sorry about that, I thought I was using the -e hex... I must have removed it at some point accidentally. Here they are 1. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-0=0x000000000000000000000000 trusted.afr.md1-client-1=0x000000000000000000000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff 2. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-0=0x000000000000000000000000 trusted.afr.md1-client-1=0x000000000000000000000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff 3. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-2=0x000000000000000000000000 trusted.afr.md1-client-3=0x000000000000000100000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x00000001000000000000000055555554 4. getfattr: Removing leading '/' from absolute path names # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-2=0x000000000000000100000000 trusted.afr.md1-client-3=0x000000000000000000000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x00000001000000000000000055555554 5. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-4=0x000000000000000000000000 trusted.afr.md1-client-5=0x000000000000000100000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9 6. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-4=0x000000000000000100000000 trusted.afr.md1-client-5=0x000000000000000000000000 trusted.gfid=0xdc398cbd2ab440ec9fed3d5937654f4b trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9 Thanks for your help, A. On Thursday 12 March 2015 07:51:40 Krutika Dhananjay wrote: Hi, Could you provide the xattrs in hex format? You can execute `getfattr -d -m . -e hex <path-to-the-directory/file-on-the-brick(s)>` -Krutika -------------------- *From: *"Alessandro Ipe" <Alessandro.Ipe at meteo.be> *To: *"Krutika Dhananjay" <kdhananj at redhat.com> *Cc: *gluster-users at gluster.org *Sent: *Thursday, March 12, 2015 5:15:08 PM *Subject: *Re: [Gluster-users] Input/output error when trying to access a file on client Hi, Actually, my gluster volume is distribute-replicate so I should provide the attributes on all the bricks. Here they are: 1. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw=trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= 2. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw=trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w= 3. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA trusted.afr.md1-client-3=0sAAAAAAAAAAEAAAAA trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw=trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA= 4. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-2=0sAAAAAAAAAAEAAAAA trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw=trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA= 5. # file: data/glusterfs/md1/brick1/root trusted.afr.md1-client-4=0sAAAAAAAAAAAAAAAA trusted.afr.md1-client-5=0sAAAAAAAAAAEAAAAA trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw=trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ= 6. # file: data/glusterfs/md1/brick1/root -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150312/7a66d743/attachment.html>