Daniel Berteaud
2017-Nov-17 10:11 UTC
[Gluster-users] ?==?utf-8?q? Help with reconnecting a faulty brick
Le Jeudi, Novembre 16, 2017 13:07 CET, Ravishankar N <ravishankar at redhat.com> a ?crit:> On 11/16/2017 12:54 PM, Daniel Berteaud wrote: > > Any way in this situation to check which file will be healed from > > which brick before reconnecting ? Using some getfattr tricks ? > Yes, there are afr xattrs that determine the heal direction for each > file. The good copy will have non-zero trusted.afr* xattrs that blame > the bad one and heal will happen from good to bad.? If both bricks have > attrs blaming the other, then the file is in split-brain.Thanks. So, say I have a file with this on the correct node # file: mnt/bricks/vmstore/prod/bilbao_sys.qcow2 security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.vmstore-client-0=0x00050f7e0000000200000000 trusted.afr.vmstore-client-1=0x000000000000000100000000 trusted.gfid=0xe86c24e5fc6b4fc6bf2b896f3cc8537d And this on the bad one # file: mnt/bricks/vmstore/prod/bilbao_sys.qcow2 security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.vmstore-client-0=0x000000000000000000000000 trusted.afr.vmstore-client-1=0x000000000000000000000000 trusted.gfid=0xe86c24e5fc6b4fc6bf2b896f3cc8537d I can guarantee Gluster will heal from the correct one to the bad. And in case of both having a non nul afr, I can manually (using setfattr) set the afr attribute to a null value before reconnecting the faulty brick, and it'll heal from the correct one. And for files which have been deleted/renamed/created on the correct node while the bad one was offline, how are those handled ? For example, I have /mnt/bricks/vmstore/prod/contis_sys.qcow2 ont btoh bricks. But, on the correct one, the file was deleted and recreated while the bad one was offline. So they haven't the same gfid now. How does gluster handle this ? Sorry for all those questions, I'm just a bit nervous :-) -- Daniel Berteaud FIREWALL-SERVICES SAS. Soci?t? de Services en Logiciels Libres Tel : 05 56 64 15 32 Visio: https://vroom.fws.fr/dani Web : http://www.firewall-services.com
On 11/17/2017 03:41 PM, Daniel Berteaud wrote:> Le Jeudi, Novembre 16, 2017 13:07 CET, Ravishankar N <ravishankar at redhat.com> a ?crit: > >> On 11/16/2017 12:54 PM, Daniel Berteaud wrote: >>> Any way in this situation to check which file will be healed from >>> which brick before reconnecting ? Using some getfattr tricks ? >> Yes, there are afr xattrs that determine the heal direction for each >> file. The good copy will have non-zero trusted.afr* xattrs that blame >> the bad one and heal will happen from good to bad.? If both bricks have >> attrs blaming the other, then the file is in split-brain. > Thanks. > > So, say I have a file with this on the correct node > # file: mnt/bricks/vmstore/prod/bilbao_sys.qcow2 > security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 > trusted.afr.vmstore-client-0=0x00050f7e0000000200000000 > trusted.afr.vmstore-client-1=0x000000000000000100000000 > trusted.gfid=0xe86c24e5fc6b4fc6bf2b896f3cc8537d > > And this on the bad one > > # file: mnt/bricks/vmstore/prod/bilbao_sys.qcow2 > security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 > trusted.afr.vmstore-client-0=0x000000000000000000000000 > trusted.afr.vmstore-client-1=0x000000000000000000000000 > trusted.gfid=0xe86c24e5fc6b4fc6bf2b896f3cc8537d > > I can guarantee Gluster will heal from the correct one to the bad. And in case of both having a non nul afr, I can manually (using setfattr) set the afr attribute to a null value before reconnecting the faulty brick, and it'll heal from the correct one.With the above xattrs, the correct node will be used as source. Ideally, good node must have trusted.afr.vmstore-client-1 as non-zero (IOWs it blames the second brick, i.e client-1) and the bad node must have all zeroes.> And for files which have been deleted/renamed/created on the correct node while the bad one was offline, how are those handled ? For example, I haveThe parent directory will have afr xattrs indicating good and back bricks. All gfids not present in good will be deleted from bad if present. All gfids present in good will be created on the bad if not present. https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-v1.md can give you an idea of how replication works in glusterfs-3.5 (which you are using) and earlier. -Ravi> > /mnt/bricks/vmstore/prod/contis_sys.qcow2 ont btoh bricks. But, on the correct one, the file was deleted and recreated while the bad one was offline. So they haven't the same gfid now. How does gluster handle this ? > > Sorry for all those questions, I'm just a bit nervous :-) >
Daniel Berteaud
2017-Nov-20 08:23 UTC
[Gluster-users] Help with reconnecting a faulty brick
Le 17/11/2017 ? 13:10, Ravishankar N a ?crit?:> The parent directory will have afr xattrs indicating good and back > bricks. All gfids not present in good will be deleted from bad if > present. All gfids present in good will be created on the bad if not > present. > https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-v1.md > can give you an idea of how replication works in glusterfs-3.5 (which > you are using) and earlier. > -Ravi >Thanks for your help, the doc you've pointed, and the time you took to explain all this. I was able to reconnect the faulty brick without a single issue :-) Cheers, Daniel -- Logo FWS *Daniel Berteaud* FIREWALL-SERVICES SAS. Soci?t? de Services en Logiciels Libres Tel : 05 56 64 15 32 <tel:0556641532> Matrix: @dani:fws.fr /www.firewall-services.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171120/1adff510/attachment.html>