Gluster users - Feb 2015 - ... i was able to produce a split brain...

On 1/31/2015 12:47 AM, Pranith Kumar Karampuri wrote:
>
> On 01/30/2015 06:28 PM, Jeff Darcy wrote:
>>> Pranith and I had a discussion regarding this issue and here is
what we have
>>> in our mind right now.
>>>
>>> We plan to provide the user commands to execute from mount so that
he can
>>> access the files in split-brain. This way he can choose which copy
is to be
>>> used as source. The user will have to perform a set of getfattrs
and
>>> setfattrs (on virtual xattrs) to decide which child to choose as
source and
>>> inform AFR with his decision.
>>>
>>> A) To know the split-brain status :
>>> getfattr -n trusted.afr.split-brain-status <path-to-file>
>>>
>>> This will provide user with the following details -
>>> 1) Whether the file is in metadata split-brain
>>> 2) Whether the file is in data split-brain
>>>
>>> It will also list the name of afr-children to choose from.
Something like :
>>> Option0: client-0
>>> Option1: client-1
>>>
>>> We also tell the user what the user could do to view metadata/data
info;
>>> like
>>> stat to get metadata etc.
>>>
>>> B) Now the user has to choose one of the options
(client-x/client-y..) to
>>> inspect the files.
>>> e.g., setfattr -n trusted.afr.split-brain-choice -v client-0
<path-to-file>
>>> We save the read-child info in inode-ctx in order to provide the
user access
>>> to the file in split-brain from that child. Once the user inspects
the file,
>>> he proceeds to do the same from the other child of replica pair and
makes an
>>> informed decision.
>>>
>>> C) Once the above steps are done, AFR is to be informed with the
final
>>> choice
>>> for source. This is achieved by -
>>> (say the fresh copy is in client-0)
>>> e.g., setfattr -n trusted.afr.split-brain-heal-finalize -v client-0
>>> <path-to-file>
>>> This child will be chosen as source and split-brain resolution will
be done.
May I suggest another possible way to get around the difficulty in 
determining which of the files is the one to keep?

What if each of the files were to be renamed by appending the name of the 
brick-host that it lives on?
For example, in a replica 2 system:
brick-1: data1
host-1: host1
brick-2: date1
host-2: host2
file name: hair-pulling.txt

after running script/command to resolve split-brain, file system would have 
two files:
hair-pulling.txt__host-1_data1
hair-pulling.txt__host-2_data1

the user would then delete the unwanted file and rename the wanted file back 
to hair-pulling.txt.

The only problem would come with a very large file with a large number of 
replicas (like the replica 5 system I am working with). You might run out of 
space for all the copies.

Otherwise, this seems (to me) to present a user-friendly way to do this.  If 
the user has doubts (and disk space), user can choose to keep the rejected 
file around for a while, "just in case" it happens to have something
useful
in it that is missing from the "accepted" file.

****************************************************************
That brought another thought to mind (have not had reason to try it):
How does gluster cope if you go behind its back and rename a
"rejected"
file?  For instance, in my example above, what if I go directly on the brick 
and rename the host-2 copy of the file to hair-pulling.txt-dud?  The ideal 
scenario would seem to be that if user does a heal it would treat the copy as 
new file, see no dupe for hair-pulling.txt, and create a new dupe on host-2.  
Since hair-pulling.txt-dud is also a new file, a dupe would be created on 
host-1.  User could then access files, verify correctness, and then delete 
hair-pulling.txt-dud.

*****************************************************************
A not-officially-sanctioned way that I dealt with a split-brain a few 
versions back:
1. decided I wanted to keep file on host-2
2. log onto host-2
3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud
4. rm /brick/data1/hair-pulling.txt
5. follow some Joe Julian blog stuff to delete the "invisible fork" of
file
6. gluster volume heal data1 all
I believe that this did work for me at that time.  I have not had to do it on 
a recent gluster version.

Ted Miller
Elkhart, IN

>
> ****************************************************************
> That brought another thought to mind (have not had reason to try it):
> How does gluster cope if you go behind its back and rename a 
> "rejected" file?  For instance, in my example above, what if I go
> directly on the brick and rename the host-2 copy of the file to 
> hair-pulling.txt-dud?  The ideal scenario would seem to be that if 
> user does a heal it would treat the copy as new file, see no dupe for 
> hair-pulling.txt, and create a new dupe on host-2.  Since 
> hair-pulling.txt-dud is also a new file, a dupe would be created on 
> host-1.  User could then access files, verify correctness, and then 
> delete hair-pulling.txt-dud.
>
> *****************************************************************
> A not-officially-sanctioned way that I dealt with a split-brain a few 
> versions back:
> 1. decided I wanted to keep file on host-2
> 2. log onto host-2
> 3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud
> 4. rm /brick/data1/hair-pulling.txt
> 5. follow some Joe Julian blog stuff to delete the "invisible
fork" of
> file
> 6. gluster volume heal data1 all
This should cause you to have two files with the same gfid. This will 
create the hardlink in .glusterfs again, and the heal will then 
re-create the .txt file also with that same gfid. Since both files will 
have the same gfid (stored in extended attributes) and be hard linked to 
the same file under .glusterfs you should then end up with both files 
being split-brain.

On 02/03/2015 10:42 PM, Ted Miller wrote:
>
> On 1/31/2015 12:47 AM, Pranith Kumar Karampuri wrote:
>>
>> On 01/30/2015 06:28 PM, Jeff Darcy wrote:
>>>> Pranith and I had a discussion regarding this issue and here is
>>>> what we have
>>>> in our mind right now.
>>>>
>>>> We plan to provide the user commands to execute from mount so
that
>>>> he can
>>>> access the files in split-brain. This way he can choose which
copy
>>>> is to be
>>>> used as source. The user will have to perform a set of
getfattrs and
>>>> setfattrs (on virtual xattrs) to decide which child to choose
as
>>>> source and
>>>> inform AFR with his decision.
>>>>
>>>> A) To know the split-brain status :
>>>> getfattr -n trusted.afr.split-brain-status <path-to-file>
>>>>
>>>> This will provide user with the following details -
>>>> 1) Whether the file is in metadata split-brain
>>>> 2) Whether the file is in data split-brain
>>>>
>>>> It will also list the name of afr-children to choose from. 
>>>> Something like :
>>>> Option0: client-0
>>>> Option1: client-1
>>>>
>>>> We also tell the user what the user could do to view
metadata/data
>>>> info; like
>>>> stat to get metadata etc.
>>>>
>>>> B) Now the user has to choose one of the options 
>>>> (client-x/client-y..) to
>>>> inspect the files.
>>>> e.g., setfattr -n trusted.afr.split-brain-choice -v client-0 
>>>> <path-to-file>
>>>> We save the read-child info in inode-ctx in order to provide
the
>>>> user access
>>>> to the file in split-brain from that child. Once the user
inspects
>>>> the file,
>>>> he proceeds to do the same from the other child of replica pair
and
>>>> makes an
>>>> informed decision.
>>>>
>>>> C) Once the above steps are done, AFR is to be informed with
the
>>>> final choice
>>>> for source. This is achieved by -
>>>> (say the fresh copy is in client-0)
>>>> e.g., setfattr -n trusted.afr.split-brain-heal-finalize -v
client-0
>>>> <path-to-file>
>>>> This child will be chosen as source and split-brain resolution
will
>>>> be done.
> May I suggest another possible way to get around the difficulty in 
> determining which of the files is the one to keep?
>
> What if each of the files were to be renamed by appending the name of 
> the brick-host that it lives on?
> For example, in a replica 2 system:
> brick-1: data1
> host-1: host1
> brick-2: date1
> host-2: host2
> file name: hair-pulling.txt
>
> after running script/command to resolve split-brain, file system would 
> have two files:
> hair-pulling.txt__host-1_data1
> hair-pulling.txt__host-2_data1
This doesn't seem so bad either. I will need to give it more thought to 
see if there are any problems.
>
> the user would then delete the unwanted file and rename the wanted 
> file back to hair-pulling.txt.
>
> The only problem would come with a very large file with a large number 
> of replicas (like the replica 5 system I am working with). You might 
> run out of space for all the copies.
>
> Otherwise, this seems (to me) to present a user-friendly way to do 
> this.  If the user has doubts (and disk space), user can choose to 
> keep the rejected file around for a while, "just in case" it
happens
> to have something useful in it that is missing from the
"accepted" file.
>
> ****************************************************************
> That brought another thought to mind (have not had reason to try it):
> How does gluster cope if you go behind its back and rename a 
> "rejected" file?  For instance, in my example above, what if I go
> directly on the brick and rename the host-2 copy of the file to 
> hair-pulling.txt-dud?  The ideal scenario would seem to be that if 
> user does a heal it would treat the copy as new file, see no dupe for 
> hair-pulling.txt, and create a new dupe on host-2.  Since 
> hair-pulling.txt-dud is also a new file, a dupe would be created on 
> host-1.  User could then access files, verify correctness, and then 
> delete hair-pulling.txt-dud.
>
> *****************************************************************
This one won't work because of the reason Joe gave about gfid-hardlinks.
> A not-officially-sanctioned way that I dealt with a split-brain a few 
> versions back:
> 1. decided I wanted to keep file on host-2
> 2. log onto host-2
> 3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud
> 4. rm /brick/data1/hair-pulling.txt
> 5. follow some Joe Julian blog stuff to delete the "invisible
fork" of
> file
> 6. gluster volume heal data1 all
> I believe that this did work for me at that time.  I have not had to 
> do it on a recent gluster version.
This would work. You can check the document written by Ravi for this in 
the official tree: 
https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md

Pranith
>
> Ted Miller
> Elkhart, IN
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users