Gluster users - Nov 2013 - Fencing FOPs on data-split-brained files

Hi,

Currenly in glusterfs, when there is a data splt-brain (only) on a file, 
we disallow the following operations from the mount-point by returning 
EIO to the application:
- Writes to the file (truncate, dd, echo, cp etc)
- Reads to the file (cat)
- Reading extended attributes (getfattr) [1]

However we do permit the following operations:
-creating hardlinks
-creating symlinks
-mv
-setattr
-chmod
-chown
--touch
-ls
-stat

While it makes sense to allow `ls` and `stat`, is it okay to  add checks 
in the FOPS to disallow the other operations? Allowing creation of links 
and changing file attributes only seems to complicate things before the 
admin can go to the backend bricks and resolve the splitbrain (by 
deleteing all but the healthy copy of the file including hardlinks). 
More so if the file is renamed before addressing the split-brain.
Please share your thoughs.

Thanks,
Ravi

[1] http://review.gluster.org/#/c/5988/

On 11/13/2013 06:01 AM, Ravishankar N wrote:
> Currenly in glusterfs, when there is a data splt-brain (only) on a
> file, we disallow the following operations from the mount-point by
> returning EIO to the application: - Writes to the file (truncate, dd,
> echo, cp etc) - Reads to the file (cat) - Reading extended attributes
> (getfattr) [1]
I see that the patch has already been cherry-picked, but it makes me
uneasy.  Setxattr etc. are inode operations, not data, so data
split-brain shouldn't matter.  Symlink is an entry operation according
to the code; seems to me like it should still be inode, but either way
it's not data so the same concern applies.  The problem seems to be that
afr_is_split_brain always checks for *both* data and metadata
split-brain, but in some cases we need an equivalent call that only
checks for one.
> However we do permit the following operations: -creating hardlinks
> -creating symlinks -mv -setattr -chmod -chown --touch -ls -stat
These are all inode/entry ops.  Because some setattr/stat fields do
interact with data operations I'd say they're the *last* ones we should
consider allowing when in data split-brain.  Some of the others might
complicate self-heal, so we could reasonably consider disallowing them
as well, but we should avoid that where feasible.
> While it makes sense to allow `ls` and `stat`, is it okay to  add
> checks in the FOPS to disallow the other operations? Allowing
> creation of links and changing file attributes only seems to
> complicate things before the admin can go to the backend bricks and
> resolve the splitbrain (by deleteing all but the healthy copy of the
> file including hardlinks). More so if the file is renamed before
> addressing the split-brain. Please share your thoughs.
I can't help but wonder whether a different kind of replication, perhaps
based on logging instead of transactions, might avoid some of this extra
complexity.  ;)

Ravi,
We should not mix up data and entry operation domains, if a file is in data
split brain that should not stop a user from rename/link/unlink operations
on the file.

Regarding your concern about complications while healing - we should change
our "manual fixing" instructions to:

- go to backend, access through gfid path or normal path
- rmxattr the afr changelogs
- truncate the file to 0 bytes (like "> filename")

Accessing the path through gfid and truncating to 0 bytes addresses your
concerns about hardlinks/renames.

Avati


On Wed, Nov 13, 2013 at 3:01 AM, Ravishankar N <ravishankar at
redhat.com>wrote:
> Hi,
>
> Currenly in glusterfs, when there is a data splt-brain (only) on a file,
> we disallow the following operations from the mount-point by returning EIO
> to the application:
> - Writes to the file (truncate, dd, echo, cp etc)
> - Reads to the file (cat)
> - Reading extended attributes (getfattr) [1]
>
> However we do permit the following operations:
> -creating hardlinks
> -creating symlinks
> -mv
> -setattr
> -chmod
> -chown
> --touch
> -ls
> -stat
>
> While it makes sense to allow `ls` and `stat`, is it okay to  add checks
> in the FOPS to disallow the other operations? Allowing creation of links
> and changing file attributes only seems to complicate things before the
> admin can go to the backend bricks and resolve the splitbrain (by deleteing
> all but the healthy copy of the file including hardlinks). More so if the
> file is renamed before addressing the split-brain.
> Please share your thoughs.
>
> Thanks,
> Ravi
>
> [1] http://review.gluster.org/#/c/5988/
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131115/6b766b21/attachment.html>