Gluster users - Aug 2014 - What to do with split-brain on / (root)

Hi,

We have a 2 server replica gluster, about 10TB (5 bricks) on each side. The
servers are fairly high-end and are connected via a private ethernet
connection, appearing to replicate with each other very quickly.

Running Ubuntu Precise (12.04) and GlusterFS 3.4.1 on the servers.

They're working quite heavily in production and downtime is difficult to
arrange.

We've had a lot of trouble with split-brain, where files/dirs on both sides
appear identical
(md5sum, datestamps, etc) but I've fixed most of them by erasing the
affected files
on one side and letting it re-sync.

However, we're left with split brain on '/' and several top level
directories.
I'm uncomfortable erasing that much of one side to solve the problem.

any advice?


- Colin
--
Colin Coghill
DevOps Engineer
Koordinates
colin.coghill at koordinates.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140805/299270c0/attachment.html>

On 08/05/2014 10:38 AM, Colin Coghill wrote:
>
> Hi,
>
> We have a 2 server replica gluster, about 10TB (5 bricks) on each 
> side. The servers are fairly high-end and are connected via a private 
> ethernet connection, appearing to replicate with each other very quickly.
>
> Running Ubuntu Precise (12.04) and GlusterFS 3.4.1 on the servers.
>
> They're working quite heavily in production and downtime is difficult 
> to arrange.
>
> We've had a lot of trouble with split-brain, where files/dirs on both 
> sides appear identical
> (md5sum, datestamps, etc) but I've fixed most of them by erasing the 
> affected files
> on one side and letting it re-sync.
>
> However, we're left with split brain on '/' and several top
level
> directories.
> I'm uncomfortable erasing that much of one side to solve the problem.
>
> any advice?
Hope this document helps:
https://github.com/gluster/glusterfs/blob/master/doc/split-brain.md

We are going to provide CLI for these issues in 3.6/3.7 versions hopefully.
Pranith
>
>
> - Colin
> -- 
> Colin Coghill
> DevOps Engineer
> Koordinates
> colin.coghill at koordinates.com <mailto:colin.coghill at
koordinates.com>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140806/11a2e7db/attachment.html>