Found the BZ https://bugzilla.redhat.com/show_bug.cgi?id=960190 - so I
restarted one of the volumes and it seems to have restarted the all daemons
again.
Self heal started again, but I seem to have split-brain issues everywhere.
There's over 100 different entries on each node, what's the best way to
restore this now? Short of having to manually go through and delete 200+
files. It looks like a full split brain as the file sizes on the different
nodes are out of balance by about 100GB or so.
Any suggestions would be much appreciated!
Cheers.
On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau <andrew at andrewklau.com>
wrote:
> Hi,
>
> Right now, I have a 2x1 replica. Ever since I had to reinstall one of the
> gluster servers, there's been issues with split-brain. The self-heal
daemon
> doesn't seem to be running on either of the nodes.
>
> To reinstall the gluster server (the original brick data was intact but
> the OS had to be reinstalled)
> - Reinstalled gluster
> - Copied over the old uuid from backup
> - gluster peer probe
> - gluster volume sync $othernode all
> - mount -t glusterfs localhost:STORAGE /mnt
> - find /mnt -noleaf -print0 | xargs --null stat >/dev/null
> 2>/var/log/glusterfs/mnt-selfheal.log
>
> I let it resync and it was working fine, atleast so I thought. I just came
> back a few days later to see there's a miss match in the brick volumes.
One
> is 50GB ahead of the other.
>
> # gluster volume heal STORAGE info
> Status: self-heal-daemon is not running on
> 966456a1-b8a6-4ca8-9da7-d0eb96997cbe
>
> /var/log/gluster/glustershd.log doesn't seem to have any recent logs,
only
> those from when the two original gluster servers were running.
>
> # gluster volume status
>
> Self-heal Daemon on localhost N/A N N/A
>
> Any suggestions would be much appreciated!
>
> Cheers
> Andrew.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130925/a14f550c/attachment.html>