Gluster users - Mar 2014 - Check Synchronization state before/after host upgrade

Hi everyone!


We are running a couple of Gluster storage servers serving mainly VM
images. We are wondering already for quite some time how to handle host OS
upgrades efficiently and safely without risking losing data on the gluster
volumes. So, it would be great if you could give me some recommendations,
how-tos, or just your opinion on how to handle host OS upgrades best.


The problem: Host OS upgrades often require restating the servers. We run
Gluster 3.4.0 and all volumes use 2x replication. So we typically restart
one server at a time, wait for re-synchronization and continue. However, we
don't know of any means to reliably identify that Gluster has fully synched
the bricks. In fact, the "volume heal info" command on our servers not
only
shows files which are out of sync, but also files which are currently being
modified - even under normal, i.e. replicated, operation. Since VM images,
log files, databases, etc. are constantly being modified, our volumes seems
to be out-of-sync all the time. Hence, after each restart, we manually
trigger re-synchronization and wait for a day before we restart the next
server, hoping that even the VM images under heavy modification have been
synched in the meantime. However, we'd like to script the upgrade to make
it faster, more automated and mostly more reliable.


My questions: What is your recommended way to reliably handle this? Is the
behavior of the "volume heal info" command supposed to be like this?


Thank you and best regards,

Georg
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140303/3c64b03c/attachment.html>