Gluster users - Apr 2014 - Need help in understanding volume heal-info behavior

Hi everyone,

I have trouble understanding the following behavior:

Suppose I have a replica 2 volume 'testvol' on two servers, server1 and 
server2, composed of server1:/bricks/testvol/brick and 
server2:/bricks/testvol/brick. Also, suppose it contains a good number 
of files.

Now, assume I remove one of the two bricks, as:

root at server1~# gluster volume remove-brick testvol replica 1 
server1:/bricks/testvol/brick

Now, I unmount and delete the logical volume supporting the brick and 
then recreate it (with a different size), and mount it the same way as 
it was mounted before (at /brick/testvol/). Then, I re-add it as:

root at server1~# gluster volume add-brick testvol replica 2 
server1:/bricks/testvol/brick

I observe that the brick on server1 does not contain any of the data 
that was in the volume.

root at server1~# ls /bricks/testvol/brick
root at server1~#

This is all right by me, since glusterfs needs some time to discover and 
sync files that are absent on the brick of server1. In fact, if I leave 
the setup undisturbed for 15 minutes to half an hour, I find that all 
data appears within the brick of server1, just as you would expect. 
Also, if I wish to speed up the process, I simply do a ls -Ra on the 
directory where the volume is mounted, and all files sync onto server1's 
brick. This is also very much as expected.

However, during the period where data on server1's brick is not 
available, if you query the heal info for the volume, gluster cli 
reports that 'Number of entries' is '0', and that too all of
'info',
'heal-failed', and 'split-brain'. This is what becomes a bit of
a
trouble for me. Fact is, we are attempting to automate the monitoring of 
our glusterfs volumes, and we depend upon heal info alone to decide 
whether data on server1 and server2 are in sync.

Could somebody, therefore, help me with the following questions?
a) Which files exactly show up in heal info?
b) What exactly should I look to monitor if we are to ascertain that 
data on our servers are in sync?

Thanks a lot for your responses!

Anirban

P.s. I am using glusterfs 3.4.2 over linux kernel version 2.6.34.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140428/d0e8acb4/attachment.html>

On 04/28/2014 01:30 AM, Chalcogen wrote:
> Hi everyone,
>
> I have trouble understanding the following behavior:
>
> Suppose I have a replica 2 volume 'testvol' on two servers, server1
> and server2, composed of server1:/bricks/testvol/brick and 
> server2:/bricks/testvol/brick. Also, suppose it contains a good number 
> of files.
>
> Now, assume I remove one of the two bricks, as:
>
> root at server1~# gluster volume remove-brick testvol replica 1 
> server1:/bricks/testvol/brick
>
> Now, I unmount and delete the logical volume supporting the brick and 
> then recreate it (with a different size), and mount it the same way as 
> it was mounted before (at /brick/testvol/). Then, I re-add it as:
>
> root at server1~# gluster volume add-brick testvol replica 2 
> server1:/bricks/testvol/brick
>
> I observe that the brick on server1 does not contain any of the data 
> that was in the volume.
>
> root at server1~# ls /bricks/testvol/brick
> root at server1~#
>
> This is all right by me, since glusterfs needs some time to discover 
> and sync files that are absent on the brick of server1. In fact, if I 
> leave the setup undisturbed for 15 minutes to half an hour, I find 
> that all data appears within the brick of server1, just as you would 
> expect. Also, if I wish to speed up the process, I simply do a ls -Ra 
> on the directory where the volume is mounted, and all files sync onto 
> server1's brick. This is also very much as expected.
>
> However, during the period where data on server1's brick is not 
> available, if you query the heal info for the volume, gluster cli 
> reports that 'Number of entries' is '0', and that too all
of 'info',
> 'heal-failed', and 'split-brain'. This is what becomes a
bit of a
> trouble for me. Fact is, we are attempting to automate the monitoring 
> of our glusterfs volumes, and we depend upon heal info alone to decide 
> whether data on server1 and server2 are in sync.
>
> Could somebody, therefore, help me with the following questions?
> a) Which files exactly show up in heal info?
The files which are healed either by the self-heal daemon or by the 
gluster heal commands.
> b) What exactly should I look to monitor if we are to ascertain that 
> data on our servers are in sync?
>
After adding a new replica brick, you need to run a full heal (gluster 
volume heal <vol-name> full). Then the results will show up in the heal 
info output.
> Thanks a lot for your responses!
>
> Anirban
>
> P.s. I am using glusterfs 3.4.2 over linux kernel version 2.6.34.
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140428/7e197fad/attachment.html>