Gluster users - Jun 2010 - health monitoring of replicated volume

Hello,

Is there any reasonable way to monitor the health of replicated volume 
and sync it, if out of sync ?

Regards,

I am curious about this - anyone?

-Jenn





On Jun 9, 2010, at 8:00 AM, Deyan Chepishev wrote:
> Hello,
> 
> Is there any reasonable way to monitor the health of replicated volume and
sync it, if out of sync ?
> 
> Regards,
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Hi Deyan,

sync can be triggered using ls -lR  (time to complete depends upon size 
of volume)

Could you expand on what all you would would like to monitor as part of 
health monitoring?

Regards,

Kamal

Deyan Chepishev wrote:
> Hello,
> 
> Is there any reasonable way to monitor the health of replicated volume 
> and sync it, if out of sync ?
> 
> Regards,
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

We plan to add a method to check if a replicated volume is out of sync. We also
plan to add a way to find out which copy is good and which out of sync - we
intend to do this for both volume/subvolume and also at the file level. The file
level is helpful to monitor vm image file copies and their currency.

There is however no planned date as yet by when this would be implemented. Will
post a note when it's done.

Let me know if you have any questions.

Regards,
Tejas.

----- Original Message -----
From: "Jenn Fountain" <jfountai at comcast.net>
To: "Gluster General Discussion List" <gluster-users at
gluster.org>
Sent: Friday, June 11, 2010 6:50:37 PM
Subject: Re: [Gluster-users] health monitoring of replicated volume

I am curious about this - anyone?

-Jenn





On Jun 9, 2010, at 8:00 AM, Deyan Chepishev wrote:
> Hello,
> 
> Is there any reasonable way to monitor the health of replicated volume and
sync it, if out of sync ?
> 
> Regards,
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Hi Deyan,

I understand your concerns. There are two things that are in the plan - 

1) A way to know a file or volume/sub-volume is out of sync with the replica.
2) Keeping track of what data changed, so that an immediate sync ( that is
triggered by ls -lR today ) can be made much faster and it does not have to
crawl through the whole data.

I hope those two things will take care of the concerns you have. Till then, you
will need to trigger off the self-heal with the current mechanism by monitoring
the nodes/network, logs etc.

We don't have exact dates as to when this feature will be implemented. Will
notify the community when we do.

It would be great to know more about your setup, how you are using Gluster and
your general experience with the product.

Regards,
Tejas Bhise.


----- Original Message -----
From: "Deyan Chepishev" <dchepishev at superhosting.bg>
To: "Gluster General Discussion List" <gluster-users at
gluster.org>
Sent: Saturday, June 12, 2010 9:54:47 PM
Subject: Re: [Gluster-users] health monitoring of replicated volume

Hello,

My main idea is that currently no one can tell you if the data is 
consistent on all nodes, for example very simple case:

2 replicated nodes, node1 and node2

I reboot node1 and during this time, someone writes a file which will be 
written on node2. At this time you have out of sync replication.

When node1 comes back up in a few minutes, there is no way to know if 
this node is with consistent data or not. Some people suggest to run ls 
-lR but you have to agree that this is not applicable if you have huge 
file system tree with millions of files.

So if I dont access the file which was written on node2 but not synced 
on node1 it will never be replicated and if something goes wrong with 
node2 I will loose data.

My question is, is there any algorithm, like wit RAID arrays, which can 
tell you if data/brick should be resynced.

as far as I understand currently my only option is to use ls -lR which 
would force resync and I should execute it on regular basis.

Regards,
Deyan.


Kamal K. Varma wrote:
> Hi Deyan,
>
> sync can be triggered using ls -lR  (time to complete depends upon 
> size of volume)
>
> Could you expand on what all you would would like to monitor as part 
> of health monitoring?
>
> Regards,
>
> Kamal
>
> Deyan Chepishev wrote:
>> Hello,
>>
>> Is there any reasonable way to monitor the health of replicated 
>> volume and sync it, if out of sync ?
>>
>> Regards,
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users