Gluster users - Nov 2015 - Debugging georeplication failures

Hi,

I've got a geo-replicated gluster volume, with a few hundred thousand
images, which get generated on demand.

I started getting replication failures in the status detail view, but it's
not obvious to me where to find the actual errors or how to actually fix
them.

The docs seem to be secretive about this as well. It seems if I tear the
geo-replication down, and do a force create from scratch, it goes back in
sync again, but as the files get generated, it starts getting failures
again at some point.

Can someone provide me with information on how to check which files are
causing failures, and what are the actual failures? Or point me to the
relevant part in the docs?

Version 3.7.5-ubuntu1~trusty1

Related SO question:
http://stackoverflow.com/questions/33839056/gluster-geo-replication-debugging-failures

Thanks,

Audrius.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151123/ffc2c065/attachment.html>

On Tue, Nov 24, 2015 at 1:23 AM, Audrius Butkevicius
<audrius.butkevicius at gmail.com> wrote:
> Hi,
>
> I've got a geo-replicated gluster volume, with a few hundred thousand
> images, which get generated on demand.
>
> I started getting replication failures in the status detail view, but
it's
> not obvious to me where to find the actual errors or how to actually fix
> them.
What's in the log file? That would help in debugging further.
>
> The docs seem to be secretive about this as well. It seems if I tear the
> geo-replication down, and do a force create from scratch, it goes back in
> sync again, but as the files get generated, it starts getting failures
again
> at some point.
>
> Can someone provide me with information on how to check which files are
> causing failures, and what are the actual failures? Or point me to the
> relevant part in the docs?
Sure. Geo-Replication maintains couple of log files (on master and
slave under /var/log/glusterfs/geo-replication). Guessing that this
problem is something on the master, providing those would help.
>
> Version 3.7.5-ubuntu1~trusty1
>
> Related SO question:
>
http://stackoverflow.com/questions/33839056/gluster-geo-replication-debugging-failures
>
> Thanks,
>
> Audrius.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

On Tue, Nov 24, 2015 at 1:23 AM, Audrius Butkevicius
<audrius.butkevicius at gmail.com> wrote:
> Hi,
>
> I've got a geo-replicated gluster volume, with a few hundred thousand
> images, which get generated on demand.
>
> I started getting replication failures in the status detail view, but
it's
> not obvious to me where to find the actual errors or how to actually fix
> them.
Chris here[1] mentioned about a bug in rsync (thanks!). Could that be
the issue here?

Mind checking rsync version used?

[1]: http://www.gluster.org/pipermail/gluster-users/2015-November/024423.html
>
> The docs seem to be secretive about this as well. It seems if I tear the
> geo-replication down, and do a force create from scratch, it goes back in
> sync again, but as the files get generated, it starts getting failures
again
> at some point.
>
> Can someone provide me with information on how to check which files are
> causing failures, and what are the actual failures? Or point me to the
> relevant part in the docs?
>
> Version 3.7.5-ubuntu1~trusty1
>
> Related SO question:
>
http://stackoverflow.com/questions/33839056/gluster-geo-replication-debugging-failures
>
> Thanks,
>
> Audrius.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users