Jeremiah Rothschild
2017-Mar-28 19:09 UTC
[Gluster-users] Geo-Replication not detecting changes
Following up on my own thread... I have spent hours and hours setting up, re-setting up, screwing with undocumented variables, upgrading from LTS to non-LTS, etc etc. Nothing seems to give. This is very much an out-of-the-box setup and core functionality just isn't working. Can anyone throw me a bone here? Please? Do I file a bug for such an open-ended issue? Is everyone assuming I've just screwed a step up? I must say the documentation is pretty clear & simple. Do you want more logs? If this is going to be a dead end then so be it but I at least need to make sure I've tried my hardest to get a working deployment. Thanks for your time and understanding! j On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote:> Hey all, > > I have a vanilla geo-replication setup running. It is comprised of two > servers, both CentOS 7 and GlusterFS 3.8.10: > > * server1: Local server. Master volume named "foo". > * server2: Remote server. Slave volume named "foo". > > Everything went fine including the initial sync. However, no new changes are > being seen or synced. > > Geo-rep status looks clean: > > # gluster volume geo-replication foo server2.franz.com::foo status > MASTER NODE: server1.x.com > MASTER VOL: foo > MASTER BRICK: /gv0/foo > SLAVE USER: root > SLAVE NODE: server2.x.com::foo > STATUS: Active > CRAWL STATUS: Changelog Crawl > LAST_SYNCED: 2017-03-23 10:12:57 > > In the geo-rep master log, I see these being triggered: > > # tail -n3 foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap] _GMaster: 20 > crawls, 0 turns > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap] _GMaster: 20 > crawls, 0 turns > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap] _GMaster: 20 > crawls, 0 turns > > I don't see any errors in any of the other logs. > > Not sure what else to poke at here. What are the possible values for the > "change_detector" config variable? Would it be worthwhile to test with a > method other than "changelog"? Other thoughts/ideas? > > Thanks in advance! > > j > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
On Wed, 29 Mar 2017 at 12:39 AM, Jeremiah Rothschild <jeremiah at franz.com> wrote:> Following up on my own thread... > > I have spent hours and hours setting up, re-setting up, screwing with > undocumented variables, upgrading from LTS to non-LTS, etc etc. > > Nothing seems to give. > > This is very much an out-of-the-box setup and core functionality just isn't > working. > > Can anyone throw me a bone here? Please? Do I file a bug for such an > open-ended issue? Is everyone assuming I've just screwed a step up? I must > say the documentation is pretty clear & simple. Do you want more logs?Response below, meantime a bug would be a good option. You can use whichever is easier for you, either bugzilla or GitHub issues at Gluster/glusterfs repo.> > If this is going to be a dead end then so be it but I at least need to make > sure I've tried my hardest to get a working deployment. > > Thanks for your time and understanding! > > jHi J, Will get back to you about this in another day. Some of the developers involved with geo-rep were travelling or on PTO in last 10days. Once I am online from work, will ping you, let's debug this together. Thanks, Amar> > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote: > > Hey all, > > > > I have a vanilla geo-replication setup running. It is comprised of two > > servers, both CentOS 7 and GlusterFS 3.8.10: > > > > * server1: Local server. Master volume named "foo". > > * server2: Remote server. Slave volume named "foo". > > > > Everything went fine including the initial sync. However, no new changes > are > > being seen or synced. > > > > Geo-rep status looks clean: > > > > # gluster volume geo-replication foo server2.franz.com::foo status > > MASTER NODE: server1.x.com > > MASTER VOL: foo > > MASTER BRICK: /gv0/foo > > SLAVE USER: root > > SLAVE NODE: server2.x.com::foo > > STATUS: Active > > CRAWL STATUS: Changelog Crawl > > LAST_SYNCED: 2017-03-23 10:12:57 > > > > In the geo-rep master log, I see these being triggered: > > > > # tail -n3 > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap] > _GMaster: 20 > > crawls, 0 turns > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap] > _GMaster: 20 > > crawls, 0 turns > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap] > _GMaster: 20 > > crawls, 0 turns > > > > I don't see any errors in any of the other logs. > > > > Not sure what else to poke at here. What are the possible values for the > > "change_detector" config variable? Would it be worthwhile to test with a > > method other than "changelog"? Other thoughts/ideas? > > > > Thanks in advance! > > > > j > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170329/dba5f4fd/attachment.html>
Kotresh Hiremath Ravishankar
2017-Mar-30 04:51 UTC
[Gluster-users] Geo-Replication not detecting changes
Hi Jeremiah,
That's really strange. Please enable DEBUG logs for geo-replication as below
and send
us the logs under
"/var/log/glusterfs/geo-replication/<mastervol>/*.log" from
master node
gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config
log-level DEBUG
Geo-rep has two ways to detect changes.
1. changelog (Changelog Crawl)
2. xsync (Hybrid Crawl):
This is good for initial sync. It has the limitation of not detecting
unlinks and renames.
So the slave would end up having unlinked files and renamed src file if it is
used after initial sync.
Thanks and Regards,
Kotresh H R
----- Original Message -----> From: "Jeremiah Rothschild" <jeremiah at franz.com>
> To: gluster-users at gluster.org
> Sent: Wednesday, March 29, 2017 12:39:11 AM
> Subject: Re: [Gluster-users] Geo-Replication not detecting changes
>
> Following up on my own thread...
>
> I have spent hours and hours setting up, re-setting up, screwing with
> undocumented variables, upgrading from LTS to non-LTS, etc etc.
>
> Nothing seems to give.
>
> This is very much an out-of-the-box setup and core functionality just
isn't
> working.
>
> Can anyone throw me a bone here? Please? Do I file a bug for such an
> open-ended issue? Is everyone assuming I've just screwed a step up? I
must
> say the documentation is pretty clear & simple. Do you want more logs?
>
> If this is going to be a dead end then so be it but I at least need to make
> sure I've tried my hardest to get a working deployment.
>
> Thanks for your time and understanding!
>
> j
>
> On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote:
> > Hey all,
> >
> > I have a vanilla geo-replication setup running. It is comprised of two
> > servers, both CentOS 7 and GlusterFS 3.8.10:
> >
> > * server1: Local server. Master volume named "foo".
> > * server2: Remote server. Slave volume named "foo".
> >
> > Everything went fine including the initial sync. However, no new
changes
> > are
> > being seen or synced.
> >
> > Geo-rep status looks clean:
> >
> > # gluster volume geo-replication foo server2.franz.com::foo status
> > MASTER NODE: server1.x.com
> > MASTER VOL: foo
> > MASTER BRICK: /gv0/foo
> > SLAVE USER: root
> > SLAVE NODE: server2.x.com::foo
> > STATUS: Active
> > CRAWL STATUS: Changelog Crawl
> > LAST_SYNCED: 2017-03-23 10:12:57
> >
> > In the geo-rep master log, I see these being triggered:
> >
> > # tail -n3
> > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log
> > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap]
_GMaster:
> > 20
> > crawls, 0 turns
> > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap]
_GMaster:
> > 20
> > crawls, 0 turns
> > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap]
_GMaster:
> > 20
> > crawls, 0 turns
> >
> > I don't see any errors in any of the other logs.
> >
> > Not sure what else to poke at here. What are the possible values for
the
> > "change_detector" config variable? Would it be worthwhile to
test with a
> > method other than "changelog"? Other thoughts/ideas?
> >
> > Thanks in advance!
> >
> > j
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>