Kotresh Hiremath Ravishankar
2017-Mar-30 09:49 UTC
[Gluster-users] Geo-Replication not detecting changes
Hi Jeremiah, I believe the bug ID is #1437244 and not #1327244.>From the geo-rep logs, the master volume is failed with "Transport Endpoint Not Connected"... [2017-03-30 07:40:57.150348] E [resource(/gv0/foo):234:errlog] Popen: command "/usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/foo/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoo.%2Fgv0%2Ffoo.gluster.log --volfile-server=localhost --volfile-id=foo --client-pid=-1 /tmp/gsyncd-aux-mount-K1j3ZD" returned with 107 .. Could you try flushing iptables on both master and slave nodes and check again? #iptables -F Thanks and Regards, Kotresh H R ----- Original Message -----> From: "Jeremiah Rothschild" <jeremiah at franz.com> > To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com> > Cc: gluster-users at gluster.org > Sent: Thursday, March 30, 2017 1:16:03 PM > Subject: Re: [Gluster-users] Geo-Replication not detecting changes > > On Thu, Mar 30, 2017 at 12:51:23AM -0400, Kotresh Hiremath Ravishankar wrote: > > Hi Jeremiah, > > Hi Kotresh! Thanks for the follow-up! > > > That's really strange. Please enable DEBUG logs for geo-replication as > > below and send > > us the logs under "/var/log/glusterfs/geo-replication/<mastervol>/*.log" > > from master node > > > > gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config log-level > > DEBUG > > Ok. > > I started from scratch & enabled debug level logging. The logs have been > attached to Bugzilla #1327244. > > > Geo-rep has two ways to detect changes. > > > > 1. changelog (Changelog Crawl) > > 2. xsync (Hybrid Crawl): > > This is good for initial sync. It has the limitation of not > > detecting unlinks and renames. > > So the slave would end up having unlinked files and renamed src file if > > it is used after initial sync. > > FYI I did try changing the changelog_detector to xsync but it made no > difference. Note that I also detailed this in the "Additional Info" section > of the Bugzilla bug. > > > Thanks and Regards, > > Thanks again! > > j > > > Kotresh H R > > > > ----- Original Message ----- > > > From: "Jeremiah Rothschild" <jeremiah at franz.com> > > > To: gluster-users at gluster.org > > > Sent: Wednesday, March 29, 2017 12:39:11 AM > > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes > > > > > > Following up on my own thread... > > > > > > I have spent hours and hours setting up, re-setting up, screwing with > > > undocumented variables, upgrading from LTS to non-LTS, etc etc. > > > > > > Nothing seems to give. > > > > > > This is very much an out-of-the-box setup and core functionality just > > > isn't > > > working. > > > > > > Can anyone throw me a bone here? Please? Do I file a bug for such an > > > open-ended issue? Is everyone assuming I've just screwed a step up? I > > > must > > > say the documentation is pretty clear & simple. Do you want more logs? > > > > > > If this is going to be a dead end then so be it but I at least need to > > > make > > > sure I've tried my hardest to get a working deployment. > > > > > > Thanks for your time and understanding! > > > > > > j > > > > > > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote: > > > > Hey all, > > > > > > > > I have a vanilla geo-replication setup running. It is comprised of two > > > > servers, both CentOS 7 and GlusterFS 3.8.10: > > > > > > > > * server1: Local server. Master volume named "foo". > > > > * server2: Remote server. Slave volume named "foo". > > > > > > > > Everything went fine including the initial sync. However, no new > > > > changes > > > > are > > > > being seen or synced. > > > > > > > > Geo-rep status looks clean: > > > > > > > > # gluster volume geo-replication foo server2.franz.com::foo status > > > > MASTER NODE: server1.x.com > > > > MASTER VOL: foo > > > > MASTER BRICK: /gv0/foo > > > > SLAVE USER: root > > > > SLAVE NODE: server2.x.com::foo > > > > STATUS: Active > > > > CRAWL STATUS: Changelog Crawl > > > > LAST_SYNCED: 2017-03-23 10:12:57 > > > > > > > > In the geo-rep master log, I see these being triggered: > > > > > > > > # tail -n3 > > > > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log > > > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap] > > > > _GMaster: > > > > 20 > > > > crawls, 0 turns > > > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap] > > > > _GMaster: > > > > 20 > > > > crawls, 0 turns > > > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap] > > > > _GMaster: > > > > 20 > > > > crawls, 0 turns > > > > > > > > I don't see any errors in any of the other logs. > > > > > > > > Not sure what else to poke at here. What are the possible values for > > > > the > > > > "change_detector" config variable? Would it be worthwhile to test with > > > > a > > > > method other than "changelog"? Other thoughts/ideas? > > > > > > > > Thanks in advance! > > > > > > > > j > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > >
Jeremiah Rothschild
2017-Mar-30 11:58 UTC
[Gluster-users] Geo-Replication not detecting changes
On Thu, Mar 30, 2017 at 05:49:32AM -0400, Kotresh Hiremath Ravishankar wrote:> Hi Jeremiah, > > I believe the bug ID is #1437244 and not #1327244.Oops! You are correct.> >From the geo-rep logs, the master volume is failed with "Transport Endpoint Not Connected" > ... > [2017-03-30 07:40:57.150348] E [resource(/gv0/foo):234:errlog] Popen: command "/usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/foo/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoo.%2Fgv0%2Ffoo.gluster.log --volfile-server=localhost --volfile-id=foo --client-pid=-1 /tmp/gsyncd-aux-mount-K1j3ZD" returned with 107 > .. > > Could you try flushing iptables on both master and slave nodes and check again? > #iptables -FDone. I then restarted glusterd on both servers and waited for a sync to happen but there is no change. Also, I believe that networking was verified as OK, because the initial sync worked? Is that not true? Well, at anyrate, here you can see that both servers can talk on 49152/tcp: [root at ill ~]# telnet aws 49152 Trying 54.165.144.9... Connected to aws. Escape character is '^]'. and [root at aws jeremiah]# telnet ill 49152 Trying 67.207.112.66... Connected to ill. Escape character is '^]'.> Thanks and Regards, > Kotresh H R > > ----- Original Message ----- > > From: "Jeremiah Rothschild" <jeremiah at franz.com> > > To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com> > > Cc: gluster-users at gluster.org > > Sent: Thursday, March 30, 2017 1:16:03 PM > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes > > > > On Thu, Mar 30, 2017 at 12:51:23AM -0400, Kotresh Hiremath Ravishankar wrote: > > > Hi Jeremiah, > > > > Hi Kotresh! Thanks for the follow-up! > > > > > That's really strange. Please enable DEBUG logs for geo-replication as > > > below and send > > > us the logs under "/var/log/glusterfs/geo-replication/<mastervol>/*.log" > > > from master node > > > > > > gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config log-level > > > DEBUG > > > > Ok. > > > > I started from scratch & enabled debug level logging. The logs have been > > attached to Bugzilla #1327244. > > > > > Geo-rep has two ways to detect changes. > > > > > > 1. changelog (Changelog Crawl) > > > 2. xsync (Hybrid Crawl): > > > This is good for initial sync. It has the limitation of not > > > detecting unlinks and renames. > > > So the slave would end up having unlinked files and renamed src file if > > > it is used after initial sync. > > > > FYI I did try changing the changelog_detector to xsync but it made no > > difference. Note that I also detailed this in the "Additional Info" section > > of the Bugzilla bug. > > > > > Thanks and Regards, > > > > Thanks again! > > > > j > > > > > Kotresh H R > > > > > > ----- Original Message ----- > > > > From: "Jeremiah Rothschild" <jeremiah at franz.com> > > > > To: gluster-users at gluster.org > > > > Sent: Wednesday, March 29, 2017 12:39:11 AM > > > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes > > > > > > > > Following up on my own thread... > > > > > > > > I have spent hours and hours setting up, re-setting up, screwing with > > > > undocumented variables, upgrading from LTS to non-LTS, etc etc. > > > > > > > > Nothing seems to give. > > > > > > > > This is very much an out-of-the-box setup and core functionality just > > > > isn't > > > > working. > > > > > > > > Can anyone throw me a bone here? Please? Do I file a bug for such an > > > > open-ended issue? Is everyone assuming I've just screwed a step up? I > > > > must > > > > say the documentation is pretty clear & simple. Do you want more logs? > > > > > > > > If this is going to be a dead end then so be it but I at least need to > > > > make > > > > sure I've tried my hardest to get a working deployment. > > > > > > > > Thanks for your time and understanding! > > > > > > > > j > > > > > > > > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote: > > > > > Hey all, > > > > > > > > > > I have a vanilla geo-replication setup running. It is comprised of two > > > > > servers, both CentOS 7 and GlusterFS 3.8.10: > > > > > > > > > > * server1: Local server. Master volume named "foo". > > > > > * server2: Remote server. Slave volume named "foo". > > > > > > > > > > Everything went fine including the initial sync. However, no new > > > > > changes > > > > > are > > > > > being seen or synced. > > > > > > > > > > Geo-rep status looks clean: > > > > > > > > > > # gluster volume geo-replication foo server2.franz.com::foo status > > > > > MASTER NODE: server1.x.com > > > > > MASTER VOL: foo > > > > > MASTER BRICK: /gv0/foo > > > > > SLAVE USER: root > > > > > SLAVE NODE: server2.x.com::foo > > > > > STATUS: Active > > > > > CRAWL STATUS: Changelog Crawl > > > > > LAST_SYNCED: 2017-03-23 10:12:57 > > > > > > > > > > In the geo-rep master log, I see these being triggered: > > > > > > > > > > # tail -n3 > > > > > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log > > > > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap] > > > > > _GMaster: > > > > > 20 > > > > > crawls, 0 turns > > > > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap] > > > > > _GMaster: > > > > > 20 > > > > > crawls, 0 turns > > > > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap] > > > > > _GMaster: > > > > > 20 > > > > > crawls, 0 turns > > > > > > > > > > I don't see any errors in any of the other logs. > > > > > > > > > > Not sure what else to poke at here. What are the possible values for > > > > > the > > > > > "change_detector" config variable? Would it be worthwhile to test with > > > > > a > > > > > method other than "changelog"? Other thoughts/ideas? > > > > > > > > > > Thanks in advance! > > > > > > > > > > j > > > > > _______________________________________________ > > > > > Gluster-users mailing list > > > > > Gluster-users at gluster.org > > > > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > >