Gluster users - Jul 2019 - Geo-replication does not send filesystem changes

Hi everyone,

I have a problem with native geo-replication setup. It successfully starts,
makes initial sync but does not send any filesystem data changes afterward.
I'm using CentOS 7.6.1810 with official glusterfs-6.3-1.el7 build on top of
ZFS on Linux.
It is a single Master node with single brick and the same Slave node.

# "gluster vol geo-rep status" command gives the following output
 MASTER NODE = gfs-alfa1
 MASTER VOL = cicd
 MASTER BRICK = /zdata/cicd/brick
 SLAVE USER = root
 SLAVE = gfs-alfa2::cicd
 SLAVE NODE = gfs-alfa2
 STATUS = Active
 CRAWL STATUS = Changelog Crawl
 LAST_SYNCED = 2019-07-05 12:08:17
 ENTRY = 0
 DATA = 0
 META = 0
 FAILURES = 0
 CHECKPOINT TIME = 2019-07-05 12:13:46
 CHECKPOINT COMPLETED = No

I enabled DEBUG level log for gsyncd.log but did not get any error messages from
it. Full log is available here: https://pastebin.com/pXL4dBhZ
On both brick I disabled ctime feature because it is incompatible with old
versions of gfs clients, enabling this feature does not help too.

# gluster volume info
 Volume Name: cicd
 Type: Distribute
 Volume ID: 8f959a35-c7ab-4484-a1e8-9fa8e3a713b4
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 1
 Transport-type: tcp
 Bricks:
 Brick1: gfs-alfa1:/zdata/cicd/brick
 Options Reconfigured:
 nfs.disable: on
 transport.address-family: inet
 features.ctime: off
 geo-replication.indexing: on
 geo-replication.ignore-pid-check: on
 changelog.changelog: on

# gluster volume get cicd rollover-time
Option                                  Value
------                                  -----
changelog.rollover-time                 15

# gluster volume get cicd fsync-interval
Option                                  Value
------                                  -----
changelog.fsync-interval                5

Could someone help me with debug of this geo-rep setup?
Thank you!

BR, Alexey


________________________________

? ?????? ??????????? ????? ? ?? ???? ?? ????????? ????? ??????????? ??????????
????????????????? ?????????, ??????????????? ????????????? ?????? ??? ????????.
???? ?? ?? ?????? ???????? ?? ??????????????? ??? ??? ?????????, ??????????,
??????????????? ??????????????? ?? ???? ???????????, ? ???? ????????? ??????????
??????????. ??????????? ??? ???????????? ????, ?? ??????????? ????????? ?????,
?????? ??????????? ?????????????????? ????????????, ?????????????, ??????????,
??????????????, ?????????? ??? ????????????? ???? ???????? ?????????? ?????
?????.

This e-mail and any attachments may contain confidential and/or privileged
information and is intended solely for the addressee. If you are not the
intended recipient (or have received this e-mail in error) please notify the
sender immediately and destroy this e-mail. Any unauthorised use, review,
retransmissions, dissemination, copying or other use of this information by
persons or entities other than the intended recipient is strictly prohibited.

The session is moved from "history crawl" to "changelog
crawl". After this
point, there are no changelogs to be synced as per the logs.
Please checking in ".processing" directories if there are any pending
changelogs to be synced at
"/var/lib/misc/gluster/gsyncd/<session>/<brickpath>/.processing"
If there are no pending changelogs, then please check if the brick is up.

On Fri, Jul 5, 2019 at 5:29 PM Pasechny Alexey <Pasechny at
iskrauraltel.ru>
wrote:
> Hi everyone,
>
> I have a problem with native geo-replication setup. It successfully
> starts, makes initial sync but does not send any filesystem data changes
> afterward.
> I'm using CentOS 7.6.1810 with official glusterfs-6.3-1.el7 build on
top
> of ZFS on Linux.
> It is a single Master node with single brick and the same Slave node.
>
> # "gluster vol geo-rep status" command gives the following output
>  MASTER NODE = gfs-alfa1
>  MASTER VOL = cicd
>  MASTER BRICK = /zdata/cicd/brick
>  SLAVE USER = root
>  SLAVE = gfs-alfa2::cicd
>  SLAVE NODE = gfs-alfa2
>  STATUS = Active
>  CRAWL STATUS = Changelog Crawl
>  LAST_SYNCED = 2019-07-05 12:08:17
>  ENTRY = 0
>  DATA = 0
>  META = 0
>  FAILURES = 0
>  CHECKPOINT TIME = 2019-07-05 12:13:46
>  CHECKPOINT COMPLETED = No
>
> I enabled DEBUG level log for gsyncd.log but did not get any error
> messages from it. Full log is available here:
> https://pastebin.com/pXL4dBhZ
> On both brick I disabled ctime feature because it is incompatible with old
> versions of gfs clients, enabling this feature does not help too.
>
> # gluster volume info
>  Volume Name: cicd
>  Type: Distribute
>  Volume ID: 8f959a35-c7ab-4484-a1e8-9fa8e3a713b4
>  Status: Started
>  Snapshot Count: 0
>  Number of Bricks: 1
>  Transport-type: tcp
>  Bricks:
>  Brick1: gfs-alfa1:/zdata/cicd/brick
>  Options Reconfigured:
>  nfs.disable: on
>  transport.address-family: inet
>  features.ctime: off
>  geo-replication.indexing: on
>  geo-replication.ignore-pid-check: on
>  changelog.changelog: on
>
> # gluster volume get cicd rollover-time
> Option                                  Value
> ------                                  -----
> changelog.rollover-time                 15
>
> # gluster volume get cicd fsync-interval
> Option                                  Value
> ------                                  -----
> changelog.fsync-interval                5
>
> Could someone help me with debug of this geo-rep setup?
> Thank you!
>
> BR, Alexey
>
>
> ________________________________
>
> ? ?????? ??????????? ????? ? ?? ???? ?? ????????? ????? ???????????
> ?????????? ????????????????? ?????????, ??????????????? ?????????????
> ?????? ??? ????????. ???? ?? ?? ?????? ???????? ?? ??????????????? ??? ???
> ?????????, ??????????, ??????????????? ??????????????? ?? ???? ???????????,
> ? ???? ????????? ?????????? ??????????. ??????????? ??? ???????????? ????,
> ?? ??????????? ????????? ?????, ?????? ??????????? ??????????????????
> ????????????, ?????????????, ??????????, ??????????????, ?????????? ???
> ????????????? ???? ???????? ?????????? ????? ?????.
>
> This e-mail and any attachments may contain confidential and/or privileged
> information and is intended solely for the addressee. If you are not the
> intended recipient (or have received this e-mail in error) please notify
> the sender immediately and destroy this e-mail. Any unauthorised use,
> review, retransmissions, dissemination, copying or other use of this
> information by persons or entities other than the intended recipient is
> strictly prohibited.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


-- 
Thanks and Regards,
Kotresh H R
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190705/050e7f07/attachment.html>