Matthew B
2019-Jun-13 18:51 UTC
[Gluster-users] Gluster Geo Replication ChangelogException Is a directory
Hello - I am having a problem with geo-replication on glusterv5 that I
hope someone can help me with.
I have a 7-server distribute cluster as the primary volume, and a 2
server distribute cluster as the secondary volume. Both are running
the same version of gluster on CentOS 7: glusterfs-5.3-2.el7.x86_64
I was able to setup the replication keys, user, groups, etc and
establish the session, but it goes faulty quickly after initializing.
I ran into the missing libgfchangelog.so error and fixed with a symlink:
[root at pcic-backup01 ~]# ln -s /usr/lib64/libgfchangelog.so.0
/usr/lib64/libgfchangelog.so
[root at pcic-backup01 ~]# ls -lh /usr/lib64/libgfchangelog.so*
lrwxrwxrwx. 1 root root 30 May 16 13:16 /usr/lib64/libgfchangelog.so
-> /usr/lib64/libgfchangelog.so.0
lrwxrwxrwx. 1 root root 23 May 16 08:58
/usr/lib64/libgfchangelog.so.0 -> libgfchangelog.so.0.0.1
-rwxr-xr-x. 1 root root 62K Feb 25 04:02 /usr/lib64/libgfchangelog.so.0.0.1
But right now, when trying to start replication it goes faulty:
[root at gluster01 ~]# gluster volume geo-replication storage
geoaccount at 10.0.231.81::pcic-backup start
Starting geo-replication session between storage &
geoaccount at 10.0.231.81::pcic-backup has been successful
[root at gluster01 ~]# gluster volume geo-replication status
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER
SLAVE SLAVE NODE STATUS
CRAWL STATUS LAST_SYNCED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.50 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.54 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.56 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.52 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.55 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.51 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
10.0.231.53 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A
Initializing... N/A N/A
[root at gluster01 ~]# gluster volume geo-replication status
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER
SLAVE SLAVE NODE STATUS
CRAWL STATUS LAST_SYNCED
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.50 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.54 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.56 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.55 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.53 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.51 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
10.0.231.52 storage /mnt/raid6-storage/storage geoaccount
ssh://geoaccount at 10.0.231.81::pcic-backup N/A Faulty
N/A N/A
[root at gluster01 ~]# gluster volume geo-replication storage
geoaccount at 10.0.231.81::pcic-backup stop
Stopping geo-replication session between storage &
geoaccount at 10.0.231.81::pcic-backup has been successful
And the
/var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.log
log file contains the error: GLUSTER: Changelog register failed
error=[Errno 21] Is a directory
[root at gluster01 ~]# cat
/var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.log
[2019-05-23 17:07:23.500781] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:23.629298] I [gsyncd(status):308:main] <top>: Using
session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:31.354005] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:31.483582] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:31.863888] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:31.994895] I [gsyncd(monitor):308:main] <top>: Using
session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:33.133888] I
[gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
Status Change status=Initializing...
[2019-05-23 17:07:33.134301] I [monitor(monitor):157:monitor] Monitor:
starting gsyncd worker brick=/mnt/raid6-storage/storage
slave_node=10.0.231.81
[2019-05-23 17:07:33.214462] I [gsyncd(agent
/mnt/raid6-storage/storage):308:main] <top>: Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:33.216737] I [changelogagent(agent
/mnt/raid6-storage/storage):72:__init__] ChangelogAgent: Agent
listining...
[2019-05-23 17:07:33.228072] I [gsyncd(worker
/mnt/raid6-storage/storage):308:main] <top>: Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:33.247236] I [resource(worker
/mnt/raid6-storage/storage):1366:connect_remote] SSH: Initializing SSH
connection between master and slave...
[2019-05-23 17:07:34.948796] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:35.73339] I [gsyncd(status):308:main] <top>: Using
session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:35.232405] I [resource(worker
/mnt/raid6-storage/storage):1413:connect_remote] SSH: SSH connection
between master and slave established. duration=1.9849
[2019-05-23 17:07:35.232748] I [resource(worker
/mnt/raid6-storage/storage):1085:connect] GLUSTER: Mounting gluster
volume locally...
[2019-05-23 17:07:36.359250] I [resource(worker
/mnt/raid6-storage/storage):1108:connect] GLUSTER: Mounted gluster
volume duration=1.1262
[2019-05-23 17:07:36.359639] I [subcmds(worker
/mnt/raid6-storage/storage):80:subcmd_worker] <top>: Worker spawn
successful. Acknowledging back to monitor
[2019-05-23 17:07:36.380975] E [repce(agent
/mnt/raid6-storage/storage):122:worker] <top>: call failed:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118,
in worker
res = getattr(self.obj, rmeth)(*in_data[2:])
File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py",
line 40, in register
return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, retries)
File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
line 45, in cl_register
cls.raise_changelog_err()
File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
line 29, in raise_changelog_err
raise ChangelogException(errn, os.strerror(errn))
ChangelogException: [Errno 21] Is a directory
[2019-05-23 17:07:36.382556] E [repce(worker
/mnt/raid6-storage/storage):214:__call__] RepceClient: call failed
call=27412:140659114579776:1558631256.38 method=register
error=ChangelogException
[2019-05-23 17:07:36.382833] E [resource(worker
/mnt/raid6-storage/storage):1266:service_loop] GLUSTER: Changelog
register failed error=[Errno 21] Is a directory
[2019-05-23 17:07:36.404313] I [repce(agent
/mnt/raid6-storage/storage):97:service_loop] RepceServer: terminating
on reaching EOF.
[2019-05-23 17:07:37.361396] I [monitor(monitor):278:monitor] Monitor:
worker died in startup phase brick=/mnt/raid6-storage/storage
[2019-05-23 17:07:37.370690] I
[gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
Status Change status=Faulty
[2019-05-23 17:07:41.526408] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:41.643923] I [gsyncd(status):308:main] <top>: Using
session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:45.722193] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:45.817210] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:46.188499] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:46.258817] I [gsyncd(config-get):308:main] <top>:
Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:47.350276] I [gsyncd(monitor-status):308:main]
<top>: Using session config file
path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf
[2019-05-23 17:07:47.364751] I
[subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor
Status Change status=Stopped
I'm not really sure where to go from here...
[root at gluster01 ~]# gluster volume geo-replication storage
geoaccount at 10.0.231.81::pcic-backup config | grep -i changelog
change_detector:changelog
changelog_archive_format:%Y%m
changelog_batch_size:727040
changelog_log_file:/var/log/glusterfs/geo-replication/storage_10.0.231.81_pcic-backup/changes-${local_id}.log
changelog_log_level:INFO
Thanks,
-Matthew