Hello,
I am trying again to establish geo-replication between a volume called
images and a local directory named /glimages. (nfs-mounted)
I get status = faulty. Looking in
/var/log/glusterfs/geo-replication/images/file%3A%2F%2F%2Fglimage.log, I
see: OSError: [Errno 107] Transport endpoint is not connected
but then " gluster peer status" shows all peers connected. Please
help!
Information and logs follow.
Thanks,
Jojo
-----------------------------------
[root at creator ~]# gluster peer status
Number of Peers: 6
Hostname: stor-003
Uuid: 3300f1a9-9252-4d39-a8dd-6ef6de66e4c3
State: Peer in Cluster (Connected)
Hostname: stor-001
Uuid: a7406cf1-c598-424e-85ab-5758016999a1
State: Peer in Cluster (Connected)
Hostname: stor-008
Uuid: 0f57c4a5-9f01-475b-b295-ebd6f63e855d
State: Peer in Cluster (Disconnected)
Hostname: stor-007
Uuid: bd966425-576c-4cba-be5c-b16eb00d10f1
State: Peer in Cluster (Disconnected)
Hostname: stor-002
Uuid: f38afa35-0c73-4c08-926f-a39953f48180
State: Peer in Cluster (Connected)
Hostname: stor-004
Uuid: 13b28d31-9eed-4052-9e45-c3baf83ce01e
State: Peer in Cluster (Connected)
[root at creator ~]# gluster volume info
Volume Name: images
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: stor-001:/glusterfs
Brick2: stor-002:/glusterfs
Brick3: stor-003:/glusterfs
Brick4: stor-004:/glusterfs
Options Reconfigured:
geo-replication.indexing: on
[root at creator ~]# gluster volume geo-replication images /glimage start
Starting geo-replication session between images & /glimage has been
successful
[root at creator ~]# gluster volume geo-replication status
MASTER SLAVE
STATUS
---------------------------------------------------------------------------
-----
images file:///glimage
faulty
[root at creator ~]# rsync
rsync version 3.0.7 protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, no symtimes
root at creator ~]# cat
/var/log/glusterfs/geo-replication-slaves/df4e1ece-61eb-47e5-8420-2d0f081ad
0fe\:file%3A%2F%2F%2Fglimage.log
[2011-09-25 17:35:59.639471] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:35:59.640634] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:36:01.677525] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:36:12.373981] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:36:12.376377] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:36:13.672760] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:36:24.898442] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:36:24.900129] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:36:26.194766] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:36:37.420826] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:36:37.421900] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:36:38.717241] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:36:49.939140] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:36:49.940651] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:36:51.241149] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:37:02.464519] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:37:02.466437] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:37:03.760050] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:37:14.985683] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:37:14.987176] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:37:16.281832] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:37:27.505346] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:37:27.506943] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:37:28.802320] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:37:40.31245] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:37:40.32705] I [resource(slave):200:service_loop] FILE:
slave listening
...
[2011-09-25 17:39:57.801064] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:39:59.96476] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:39:59.643095] I [resource(slave):206:service_loop] FILE:
connection inactive for 120 seconds, stopping
[2011-09-25 17:39:59.643385] I [gsyncd(slave):298:main_i] <top>: exiting.
[2011-09-25 17:40:10.320939] I [gsyncd(slave):286:main_i] <top>: syncing:
file:///glimage
[2011-09-25 17:40:10.322839] I [resource(slave):200:service_loop] FILE:
slave listening
[2011-09-25 17:40:11.616868] I [repce(slave):61:service_loop] RepceServer:
terminating on reaching EOF.
[2011-09-25 17:40:12.377886] I [resource(slave):206:service_loop] FILE:
connection inactive for 120 seconds, stopping
[2011-09-25 17:40:12.378180] I [gsyncd(slave):298:main_i] <top>: exiting.
[root at creator ~]# ls -lh
/var/log/glusterfs/geo-replication/images/file%3A%2F%2F%2Fglimage.gluster.l
og
-rw-r--r-- 1 root root 3.5K Sep 25 17:42
/var/log/glusterfs/geo-replication/images/file%3A%2F%2F%2Fglimage.gluster.l
og
[root at creator ~]# cat
/var/log/glusterfs/geo-replication/images/file%3A%2F%2F%2Fglimage.gluster.l
og
[2011-09-25 17:35:59.860493] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:36:12.412180] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:36:24.934716] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:36:37.457013] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:36:49.978919] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:37:02.500859] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:37:15.22302] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:37:27.543106] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:37:40.67778] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:37:52.588007] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:38:05.109423] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:38:17.632551] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:38:30.156666] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:38:42.683171] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:38:55.207246] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:39:07.732034] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:39:20.257885] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:39:32.781833] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:39:45.311328] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:39:57.836471] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:40:10.357471] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:40:22.885769] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:40:35.409369] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:40:47.932260] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:41:00.454678] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:41:12.978558] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:41:25.503510] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:41:38.29328] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[2011-09-25 17:41:50.552532] E [glusterfsd.c:1356:daemonize] 0-daemonize:
Daemonization failed: No such device
[root at creator ~]# cat
/var/log/glusterfs/geo-replication/images/file%3A%2F%2F%2Fglimage.log
[2011-09-25 17:35:59.327866] I [monitor(monitor):19:set_state] Monitor:
new state: starting...
[2011-09-25 17:35:59.340078] I [monitor(monitor):42:monitor] Monitor:
------------------------------------------------------------
[2011-09-25 17:35:59.340276] I [monitor(monitor):43:monitor] Monitor:
starting gsyncd worker
[2011-09-25 17:35:59.418025] I [gsyncd:286:main_i] <top>: syncing:
gluster://localhost:images -> file:///glimage
[2011-09-25 17:36:01.673922] E [syncdutils:131:exception] <top>: FAIL:
Traceback (most recent call last):
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 102, in main
main_i()
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 296, in main_i
local.service_loop(*[r for r in [remote] if r])
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 401, in service_loop
GMaster(self, args[0]).crawl_loop()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 113, in crawl_loop
self.crawl()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 173, in crawl
volinfo_sys = self.get_sys_volinfo()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 22, in get_sys_volinfo
fgn_vis, nat_vi = self.master.server.foreign_volume_infos(), \
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 330, in foreign_volume_infos
xattr_list = Xattr.llistxattr_buf('.')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 57, in llistxattr_buf
size = cls.llistxattr(path)
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 38, in llistxattr
ret = cls._query_xattr(path, siz, 'llistxattr')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 26, in _query_xattr
cls.raise_oserr()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 16, in raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 107] Transport endpoint is not connected
[2011-09-25 17:36:02.87781] I [monitor(monitor):19:set_state] Monitor: new
state: faulty
[2011-09-25 17:36:12.96443] I [monitor(monitor):42:monitor] Monitor:
------------------------------------------------------------
[2011-09-25 17:36:12.96780] I [monitor(monitor):43:monitor] Monitor:
starting gsyncd worker
[2011-09-25 17:36:12.175059] I [gsyncd:286:main_i] <top>: syncing:
gluster://localhost:images -> file:///glimage
[2011-09-25 17:36:13.669708] E [syncdutils:131:exception] <top>: FAIL:
Traceback (most recent call last):
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 102, in main
main_i()
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 296, in main_i
local.service_loop(*[r for r in [remote] if r])
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 401, in service_loop
GMaster(self, args[0]).crawl_loop()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 113, in crawl_loop
self.crawl()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 173, in crawl
volinfo_sys = self.get_sys_volinfo()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 22, in get_sys_volinfo
fgn_vis, nat_vi = self.master.server.foreign_volume_infos(), \
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 330, in foreign_volume_infos
xattr_list = Xattr.llistxattr_buf('.')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 57, in llistxattr_buf
size = cls.llistxattr(path)
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 38, in llistxattr
ret = cls._query_xattr(path, siz, 'llistxattr')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 26, in _query_xattr
cls.raise_oserr()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 16, in raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 107] Transport endpoint is not connected
[2011-09-25 17:36:24.618252] I [monitor(monitor):42:monitor] Monitor:
------------------------------------------------------------
[2011-09-25 17:36:24.618722] I [monitor(monitor):43:monitor] Monitor:
starting gsyncd worker
[2011-09-25 17:36:24.699768] I [gsyncd:286:main_i] <top>: syncing:
gluster://localhost:images -> file:///glimage
[2011-09-25 17:36:26.191740] E [syncdutils:131:exception] <top>: FAIL:
Traceback (most recent call last):
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 102, in main
main_i()
File
"/opt/glusterfs/3.2.2/local/libexec//glusterfs/python/syncdaemon/gsyncd.py"
, line 296, in main_i
local.service_loop(*[r for r in [remote] if r])
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 401, in service_loop
GMaster(self, args[0]).crawl_loop()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 113, in crawl_loop
self.crawl()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 173, in crawl
volinfo_sys = self.get_sys_volinfo()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/master.py",
line 22, in get_sys_volinfo
fgn_vis, nat_vi = self.master.server.foreign_volume_infos(), \
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/resource.py
", line 330, in foreign_volume_infos
xattr_list = Xattr.llistxattr_buf('.')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 57, in llistxattr_buf
size = cls.llistxattr(path)
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 38, in llistxattr
ret = cls._query_xattr(path, siz, 'llistxattr')
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 26, in _query_xattr
cls.raise_oserr()
File
"/opt/glusterfs/3.2.2/local/libexec/glusterfs/python/syncdaemon/libcxattr.p
y", line 16, in raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 107] Transport endpoint is not connected