Naranderan Ramakrishnan
2020-May-26 07:38 UTC
[Gluster-users] Faulty staus in geo-replication session of a sub-volume
Dear Developers/Users, A geo-rep session of a sub-volume is in 'faulty' status. Please find the setup and log details below. Setup Details: Gluster version - 7.0 Volume configuration - 2x3 (DxR) gysncd permission(master) - root gysncd permission(slave) - sas (non-root) glusterd, glusterfsd permissions(master) - root glusterd, glusterfsd permissions(slave) - root Log details: In the *master gyncd* log, this traceback is printed *repeatedly*.> [2020-05-22 12:09:43.838727] I [master(worker > /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken > duration=0.4240 num_files=1 job=1 return_code=0 > [2020-05-22 12:09:43.944392] E [repce(worker > /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed > call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError > [2020-05-22 12:09:43.944746] E [syncdutils(worker > /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL: > Traceback (most recent call last): > File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in > main > func(args) > File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in > subcmd_worker > local.service_loop(remote) > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, > in service_loop > g3.crawlwrap(oneshot=True) > File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in > crawlwrap > self.crawl() > File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in > crawl > self.changelogs_batch_process(changes) > File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in > changelogs_batch_process > self.process(batch) > File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in > process > self.process_change(change, done, retry) > File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in > process_change > failures = self.slave.server.entry_ops(entries) > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in > __call__ > return self.ins(self.meth, *a) > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in > __call__ > raise res > OSError: [Errno 13] Permission denied: > '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' > [2020-05-22 12:09:43.968710] I [repce(agent > /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating > on reaching EOF. > [2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: > worker died in startup phase brick=/home/sas/gluster/data/code-ide > [2020-05-22 12:09:44.913692] I > [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status > Change status=Faulty >In salve end, these are printed *repeatedly*.> > [2020-05-22 11:23:26.65115] W [gsyncd(slave > 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session > config file not exists, using the default config > path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf > [2020-05-22 11:23:26.77414] I [resource(slave > 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: > Mounting gluster volume locally... > [2020-05-22 11:23:27.297466] I [resource(slave > 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: > Mounted gluster volume duration=1.2199 > [2020-05-22 11:23:27.298125] I [resource(slave > 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: > slave listening > [2020-05-22 11:23:32.654939] E [repce(slave > 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call > failed: > Traceback (most recent call last): > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in > worker > res = getattr(self.obj, rmeth)(*in_data[2:]) > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, > in entry_ops > collect_failure(e, cmd_ret, uid, gid) > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, > in collect_failure > disk_gfid) > File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, > in get_slv_dir_path > [ENOENT], [ESTALE]) > File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, > in errno_wrap > return call(*arg) > OSError: [Errno 13] Permission denied: > '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' > [2020-05-22 11:23:32.741317] I [repce(slave > 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: > terminating on reaching EOF. >Additional info: Parallel to this GFID(/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e) mentioned in master gyscnd log, there are some files with ---------T permission & *trusted.glusterfs.dht.linkto* extended attribute in the master subvolume for which geo-rep session is in faulty status. Not sure whether this is related to this geo-rep issue or not. I have attached a few screenshots and log stmts for further info. Please let us know how we should solve this. Thanks in advance. Regards, Naranderan R -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200526/49963943/attachment-0001.html> -------------- next part -------------- sas at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops collect_failure(e, cmd_ret, uid, gid) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure disk_gfid) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path [ENOENT], [ESTALE]) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap return call(*arg) OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' [2020-05-22 11:23:14.762231] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. [2020-05-22 11:23:26.65115] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf [2020-05-22 11:23:26.77414] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting gluster volume locally... [2020-05-22 11:23:27.297466] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted gluster volume duration=1.2199 [2020-05-22 11:23:27.298125] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: slave listening [2020-05-22 11:23:32.654939] E [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops collect_failure(e, cmd_ret, uid, gid) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure disk_gfid) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path [ENOENT], [ESTALE]) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap return call(*arg) OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' [2020-05-22 11:23:32.741317] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. [2020-05-22 11:23:44.797867] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf -------------- next part -------------- A non-text attachment was scrubbed... Name: Slave-mnt-10.47.8.153-home-sas-gluster-data-code-ide-log-tailf.png Type: image/png Size: 1827884 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200526/49963943/attachment-0001.png> -------------- next part -------------- root at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#ls gsyncd.log mnt-10.47.8.153-home-sas-gluster-data-code-ide.log root at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#tail -f mnt-10.47.8.153-home-sas-gluster-data-code-ide.log [2020-05-22 11:31:10.001901] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:10.001955] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met [2020-05-22 11:31:10.002114] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:10.003158] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:10.003215] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f8ec76fb244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f8ed62782f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f8ed6274c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:10.003227] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg [2020-05-22 11:31:10.003564] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:10.004717] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2020-05-22 11:31:10.004746] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0 [2020-05-22 11:31:10.007121] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5 [2020-05-22 11:31:16.385568] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-IgRBaT [2020-05-22 11:31:16.385666] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f8ed50c2dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56136fdfa625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56136fdfa48b] ) 0-: received signum (15), shutting down [2020-05-22 11:31:16.385687] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'. [2020-05-22 11:31:16.385698] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'. [2020-05-22 11:31:27.695486] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-gXOnl6) [2020-05-22 11:31:27.698895] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230635 [2020-05-22 11:31:27.707687] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0 [2020-05-22 11:31:27.707741] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2020-05-22 11:31:27.714024] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.716381] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.716834] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0) [2020-05-22 11:31:27.716863] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12) [2020-05-22 11:31:27.718763] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.721166] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.721451] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.721476] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg [2020-05-22 11:31:27.721587] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0) [2020-05-22 11:31:27.721606] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13) [2020-05-22 11:31:27.723607] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.725797] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.725826] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online. [2020-05-22 11:31:27.725971] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:27.726190] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.726263] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.726279] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg [2020-05-22 11:31:27.726404] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0) [2020-05-22 11:31:27.726419] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12) [2020-05-22 11:31:27.728149] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.730264] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.730284] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met [2020-05-22 11:31:27.730395] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:27.730634] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.730714] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.730731] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg [2020-05-22 11:31:27.730867] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0) [2020-05-22 11:31:27.730883] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13) [2020-05-22 11:31:27.732462] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport [2020-05-22 11:31:27.734593] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.734935] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-3: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.735007] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.735026] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-3: failed to set process-name in handshake msg [2020-05-22 11:31:27.735187] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-4: changing port to 49152 (from 0) [2020-05-22 11:31:27.735203] I [socket.c:864:__socket_shutdown] 0-code-ide-client-4: intentional socket shutdown(12) Final graph: +------------------------------------------------------------------------------+ 1: volume code-ide-client-0 2: type protocol/client 3: option opversion 70000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key code-ide 7: option client-version 7.0 8: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-0-RECON_NO:-0 9: option fops-version 1298437 10: option ping-timeout 42 11: option remote-host 10.37.11.253 12: option remote-subvolume /home/sas/gluster/data/code-ide 13: option transport-type socket 14: option transport.address-family inet 15: option username 57f0d708-564c-4762-bcba-39bbbd32a314 16: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 17: option transport.socket.ssl-enabled off 18: option transport.tcp-user-timeout 0 19: option transport.socket.keepalive-time 20 20: option transport.socket.keepalive-interval 2 21: option transport.socket.keepalive-count 9 22: option send-gids true 23: end-volume 24: 25: volume code-ide-client-1 26: type protocol/client 27: option opversion 70000 28: option clnt-lk-version 1 29: option volfile-checksum 0 30: option volfile-key code-ide 31: option client-version 7.0 32: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-1-RECON_NO:-0 33: option fops-version 1298437 34: option ping-timeout 42 35: option remote-host 10.37.11.252 36: option remote-subvolume /home/sas/gluster/data/code-ide 37: option transport-type socket 38: option transport.address-family inet 39: option username 57f0d708-564c-4762-bcba-39bbbd32a314 40: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 41: option transport.socket.ssl-enabled off 42: option transport.tcp-user-timeout 0 43: option transport.socket.keepalive-time 20 44: option transport.socket.keepalive-interval 2 45: option transport.socket.keepalive-count 9 46: option send-gids true 47: end-volume 48: 49: volume code-ide-client-2 50: type protocol/client 51: option opversion 70000 52: option clnt-lk-version 1 53: option volfile-checksum 0 54: option volfile-key code-ide 55: option client-version 7.0 56: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-2-RECON_NO:-0 57: option fops-version 1298437 58: option ping-timeout 42 59: option remote-host 10.37.11.251 60: option remote-subvolume /home/sas/gluster/data/code-ide 61: option transport-type socket 62: option transport.address-family inet 63: option username 57f0d708-564c-4762-bcba-39bbbd32a314 64: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 65: option transport.socket.ssl-enabled off 66: option transport.tcp-user-timeout 0 67: option transport.socket.keepalive-time 20 68: option transport.socket.keepalive-interval 2 69: option transport.socket.keepalive-count 9 70: option send-gids true 71: end-volume 72: 73: volume code-ide-replicate-0 74: type cluster/replicate 75: option afr-pending-xattr code-ide-client-0,code-ide-client-1,code-ide-client-2 76: option use-compound-fops off 77: subvolumes code-ide-client-0 code-ide-client-1 code-ide-client-2 78: end-volume 79: 80: volume code-ide-client-3 81: type protocol/client 82: option opversion 70000 83: option clnt-lk-version 1 84: option volfile-checksum 0 85: option volfile-key code-ide 86: option client-version 7.0 87: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-3-RECON_NO:-0 88: option fops-version 1298437 89: option ping-timeout 42 90: option remote-host 10.37.11.250 91: option remote-subvolume /home/sas/gluster/data/code-ide 92: option transport-type socket 93: option transport.address-family inet 94: option username 57f0d708-564c-4762-bcba-39bbbd32a314 95: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 96: option transport.socket.ssl-enabled off 97: option transport.tcp-user-timeout 0 98: option transport.socket.keepalive-time 20 99: option transport.socket.keepalive-interval 2 100: option transport.socket.keepalive-count 9 101: option send-gids true 102: end-volume 103: 104: volume code-ide-client-4 105: type protocol/client 106: option ping-timeout 42 107: option remote-host 10.37.11.249 108: option remote-subvolume /home/sas/gluster/data/code-ide 109: option transport-type socket 110: option transport.address-family inet 111: option username 57f0d708-564c-4762-bcba-39bbbd32a314 112: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 113: option transport.socket.ssl-enabled off 114: option transport.tcp-user-timeout 0 115: option transport.socket.keepalive-time 20 116: option transport.socket.keepalive-interval 2 117: option transport.socket.keepalive-count 9 118: option send-gids true 119: end-volume 120: 121: volume code-ide-client-5 122: type protocol/client 123: option ping-timeout 42 124: option remote-host 10.37.11.246 125: option remote-subvolume /home/sas/gluster/data/code-ide 126: option transport-type socket 127: option transport.address-family inet 128: option username 57f0d708-564c-4762-bcba-39bbbd32a314 129: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1 130: option transport.socket.ssl-enabled off 131: option transport.tcp-user-timeout 0 132: option transport.socket.keepalive-time 20 133: option transport.socket.keepalive-interval 2 134: option transport.socket.keepalive-count 9 135: option send-gids true 136: end-volume 137: 138: volume code-ide-replicate-1 139: type cluster/replicate 140: option afr-pending-xattr code-ide-client-3,code-ide-client-4,code-ide-client-5 141: option use-compound-fops off 142: subvolumes code-ide-client-3 code-ide-client-4 code-ide-client-5 143: end-volume 144: 145: volume code-ide-dht 146: type cluster/distribute 147: option lock-migration off 148: option force-migration off 149: subvolumes code-ide-replicate-0 code-ide-replicate-1 150: end-volume 151: 152: volume code-ide-utime 153: type features/utime 154: option noatime on 155: subvolumes code-ide-dht 156: end-volume 157: 158: volume code-ide-write-behind 159: type performance/write-behind 160: subvolumes code-ide-utime 161: end-volume 162: 163: volume code-ide-read-ahead 164: type performance/read-ahead 165: subvolumes code-ide-write-behind 166: end-volume 167: 168: volume code-ide-readdir-ahead 169: type performance/readdir-ahead 170: option parallel-readdir off 171: option rda-request-size 131072 172: option rda-cache-limit 10MB 173: subvolumes code-ide-read-ahead 174: end-volume 175: 176: volume code-ide-io-cache 177: type performance/io-cache 178: subvolumes code-ide-readdir-ahead 179: end-volume 180: 181: volume code-ide-open-behind 182: type performance/open-behind 183: subvolumes code-ide-io-cache 184: end-volume 185: 186: volume code-ide-quick-read 187: type performance/quick-read 188: subvolumes code-ide-open-behind 189: end-volume 190: 191: volume code-ide-md-cache 192: type performance/md-cache 193: option cache-posix-acl true 194: subvolumes code-ide-quick-read 195: end-volume 196: 197: volume code-ide 198: type debug/io-stats 199: option log-level INFO 200: option threads 16 201: option latency-measurement off 202: option count-fop-hits off 203: option global-threading off 204: subvolumes code-ide-md-cache 205: end-volume 206: 207: volume posix-acl-autoload 208: type system/posix-acl 209: subvolumes code-ide 210: end-volume 211: 212: volume gfid-access-autoload 213: type features/gfid-access 214: subvolumes posix-acl-autoload 215: end-volume 216: 217: volume meta-autoload 218: type meta 219: subvolumes gfid-access-autoload 220: end-volume 221: +------------------------------------------------------------------------------+ [2020-05-22 11:31:27.737910] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-3: Connected to code-ide-client-3, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.737926] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-1: Subvolume 'code-ide-client-3' came back up; going online. [2020-05-22 11:31:27.738026] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:27.738195] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-5: changing port to 49152 (from 0) [2020-05-22 11:31:27.738212] I [socket.c:864:__socket_shutdown] 0-code-ide-client-5: intentional socket shutdown(13) [2020-05-22 11:31:27.739521] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-4: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.739592] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.739608] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-4: failed to set process-name in handshake msg [2020-05-22 11:31:27.740090] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.740105] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met [2020-05-22 11:31:27.740222] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:27.741492] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:27.741547] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:27.741560] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg [2020-05-22 11:31:27.741931] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:27.743285] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2020-05-22 11:31:27.743338] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0 [2020-05-22 11:31:27.746010] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5 [2020-05-22 11:31:34.214315] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-gXOnl6 [2020-05-22 11:31:34.214450] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f9309abfdd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55dcea04e625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55dcea04e48b] ) 0-: received signum (15), shutting down [2020-05-22 11:31:34.214485] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'. [2020-05-22 11:31:34.214499] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'. [2020-05-22 11:31:46.446624] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-oq0aUM) [2020-05-22 11:31:46.450189] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230980 [2020-05-22 11:31:46.458835] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2020-05-22 11:31:46.458826] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0 [2020-05-22 11:31:46.465054] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.467356] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.467849] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0) [2020-05-22 11:31:46.467872] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12) [2020-05-22 11:31:46.469623] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.472059] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0) [2020-05-22 11:31:46.472078] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13) [2020-05-22 11:31:46.473883] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.475942] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:46.476169] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:46.476186] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg [2020-05-22 11:31:46.476472] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:46.476555] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:46.476574] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg [2020-05-22 11:31:46.476837] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:46.476858] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online. [2020-05-22 11:31:46.476979] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:46.477009] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0) [2020-05-22 11:31:46.477025] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12) [2020-05-22 11:31:46.478210] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.480275] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'. [2020-05-22 11:31:46.480294] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met [2020-05-22 11:31:46.480406] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument] [2020-05-22 11:31:46.480644] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) [2020-05-22 11:31:46.480719] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument] [2020-05-22 11:31:46.480744] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg [2020-05-22 11:31:46.480870] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0) [2020-05-22 11:31:46.480887] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13) [2020-05-22 11:31:46.482509] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport [2020-05-22 11:31:46.484519] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'. -------------- next part -------------- sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date Fri May 22 05:09:23 PDT 2020 sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide> sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' [2020-05-22 12:09:25.176702] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. [2020-05-22 12:09:26.144371] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide [2020-05-22 12:09:26.145390] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2020-05-22 12:09:36.496685] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2020-05-22 12:09:36.496835] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker brick=/home/sas/gluster/data/code-ide slave_node=10.37.11.246 [2020-05-22 12:09:36.541825] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf [2020-05-22 12:09:36.545790] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining... [2020-05-22 12:09:36.554223] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf [2020-05-22 12:09:36.570023] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave... [2020-05-22 12:09:38.784018] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established. duration=2.2138 [2020-05-22 12:09:38.784291] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally... [2020-05-22 12:09:39.906485] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume duration=1.1220 [2020-05-22 12:09:39.906748] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor [2020-05-22 12:09:42.84068] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide [2020-05-22 12:09:42.84353] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time time=1590149382 [2020-05-22 12:09:42.88851] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change status=Active [2020-05-22 12:09:42.149405] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2020-05-22 12:09:42.149740] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl turns=1 stime=(1588918750, 0) entry_stime=(1588918779, 0) etime=1590149382 [2020-05-22 12:09:43.163168] I [master(worker /home/sas/gluster/data/code-ide):1583:crawl] _GMaster: slave's time stime=(1588918750, 0) [2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken duration=0.4240 num_files=1 job=1 return_code=0 [2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError [2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in crawl self.changelogs_batch_process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in changelogs_batch_process self.process(batch) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' [2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. [2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide [2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty ^C sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide> sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date Fri May 22 05:09:59 PDT 2020 sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' [2020-05-22 12:10:02.710613] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. [2020-05-22 12:10:03.681055] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide [2020-05-22 12:10:03.682048] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2020-05-22 12:10:14.31925] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2020-05-22 12:10:14.32105] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker brick=/home/sas/gluster/data/code-ide slave_node=10.37.11.246 [2020-05-22 12:10:14.76786] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf [2020-05-22 12:10:14.81006] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining... [2020-05-22 12:10:14.91418] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf [2020-05-22 12:10:14.109091] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave... [2020-05-22 12:10:16.334167] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established. duration=2.2248 [2020-05-22 12:10:16.334379] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally... [2020-05-22 12:10:17.444447] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume duration=1.1099 [2020-05-22 12:10:17.444658] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor [2020-05-22 12:10:19.612169] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide [2020-05-22 12:10:19.612401] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time time=1590149419 [2020-05-22 12:10:19.616991] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change status=Active [2020-05-22 12:10:19.669262] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2020-05-22 12:10:19.669564] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl turns=1 stime=(1588918750, 0) entry_stime=(1588918779, 0) etime=1590149419
Strahil Nikolov
2020-May-30 14:51 UTC
[Gluster-users] Faulty staus in geo-replication session of a sub-volume
Hello Naranderan, what OS are you using ? Do you have SELINUX in enforcing mode (verify via 'sestatus') ? Best Regards, Strahil Nikolov ? ??????, 30 ??? 2020 ?., 13:33:05 ?. ???????+3, Naranderan Ramakrishnan <rnaranbe at gmail.com> ??????: Dear Developers/Users, A geo-rep session of a sub-volume is in 'faulty' status. Please find the setup and log details?below. Setup Details:> Gluster version - 7.0 > Volume configuration - 2x3 (DxR) > gysncd?permission(master)? - root > gysncd?permission(slave)? - sas (non-root) > glusterd, glusterfsd permissions(master) - root > glusterd, glusterfsd permissions(slave) - rootLog details: In the master gyncd log, this traceback is printed repeatedly.> [2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken duration=0.4240 num_files=1 job=1 return_code=0 > [2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError > [2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL: > Traceback (most recent call last): > ? File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in main > ? ? func(args) > ? File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker > ? ? local.service_loop(remote) > ? File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, in service_loop > ? ? g3.crawlwrap(oneshot=True) > ? File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap > ? ? self.crawl() > ? File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in crawl > ? ? self.changelogs_batch_process(changes) > ? File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in changelogs_batch_process > ? ? self.process(batch) > ? File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process > ? ? self.process_change(change, done, retry) > ? File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change > ? ? failures = self.slave.server.entry_ops(entries) > ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ > ? ? return self.ins(self.meth, *a) > ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ > ? ? raise res > OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' > [2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF. > [2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide > [2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=FaultyIn salve end, these?are printed repeatedly.?> > [2020-05-22 11:23:26.65115] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf > [2020-05-22 11:23:26.77414] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting gluster volume locally... > [2020-05-22 11:23:27.297466] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted gluster volume duration=1.2199 > [2020-05-22 11:23:27.298125] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: slave listening > [2020-05-22 11:23:32.654939] E [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call failed: > Traceback (most recent call last): > ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker > ? ? res = getattr(self.obj, rmeth)(*in_data[2:]) > ? File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops > ? ? collect_failure(e, cmd_ret, uid, gid) > ? File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure > ? ? disk_gfid) > ? File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path > ? ? [ENOENT], [ESTALE]) > ? File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap > ? ? return call(*arg) > OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e' > [2020-05-22 11:23:32.741317] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.Additional info: Parallel to this GFID(/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e) mentioned in master gyscnd?log, there are some files with ---------T permission &?trusted.glusterfs.dht.linkto?extended attribute in the master subvolume for which geo-rep session is in faulty status. Not sure whether this is related to this geo-rep issue or not. I have attached a few screenshots and log stmts for further info. Please let us know how we should solve this. Thanks in advance. Regards, Naranderan R ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users