Nico van Royen
2018-Sep-06 09:24 UTC
[Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup
Hello, On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails. For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc). Installed packages, both on master and slaves: [root at clrv0000110367 geo-replication]# rpm -qa | grep gluster glusterfs-client-xlators-4.1.3-1.el7.x86_64 glusterfs-events-4.1.3-1.el7.x86_64 glusterfs-geo-replication-4.1.3-1.el7.x86_64 glusterfs-4.1.3-1.el7.x86_64 glusterfs-api-4.1.3-1.el7.x86_64 glusterfs-fuse-4.1.3-1.el7.x86_64 glusterfs-server-4.1.3-1.el7.x86_64 glusterfs-rdma-4.1.3-1.el7.x86_64 glusterfs-extra-xlators-4.1.3-1.el7.x86_64 glusterfs-libs-4.1.3-1.el7.x86_64 glusterfs-cli-4.1.3-1.el7.x86_64 python2-gluster-4.1.3-1.el7.x86_64 glusterfs-coreutils-0.2.0-1.el7.x86_64 Master volume setup: # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export clrv0000110389:/gluster/VOLUME2/export # gluster v start VOLUME2 # gluster volume set all cluster.enable-shared-storage enable Slave volume setup # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export clrv0000110606:/gluster/VOLUME2/export # gluster v start VOLUME2 # gluster volume set all cluster.enable-shared-storage enable On master server: # ssh-keygen (accepting all defaults) # ssh-copy-id clrv0000110605 (one of the slave servers) # gluster-georep-sshkey generate # gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2 create push-pem Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.log such as: [2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or directory] [2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/geo-replication/gsyncd_template.conf). [2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume VOLUME1 [2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped [2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed [2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886 [2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped [2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service [2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped [2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped [2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22 [2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). [2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110389.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. [2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110371.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. [2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110372.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. [2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size) The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd" repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241] [2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). [2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC == header->magic [2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key pending frames: frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2018-09-06 08:50:36 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 4.1.3 /lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874] /lib64/libc.so.6(+0x36280)[0x7fae3cb6e280] /lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207] /lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8] /lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7] /lib64/libc.so.6(+0x81429)[0x7fae3cbb9429] /lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d] /lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0] /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fae32f93254] /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fae32f9536e] /lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0] /lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0] --------- Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?) Anyone any idea's ? Regards, Nico van Roijen -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180906/f3d1159b/attachment.html>
Kotresh Hiremath Ravishankar
2018-Sep-06 10:58 UTC
[Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup
Hi Nico, The glusterd has crashed on this node. Please raise a bug with core file? Please use the following tool [1] to setup geo-rep by bringing back the glusterd if you are finding it difficult with geo-rep setup steps and let us know if if it still crashes? [1] http://aravindavk.in/blog/introducing-georepsetup/ On Thu, Sep 6, 2018 at 2:54 PM, Nico van Royen <nico at van-royen.nl> wrote:> Hello, > > On our dev environment we want to test GeoReplication with GlusterFS 4.1 > and every attempt so far fails. > For now, we don't care (yet) about running it as a non-root user (not > using the mountbroker etc). > > Installed packages, both on master and slaves: > [root at clrv0000110367 geo-replication]# rpm -qa | grep gluster > glusterfs-client-xlators-4.1.3-1.el7.x86_64 > glusterfs-events-4.1.3-1.el7.x86_64 > glusterfs-geo-replication-4.1.3-1.el7.x86_64 > glusterfs-4.1.3-1.el7.x86_64 > glusterfs-api-4.1.3-1.el7.x86_64 > glusterfs-fuse-4.1.3-1.el7.x86_64 > glusterfs-server-4.1.3-1.el7.x86_64 > glusterfs-rdma-4.1.3-1.el7.x86_64 > glusterfs-extra-xlators-4.1.3-1.el7.x86_64 > glusterfs-libs-4.1.3-1.el7.x86_64 > glusterfs-cli-4.1.3-1.el7.x86_64 > python2-gluster-4.1.3-1.el7.x86_64 > glusterfs-coreutils-0.2.0-1.el7.x86_64 > > Master volume setup: > # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp > clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export > clrv0000110389:/gluster/VOLUME2/export > # gluster v start VOLUME2 > # gluster volume set all cluster.enable-shared-storage enable > > Slave volume setup > # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp > clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export > clrv0000110606:/gluster/VOLUME2/export > # gluster v start VOLUME2 > # gluster volume set all cluster.enable-shared-storage enable > > On master server: > # ssh-keygen (accepting all defaults) > # ssh-copy-id clrv0000110605 (one of the slave servers) > # gluster-georep-sshkey generate > # gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2 > create push-pem > > Several seconds later, all of the glusterd instances on the master side > crash, with /var/log/glusterfs/glusterd.log such as: > > [2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] > 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_ > clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config > file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such > file or directory] > [2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] > 0-management: Using default config template(/var/lib/glusterd/ > geo-replication/gsyncd_template.conf). > [2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__ > glusterd_handle_cli_profile_volume] 0-management: Received volume profile > req for volume VOLUME1 > [2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__ > glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req > [2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] > 0-management: nfs already stopped > [2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] > 0-management: nfs service is stopped > [2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] > 0-management: nfs/server.so xlator is not installed > [2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] > 0-management: Stopping glustershd daemon running in pid: 40886 > [2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] > 0-management: glustershd service is stopped > [2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] > 0-management: Starting glustershd service > [2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] > 0-management: bitd already stopped > [2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] > 0-management: bitd service is stopped > [2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] > 0-management: scrub already stopped > [2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] > 0-management: scrub service is stopped > [2018-09-06 08:50:28.391332] I [run.c:241:runner_log] > (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) > [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) > [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) > [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/ > gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 > is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/ > common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol> VOLUME2,ssh_port=22 > [2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] > 0-management: Using passed config template(/var/lib/glusterd/ > geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). > [2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] > 0-glusterd: Staging failed on clrv0000110389.ic.ing.net. Error: > Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does > not exist. > [2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] > 0-glusterd: Staging failed on clrv0000110371.ic.ing.net. Error: > Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does > not exist. > [2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] > 0-glusterd: Staging failed on clrv0000110372.ic.ing.net. Error: > Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does > not exist. > [2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] > (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) > [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) > [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) > [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t > *)((char *)free_ptr + header->size) > The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] > 0-management: Received request to get state for glusterd" > repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 > 08:50:35.690241] > [2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] > 0-management: Using passed config template(/var/lib/glusterd/ > geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). > [2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] > (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] > -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] > -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: > Assertion failed: GF_MEM_HEADER_MAGIC => header->magic > [2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886: > glusterd_get_value_for_vme_entry] 0-management: Failed to get option for > xattr-user-namespace-mode key > pending frames: > frame : type(0) op(0) > patchset: git://git.gluster.org/glusterfs.git > signal received: 6 > time of crash: > 2018-09-06 08:50:36 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 4.1.3 > /lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920] > /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874] > /lib64/libc.so.6(+0x36280)[0x7fae3cb6e280] > /lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207] > /lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8] > /lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7] > /lib64/libc.so.6(+0x81429)[0x7fae3cbb9429] > /lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d] > /lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0] > /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+ > 0x22254)[0x7fae32f93254] > /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+ > 0x2436e)[0x7fae32f9536e] > /lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0] > /lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0] > --------- > > Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) > is empty, and was (re)created when the gluster volume geo-replication > command was issued. (is it supposed to be empty ?) > > Anyone any idea's ? > > Regards, Nico van Roijen > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180906/f5473a7f/attachment.html>