Christos Tsalidis
2018-Oct-18 11:06 UTC
[Gluster-users] Adding a new node receives State: Peer Rejected (Connected)
Hi all, I am testing to extend my gluster cluster and add a new node in glusterfs 3.12.14 version on CentOS Linux release 7.5.1804 and receives State: Peer Rejected (Connected) [root at serverc ~]# gluster peer status Number of Peers: 2 Hostname: servera Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 State: Peer Rejected (Connected) Hostname: serverb.lab.local Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Peer Rejected (Connected) [root at servera ~]# gluster peer status Number of Peers: 2 Hostname: serverb.lab.local Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Peer in Cluster (Connected) Hostname: serverc Uuid: 4dc48498-bb24-446e-a3c2-e2855beaa02e State: Peer Rejected (Connected) [root at servera ~]# gluster volume info mastervol Volume Name: mastervol Type: Distribute Volume ID: 22fd33cf-ee41-4eef-94c3-a1627258b0e0 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: servera:/bricks/brick-a1/brick Brick2: serverb:/bricks/brick-b1/brick Options Reconfigured: server.ssl: on client.ssl: on auth.ssl-allow: servera,serverb,workstation,servere,serverc features.scrub-throttle: aggressive features.scrub-freq: hourly features.scrub: Active features.bitrot: on features.hard-timeout: 1s features.soft-timeout: 5s features.quota-deem-statfs: on features.inode-quota: on features.quota: on storage.batch-fsync-delay-usec: 0 server.allow-insecure: on performance.stat-prefetch: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on cluster.rebal-throttle: lazy geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on changelog.rollover-time: 5 cluster.enable-shared-storage: enable auto-delete: enable snap-activate-on-create: enable root at serverc ~]# systemctl stop glusterd [root at serverc ~]# cd /var/lib/glusterd/ [root at serverc glusterd]# ll total 8 drwxr-xr-x. 2 root root 6 Oct 18 11:19 bitd -rw-------. 1 root root 66 Oct 18 11:25 glusterd.info drwxr-xr-x. 2 root root 35 Oct 18 11:22 glustershd drwxr-xr-x. 2 root root 6 Oct 18 11:21 groups drwxr-xr-x. 3 root root 15 Oct 18 11:21 hooks drwxr-xr-x. 2 root root 6 Oct 18 11:21 nfs -rw-------. 1 root root 157 Oct 18 11:22 options drwxr-xr-x. 2 root root 94 Oct 18 11:25 peers drwxr-xr-x. 2 root root 24 Oct 18 11:22 quotad drwxr-xr-x. 2 root root 6 Oct 18 11:19 scrub -rw-r--r--. 1 root root 0 Oct 18 11:01 secure-access drwxr-xr-x. 2 root root 31 Oct 18 11:22 snaps drwxr-xr-x. 2 root root 6 Oct 18 11:19 ss_brick drwxr-xr-x. 3 root root 63 Oct 18 11:22 trash drwxr-xr-x. 5 root root 68 Oct 18 11:22 vols [root at serverc glusterd]# rm -rf bitd glustershd groups hooks nfs options peers quotad scrub snaps ss_brick trash vols [root at serverc glusterd]# ll total 4 -rw-------. 1 root root 66 Oct 18 11:25 glusterd.info -rw-r--r--. 1 root root 0 Oct 18 11:01 secure-access [root at serverc glusterd]# [root at serverc glusterd]# systemctl start glusterd [root at serverc glusterd]# gluster peer staus unrecognized word: staus (position 1) [root at serverc glusterd]# gluster peer status Number of Peers: 0 [root at serverc glusterd]# gluster peer probe servera peer probe: success. [root at serverc glusterd]# gluster peer probe serverb peer probe: success. [root at serverc glusterd]# gluster peer status Number of Peers: 2 Hostname: servera Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 State: Accepted peer request (Connected) Hostname: serverb Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Accepted peer request (Connected) [root at serverc glusterd]# systemctl restart glusterd [root at serverc glusterd]# gluster peer status Number of Peers: 2 Hostname: servera Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 State: Sent and Received peer request (Connected) Hostname: serverb Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Sent and Received peer request (Connected) [root at serverc glusterd]# [root at serverc glusterd]# gluster peer status Number of Peers: 2 Hostname: servera Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 State: Peer in Cluster (Connected) Hostname: serverb.lab.local Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Peer in Cluster (Connected) Other names: serverb.lab.local [root at serverc glusterd]# gluster peer status Number of Peers: 2 Hostname: servera Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 State: Peer in Cluster (Connected) Hostname: serverb.lab.local Uuid: e88e454a-f85a-472a-8920-a541c8615d03 State: Peer Rejected (Connected) Other names: serverb.lab.local [2018-10-18 10:44:06.424524] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-10-18 10:44:06.430245] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 4dc48498-bb24-446e-a3c2-e2855beaa02e [2018-10-18 10:44:06.440524] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera [2018-10-18 10:44:06.440557] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2018-10-18 10:44:06.448906] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb [2018-10-18 10:44:06.448941] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2018-10-18 10:44:08.785483] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req The message "I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-10-18 10:44:08.785483] and [2018-10-18 10:44:14.411018] [2018-10-18 10:44:17.014898] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera, port: 0 [2018-10-18 10:44:17.017750] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:17.019991] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2018-10-18 10:44:17.020433] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:17.023102] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb, port: 0 [2018-10-18 10:44:17.030879] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e88e454a-f85a-472a-8920-a541c8615d03 [2018-10-18 10:44:17.104044] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-10-18 10:44:18.502357] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 [2018-10-18 10:44:18.506876] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:18.531884] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:18.542340] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 from peer servera missing on localhost [2018-10-18 10:44:18.542494] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.12.01 from peer servera missing on localhost [2018-10-18 10:44:18.542561] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.14.01 from peer servera missing on localhost [2018-10-18 10:44:18.542634] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.16.01 from peer servera missing on localhost [2018-10-18 10:44:18.542712] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.18.01 from peer servera missing on localhost [2018-10-18 10:44:18.545596] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to servera (0), ret: 0, op_ret: 0 [2018-10-18 10:44:18.550763] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:18.552681] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2018-10-18 10:44:18.579609] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory" repeated 5 times between [2018-10-18 10:44:18.579609] and [2018-10-18 10:44:18.605927] [2018-10-18 10:44:18.607656] E [MSGID: 106049] [glusterd-snapshot-utils.c:1529:glusterd_gen_snap_volfiles] 0-management: Parent volinfo not found for 1ec76d1dbc1c4209b774b13c07b7b337 volume of snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 [2018-10-18 10:44:18.607689] E [MSGID: 106069] [glusterd-snapshot-utils.c:1678:glusterd_import_friend_snap] 0-management: Failed to generate snap vol files for snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 [2018-10-18 10:44:18.607819] E [MSGID: 106100] [glusterd-snapshot-utils.c:2106:glusterd_update_snaps_synctask] 0-management: Failed to import snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 from peer servera [2018-10-18 10:44:18.664472] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2018-10-18 10:44:18.664613] I [socket.c:4315:socket_init] 0-snapd: SSL support for glusterd is ENABLED [2018-10-18 10:44:18.664776] E [socket.c:4383:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2018-10-18 10:44:18.665723] E [MSGID: 106029] [glusterd-volgen.c:974:volgen_write_volfile] 0-management: file creation failed [No such file or directory] [2018-10-18 10:44:18.665753] E [MSGID: 106069] [glusterd-volgen.c:1020:volgen_write_volfile] 0-management: failed to create volfile /var/lib/glusterd/vols/replvol/replvol-snapd.vol [2018-10-18 10:44:18.665775] E [MSGID: 106114] [glusterd-snapd-svc.c:175:glusterd_snapdsvc_manager] 0-management: Couldn't create snapd volfile for volume: replvol [2018-10-18 10:44:18.693863] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory [2018-10-18 10:44:18.712003] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2018-10-18 10:44:18.712208] I [socket.c:4315:socket_init] 0-nfs: SSL support for glusterd is ENABLED [2018-10-18 10:44:18.712392] E [socket.c:4383:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory" repeated 5 times between [2018-10-18 10:44:18.693863] and [2018-10-18 10:44:18.710617] [2018-10-18 10:44:18.713052] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped [2018-10-18 10:44:18.713091] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped [2018-10-18 10:44:18.713106] I [MSGID: 106600] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed [2018-10-18 10:44:18.713147] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2018-10-18 10:44:18.713228] I [socket.c:4315:socket_init] 0-glustershd: SSL support for glusterd is ENABLED [2018-10-18 10:44:18.713343] E [socket.c:4383:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2018-10-18 10:44:18.715967] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory [2018-10-18 10:44:18.718305] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory [2018-10-18 10:44:18.720532] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped [2018-10-18 10:44:18.720636] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped [2018-10-18 10:44:18.720722] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service [2018-10-18 10:44:19.736067] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2018-10-18 10:44:19.736205] I [socket.c:4315:socket_init] 0-quotad: SSL support for glusterd is ENABLED [2018-10-18 10:44:19.736312] E [socket.c:4383:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2018-10-18 10:44:19.737481] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped [2018-10-18 10:44:19.737504] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped [2018-10-18 10:44:19.737533] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting quotad service [2018-10-18 10:44:20.746148] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2018-10-18 10:44:20.746274] I [socket.c:4315:socket_init] 0-bitd: SSL support for glusterd is ENABLED [2018-10-18 10:44:20.746388] E [socket.c:4383:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2018-10-18 10:44:20.747092] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped [2018-10-18 10:44:20.747120] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped [2018-10-18 10:44:20.747159] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2018-10-18 10:44:20.747239] I [socket.c:4315:socket_init] 0-scrub: SSL support for glusterd is ENABLED [2018-10-18 10:44:20.747323] E [socket.c:4383:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled [2018-10-18 10:44:20.747955] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped [2018-10-18 10:44:20.747978] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped [2018-10-18 10:44:20.748550] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549 [2018-10-18 10:44:20.786840] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 [2018-10-18 10:44:20.836517] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: e88e454a-f85a-472a-8920-a541c8615d03 [2018-10-18 10:44:20.839057] E [MSGID: 106010] [glusterd-utils.c:3383:glusterd_compare_friend_volume] 0-management: Version of Cksums replvol differ. local cksum = 3139705045, remote cksum 3139442900 on peer serverb.lab.local [2018-10-18 10:44:20.839450] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to serverb.lab.local (0), ret: 0, op_ret: -1 [2018-10-18 10:44:22.822516] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [root at serverc glusterd]# -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181018/bb6c765f/attachment.html>