Tom-
That looks like the same problem I had after upgrading from GlusterFS
3.2.6 to 3.3. Here is a link to a thread that includes the diagnosis
(missing username and password entries in vol files) and solution
suggested by Rahul Hinduja.
http://gluster.org/pipermail/gluster-users/2012-August/011055.html
-Dan.
On 09/15/2012 02:32 AM, gluster-users-request at gluster.org
wrote:> Message: 2
> Date: Fri, 14 Sep 2012 20:57:25 +0000
> From: Thomas Indelli<tindelli at uchicago.edu>
> Subject: [Gluster-users] Problems adding new server to pool
> To: "gluster-users at gluster.org"<gluster-users at
gluster.org>
> Message-ID:
> <A4C9147B95CDDC4097015BC5551AEB6E0473AE80 at
xm-mbx-01-prod.ad.uchicago.edu>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Hello,
>
> I don't have a great deal of experience yet with Gluster, and I'm
having some tremendous difficulties adding an additional (3rd) server to the
pool I have set up. This is running Gluster 3.3.0 on RHEL 6.3, using RPMs:
> # rpm -qa | grep gluster
> glusterfs-rdma-3.3.0-1.el6.x86_64
> glusterfs-server-3.3.0-1.el6.x86_64
> glusterfs-3.3.0-1.el6.x86_64
> glusterfs-fuse-3.3.0-1.el6.x86_64
>
> When I first set it up, I was able to add the second server without an
issue using gluster peer probe vothgluster2, however when I do the same here, it
just hangs for a couple minutes, then returns to the prompt without successfully
probing the server.
>> From the first server:
> # gluster volume info
>
> Volume Name: home
> Type: Distribute
> Volume ID: a6f62691-a876-431a-a941-d3b76e197eeb
> Status: Started
> Number of Bricks: 4
> Transport-type: tcp,rdma
> Bricks:
> Brick1: vothgluster1:/export/brick1
> Brick2: vothgluster1:/export/brick2
> Brick3: vothgluster2:/export/brick1
> Brick4: vothgluster2:/export/brick2
> Options Reconfigured:
> nfs.rpc-auth-allow:
128.135.10.*,205.208.111.*,128.135.233.*,128.135.186.*,10.135.10.*,205.208.76.203
> nfs.disable: on
> nfs.register-with-portmap: on
> performance.io-cache: on
> nfs.port: 2049
> features.quota: off
> performance.cache-size: 64MB
> performance.read-ahead: off
> performance.write-behind-window-size: 1MB
> nfs.ports-insecure: on
>
> # gluster peer status
> Number of Peers: 2
>
> Hostname: vothgluster2
> Uuid: 44033fcb-f6da-4491-b860-d05bb3ce23f7
> State: Peer in Cluster (Connected)
>
> Hostname: vothgluster3
> Uuid: 924322ef-5e55-427b-a393-90903ccb7d16
> State: Establishing Connection (Connected)
>> From the second server, gluster volume info reports the same.
> # gluster peer status
> Number of Peers: 1
>
> Hostname: 205.208.111.14
> Uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47
> State: Peer in Cluster (Connected)
>> From the new (problematic) server:
> # gluster volume info
> No volumes present
>
> # gluster peer status
> Number of Peers: 1
>
> Hostname: 205.208.111.14
> Uuid: 00000000-0000-0000-0000-000000000000
> State: Connected to Peer (Connected)
>
> Forward and Reverse DNS lookups between all 3 servers work fine. Ping
works fine. IB connectivity works fine (using tcp,rdma transport type).
>
> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on vothgluster3
> [2012-09-14 15:16:27.528532] I [glusterfsd.c:1666:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.3.0
> [2012-09-14 15:16:27.530778] I [glusterd.c:807:init] 0-management: Using
/var/lib/glusterd as working directory
> [2012-09-14 15:16:28.088422] I [glusterd.c:95:glusterd_uuid_init]
0-glusterd: retrieved UUID: 924322ef-5e55-427b-a393-90903ccb7d16
> [2012-09-14 15:16:28.097455] I
[glusterd.c:265:glusterd_check_gsync_present] 0-glusterd: geo-replication module
not installed in the system
> Given volfile:
>
+------------------------------------------------------------------------------+
> 1: volume management
> 2: type mgmt/glusterd
> 3: option working-directory /var/lib/glusterd
> 4: option transport-type socket,rdma
> 5: option transport.socket.keepalive-time 10
> 6: option transport.socket.keepalive-interval 2
> 7: option transport.socket.read-fail-log off
> 8: end-volume
>
>
+------------------------------------------------------------------------------+
> [2012-09-14 15:16:32.760625] I
[glusterd-handler.c:1753:glusterd_handle_probe_query] 0-glusterd: Received probe
from uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47
> [2012-09-14 15:16:32.760707] I
[glusterd-handler.c:411:glusterd_friend_find] 0-glusterd: Unable to find peer by
uuid
> [2012-09-14 15:16:32.760929] I
[glusterd-handler.c:423:glusterd_friend_find] 0-glusterd: Unable to find
hostname: 205.208.111.14
> [2012-09-14 15:16:32.760950] I
[glusterd-handler.c:1767:glusterd_handle_probe_query] 0-glusterd: Unable to find
peerinfo for host: 205.208.111.14 (24007)
> [2012-09-14 15:16:32.766007] I
[glusterd-handler.c:2204:glusterd_friend_add] 0-management: connect returned 0
> [2012-09-14 15:16:32.766067] I
[glusterd-handler.c:1793:glusterd_handle_probe_query] 0-glusterd: Responded to
205.208.111.14, op_ret: 0, op_errno: 0, ret: 0
> [2012-09-14 15:16:32.766572] I
[glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using Program
glusterd mgmt, Num (1238433), Version (2)
> [2012-09-14 15:16:32.766620] I
[glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using Program Peer
mgmt, Num (1238437), Version (2)
> [2012-09-14 15:18:44.916221] I
[glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received
cli list req
> [2012-09-14 15:19:41.020394] I
[glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get
vol req
> [2012-09-14 15:22:40.807457] I
[glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get
vol req
> [2012-09-14 15:22:48.655202] I
[glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received
cli list req
>
> -----
>
> Please let me know any other information to provide.
>
> Thanks in advance,
> Tom
>
>
>
> --
> Tom Indelli
> tindelli at uchicago.edu<mailto:tindelli at uchicago.edu>
> Systems Administrator
> Physical Sciences Division
> The University of Chicago