Gluster users - Mar 2014 - glusterd service fails to start from AWS AMI

Hello all.

I have a working replica 2 cluster (4 nodes) up and running happily over Amazon
EC2.  My end goal is to create AMIs of each machine and then quickly reproduce
the same, but new, cluster from those AMIs.  Essentially, I'd like a cluster
"template".

-Assigned original instances' Elastic IPs to new machines to reduce
resolution issues.
-Passwordless SSH works on initial boot across all machines
-Node1: has no evident issue.  Starts with glusterd running.
-Node1: 'gluster peer status' returns correct public DNS / hostnames for
peer nodes.  Status: (Disconnected)  --since the service is off on them

Since my goal is to create a cluster template, reinstalling gluster for each
node, though it'll probably work, isn't an optimal solution.

Thank You

#  Node2: etc-glusterfs-glusterd.vol.log
#  Begins at 'service glusterd start' command entry

[2014-03-04 21:20:30.532138] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterd:
Started running /usr/sbin/glusterd version 3.4.0.44rhs (/usr/sbin/glusterd
--pid-file=/var/run/glusterd.pid)
[2014-03-04 21:20:30.539331] I [glusterd.c:1020:init] 0-management: Using
/var/lib/glusterd as working directory
[2014-03-04 21:20:30.542578] I [socket.c:3485:socket_init] 0-socket.management:
SSL support is NOT enabled
[2014-03-04 21:20:30.542603] I [socket.c:3500:socket_init] 0-socket.management:
using system polling thread
[2014-03-04 21:20:30.543203] C [rdma.c:4099:gf_rdma_init] 0-rpc-transport/rdma:
Failed to get IB devices
[2014-03-04 21:20:30.543342] E [rdma.c:4990:init] 0-rdma.management: Failed to
initialize IB Device
[2014-03-04 21:20:30.543375] E [rpc-transport.c:320:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2014-03-04 21:20:30.543471] W [rpcsvc.c:1387:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2014-03-04 21:20:37.116571] I
[glusterd-store.c:1388:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
[2014-03-04 21:20:37.120082] E
[glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2014-03-04 21:20:37.120118] E
[glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2014-03-04 21:20:37.120137] E
[glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2014-03-04 21:20:37.120154] E
[glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2014-03-04 21:20:37.761785] I [glusterd-handler.c:2886:glusterd_friend_add]
0-management: connect returned 0
[2014-03-04 21:20:37.765059] I [glusterd-handler.c:2886:glusterd_friend_add]
0-management: connect returned 0
[2014-03-04 21:20:37.767677] I [glusterd-handler.c:2886:glusterd_friend_add]
0-management: connect returned 0
[2014-03-04 21:20:37.767783] I [rpc-clnt.c:974:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.767850] I [socket.c:3485:socket_init] 0-management: SSL
support is NOT enabled
[2014-03-04 21:20:37.767866] I [socket.c:3500:socket_init] 0-management: using
system polling thread
[2014-03-04 21:20:37.772356] I [rpc-clnt.c:974:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.772441] I [socket.c:3485:socket_init] 0-management: SSL
support is NOT enabled
[2014-03-04 21:20:37.772459] I [socket.c:3500:socket_init] 0-management: using
system polling thread
[2014-03-04 21:20:37.776131] I [rpc-clnt.c:974:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.776185] I [socket.c:3485:socket_init] 0-management: SSL
support is NOT enabled
[2014-03-04 21:20:37.776201] I [socket.c:3500:socket_init] 0-management: using
system polling thread
[2014-03-04 21:20:37.780363] E
[glusterd-store.c:2548:glusterd_resolve_all_bricks] 0-glusterd: resolve brick
failed in restore
[2014-03-04 21:20:37.780395] E [xlator.c:423:xlator_init] 0-management:
Initialization of volume 'management' failed, review your volfile again
[2014-03-04 21:20:37.780410] E [graph.c:292:glusterfs_graph_init] 0-management:
initializing translator failed
[2014-03-04 21:20:37.780422] E [graph.c:479:glusterfs_graph_activate] 0-graph:
init failed
[2014-03-04 21:20:37.780723] W [glusterfsd.c:1097:cleanup_and_exit]
(-->/usr/sbin/glusterd(main+0x6b1) [0x406a91]
(-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x405247]
(-->/usr/sbin/glusterd(glusterfs_process_volfp+0x106) [0x405156]))) 0-:
received signum (0), shutting down

I don't want to sound simplistic, but seems to be name resolution/network
related.

Again, I DO know you email ends with redhat.com, but just to make sure,
Gluster is running on what distro ? I never dealt with Amazon's platform,
so ignorance here is abundant.

The reason why I am asking is that I am stress-testing my first (on
premisses) install, and I ran into a problem that I am choosing to ignore
for now, but will have to solve in the future: DNS resolution stops working
after a while.

I am using CentOS 6.5, with Gluster 3.4.2. I have a bonded NIC, made out of
two physical ones and a third NIC for management.

I realized that, despite the fact that I have manually configured all
interfaces, disabled user control (may be this), disabled NM access to
them, and even tried to update resolv.conf, after a reboot, name resolution
does not work.

While the NICs were working with NM and/or DHCP, all went file, but after
tailoring my ifcfg-* files, DNS went south.

You said your name resolution does work. Maybe an entry on your hosts file
just to test ?

Another thought would be using 3.4.2, instead of 3.4.0.

Just wanted to share.

KR,

Carlos


On Tue, Mar 4, 2014 at 10:45 PM, Jon Cope <jcope at redhat.com> wrote:
> Hello all.
>
> I have a working replica 2 cluster (4 nodes) up and running happily over
> Amazon EC2.  My end goal is to create AMIs of each machine and then quickly
> reproduce the same, but new, cluster from those AMIs.  Essentially, I'd
> like a cluster "template".
>
> -Assigned original instances' Elastic IPs to new machines to reduce
> resolution issues.
> -Passwordless SSH works on initial boot across all machines
> -Node1: has no evident issue.  Starts with glusterd running.
> -Node1: 'gluster peer status' returns correct public DNS /
hostnames for
> peer nodes.  Status: (Disconnected)  --since the service is off on them
>
> Since my goal is to create a cluster template, reinstalling gluster for
> each node, though it'll probably work, isn't an optimal solution.
>
> Thank You
>
> #  Node2: etc-glusterfs-glusterd.vol.log
> #  Begins at 'service glusterd start' command entry
>
> [2014-03-04 21:20:30.532138] I [glusterfsd.c:2024:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version
> 3.4.0.44rhs (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
> [2014-03-04 21:20:30.539331] I [glusterd.c:1020:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2014-03-04 21:20:30.542578] I [socket.c:3485:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2014-03-04 21:20:30.542603] I [socket.c:3500:socket_init]
> 0-socket.management: using system polling thread
> [2014-03-04 21:20:30.543203] C [rdma.c:4099:gf_rdma_init]
> 0-rpc-transport/rdma: Failed to get IB devices
> [2014-03-04 21:20:30.543342] E [rdma.c:4990:init] 0-rdma.management:
> Failed to initialize IB Device
> [2014-03-04 21:20:30.543375] E [rpc-transport.c:320:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2014-03-04 21:20:30.543471] W [rpcsvc.c:1387:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2014-03-04 21:20:37.116571] I
> [glusterd-store.c:1388:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 2
> [2014-03-04 21:20:37.120082] E
> [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-0
> [2014-03-04 21:20:37.120118] E
> [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-1
> [2014-03-04 21:20:37.120137] E
> [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-2
> [2014-03-04 21:20:37.120154] E
> [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-3
> [2014-03-04 21:20:37.761785] I
> [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect
> returned 0
> [2014-03-04 21:20:37.765059] I
> [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect
> returned 0
> [2014-03-04 21:20:37.767677] I
> [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect
> returned 0
> [2014-03-04 21:20:37.767783] I [rpc-clnt.c:974:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-03-04 21:20:37.767850] I [socket.c:3485:socket_init] 0-management:
> SSL support is NOT enabled
> [2014-03-04 21:20:37.767866] I [socket.c:3500:socket_init] 0-management:
> using system polling thread
> [2014-03-04 21:20:37.772356] I [rpc-clnt.c:974:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-03-04 21:20:37.772441] I [socket.c:3485:socket_init] 0-management:
> SSL support is NOT enabled
> [2014-03-04 21:20:37.772459] I [socket.c:3500:socket_init] 0-management:
> using system polling thread
> [2014-03-04 21:20:37.776131] I [rpc-clnt.c:974:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-03-04 21:20:37.776185] I [socket.c:3485:socket_init] 0-management:
> SSL support is NOT enabled
> [2014-03-04 21:20:37.776201] I [socket.c:3500:socket_init] 0-management:
> using system polling thread
> [2014-03-04 21:20:37.780363] E
> [glusterd-store.c:2548:glusterd_resolve_all_bricks] 0-glusterd: resolve
> brick failed in restore
> [2014-03-04 21:20:37.780395] E [xlator.c:423:xlator_init] 0-management:
> Initialization of volume 'management' failed, review your volfile
again
> [2014-03-04 21:20:37.780410] E [graph.c:292:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2014-03-04 21:20:37.780422] E [graph.c:479:glusterfs_graph_activate]
> 0-graph: init failed
> [2014-03-04 21:20:37.780723] W [glusterfsd.c:1097:cleanup_and_exit]
> (-->/usr/sbin/glusterd(main+0x6b1) [0x406a91]
> (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x405247]
> (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x106) [0x405156]))) 0-:
> received signum (0), shutting down
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140304/9d04cb03/attachment.html>