Hello,
while playing around with the new elastic glusterfs system (via
'glusterd', previously have been using glusterfs with static
configuration) I have stumbled upon such problem:
1. I have a test system with 12 nodes in a replicated/distributed way (replica
count 3):
Volume Name: storage
Type: Distributed-Replicate
Status: Started
Number of Bricks: 4 x 3 = 12
Transport-type: tcp
2. One of the brick systems/servers had a simulated hardware failure (disks have
been wiped) and restarted a new.
3. When the server ('glusterd') came up the rest of the bricks received
something like:
Jul 25 17:10:45 snode182 GlusterFS[3371]: [2011-07-25 17:10:45.435786] C
[glusterd-rpc-ops.c:748:glusterd3_1_cluster_lock_cbk] 0-:
Lock response received from unknown peer: 4ecec354-1d02-4709-8f1e-607a735dbe62
Obviously the peer UID in glusterd.info (because of the full
"crash/reinstall") seems to be different from the UID which is in the
cluster configuration.
Peer status shows:
Hostname: 10.0.0.149
Uuid: f9ea651e-68da-40fa-80d9-6bee7779aa97
State: Peer Rejected (Connected)
4. While the info commands work fine anything that involves changing the volume
settings return that the volume doesn't exist (from
the logs seem to be coming from the reinstalled node):
[2011-07-25 17:08:54.579631] E
[glusterd-op-sm.c:1237:glusterd_op_stage_set_volume] 0-: Volume storage does not
exist
[2011-07-25 17:08:54.579769] E [glusterd-op-sm.c:7107:glusterd_op_ac_stage_op]
0-: Validate failed: -1
So my question is how to correctly reintroduce the box to the glusterfs cluster
since:
1. I can't run 'peer probe 10.0.0.149' as gluster says the peer is
already in cluster
2. I can't remove the peer because it is part of a volume.
3. I can't remove the brick from the volume because gluster asks me to
remove 3 bricks (eg the replica count and also would mean
data loss).
4. I imagine that the replace-brick won't work even if I fake the new node
with a different ip/hostname (since the source brick will
be down) or will it replicate from the alive ones?
I tried just to manually change the UID back to the the one which is listed in
the rest of the nodes (peer status) but apparently it
was not enough (the node itself didn't see any other servers and wasn't
able to sync volume information from remote brick(s)
complaining that it is not his friend).
Then when I manually placed all the peers/* files from a running bricknode and
restarted glusterd the node reverted to 'Peer in
Cluster' state.
Is this the way?
Or am I doing something totally wrong?
wbr
rr