Hello All, I need to migrate some data from a pair of mirrored bricks (i.e. a brick and it's mirror). Unfortunately the "... replace-brick ... start" and "... replace-brick ... status" commands give a null response. The commands I used followed by the relevant log file errors are shown below. The bdan11 server (where I want the data to go) is new and doesn't have any data on it at the moment. However glusterd is running and the host is part of the peer group. The fuse module is loaded on all servers. I tried restarting all the gluster daemons on all the servers but it didn't make any difference. I also tried a similar operation on another volume using other servers (all with existing GlusterFS bricks this time) with the same result. Any suggestions would be much appreciated. [root at bdan0 ~]# gluster volume replace-brick atmos bdan5:/vg2lv1/glusterfs2 bdan11:/atmos/glusterfs start [root at bdan0 ~]# gluster volume replace-brick atmos bdan5:/vg2lv1/glusterfs2 bdan11:/atmos/glusterfs status [root at bdan0 ~]# ---- on bdan11 (where I want the data to go to) --- [2011-02-14 16:57:52.296610] I [glusterd-handler.c:673:glusterd_handle_cli_list_friends] glusterd: Received cli list req [2011-02-14 16:58:21.56781] I [glusterd-handler.c:425:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:21.56828] I [glusterd-utils.c:236:glusterd_lock] glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:21.56868] I [glusterd-handler.c:1973:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0 [2011-02-14 16:58:21.96404] I [glusterd-handler.c:464:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:21.96466] I [glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: bdan5:/vg2lv1/glusterfs2 [2011-02-14 16:58:21.99679] I [glusterd-utils.c:2144:glusterd_friend_find_by_hostname] glusterd: Friend bdan5 found.. state: 3 [2011-02-14 16:58:21.99720] I [glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick [2011-02-14 16:58:21.102568] I [glusterd-handler.c:2065:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0 ---- on bdan5 (current location of data) --- [2011-02-14 16:58:59.159290] I [glusterd-handler.c:425:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:59.159567] I [glusterd-utils.c:236:glusterd_lock] glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:59.159621] I [glusterd-handler.c:1973:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0 [2011-02-14 16:58:59.198752] I [glusterd-handler.c:464:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:59.198799] I [glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: bdan5:/vg2lv1/glusterfs2 [2011-02-14 16:58:59.199123] I [glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick [2011-02-14 16:58:59.199183] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199201] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199214] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199227] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199240] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199251] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199263] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199275] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199287] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199299] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199312] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.201616] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.201667] I [glusterd-handler.c:2065:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0 ---- on bdan0 (where volume commands were executed)--- [2011-02-14 16:58:59.159586] I [glusterd-handler.c:1222:glusterd_handle_replace_brick] glusterd: Received replace brick req [2011-02-14 16:58:59.159694] I [glusterd-utils.c:236:glusterd_lock] glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317 [2011-02-14 16:58:59.159717] I [glusterd-handler.c:2832:glusterd_op_txn_begin] glusterd: Acquired local lock [2011-02-14 16:58:59.160085] I [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock req to 14 peers [2011-02-14 16:58:59.160168] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: cc81118c-ff50-4823-8e75-f710b88f9d9c [2011-02-14 16:58:59.160204] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160259] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 66f88fb4-5aa7-44c0-870f-a2d41fd88c47 [2011-02-14 16:58:59.160282] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160323] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 29566f7b-fa3a-43fd-9751-086479cedd01 [2011-02-14 16:58:59.160349] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160396] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 8e037099-d853-4fc3-a4b7-28980fd2b131 [2011-02-14 16:58:59.160419] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160468] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 814b274a-8dbe-40d2-a406-c41d347f8ad1 [2011-02-14 16:58:59.160538] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160577] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 80363690-44c6-4d86-9607-5af0764e8900 [2011-02-14 16:58:59.160600] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.160638] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 344ba772-9c6f-4263-a380-79a24fb6f741 [2011-02-14 16:58:59.160660] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.169445] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 6753c5fa-e33c-4724-930e-53efa6e95563 [2011-02-14 16:58:59.169481] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.172089] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 39422893-699c-4f4a-a1c4-e038a4c928b8 [2011-02-14 16:58:59.172121] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.172382] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 681f3a0d-4fad-4b18-a625-da747f551155 [2011-02-14 16:58:59.172406] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.177922] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: fca1393e-ee59-4b4c-9205-b3dcdd2ee1ae [2011-02-14 16:58:59.177946] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.181466] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: a2cd8e72-151a-4fb1-936d-1fa9ef9c1ca4 [2011-02-14 16:58:59.181506] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.181548] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: fa313e90-3e8e-4f4d-a266-43f34777ed07 [2011-02-14 16:58:59.181570] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.195991] I [glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 551ca306-46a9-423a-92a8-0e33ccbb3b2b [2011-02-14 16:58:59.196023] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.196072] I [glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: bdan5:/vg2lv1/glusterfs2 [2011-02-14 16:58:59.197837] I [glusterd-utils.c:2144:glusterd_friend_find_by_hostname] glusterd: Friend bdan5 found.. state: 3 [2011-02-14 16:58:59.197875] I [glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick [2011-02-14 16:58:59.197950] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.197976] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.197999] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198023] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198046] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198068] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198091] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198155] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198180] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198204] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.198228] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199251] I [glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: Friend bdan11 found.. state: 3 [2011-02-14 16:58:59.199580] I [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 14 peers [2011-02-14 16:58:59.202555] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 814b274a-8dbe-40d2-a406-c41d347f8ad1 [2011-02-14 16:58:59.202582] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.203934] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 6753c5fa-e33c-4724-930e-53efa6e95563 [2011-02-14 16:58:59.203957] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204128] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 8e037099-d853-4fc3-a4b7-28980fd2b131 [2011-02-14 16:58:59.204156] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204281] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: fca1393e-ee59-4b4c-9205-b3dcdd2ee1ae [2011-02-14 16:58:59.204303] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204528] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: fa313e90-3e8e-4f4d-a266-43f34777ed07 [2011-02-14 16:58:59.204564] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204608] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 681f3a0d-4fad-4b18-a625-da747f551155 [2011-02-14 16:58:59.204631] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204727] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 66f88fb4-5aa7-44c0-870f-a2d41fd88c47 [2011-02-14 16:58:59.204763] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204806] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 344ba772-9c6f-4263-a380-79a24fb6f741 [2011-02-14 16:58:59.204829] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.204939] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 29566f7b-fa3a-43fd-9751-086479cedd01 [2011-02-14 16:58:59.204970] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.205012] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: a2cd8e72-151a-4fb1-936d-1fa9ef9c1ca4 [2011-02-14 16:58:59.205034] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.206075] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 80363690-44c6-4d86-9607-5af0764e8900 [2011-02-14 16:58:59.206099] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.206724] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 551ca306-46a9-423a-92a8-0e33ccbb3b2b [2011-02-14 16:58:59.206747] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 16:58:59.227190] I [glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 39422893-699c-4f4a-a1c4-e038a4c928b8 [2011-02-14 16:58:59.227214] I [glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster [2011-02-14 17:07:16.354075] E [glusterd3_1-mops.c:1357:glusterd_handle_rpc_msg] : Unable to set cli op: 16 [2011-02-14 17:07:16.354152] E [glusterd-utils.c:399:glusterd_serialize_reply] : Failed to encode message [2011-02-14 17:07:16.354179] E [glusterd-utils.c:441:glusterd_submit_reply] : Failed to serialize reply [2011-02-14 17:07:16.354200] W [glusterd3_1-mops.c:1498:glusterd_handle_rpc_msg] : Returning -1 -Dan.