thr3ads.net - Gluster users - [Gluster-users] brick addition failed when volume is changed from distributed to replicated [Aug 2014]

If this information is useful, please help other people find it:
Share via:

Suvendu Mitra

2014-Aug-07 12:07 UTC

[Gluster-users] brick addition failed when volume is changed from distributed to replicated

Hi,
Following command failed without any proper error. Please find the attached
tar which contain strace output + glusterfs log files.

*# gluster volume add-brick mstate replica 2
169.254.0.91:/mnt/bricks/mstate/brick force*
*volume add-brick: failed: *

*version -- glusterfs 3.4.2*
We have started glusterd by "--log-level=DEBUG" but no traceable
error.
>From '/var/log/glusterfs/bricks/mnt-bricks-mstate-brick.log" we can
foundfollowing error

*[2014-08-07 11:31:40.739136] E [posix.c:2855:posix_fgetxattr]
0-mstate-posix: fgetxattr failed on key trusted.glusterfs.dht.linkto (No
data available)*
*[2014-08-07 11:31:40.739159] I
[server-rpc-fops.c:862:server_fgetxattr_cbk] 0-mstate-server: 23122:
FGETXATTR 0 (43da77f0-6137-451e-85f6-7b066bbee53a)
(trusted.glusterfs.dht.linkto) ==> (No data available)*

-----------
Seems peer into the system

# gluster peer status
Number of Peers: 1

Hostname: 169.254.0.91
Port: 24007
Uuid: 7301bb4b-5c4b-4f20-b0d4-da977adf1fb0
State: Peer in Cluster (Connected)
[root at SN-0(BUNGABUNGA) /root]

--------------
Also '/var/log/glusterfs/etc-glusterfs-glusterd.vol.log' has no error
information

173380 [2014-08-07 11:31:53.095289] I
[glusterd-brick-ops.c:193:gd_addbr_validate_replica_count] 0-management:
Changing the type of volume mstate from 'distribute' to
'replica'
173381 [2014-08-07 11:31:53.095306] D [glusterd-utils.c:340:glusterd_lock]
0-management: Cluster lock held by 3ba43bbc-608d-4ba4-9b99-e3c65f2a8f65
173382 [2014-08-07 11:31:53.096566] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
173383 [2014-08-07 11:31:53.096589] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173384 [2014-08-07 11:31:53.096615] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
173385 [2014-08-07 11:31:53.096631] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173386 [2014-08-07 11:31:53.096648] D
[glusterd-utils.c:5223:glusterd_is_rb_started] 0-: is_rb_started:status=0
173387 [2014-08-07 11:31:53.096663] D
[glusterd-utils.c:5232:glusterd_is_rb_paused] 0-: is_rb_paused:status=0
173388 [2014-08-07 11:31:53.096685] D
[glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
173389 [2014-08-07 11:31:53.096703] D
[glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
173390 [2014-08-07 11:31:53.096719] D
[glusterd-utils.c:665:glusterd_volinfo_new] 0-management: Returning 0
173391 [2014-08-07 11:31:53.096737] D
[glusterd-utils.c:749:glusterd_volume_brickinfos_delete] 0-management:
Returning 0
173392 [2014-08-07 11:31:53.096750] D
[glusterd-utils.c:777:glusterd_volinfo_delete] 0-management: Returning 0
173393 [2014-08-07 11:31:53.096759] D
[glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
173394 [2014-08-07 11:31:53.096768] D
[glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
173395 [2014-08-07 11:31:53.096784] D
[glusterd-utils.c:665:glusterd_volinfo_new] 0-management: Returning 0
173396 [2014-08-07 11:31:53.096795] D
[glusterd-utils.c:749:glusterd_volume_brickinfos_delete] 0-management:
Returning 0
173397 [2014-08-07 11:31:53.096802] D
[glusterd-utils.c:777:glusterd_volinfo_delete] 0-management: Returning 0
173398 [2014-08-07 11:31:53.096810] D
[glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
173399 [2014-08-07 11:31:53.096817] D
[glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
173400 [2014-08-07 11:31:53.096825] D
[glusterd-utils.c:4980:glusterd_friend_find_by_hostname] 0-management:
Friend 169.254.0.91 found.. state: 3
173401 [2014-08-07 11:31:53.096833] D
[glusterd-utils.c:5064:glusterd_hostname_to_uuid] 0-management: returning 0
173402 [2014-08-07 11:31:53.096840] D
[glusterd-utils.c:819:glusterd_resolve_brick] 0-management: Returning 0
173403 [2014-08-07 11:31:53.096847] D
[glusterd-utils.c:4936:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
173404 [2014-08-07 11:31:53.096854] D
[glusterd-utils.c:5215:glusterd_new_brick_validate] 0-management: returning
0
173405 [2014-08-07 11:31:53.096862] D
[glusterd-brick-ops.c:1177:glusterd_op_stage_add_brick] 0-management:
Returning 0
173406 [2014-08-07 11:31:53.096868] D
[glusterd-op-sm.c:3975:glusterd_op_stage_validate] 0-management: Returning 0
173407 [2014-08-07 11:31:53.098922] D
[glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management:
Returning 0
173408 [2014-08-07 11:31:53.099745] D [socket.c:486:__socket_rwv]
0-socket.management: EOF on socket
173409 [2014-08-07 11:31:53.099764] D [socket.c:2236:socket_event_handler]
0-transport: disconnecting now
173410 [2014-08-07 11:33:23.223682] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173411 [2014-08-07 11:33:23.224223] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173412 [2014-08-07 11:33:23.224257] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume services found
173413 [2014-08-07 11:33:23.224268] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173414 [2014-08-07 11:33:23.224627] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173415 [2014-08-07 11:33:23.224649] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume home found
173416 [2014-08-07 11:33:23.224657] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173417 [2014-08-07 11:33:23.225033] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173418 [2014-08-07 11:33:23.225056] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume config found
173419 [2014-08-07 11:33:23.225064] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173420 [2014-08-07 11:33:23.225456] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173421 [2014-08-07 11:33:23.225478] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume log found
173422 [2014-08-07 11:33:23.225486] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173423 [2014-08-07 11:33:23.225838] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
173424 [2014-08-07 11:33:23.225860] D
[glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
173425 [2014-08-07 11:33:23.225868] D
[glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
173426 [2014-08-07 11:33:23.226477] D [socket.c:486:__socket_rwv]
0-socket.management: EOF on socket
173427 [2014-08-07 11:33:23.226497] D [socket.c:2236:socket_event_handler]
0-transport: disconnecting now


-- 
Suvendu Mitra
GSM - +358504821066
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140807/71af51c4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: issue.tar.gz
Type: application/x-gzip
Size: 1956765 bytes
Desc: not available
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140807/71af51c4/attachment-0001.gz>

Juan José Pavlik Salles

2014-Aug-07 13:57 UTC

head link

[Gluster-users] brick addition failed when volume is changed from distributed to replicated

Hi Suvendu, I think you have to add two bricks every time you want to
add-brick to a replicate 2 volume. You can't add just one brick, you have
to add two.


2014-08-07 9:07 GMT-03:00 Suvendu Mitra <suvendu.mitra at gmail.com>:
> Hi,
> Following command failed without any proper error. Please find the
> attached tar which contain strace output + glusterfs log files.
>
> *# gluster volume add-brick mstate replica 2
> 169.254.0.91:/mnt/bricks/mstate/brick force*
> *volume add-brick: failed: *
>
> *version -- glusterfs 3.4.2*
> We have started glusterd by "--log-level=DEBUG" but no traceable
error.
>
> From '/var/log/glusterfs/bricks/mnt-bricks-mstate-brick.log" we
can found
> following error
>
> *[2014-08-07 11:31:40.739136] E [posix.c:2855:posix_fgetxattr]
> 0-mstate-posix: fgetxattr failed on key trusted.glusterfs.dht.linkto (No
> data available)*
> *[2014-08-07 11:31:40.739159] I
> [server-rpc-fops.c:862:server_fgetxattr_cbk] 0-mstate-server: 23122:
> FGETXATTR 0 (43da77f0-6137-451e-85f6-7b066bbee53a)
> (trusted.glusterfs.dht.linkto) ==> (No data available)*
>
> -----------
> Seems peer into the system
>
> # gluster peer status
> Number of Peers: 1
>
> Hostname: 169.254.0.91
> Port: 24007
> Uuid: 7301bb4b-5c4b-4f20-b0d4-da977adf1fb0
> State: Peer in Cluster (Connected)
> [root at SN-0(BUNGABUNGA) /root]
>
> --------------
> Also '/var/log/glusterfs/etc-glusterfs-glusterd.vol.log' has no
error
> information
>
> 173380 [2014-08-07 11:31:53.095289] I
> [glusterd-brick-ops.c:193:gd_addbr_validate_replica_count] 0-management:
> Changing the type of volume mstate from 'distribute' to
'replica'
> 173381 [2014-08-07 11:31:53.095306] D [glusterd-utils.c:340:glusterd_lock]
> 0-management: Cluster lock held by 3ba43bbc-608d-4ba4-9b99-e3c65f2a8f65
> 173382 [2014-08-07 11:31:53.096566] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
> 173383 [2014-08-07 11:31:53.096589] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173384 [2014-08-07 11:31:53.096615] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
> 173385 [2014-08-07 11:31:53.096631] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173386 [2014-08-07 11:31:53.096648] D
> [glusterd-utils.c:5223:glusterd_is_rb_started] 0-: is_rb_started:status=0
> 173387 [2014-08-07 11:31:53.096663] D
> [glusterd-utils.c:5232:glusterd_is_rb_paused] 0-: is_rb_paused:status=0
> 173388 [2014-08-07 11:31:53.096685] D
> [glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
> 173389 [2014-08-07 11:31:53.096703] D
> [glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
> Returning 0
> 173390 [2014-08-07 11:31:53.096719] D
> [glusterd-utils.c:665:glusterd_volinfo_new] 0-management: Returning 0
> 173391 [2014-08-07 11:31:53.096737] D
> [glusterd-utils.c:749:glusterd_volume_brickinfos_delete] 0-management:
> Returning 0
> 173392 [2014-08-07 11:31:53.096750] D
> [glusterd-utils.c:777:glusterd_volinfo_delete] 0-management: Returning 0
> 173393 [2014-08-07 11:31:53.096759] D
> [glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
> 173394 [2014-08-07 11:31:53.096768] D
> [glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
> Returning 0
> 173395 [2014-08-07 11:31:53.096784] D
> [glusterd-utils.c:665:glusterd_volinfo_new] 0-management: Returning 0
> 173396 [2014-08-07 11:31:53.096795] D
> [glusterd-utils.c:749:glusterd_volume_brickinfos_delete] 0-management:
> Returning 0
> 173397 [2014-08-07 11:31:53.096802] D
> [glusterd-utils.c:777:glusterd_volinfo_delete] 0-management: Returning 0
> 173398 [2014-08-07 11:31:53.096810] D
> [glusterd-utils.c:803:glusterd_brickinfo_new] 0-management: Returning 0
> 173399 [2014-08-07 11:31:53.096817] D
> [glusterd-utils.c:865:glusterd_brickinfo_new_from_brick] 0-management:
> Returning 0
> 173400 [2014-08-07 11:31:53.096825] D
> [glusterd-utils.c:4980:glusterd_friend_find_by_hostname] 0-management:
> Friend 169.254.0.91 found.. state: 3
> 173401 [2014-08-07 11:31:53.096833] D
> [glusterd-utils.c:5064:glusterd_hostname_to_uuid] 0-management: returning 0
> 173402 [2014-08-07 11:31:53.096840] D
> [glusterd-utils.c:819:glusterd_resolve_brick] 0-management: Returning 0
> 173403 [2014-08-07 11:31:53.096847] D
> [glusterd-utils.c:4936:glusterd_friend_find_by_uuid] 0-management: Friend
> found... state: Peer in Cluster
> 173404 [2014-08-07 11:31:53.096854] D
> [glusterd-utils.c:5215:glusterd_new_brick_validate] 0-management: returning
> 0
> 173405 [2014-08-07 11:31:53.096862] D
> [glusterd-brick-ops.c:1177:glusterd_op_stage_add_brick] 0-management:
> Returning 0
> 173406 [2014-08-07 11:31:53.096868] D
> [glusterd-op-sm.c:3975:glusterd_op_stage_validate] 0-management: Returning
0
> 173407 [2014-08-07 11:31:53.098922] D
> [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management:
> Returning 0
> 173408 [2014-08-07 11:31:53.099745] D [socket.c:486:__socket_rwv]
> 0-socket.management: EOF on socket
> 173409 [2014-08-07 11:31:53.099764] D [socket.c:2236:socket_event_handler]
> 0-transport: disconnecting now
> 173410 [2014-08-07 11:33:23.223682] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173411 [2014-08-07 11:33:23.224223] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173412 [2014-08-07 11:33:23.224257] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume services found
> 173413 [2014-08-07 11:33:23.224268] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173414 [2014-08-07 11:33:23.224627] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173415 [2014-08-07 11:33:23.224649] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume home found
> 173416 [2014-08-07 11:33:23.224657] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173417 [2014-08-07 11:33:23.225033] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173418 [2014-08-07 11:33:23.225056] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume config found
> 173419 [2014-08-07 11:33:23.225064] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173420 [2014-08-07 11:33:23.225456] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173421 [2014-08-07 11:33:23.225478] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume log found
> 173422 [2014-08-07 11:33:23.225486] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173423 [2014-08-07 11:33:23.225838] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> 173424 [2014-08-07 11:33:23.225860] D
> [glusterd-utils.c:1209:glusterd_volinfo_find] 0-: Volume mstate found
> 173425 [2014-08-07 11:33:23.225868] D
> [glusterd-utils.c:1216:glusterd_volinfo_find] 0-: Returning 0
> 173426 [2014-08-07 11:33:23.226477] D [socket.c:486:__socket_rwv]
> 0-socket.management: EOF on socket
> 173427 [2014-08-07 11:33:23.226497] D [socket.c:2236:socket_event_handler]
> 0-transport: disconnecting now
>
>
> --
> Suvendu Mitra
> GSM - +358504821066
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pavlik Salles Juan Jos?
Blog - http://viviendolared.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140807/6dee9cb1/attachment.html>

Gluster users - Aug 2014 - brick addition failed when volume is changed from distributed to replicated

[Gluster-users] brick addition failed when volume is changed from distributed to replicated

[Gluster-users] brick addition failed when volume is changed from distributed to replicated