thr3ads.net - Gluster users - [Gluster-users] Glusterd fails to start after removing /brick1/gv2/.glusterfs [Apr 2015]

If this information is useful, please help other people find it:
Share via:

Ernie Dunbar

2015-Apr-24 18:29 UTC

[Gluster-users] Glusterd fails to start after removing /brick1/gv2/.glusterfs

I've nuked my gluster brick, configuration, and files with the intent to 
rebuild them. This is what happens when I try to start glusterd with the 
--debug option:

root at nfs1:/etc/apt/sources.list.d# glusterd --debug
[2015-04-24 18:26:07.978598] I [MSGID: 100030] [glusterfsd.c:2018:main] 
0-glusterd: Started running glusterd version 3.6.2 (args: glusterd 
--debug)
[2015-04-24 18:26:07.978685] D 
[logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting 
timer now. Timeout = 120, current buf size = 5
[2015-04-24 18:26:07.979021] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 
0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
[2015-04-24 18:26:07.982180] I [glusterd.c:1214:init] 0-management: 
Maximum allowed open file descriptors set to 65536
[2015-04-24 18:26:07.982238] I [glusterd.c:1259:init] 0-management: 
Using /var/lib/glusterd as working directory
[2015-04-24 18:26:07.982320] D 
[glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 
128
[2015-04-24 18:26:07.982532] D [rpcsvc.c:2198:rpcsvc_init] 
0-rpc-service: RPC service inited.
[2015-04-24 18:26:07.982559] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, 
Port: 0
[2015-04-24 18:26:07.982595] D [rpc-transport.c:262:rpc_transport_load] 
0-rpc-transport: attempt to load file 
/usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/socket.so
[2015-04-24 18:26:07.983291] D [socket.c:3799:socket_init] 
0-socket.management: SSL support on the I/O path is NOT enabled
[2015-04-24 18:26:07.983325] D [socket.c:3802:socket_init] 
0-socket.management: SSL support for glusterd is NOT enabled
[2015-04-24 18:26:07.983345] D [socket.c:3819:socket_init] 
0-socket.management: using system polling thread
[2015-04-24 18:26:07.983369] D [name.c:550:server_fill_address_family] 
0-socket.management: option address-family not specified, defaulting to 
inet
[2015-04-24 18:26:07.983457] D [rpc-transport.c:262:rpc_transport_load] 
0-rpc-transport: attempt to load file 
/usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/rdma.so
[2015-04-24 18:26:07.983895] D [rpc-transport.c:300:rpc_transport_load] 
0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on 
/usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/rdma.so: 
undefined symbol: reconfigure
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
[2015-04-24 18:26:07.984165] W [rdma.c:4221:__gf_rdma_ctx_create] 
0-rpc-transport/rdma: rdma_cm event channel creation failed (No such 
device)
[2015-04-24 18:26:07.984191] E [rdma.c:4519:init] 0-rdma.management: 
Failed to initialize IB Device
[2015-04-24 18:26:07.984215] E [rpc-transport.c:333:rpc_transport_load] 
0-rpc-transport: 'rdma' initialization failed
[2015-04-24 18:26:07.984280] W [rpcsvc.c:1524:rpcsvc_transport_create] 
0-rpc-service: cannot create listener, initing the transport failed
[2015-04-24 18:26:07.984310] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, 
Ver: 2, Port: 0
[2015-04-24 18:26:07.984330] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 
1238463, Ver: 2, Port: 0
[2015-04-24 18:26:07.984351] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, 
Ver: 2, Port: 0
[2015-04-24 18:26:07.984378] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 
1238433, Ver: 3, Port: 0
[2015-04-24 18:26:07.984398] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, 
Ver: 1, Port: 0
[2015-04-24 18:26:07.984418] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, 
Ver: 2, Port: 0
[2015-04-24 18:26:07.984440] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 
1239873, Ver: 1, Port: 0
[2015-04-24 18:26:07.984495] D [rpcsvc.c:2198:rpcsvc_init] 
0-rpc-service: RPC service inited.
[2015-04-24 18:26:07.984518] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, 
Port: 0
[2015-04-24 18:26:07.984544] D [rpc-transport.c:262:rpc_transport_load] 
0-rpc-transport: attempt to load file 
/usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/socket.so
[2015-04-24 18:26:07.984629] D [socket.c:3684:socket_init] 
0-socket.management: disabling nodelay
[2015-04-24 18:26:07.984657] D [socket.c:3799:socket_init] 
0-socket.management: SSL support on the I/O path is NOT enabled
[2015-04-24 18:26:07.984676] D [socket.c:3802:socket_init] 
0-socket.management: SSL support for glusterd is NOT enabled
[2015-04-24 18:26:07.984696] D [socket.c:3819:socket_init] 
0-socket.management: using system polling thread
[2015-04-24 18:26:07.984788] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, 
Ver: 2, Port: 0
[2015-04-24 18:26:07.984818] D [rpcsvc.c:1801:rpcsvc_program_register] 
0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), 
Num: 14398633, Ver: 2, Port: 0
[2015-04-24 18:26:07.984865] D 
[glusterd-utils.c:8288:glusterd_sm_tr_log_init] 0-: returning 0
[2015-04-24 18:26:07.984893] D [glusterd.c:1484:init] 0-management: 
cannot get run-with-valgrind value
[2015-04-24 18:26:08.063588] D 
[glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning 0
[2015-04-24 18:26:08.063710] D 
[glusterd.c:572:glusterd_crt_georep_folders] 0-: Returning 0
[2015-04-24 18:26:10.094132] E [store.c:432:gf_store_handle_retrieve] 
0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned 
error: (No such file or directory)
[2015-04-24 18:26:10.094197] D [store.c:437:gf_store_handle_retrieve] 
0-: Returning -1
[2015-04-24 18:26:10.094221] D 
[glusterd-store.c:1935:glusterd_retrieve_op_version] 0-: Unable to get 
store handle!
[2015-04-24 18:26:10.094251] E [store.c:432:gf_store_handle_retrieve] 
0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned 
error: (No such file or directory)
[2015-04-24 18:26:10.094272] D [store.c:437:gf_store_handle_retrieve] 
0-: Returning -1
[2015-04-24 18:26:10.094290] D 
[glusterd-store.c:2093:glusterd_retrieve_uuid] 0-: Unable to get 
storehandle!
[2015-04-24 18:26:10.094308] D 
[glusterd-store.c:2112:glusterd_retrieve_uuid] 0-: Returning -1
[2015-04-24 18:26:10.094325] I 
[glusterd-store.c:2063:glusterd_restore_op_version] 0-management: 
Detected new install. Setting op-version to maximum : 30600
[2015-04-24 18:26:10.094409] D 
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
[2015-04-24 18:26:10.094448] E [store.c:432:gf_store_handle_retrieve] 
0-: Path corresponding to /var/lib/glusterd/vols/gv2/info, returned 
error: (No such file or directory)
[2015-04-24 18:26:10.094470] D [store.c:437:gf_store_handle_retrieve] 
0-: Returning -1
[2015-04-24 18:26:10.094488] E 
[glusterd-store.c:2623:glusterd_store_update_volinfo] 0-management: 
volinfo handle is NULL
[2015-04-24 18:26:10.094506] E 
[glusterd-store.c:2868:glusterd_store_retrieve_volume] 0-management: 
Failed to update volinfo for gv2 volume
[2015-04-24 18:26:10.094525] D 
[glusterd-utils.c:950:glusterd_volume_brickinfos_delete] 0-management: 
Returning 0
[2015-04-24 18:26:10.094546] D [store.c:458:gf_store_handle_destroy] 0-: 
Returning 0
[2015-04-24 18:26:10.094565] D 
[glusterd-utils.c:994:glusterd_volinfo_delete] 0-management: Returning 0
[2015-04-24 18:26:10.094590] E 
[glusterd-store.c:3062:glusterd_store_retrieve_volumes] 0-: Unable to 
restore volume: gv2
[2015-04-24 18:26:10.094612] D 
[glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning 
with -1
[2015-04-24 18:26:10.094634] D [glusterd-store.c:4321:glusterd_restore] 
0-: Returning -1
[2015-04-24 18:26:10.094659] E [xlator.c:425:xlator_init] 0-management: 
Initialization of volume 'management' failed, review your volfile again
[2015-04-24 18:26:10.094683] E [graph.c:322:glusterfs_graph_init] 
0-management: initializing translator failed
[2015-04-24 18:26:10.094702] E [graph.c:525:glusterfs_graph_activate] 
0-graph: init failed
[2015-04-24 18:26:10.095143] D [logging.c:1740:gf_log_flush_extra_msgs] 
0-logging-infra: Log buffer size reduced. About to flush 2 extra log 
messages
[2015-04-24 18:26:10.095176] D [logging.c:1743:gf_log_flush_extra_msgs] 
0-logging-infra: Just flushed 2 extra log messages
[2015-04-24 18:26:10.095395] W [glusterfsd.c:1194:cleanup_and_exit] (--> 
0-: received signum (0), shutting down
[2015-04-24 18:26:10.095426] D 
[glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: 
portmapper signout arguments not given

Ernie Dunbar

2015-Apr-24 18:34 UTC

head link

[Gluster-users] Glusterd fails to start after removing /brick1/gv2/.glusterfs

Note: this was fixed by removing the files in /var/lib/glusterd like 
this:

for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks' 
 >/dev/null 2>&1;then rm -rf $file; fi; done

I could swear I'd done this once before during the reinstall, but 
whatever.

On 2015-04-24 11:29, Ernie Dunbar wrote:> I've nuked my gluster brick, configuration, and files with the intent
> to rebuild them. This is what happens when I try to start glusterd
> with the --debug option:
> 
> root at nfs1:/etc/apt/sources.list.d# glusterd --debug
> [2015-04-24 18:26:07.978598] I [MSGID: 100030]
> [glusterfsd.c:2018:main] 0-glusterd: Started running glusterd version
> 3.6.2 (args: glusterd --debug)
> [2015-04-24 18:26:07.978685] D
> [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting
> timer now. Timeout = 120, current buf size = 5
> [2015-04-24 18:26:07.979021] D [MSGID: 0] [glusterfsd.c:613:get_volfp]
> 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
> [2015-04-24 18:26:07.982180] I [glusterd.c:1214:init] 0-management:
> Maximum allowed open file descriptors set to 65536
> [2015-04-24 18:26:07.982238] I [glusterd.c:1259:init] 0-management:
> Using /var/lib/glusterd as working directory
> [2015-04-24 18:26:07.982320] D
> [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog
> value: 128
> [2015-04-24 18:26:07.982532] D [rpcsvc.c:2198:rpcsvc_init]
> 0-rpc-service: RPC service inited.
> [2015-04-24 18:26:07.982559] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver:
> 1, Port: 0
> [2015-04-24 18:26:07.982595] D
> [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
> load file
> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/socket.so
> [2015-04-24 18:26:07.983291] D [socket.c:3799:socket_init]
> 0-socket.management: SSL support on the I/O path is NOT enabled
> [2015-04-24 18:26:07.983325] D [socket.c:3802:socket_init]
> 0-socket.management: SSL support for glusterd is NOT enabled
> [2015-04-24 18:26:07.983345] D [socket.c:3819:socket_init]
> 0-socket.management: using system polling thread
> [2015-04-24 18:26:07.983369] D [name.c:550:server_fill_address_family]
> 0-socket.management: option address-family not specified, defaulting
> to inet
> [2015-04-24 18:26:07.983457] D
> [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
> load file
> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/rdma.so
> [2015-04-24 18:26:07.983895] D
> [rpc-transport.c:300:rpc_transport_load] 0-rpc-transport: dlsym
> (gf_rpc_transport_reconfigure) on
> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/rdma.so:
> undefined symbol: reconfigure
> librdmacm: couldn't read ABI version.
> librdmacm: assuming: 4
> CMA: unable to get RDMA device list
> [2015-04-24 18:26:07.984165] W [rdma.c:4221:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such
> device)
> [2015-04-24 18:26:07.984191] E [rdma.c:4519:init] 0-rdma.management:
> Failed to initialize IB Device
> [2015-04-24 18:26:07.984215] E
> [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma'
> initialization failed
> [2015-04-24 18:26:07.984280] W [rpcsvc.c:1524:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2015-04-24 18:26:07.984310] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc peer, Num:
> 1238437, Ver: 2, Port: 0
> [2015-04-24 18:26:07.984330] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc cli read-only,
> Num: 1238463, Ver: 2, Port: 0
> [2015-04-24 18:26:07.984351] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc mgmt, Num:
> 1238433, Ver: 2, Port: 0
> [2015-04-24 18:26:07.984378] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num:
> 1238433, Ver: 3, Port: 0
> [2015-04-24 18:26:07.984398] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456,
> Ver: 1, Port: 0
> [2015-04-24 18:26:07.984418] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Handshake, Num:
> 14398633, Ver: 2, Port: 0
> [2015-04-24 18:26:07.984440] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster MGMT Handshake, Num:
> 1239873, Ver: 1, Port: 0
> [2015-04-24 18:26:07.984495] D [rpcsvc.c:2198:rpcsvc_init]
> 0-rpc-service: RPC service inited.
> [2015-04-24 18:26:07.984518] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver:
> 1, Port: 0
> [2015-04-24 18:26:07.984544] D
> [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
> load file
> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.2/rpc-transport/socket.so
> [2015-04-24 18:26:07.984629] D [socket.c:3684:socket_init]
> 0-socket.management: disabling nodelay
> [2015-04-24 18:26:07.984657] D [socket.c:3799:socket_init]
> 0-socket.management: SSL support on the I/O path is NOT enabled
> [2015-04-24 18:26:07.984676] D [socket.c:3802:socket_init]
> 0-socket.management: SSL support for glusterd is NOT enabled
> [2015-04-24 18:26:07.984696] D [socket.c:3819:socket_init]
> 0-socket.management: using system polling thread
> [2015-04-24 18:26:07.984788] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463,
> Ver: 2, Port: 0
> [2015-04-24 18:26:07.984818] D [rpcsvc.c:1801:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Handshake (CLI
> Getspec), Num: 14398633, Ver: 2, Port: 0
> [2015-04-24 18:26:07.984865] D
> [glusterd-utils.c:8288:glusterd_sm_tr_log_init] 0-: returning 0
> [2015-04-24 18:26:07.984893] D [glusterd.c:1484:init] 0-management:
> cannot get run-with-valgrind value
> [2015-04-24 18:26:08.063588] D
> [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning 0
> [2015-04-24 18:26:08.063710] D
> [glusterd.c:572:glusterd_crt_georep_folders] 0-: Returning 0
> [2015-04-24 18:26:10.094132] E [store.c:432:gf_store_handle_retrieve]
> 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned
> error: (No such file or directory)
> [2015-04-24 18:26:10.094197] D [store.c:437:gf_store_handle_retrieve]
> 0-: Returning -1
> [2015-04-24 18:26:10.094221] D
> [glusterd-store.c:1935:glusterd_retrieve_op_version] 0-: Unable to get
> store handle!
> [2015-04-24 18:26:10.094251] E [store.c:432:gf_store_handle_retrieve]
> 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned
> error: (No such file or directory)
> [2015-04-24 18:26:10.094272] D [store.c:437:gf_store_handle_retrieve]
> 0-: Returning -1
> [2015-04-24 18:26:10.094290] D
> [glusterd-store.c:2093:glusterd_retrieve_uuid] 0-: Unable to get
> storehandle!
> [2015-04-24 18:26:10.094308] D
> [glusterd-store.c:2112:glusterd_retrieve_uuid] 0-: Returning -1
> [2015-04-24 18:26:10.094325] I
> [glusterd-store.c:2063:glusterd_restore_op_version] 0-management:
> Detected new install. Setting op-version to maximum : 30600
> [2015-04-24 18:26:10.094409] D
> [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
> [2015-04-24 18:26:10.094448] E [store.c:432:gf_store_handle_retrieve]
> 0-: Path corresponding to /var/lib/glusterd/vols/gv2/info, returned
> error: (No such file or directory)
> [2015-04-24 18:26:10.094470] D [store.c:437:gf_store_handle_retrieve]
> 0-: Returning -1
> [2015-04-24 18:26:10.094488] E
> [glusterd-store.c:2623:glusterd_store_update_volinfo] 0-management:
> volinfo handle is NULL
> [2015-04-24 18:26:10.094506] E
> [glusterd-store.c:2868:glusterd_store_retrieve_volume] 0-management:
> Failed to update volinfo for gv2 volume
> [2015-04-24 18:26:10.094525] D
> [glusterd-utils.c:950:glusterd_volume_brickinfos_delete] 0-management:
> Returning 0
> [2015-04-24 18:26:10.094546] D [store.c:458:gf_store_handle_destroy]
> 0-: Returning 0
> [2015-04-24 18:26:10.094565] D
> [glusterd-utils.c:994:glusterd_volinfo_delete] 0-management: Returning
> 0
> [2015-04-24 18:26:10.094590] E
> [glusterd-store.c:3062:glusterd_store_retrieve_volumes] 0-: Unable to
> restore volume: gv2
> [2015-04-24 18:26:10.094612] D
> [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning
> with -1
> [2015-04-24 18:26:10.094634] D
> [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1
> [2015-04-24 18:26:10.094659] E [xlator.c:425:xlator_init]
> 0-management: Initialization of volume 'management' failed, review
> your volfile again
> [2015-04-24 18:26:10.094683] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2015-04-24 18:26:10.094702] E [graph.c:525:glusterfs_graph_activate]
> 0-graph: init failed
> [2015-04-24 18:26:10.095143] D
> [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer
> size reduced. About to flush 2 extra log messages
> [2015-04-24 18:26:10.095176] D
> [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed
> 2 extra log messages
> [2015-04-24 18:26:10.095395] W [glusterfsd.c:1194:cleanup_and_exit]
> (--> 0-: received signum (0), shutting down
> [2015-04-24 18:26:10.095426] D
> [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt:
> portmapper signout arguments not given
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

Gluster users - Apr 2015 - Glusterd fails to start after removing /brick1/gv2/.glusterfs

[Gluster-users] Glusterd fails to start after removing /brick1/gv2/.glusterfs

[Gluster-users] Glusterd fails to start after removing /brick1/gv2/.glusterfs