Franco Broi
2014-Apr-28 02:52 UTC
[Gluster-users] Gluster 3.5 wont start if disks are full
Hi I have a 2 node test system, on one of the nodes the volume disks are full and gluster will not start after an upgrade to 3.5. Previous version was 3.4.3. I am assuming that gluster wont start because the disks are full although I can't see anything in the log that might suggest that that is the case. Here's a grep of the errors from the log file: grep ' E ' /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2014-04-28 09:19:24.616070] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x7fb38cd01c8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fb38cd017f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fb38cd0170e]))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2014-04-28 09:19:24.614255 (xid=0x28x) [2014-04-28 09:19:24.616084] E [glusterd-handshake.c:862:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later [2014-04-28 09:19:27.234725] E [socket.c:2157:socket_connect_finish] 0-management: connection to 192.168.34.6:24007 failed (Connection refused) [2014-04-28 09:21:47.063528] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2014-04-28 09:21:47.104301] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-04-28 09:21:47.104329] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-04-28 09:21:47.104342] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2014-04-28 09:21:47.104353] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2014-04-28 09:21:47.104364] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2014-04-28 09:21:47.104375] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2014-04-28 09:21:47.104386] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2014-04-28 09:21:47.104397] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2014-04-28 09:22:09.971037] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2014-04-28 09:22:10.185569] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-04-28 09:22:10.185611] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-04-28 09:22:10.185624] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2014-04-28 09:22:10.185634] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2014-04-28 09:22:10.185645] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2014-04-28 09:22:10.185655] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2014-04-28 09:22:10.185666] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2014-04-28 09:22:10.185677] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2014-04-28 09:22:13.519169] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/61ba1ffcacede69af7df2ca0a0000f0c.socket error: Permission denied Status of volume: gluster_test Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick nas5-10g:/data17/gvol N/A N N/A Brick nas5-10g:/data18/gvol N/A N N/A Brick nas5-10g:/data19/gvol N/A N N/A Brick nas5-10g:/data20/gvol N/A N N/A Brick nas6-10g:/data21/gvol 49152 Y 447 Brick nas6-10g:/data22/gvol 49153 Y 453 Brick nas6-10g:/data23/gvol 49154 Y 460 Brick nas6-10g:/data24/gvol 49155 Y 465 NFS Server on localhost 2049 Y 473 NFS Server on nas5-10g 2049 Y 20880 Task Status of Volume gluster_test ------------------------------------------------------------------------------ There are no active volume tasks Cheers,