ML mail
2015-Feb-05 13:16 UTC
[Gluster-users] glusterfsd not starting at boot with Debian 7
Dear Pranith As asked you will find below the glusterd logfile including the log entries at the boot of the server. On Debian the logfile is located in the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option rpc-auth-allow-insecure on 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote: On 02/05/2015 04:08 AM, ML mail wrote:> Hi, > > I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running. > > > After a boot running "gluster volume status" shows: > Status of volume: myvol > Gluster process Port Online Pid > ------------------------------------------------------------------------------ > Brick gfs1a:/data/myvol/brick N/A N N/A > > Task Status of Volume myvol > ------------------------------------------------------------------------------ > > There are no active volume tasks > > as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy?If glusterd is starting then there shouldn't be any bug in the init scripts. Could you attach the log file of glusterd? I am not sure where it is stored in debian may be /var/log/glusterfs or /usr/local/var/log/glusterfs it will have glusterd in the name of the file. Pranith> > Best regards > ML > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Niels de Vos
2015-Feb-05 15:33 UTC
[Gluster-users] glusterfsd not starting at boot with Debian 7
On Thu, Feb 05, 2015 at 01:16:54PM +0000, ML mail wrote:> Dear Pranith > > As asked you will find below the glusterd logfile including the log > entries at the boot of the server. On Debian the logfile is located in > the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file.This is the log file of the glusterd process, whereas the issue you reported is with the glusterfsd (brick) process. There should be a log for each brick under /var/log/glusterfs/bricks/. The most common problem, is that the mount point is not available yet when glusterd starts the brick processes. You mention that you can get the processes started with a 'gluster volume stop/start' command. An easier way would be 'gluster volume start $VOLUME force' which start only the missing processes (and does not stop any brick processes on other storage servers). Depending on how you mount the filesystems for your bricks, you may be able to add the x-systemd.automount option to /etc/fstab to trigger a mount of the filesystem upon access (when the brick process starts). See 'man systemd.mount' for some more details. HTH, Niels> > > [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory > [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread > [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device > [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed > [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread > [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 > [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 > [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 > Final graph: > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option rpc-auth-allow-insecure on > 8: option transport.socket.read-fail-log off > 9: option transport.socket.keepalive-interval 2 > 10: option transport.socket.keepalive-time 10 > 11: option transport-type rdma > 12: option working-directory /var/lib/glusterd > 13: end-volume > 14: > +------------------------------------------------------------------------------+ > [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled > [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread > [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now > [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled > [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread > [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick > [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now > > > > > > On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote: > > On 02/05/2015 04:08 AM, ML mail wrote: > > Hi, > > > > I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running. > > > > > > After a boot running "gluster volume status" shows: > > Status of volume: myvol > > Gluster process Port Online Pid > > ------------------------------------------------------------------------------ > > Brick gfs1a:/data/myvol/brick N/A N N/A > > > > Task Status of Volume myvol > > ------------------------------------------------------------------------------ > > > > There are no active volume tasks > > > > as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy? > If glusterd is starting then there shouldn't be any bug in the init > scripts. Could you attach the log file of glusterd? I am not sure where > it is stored in debian may be /var/log/glusterfs or > /usr/local/var/log/glusterfs it will have glusterd in the name of the file. > > Pranith > > > > > Best regards > > ML > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150205/2b27d802/attachment.sig>
Pranith Kumar Karampuri
2015-Feb-05 16:03 UTC
[Gluster-users] glusterfsd not starting at boot with Debian 7
On 02/05/2015 06:46 PM, ML mail wrote:> Dear Pranith > > As asked you will find below the glusterd logfile including the log entries at the boot of the server. On Debian the logfile is located in the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. > > > [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory > [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread > [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device > [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed > [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread > [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 > [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 > [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 > Final graph: > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option rpc-auth-allow-insecure on > 8: option transport.socket.read-fail-log off > 9: option transport.socket.keepalive-interval 2 > 10: option transport.socket.keepalive-time 10 > 11: option transport-type rdma > 12: option working-directory /var/lib/glusterd > 13: end-volume > 14: > +------------------------------------------------------------------------------+ > [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled > [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread > [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now > [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled > [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread > [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick > [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting nowLog above suggests that glusterd launched the brick but it failed to come up. Let us see the logfile of the brick as well. It should be present in /var/log/glusterfs/bricks Pranith> > > > > > On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote: > > On 02/05/2015 04:08 AM, ML mail wrote: >> Hi, >> >> I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running. >> >> >> After a boot running "gluster volume status" shows: >> Status of volume: myvol >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick gfs1a:/data/myvol/brick N/A N N/A >> >> Task Status of Volume myvol >> ------------------------------------------------------------------------------ >> >> There are no active volume tasks >> >> as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy? > If glusterd is starting then there shouldn't be any bug in the init > scripts. Could you attach the log file of glusterd? I am not sure where > it is stored in debian may be /var/log/glusterfs or > /usr/local/var/log/glusterfs it will have glusterd in the name of the file. > > Pranith > >> Best regards >> ML >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users