Joseph Lorenzini
2017-Jan-25 11:45 UTC
[Gluster-users] possible kernel panic with glusterd
Hi all, I have recently started exploring the DFS solution space and was doing some basic setup and testing with gluster. I set up a pool of three nodes following the quick start guide. That seemed to work fine. However, shortly after that, I noticed that one of the servers in the pool was becoming non-responsive -- as in the entire VM was completely hung and i had to use the hypervisor to force a reboot. I sshed into the server and started poking around. glusterd was shut off. I started it up ....and the following happened: Message from syslogd at Jan 25 05:20:47 ... kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [glusterd:2374] At which point, the VM became completely unresponsive again. All servers are the same. They are running centos 7.3, linux kernel 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. Since I just started investigating gluster, it is certainly possible that I misconfigured something on that one node. However, a kernel hang/panic seems like an excessive response :). If anyone would have any ideas or suggestions about what may be happening here or additional places I should look into to find out what is going on, I am all ears. Thanks, Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170125/d0ffe872/attachment.html>
On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini <jaloren at gmail.com> wrote:> Hi all, > > I have recently started exploring the DFS solution space and was doing > some basic setup and testing with gluster. I set up a pool of three nodes > following the quick start guide. That seemed to work fine. > > However, shortly after that, I noticed that one of the servers in the pool > was becoming non-responsive -- as in the entire VM was completely hung and > i had to use the hypervisor to force a reboot. I sshed into the server and > started poking around. glusterd was shut off. I started it up ....and the > following happened: > > Message from syslogd at Jan 25 05:20:47 ... > kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for > 22s! [glusterd:2374] > >Could you attach the glusterd log file to enable us to look at why glusterd got shutdown?> > At which point, the VM became completely unresponsive again. > > All servers are the same. They are running centos 7.3, linux kernel > 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. > > Since I just started investigating gluster, it is certainly possible that > I misconfigured something on that one node. However, a kernel hang/panic > seems like an excessive response :). If anyone would have any ideas or > suggestions about what may be happening here or additional places I should > look into to find out what is going on, I am all ears. > > Thanks, > Joe > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-- ~ Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170125/73b106be/attachment.html>