Samikshan Bairagya
2017-Jan-25 13:44 UTC
[Gluster-users] possible kernel panic with glusterd
On 01/25/2017 06:57 PM, Joseph Lorenzini wrote:> Hi Atin, > > I assume you are referring to the /var/log/glustershd.log. If so, that file > never gets created.You'd find the glusterd log file as /var/log/glusterfs/etc-glusterfs-glusterd.vol.log AFAR for glusterfs-3.8. ~ Samikshan> > Joe > > On Wed, Jan 25, 2017 at 6:14 AM Atin Mukherjee <amukherj at redhat.com> wrote: > >> On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini <jaloren at gmail.com> >> wrote: >> >> Hi all, >> >> I have recently started exploring the DFS solution space and was doing >> some basic setup and testing with gluster. I set up a pool of three nodes >> following the quick start guide. That seemed to work fine. >> >> However, shortly after that, I noticed that one of the servers in the pool >> was becoming non-responsive -- as in the entire VM was completely hung and >> i had to use the hypervisor to force a reboot. I sshed into the server and >> startd poking around. glusterd was shut off. I started it up ....and the >> following happened: >> >> Message from syslogd at Jan 25 05:20:47 ... >> kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for >> 22s! [glusterd:2374] >> >> >> Could you attach the glusterd log file to enable us to look at why >> glusterd got shutdown? >> >> >> >> At which point, the VM became completely unresponsive again. >> >> All servers are the same. They are running centos 7.3, linux kernel >> 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. >> >> Since I just started investigating gluster, it is certainly possible that >> I misconfigured something on that one node. However, a kernel hang/panic >> seems like an excessive response :). If anyone would have any ideas or >> suggestions about what may be happening here or additional places I should >> look into to find out what is going on, I am all ears. >> >> Thanks, >> Joe >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> >> ~ Atin (atinm) >> > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >
Joseph Lorenzini
2017-Jan-25 17:43 UTC
[Gluster-users] possible kernel panic with glusterd
That file exists but is empty. Joe On Wed, Jan 25, 2017 at 7:44 AM Samikshan Bairagya <sbairagy at redhat.com> wrote:> > > On 01/25/2017 06:57 PM, Joseph Lorenzini wrote: > > Hi Atin, > > > > I assume you are referring to the /var/log/glustershd.log. If so, that > file > > never gets created. > > You'd find the glusterd log file as > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log AFAR for glusterfs-3.8. > > ~ Samikshan > > > > > Joe > > > > On Wed, Jan 25, 2017 at 6:14 AM Atin Mukherjee <amukherj at redhat.com> > wrote: > > > >> On Wed, Jan 25, 2017 at 5:15 PM, Joseph Lorenzini <jaloren at gmail.com> > >> wrote: > >> > >> Hi all, > >> > >> I have recently started exploring the DFS solution space and was doing > >> some basic setup and testing with gluster. I set up a pool of three > nodes > >> following the quick start guide. That seemed to work fine. > >> > >> However, shortly after that, I noticed that one of the servers in the > pool > >> was becoming non-responsive -- as in the entire VM was completely hung > and > >> i had to use the hypervisor to force a reboot. I sshed into the server > and > >> startd poking around. glusterd was shut off. I started it up ....and the > >> following happened: > >> > >> Message from syslogd at Jan 25 05:20:47 ... > >> kernel:[ 288.145027] NMI watchdog: BUG: soft lockup - CPU#1 stuck for > >> 22s! [glusterd:2374] > >> > >> > >> Could you attach the glusterd log file to enable us to look at why > >> glusterd got shutdown? > >> > >> > >> > >> At which point, the VM became completely unresponsive again. > >> > >> All servers are the same. They are running centos 7.3, linux kernel > >> 3.10.0-514.2.2.el7.x86_64. The glusterfs-server is 3.8. > >> > >> Since I just started investigating gluster, it is certainly possible > that > >> I misconfigured something on that one node. However, a kernel hang/panic > >> seems like an excessive response :). If anyone would have any ideas or > >> suggestions about what may be happening here or additional places I > should > >> look into to find out what is going on, I am all ears. > >> > >> Thanks, > >> Joe > >> > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> http://lists.gluster.org/mailman/listinfo/gluster-users > >> > >> > >> > >> > >> -- > >> > >> ~ Atin (atinm) > >> > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170125/ba83e8bf/attachment.html>