chamara samarakoon
2015-Jan-20 05:59 UTC
[Gluster-users] CentOS Freeze with GlusterFS Error
Hi All, We have setup Openstack cloud as below. And the "/va/lib/nova/instances" is a Gluster volume. CentOS - 6.5 Kernel - 2.6.32-431.29.2.el6.x86_64 GlusterFS - glusterfs 3.5.2 built on Jul 31 2014 18:47:54 OpenStack - RDO using Packstack ? Recently Controller node freezes with following error (Which required hard reboot), as a result Gluster volumes on compute node can not reach the controller and due to that all the instances on compute nodes become to read-only status which causes to restart all instances. *BUG: scheduling while atomic : glusterfsd/42725/0xffffffff* *BUG: unable to handle kernel paging request at 0000000038a60d0a8* *IP: [<fffffffff81058e5d>] task_rq_lock+0x4d/0xa0* *PGD 1065525067 PUD 0* *Oops: 0000 [#1] SMP* *last sysfs file : /sys/device/pci0000:80/0000:80:02.0/0000:86:00.0/host2/port-2:0/end_device-2:0/target2:0:0/2:0:0:1/state* *CPU 0* *Modules linked in : xtconntrack iptable_filter ip_tables ipt_REDIRECT fuse ipv openvswitch vxlan iptable_mangle * Please advice on above incident , also feedback on the Openstack + GlusterFS setup is appreciated. Thank You, Chamara -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150120/ecf8e7b9/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: openstak-gluster.png Type: image/png Size: 240027 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150120/ecf8e7b9/attachment.png>
What does "Controller" mean, the openstack controller node or somethign else (like HBA ) ? You picture says its SAN but the text says multi-path mount.. SAN would mean block devices, so I am assuming you have redundant block devices on the compute host, mkfs'ing it and then creating bricks for gluster ? The stack trace looks like you hit a kernel bug and glusterfsd happens to be running on the CPU at the time... my 2 cents thanx, deepak On Tue, Jan 20, 2015 at 11:29 AM, chamara samarakoon <chthsa123 at gmail.com> wrote:> Hi All, > > > We have setup Openstack cloud as below. And the "/va/lib/nova/instances" > is a Gluster volume. > > CentOS - 6.5 > Kernel - 2.6.32-431.29.2.el6.x86_64 > GlusterFS - glusterfs 3.5.2 built on Jul 31 2014 18:47:54 > OpenStack - RDO using Packstack > > > > > ? > > > Recently Controller node freezes with following error (Which required hard > reboot), as a result Gluster volumes on compute node can not reach the > controller and due to that all the instances on compute nodes become to > read-only status which causes to restart all instances. > > > > > *BUG: scheduling while atomic : glusterfsd/42725/0xffffffff* > *BUG: unable to handle kernel paging request at 0000000038a60d0a8* > *IP: [<fffffffff81058e5d>] task_rq_lock+0x4d/0xa0* > *PGD 1065525067 PUD 0* > *Oops: 0000 [#1] SMP* > *last sysfs file : > /sys/device/pci0000:80/0000:80:02.0/0000:86:00.0/host2/port-2:0/end_device-2:0/target2:0:0/2:0:0:1/state* > *CPU 0* > *Modules linked in : xtconntrack iptable_filter ip_tables ipt_REDIRECT > fuse ipv openvswitch vxlan iptable_mangle * > > Please advice on above incident , also feedback on the Openstack + > GlusterFS setup is appreciated. > > Thank You, > Chamara > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150120/98c29d0e/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: openstak-gluster.png Type: image/png Size: 240027 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150120/98c29d0e/attachment-0001.png>