CentOS - Sep 2009 - Centos 5.3 Kernel panic with fuse and glusterfs.

Hi List,

We currently have a very irritating problem with Centos 5.3 x86_64 
running on a Dell Poweredge SC1435.  The problem is this: We are 
experiencing frequent kernel panics while using glusterfs and Fuse. 
Across the cluster of servers, we are experiencing roughly 1 panic every 
1-2 days.  This wasn't a problem with earlier servers where we used 
Fedora 6. 

Here's a kernel panic screenshot:
http://imagehost.gr/images/c5ad2d5jzgpgoq91v24y.png

Here's some general info:
Linux server6 2.6.18-128.1.10.el5 #1 SMP Thu May 7 10:35:59 EDT 2009 
x86_64 x86_64 x86_64 GNU/Linux
Using:
fuse-2.7.4-8_10.el5
fuse-kmdl-2.6.18-128.1.10.el5-2.7.4-8_10.el5
fuse-libs-2.7.4-8_10.el5
glusterfs-common-2.0.1-1.el5
glusterfs-client-2.0.1-1.el5
glusterfs-server-2.0.1-1.el5

I've straced glusterfs while it dies, and there's nothing seriously 
spurious, just it stops working as soon as the kernel locks up.

A little background, Gluster is used to share some directories which are 
used by apache to serve files from.  I've managed to replicate the live 
environment inside a virtual machine, and also to replicate the kernel 
panic by loading the virtual machine's apache with ApacheBench, at as 
few as 3 concurrent requests, the kernel locks up.
However, i have been unable to reproduce this exact behavior on the live 
cluster, and have tried up to 10,000 concurrent requests which max out 
the network more than anything.

I've tried latest versions of gluster and fuse from development 
snapshots and stable releases, I've tried patched versions of fuse 
released by Gluster.  Nothing seems to improve this problem.

If anyone has any ideas for further debugging, or other routes for 
support.  I'm running out of ideas. 

Thanks in advance

Tom O'Connor

-- 
Tom O'Connor
--------------------------
Assanka: Every possibility
w: http://www.assanka.net/
t: 0870 085 2038
f: 0871 433 0919
e: tom.oconnor at assanka.net

On Tue, Sep 15, 2009 at 03:24:52PM +0100, Tom O'Connor
wrote:
> If anyone has any ideas for further debugging, or other routes for 
> support.  I'm running out of ideas. 
Enterprise Linux 5.4 with included official FUSE support seems like the next
place to look. 

-- 
Matthew Miller           mattdm at mattdm.org         
<http://mattdm.org/>