Gluster users - Jul 2009 - glusterfs 2.0.3 core dump

Hi All,

I'm running into a consistent crash with gluster 2.0.3

RHEL5.3
Fuse 2.7.4
Kernel 2.6.18-128.1.6.el5xen

Vol spec is basic AFR over GigE:
--------------------------------

volume u0-2
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.200.2
    option remote-subvolume u0
end-volume

volume u0-1
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.200.1
    option remote-subvolume u0
end-volume

volume afr
   type cluster/afr
   option read-subvolume u0-2
   subvolumes u0-1 u0-2
end-volume

Server spec:
------------

volume export
  type storage/posix
  option directory /export/u0
end-volume

volume posix-locks
  type features/posix-locks
 subvolumes export
end-volume

volume io-thr
 type performance/io-threads
  subvolumes posix-locks
end-volume

volume u0
 type performance/read-ahead
 subvolumes io-thr
end-volume

volume server
  type protocol/server
  option transport-type tcp/server
  option auth.addr.u0.allow 192.*
  subvolumes u0
end-volume

Core dump BT gives:
-------------------

Core was generated by `/sbin/glusterfs --log-level=NORMAL
--disable-direct-io-mode
--volfile=/etc/glusterfs/glusterfs-client.vol'.Program terminated with
signal 11, Segmentation fault.
[New process 3663]
[New process 3668]
[New process 3664]
#0  client_readv_cbk (frame=0x2aaaac00bb10, hdr=0x9629d40,
    hdrlen=<value optimized out>, iobuf=0x0) at client-protocol.c:4319
4319                            vector.iov_base = iobuf->ptr;
(gdb) bt
#0  client_readv_cbk (frame=0x2aaaac00bb10, hdr=0x9629d40,
    hdrlen=<value optimized out>, iobuf=0x0) at client-protocol.c:4319
#1  0x00002b1089e807ca in protocol_client_pollin (this=0x961cc40,
    trans=0x9621cc0) at client-protocol.c:6169
#2  0x00002b1089e87b42 in notify (this=0xb2ee594a, event=2,
data=0x9621cc0)
    at client-protocol.c:6213
#3  0x00002aaaaaaafcf3 in socket_event_handler (fd=<value optimized
out>,    idx=2, data=0x9621cc0, poll_in=1, poll_out=0, poll_err=0) at
socket.c:814
#4  0x00002b10893f5135 in event_dispatch_epoll (event_pool=0x96184b0)
    at event.c:804
#5  0x00000000004039ea in main (argc=5, argv=0x7fff216f9158)
    at glusterfsd.c:1226

Any ideas?

The easiest way I've found to reproduce this is by attempting to build
software within the mounted glusterfs.

I'm more than happy to dig deeper if required.

Thanks, Jeff.

----- "Jeff Evans" <jeffe at tricab.com> wrote:
> Any ideas?
 
> The easiest way I've found to reproduce this is by attempting to
> build software within the mounted glusterfs.
Do you still have the core file? Can you give us the value of
op_ret in frame 0? ("frame 0; print op_ret").

Vikas

On the 15-07-2009 at 17:58pm, Vikas Gorur wrote:
> Do you still have the core file? Can you give us the value of
> op_ret in frame 0? ("frame 0; print op_ret").
Sure do, here's the output:

#0  client_readv_cbk (frame=0x2aaaac00bb10, hdr=0x9629d40,
    hdrlen=<value optimized out>, iobuf=0x0) at
    client-protocol.c:4319
4319                            vector.iov_base = iobuf->ptr;
(gdb) frame 0
#0  client_readv_cbk (frame=0x2aaaac00bb10, hdr=0x9629d40,
    hdrlen=<value optimized out>, iobuf=0x0) at
    client-protocol.c:4319
4319                            vector.iov_base = iobuf->ptr;
(gdb) print op_ret
$1 = 4096

Thanks, Jeff.