Gluster users - Jun 2011 - gluster fuse disk state problem (reproducible )

I think I have the same problems as:

http://gluster.org/pipermail/gluster-users/2011-June/007980.html

and

http://gluster.org/pipermail/gluster-users/2011-May/007697.html

Recently I installed Glusterfs 3.2.0 on four workstations which share a
total of 28TB of disk-space between them for batch processing of FMRI
and DTI data. Had the same setup between three workstations and 9TB with
Glusterfs 3.0.4 for over a year, but the hard-disk pool was to small.
The three "old" workstations from the old cluster and two more will
also
access the 28TB pool.

OS: Scientific Linux 6
Glusterfs 3.2.0 and 3.2.1 compiled from source rpm

I've been testing the new setup for over a week now, but it seemed
unstable, Glusterfs clients would stop responding and tasks hung without
being able to kill -9. Only a reboot would stop the tasks.
a kill of the Glusterfs client and a mount afterwards would make the
gluster storage accessible again.

One of the tasks is converting a medical image format called nifti to
another format called analyze. We use a application included in a
package called mrtrix (2.9.0) for this. The application is mrconvert.

-A conversion of a large nifti file (256MB) to a analyze file will hang
application and the Gluster fuse client
- A small one (10MB) will work. 
- If I convert it reading from the Gluster storage to a local disk
everything works fine. 
- Reading from a local disk to the Gluster storage will hang the
application and gluster fuse client
- disabling the quick-read does nothing.

Errors are the same as a in the second post above, I can post them
Monday. I did a strace on a functioning and a none functioning convert
and the application hangs on the last close like:

munmap .............
close(4

The conversion was completed.
I tried 3.2.0, 3.1.4 and 3.2.1 and all of them have the same problem.
I went back to 3.0.8 and this is functioning.

-- 
Marc Geerlings 

                     Application Manager Research 
                        Department of Radiology

                  Maastricht University Medical Center
                        P.O. Box: 5800, 6202 AZ 
                      Maastricht, The Netherlands

                            P. Debyelaan 25
                          6229 HX, Maastricht
                             Room: 0.C1.037

                        Tel. + 31 43 ? 387 49 50
                        Fax. + 31 43 ? 387 69 09

                       email: m.geerlings at mumc.nl
                  m.geerlings at maastrichtuniversity.nl

Marc,
 Can you get us a process state dump of the glusterfs client while it is in
the hung state? This will be very useful for us to debug the issue.

Avati

On Fri, Jun 17, 2011 at 11:17 PM, Marc Geerlings
<marc.geerlings at gmail.com>wrote:
> I think I have the same problems as:
>
> http://gluster.org/pipermail/gluster-users/2011-June/007980.html
>
> and
>
> http://gluster.org/pipermail/gluster-users/2011-May/007697.html
>
> Recently I installed Glusterfs 3.2.0 on four workstations which share a
> total of 28TB of disk-space between them for batch processing of FMRI
> and DTI data. Had the same setup between three workstations and 9TB with
> Glusterfs 3.0.4 for over a year, but the hard-disk pool was to small.
> The three "old" workstations from the old cluster and two more
will also
> access the 28TB pool.
>
> OS: Scientific Linux 6
> Glusterfs 3.2.0 and 3.2.1 compiled from source rpm
>
> I've been testing the new setup for over a week now, but it seemed
> unstable, Glusterfs clients would stop responding and tasks hung without
> being able to kill -9. Only a reboot would stop the tasks.
> a kill of the Glusterfs client and a mount afterwards would make the
> gluster storage accessible again.
>
> One of the tasks is converting a medical image format called nifti to
> another format called analyze. We use a application included in a
> package called mrtrix (2.9.0) for this. The application is mrconvert.
>
> -A conversion of a large nifti file (256MB) to a analyze file will hang
> application and the Gluster fuse client
> - A small one (10MB) will work.
> - If I convert it reading from the Gluster storage to a local disk
> everything works fine.
> - Reading from a local disk to the Gluster storage will hang the
> application and gluster fuse client
> - disabling the quick-read does nothing.
>
> Errors are the same as a in the second post above, I can post them
> Monday. I did a strace on a functioning and a none functioning convert
> and the application hangs on the last close like:
>
> munmap .............
> close(4
>
> The conversion was completed.
> I tried 3.2.0, 3.1.4 and 3.2.1 and all of them have the same problem.
> I went back to 3.0.8 and this is functioning.
>
> --
> Marc Geerlings
>
>                     Application Manager Research
>                        Department of Radiology
>
>                  Maastricht University Medical Center
>                        P.O. Box: 5800, 6202 AZ
>                      Maastricht, The Netherlands
>
>                            P. Debyelaan 25
>                          6229 HX, Maastricht
>                             Room: 0.C1.037
>
>                        Tel. + 31 43 ? 387 49 50
>                        Fax. + 31 43 ? 387 69 09
>
>                       email: m.geerlings at mumc.nl
>                  m.geerlings at maastrichtuniversity.nl
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110620/4f91c6db/attachment.html>