Gluster users - Jan 2013 - glusterfs performance issues

Hi,
I'm running into performance trouble when using a simple replicated
glusterfs volume for a webserver. Specifically I set up two webservers with
an additional partition each for glusterfs 3.3.1. I formated the partitions
with xfs, created a simple replicated volume and mounted it on /mnt/data.

The issue is that the CMS running on the webserver creates resized versions
for images for the pages using GraphicsMagick and I see lots of "gm
convert" processes piling up and the gluster daemons show high cpu usage.
When I cd into one of the image directories and then type e.g.
"image1234.jp" <tab> the shell freezes for a minute before
finally
completing the filename with the expected "g" at the end.

The "gm convert" processes make almost no progress even though on a
regular
filesystem each call takes only a fraction of a second.

I looks like some sort of locking issue or something else that prevents
these processes or the bash completion from finishing in a reasonable
amount of time.

I've also written a small script that uses 50 parallel dd write processes
but cannot replicate the behavior with that. Here all the processes seem to
behave normally.

Any ideas what the problem could be and what I could do to make glusterfs
behave in a less erratic manner?

Regards,
  Dennis

On 01/07/2013 12:03 PM, Dennis Jacobfeuerborn wrote:
> The "gm convert" processes make almost no progress even though on
a regular
> filesystem each call takes only a fraction of a second.
Can you run gm_convert under strace?  That will give us a more accurate
idea of what kind of I/O it's generating.  I recommend both -t and -T to
get timing information as well.  Also, it never hurts to file a bug so
we can track/prioritize/etc.  Thanks.

https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

On Tue, 08 Jan 2013 07:55:41 -0500
Jeff Darcy <jdarcy at redhat.com> wrote:
> On 1/8/13 7:35 AM, Stephan von Krawczynski wrote:
> > In fact I already saw just about every possibility you can think of
when
> > accessing files, be it a simple "ls" or writing or reading a
file.
> 
> Would you mind citing the bug IDs for the problems you found?
Yes, I mind.
The problem with this kind of bugs is that you cannot describe reproduction.
Which makes them pretty useless as bug reports.
They can therefore only contain the information that such situations are seen,
but not much else. And me and others have told that continously over the years
on the lists.
Take 4 physical boxes and check out some damage situations (switch bricks off
and on), you will see the described problems within a day.
You only need bonnie and ls to find out.

-- 
Regards,
Stephan