Gluster users - Sep 2014 - glusterfs replica volume self heal lots of small file very very slow?how to improve? why slow?

Hi Just Gluster,

Sorry for the bad formatting of the reply, I just copy/pasted the values from
the daily digest

As for using GlusterFS with small files, it is also the use case in which
we're trying to use it.
Unfortunately, we weren't successful yet in having a smooth ride with a
Gluster cluster with a lot of small files.

Our use case is storing several million files, averaging under 1mb in size (most
of them around 10kb). We still have some files at about 1gb, but very few.

When a self-heal hits in our use-case, it is a direct impact in performance for
the users. The CPU of the Gluster nodes hits 100%, and maintains this for
usually 1 hour, but sometimes goes up to 4-5 hours.
This usually renders the Gluster cluster unusable, with a high impact for us.

We're hoping the refactoring of the AFR in version 3.6 will help. We're
also looking in testing some values in here:
http://www.gluster.org/community/documentation/index.php/Documenting_the_undocumented

But as of yet, we haven't found anything that makes our use case work.
If you do find something though, please share it with the community, I would be
thrilled to make it work :)

Jocelyn

On Mon, Sep 22, 2014 at 10:04 AM, Jocelyn Hotte
<jocelyn.hotte at ubisoft.com> wrote:
> When a self-heal hits in our use-case, it is a direct impact in performance
for the users. The CPU of the Gluster nodes hits 100%, and maintains this for
usually 1 hour, but sometimes goes up to 4-5 hours.
> This usually renders the Gluster cluster unusable, with a high impact for
us.

Have you tried using cgroups to limit the effect of self-heal?