Gluster users - Jul 2015 - How to regenerate all metadata with brick content

Dear all,

We are currently in a very critical situation because all our scientific
production in our computing center is stopped since more than a week for
maintenance (because of metadata "corruption"), I tried some scripts
provided my GlusterFS support team with no effect and I can not continue in this
way.

So,I wonder if it exists a way to regenerate all bricks metadata (.glusterfs
directories) after having remove ".glusterfs" directory on each brick
(probably after having upgraded GlusterFS from v3.5.3 to v3.7.2)

Your volume is distributed and replicated on 4 servers with 2 bricks each (i.e.
distributed on 2 servers and replicated on the 2 others). My volume is not
stripped

In addition, because my the past I noticed a lot of performance issue, probably
because some users play with a big files and some others with a lot (thousands,
hundreds of thousands nay more) small files (hundreds of kB), could you advice
me about how to best tune the volume with performance parameters?

Thanks in advance,
Geoffrey
------------------------------------------------------
Geoffrey Letessier
Responsable informatique & ing?nieur syst?me
UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
Institut de Biologie Physico-Chimique
13, rue Pierre et Marie Curie - 75005 Paris
Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr

Hello Geoffrey,

If gluster dev couldn't fix it, I would go the dumb way:

   1. Remove the 2 replicated brick
   2. Reset these 2 bricks using new uuid/volumes (like if they were just
   installed)
   3. rsync data directly from current nodes to this new 2 nodes cluster
   4. reset the 2 remaining bricks
   5. setup them to replicate the others one.

If you are installing 3.7.2, all clients should be updated before

Using fuse (native gluster protocol/client), load is pushed on clients
(they write to both replica). So they have to be tuned too.

just my 2cents on how I would try to resolve this issue


Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2015-07-07 6:43 GMT+02:00 Geoffrey Letessier <geoffrey.letessier at
cnrs.fr>:
> Dear all,
>
> We are currently in a very critical situation because all our scientific
> production in our computing center is stopped since more than a week for
> maintenance (because of metadata "corruption"), I tried some
scripts
> provided my GlusterFS support team with no effect and I can not continue in
> this way.
>
> So,I wonder if it exists a way to regenerate all bricks metadata
> (.glusterfs directories) after having remove ".glusterfs"
directory on each
> brick (probably after having upgraded GlusterFS from v3.5.3 to v3.7.2)
>
> Your volume is distributed and replicated on 4 servers with 2 bricks each
> (i.e. distributed on 2 servers and replicated on the 2 others). My volume
> is not stripped
>
> In addition, because my the past I noticed a lot of performance issue,
> probably because some users play with a big files and some others with a
> lot (thousands, hundreds of thousands nay more) small files (hundreds of
> kB), could you advice me about how to best tune the volume with performance
> parameters?
>
> Thanks in advance,
> Geoffrey
> ------------------------------------------------------
> Geoffrey Letessier
> Responsable informatique & ing?nieur syst?me
> UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
> Institut de Biologie Physico-Chimique
> 13, rue Pierre et Marie Curie - 75005 Paris
> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150707/6c59a36e/attachment.html>

Is it possible for you to come to #gluster on IRC if you are not 
already? My nick is pranithk

Pranith
On 07/07/2015 10:13 AM, Geoffrey Letessier wrote:
> Dear all,
>
> We are currently in a very critical situation because all our scientific
production in our computing center is stopped since more than a week for
maintenance (because of metadata "corruption"), I tried some scripts
provided my GlusterFS support team with no effect and I can not continue in this
way.
>
> So,I wonder if it exists a way to regenerate all bricks metadata
(.glusterfs directories) after having remove ".glusterfs" directory on
each brick (probably after having upgraded GlusterFS from v3.5.3 to v3.7.2)
>
> Your volume is distributed and replicated on 4 servers with 2 bricks each
(i.e. distributed on 2 servers and replicated on the 2 others). My volume is not
stripped
>
> In addition, because my the past I noticed a lot of performance issue,
probably because some users play with a big files and some others with a lot
(thousands, hundreds of thousands nay more) small files (hundreds of kB), could
you advice me about how to best tune the volume with performance parameters?
>
> Thanks in advance,
> Geoffrey
> ------------------------------------------------------
> Geoffrey Letessier
> Responsable informatique & ing?nieur syst?me
> UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
> Institut de Biologie Physico-Chimique
> 13, rue Pierre et Marie Curie - 75005 Paris
> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users