Gluster users - May 2014 - Decomissioned brick, mismatching layouts, EINVAL on file/folder create

Hello,

  sorry for the strange title, but it might be possible that those
symptoms are related...

  I'm running a Gluster 3.4.2 volume with 8 bricks in distribute mode.

  Easiest problem to diagnose : sometimes a folder goes into a state
where any attempt to create a new file or folder within ends up with
EINVAL (being root or not).

  Those errors may persist or not (sometimes it's fixed by itself a
while later), and it seems to be related to this kind of information in
the client's log :

[2014-05-14 19:17:12.937364] I [dht-common.c:623:dht_revalidate_cbk]
0-xxx-prod-dht: mismatching layouts for /uploads/images/5373
[2014-05-14 19:17:12.938960] I
[dht-layout.c:726:dht_layout_dir_mismatch] 0-xxx-prod-dht:
/uploads/images/5373/9c8b - disk layout missing

  ... where /uploads/images/5373 is a folder which I can list with
readable file, but where creating new files and folder is impossible.


  Meanwhile I historically tried to remove a brick from this volume
which has 8 of them. The rebalance operation took ages and I had to
cancel it because I was suffering from performance problems. Now the 8th
brick is shown in the volume status but marked 'decomissioned' in the
internal metadata, and has this property : while all 8 bricks have the
same inode count (roughly 5 millions), only the 7th first have a
balanced block usage with 800 GB, while the 8th has been stuck 270 GB
(the point where I tried to remove it).

  It looks expected. The 8th brick is still part of the volume but has
only 'historical' data which is still needed. I'd like to re-add
this
brick into my volume, is it as simple as issuing a 'volume add-brick'
and then a 'volume rebalance' ?

  In any case if there's a way to avoid those bugs, I'd be interested to
hear about it. Thanks for your experience and your help !

On Thu, May 15, 2014 at 1:16 AM, Vincent Caron <vcaron at bearstech.com>
wrote:
> Hello,
>
>   sorry for the strange title, but it might be possible that those
> symptoms are related...
>
>   I'm running a Gluster 3.4.2 volume with 8 bricks in distribute mode.
>
>   Easiest problem to diagnose : sometimes a folder goes into a state
> where any attempt to create a new file or folder within ends up with
> EINVAL (being root or not).
>
>   Those errors may persist or not (sometimes it's fixed by itself a
> while later), and it seems to be related to this kind of information in
> the client's log :
>
> [2014-05-14 19:17:12.937364] I [dht-common.c:623:dht_revalidate_cbk]
> 0-xxx-prod-dht: mismatching layouts for /uploads/images/5373
> [2014-05-14 19:17:12.938960] I
> [dht-layout.c:726:dht_layout_dir_mismatch] 0-xxx-prod-dht:
> /uploads/images/5373/9c8b - disk layout missing
>
>   ... where /uploads/images/5373 is a folder which I can list with
> readable file, but where creating new files and folder is impossible.
>
>
>   Meanwhile I historically tried to remove a brick from this volume
> which has 8 of them. The rebalance operation took ages and I had to
> cancel it because I was suffering from performance problems. Now the 8th
> brick is shown in the volume status but marked 'decomissioned' in
the
> internal metadata, and has this property : while all 8 bricks have the
> same inode count (roughly 5 millions), only the 7th first have a
> balanced block usage with 800 GB, while the 8th has been stuck 270 GB
> (the point where I tried to remove it).
>
>   It looks expected. The 8th brick is still part of the volume but has
> only 'historical' data which is still needed. I'd like to
re-add this
> brick into my volume, is it as simple as issuing a 'volume
add-brick'
> and then a 'volume rebalance' ?
>
If you haven't yet commited the remove-brick operation, you can do
remove-brick stop followed by rebalance start operation.

i.e. gluster volume remove-brick <vol> <brick> stop
     gluster volume rebalance <vol> start
>
>   In any case if there's a way to avoid those bugs, I'd be
interested to
> hear about it. Thanks for your experience and your help !
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


-- 
Thanks & Regards
Shylesh Kumar M
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140515/7a3a666b/attachment.html>