Gluster users - Aug 2011 - cluster.min-free-disk separate for each, brick

On 15/08/11 20:00, gluster-users-request at gluster.org
wrote:
> Message: 1
> Date: Sun, 14 Aug 2011 23:24:46 +0300
> From: "Deyan Chepishev - SuperHosting.BG"<dchepishev at
superhosting.bg>
> Subject: [Gluster-users] cluster.min-free-disk  separate for each
> 	brick
> To: gluster-users at gluster.org
> Message-ID:<4E482F0E.3030604 at superhosting.bg>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hello,
>
> I have a gluster set up with very different brick sizes.
>
> brick1: 9T
> brick2: 9T
> brick3: 37T
>
> with this configuration if I set the parameter cluster.min-free-disk to 10%
it
> applies to all bricks which is quite uncomfortable with these brick sizes,
> because 10% for the small bricks are ~ 1T but for the big brick it is ~3.7T
and
> what happens at the end is that if all brick go to 90% usage and I continue
> writing, the small ones eventually fill up to 100% while the big one has
enough
> free space.
>
> My question is, is there a way to set cluster.min-free-disk per brick
instead
> setting it for the entire volume or any other way to work around this
problem ?
>
> Thank you in advance
>
> Regards,
> Deyan
>
Hello Deyan,

I have exactly the same problem and I have asked about it before - see 
links below.

http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/
http://gluster.org/pipermail/gluster-users/2011-May/007788.html

My understanding is that the patch referred to in Amar's reply in the 
May thread prevents a "migrate-data" rebalance operation failing by 
running out of space on smaller bricks, but that doesn't solve the 
problem we are having.  Being able to set min-free-disk for each brick 
separately would be useful, as would being able to set this value as a 
number of bytes rather than a percentage.  However, even if these 
features were present we would still have a problem when the amount of 
free space becomes less than min-free-disk, because this just results in 
a warning message in the logs and doesn't actually prevent more files 
from being written.  In other words, min-free-disk is a soft limit 
rather than a hard limit.  When a volume is more than 90% full there may 
still be hundreds of gigabytes of free space spread over the large 
bricks, but the small bricks may each only have a few gigabytes left of 
even less.  Users do "df" and see lots of free space in the volume so 
they continue writing files.  However, when GlusterFS chooses to write a 
file to a small brick, the write fails with "device full" errors if
the
file grows too large, which is often the case here with files typically 
several gigabytes in size for some applications.

I would really like to know if there is a way to make min-free-disk a 
hard limit.  Ideally, GlusterFS would chose a brick on which to write a 
file based on how much free space it has left rather than choosing a 
brick at random (or however it is done now).  That would solve the 
problem of non-uniform brick sizes without the need for a hard 
min-free-disk limit.

Amar's comment in the May thread about QA testing being done only on 
volumes with uniform brick sizes prompted me to start standardising on a 
uniform brick size for each volume in my cluster.  My impression is that 
implementing the features needed for users with non-uniform brick sizes 
is not a priority for Gluster, and that users are all expected to use 
uniform brick sizes.  I really think this fact should be stated clearly 
in the GlusterFS documentation, in the sections on creating volumes in 
the Administration Guide for example.  That would stop other users from 
going down the path that I did initially, which has given me a real 
headache because I am now having to move tens of terabytes of data off 
bricks that are larger than the new standard size.

Regards
Dan.

-- 
Mr. D.A. Bretherton
Computer System Manager
Environmental Systems Science Centre
Harry Pitt Building
3 Earley Gate
University of Reading
Reading, RG6 6AL
UK

Tel. +44 118 378 5205
Fax: +44 118 378 6413

Hello,

This is really bad news, because I already migrated my data and I just realized 
that I am screwed because Gluster just does not care about the brick sizes.
It is impossible to move to uniform brick sizes.

Currently we use 2TB  HDDs, but the disks are growing and soon we will probably 
use 3TB hdds or whatever other larges sizes appear on the market. So if we 
choose to use raid5 and some level of redundancy (for example 6hdds in raid5, no
matter what their size is) this sooner or later will lead us to non uniform 
bricks which is a problem and it is not correct to expect that we always can or 
want to provide uniform size bricks.

With this way of thinking if we currently have 10T from 6x2T in hdd5, at some 
point when there is a 10T on a single disk we will have to use no raid just 
because gluster can not handle non uniform bricks.

Regards,
Deyan






Dan Bretherton wrote:
>
> On 15/08/11 20:00, gluster-users-request at gluster.org wrote:
>> Message: 1
>> Date: Sun, 14 Aug 2011 23:24:46 +0300
>> From: "Deyan Chepishev - SuperHosting.BG"<dchepishev at
superhosting.bg>
>> Subject: [Gluster-users] cluster.min-free-disk  separate for each
>>     brick
>> To: gluster-users at gluster.org
>> Message-ID:<4E482F0E.3030604 at superhosting.bg>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> Hello,
>>
>> I have a gluster set up with very different brick sizes.
>>
>> brick1: 9T
>> brick2: 9T
>> brick3: 37T
>>
>> with this configuration if I set the parameter cluster.min-free-disk to
10% it
>> applies to all bricks which is quite uncomfortable with these brick
sizes,
>> because 10% for the small bricks are ~ 1T but for the big brick it is
~3.7T and
>> what happens at the end is that if all brick go to 90% usage and I
continue
>> writing, the small ones eventually fill up to 100% while the big one
has enough
>> free space.
>>
>> My question is, is there a way to set cluster.min-free-disk per brick
instead
>> setting it for the entire volume or any other way to work around this
problem ?
>>
>> Thank you in advance
>>
>> Regards,
>> Deyan
>>
> Hello Deyan,
>
> I have exactly the same problem and I have asked about it before - see
links
> below.
>
>
http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/
>
> http://gluster.org/pipermail/gluster-users/2011-May/007788.html
>
> My understanding is that the patch referred to in Amar's reply in the
May
> thread prevents a "migrate-data" rebalance operation failing by
running out of
> space on smaller bricks, but that doesn't solve the problem we are
having.
> Being able to set min-free-disk for each brick separately would be useful,
as
> would being able to set this value as a number of bytes rather than a 
> percentage.  However, even if these features were present we would still
have
> a problem when the amount of free space becomes less than min-free-disk, 
> because this just results in a warning message in the logs and doesn't 
> actually prevent more files from being written.  In other words,
min-free-disk
> is a soft limit rather than a hard limit.  When a volume is more than 90%
full
> there may still be hundreds of gigabytes of free space spread over the
large
> bricks, but the small bricks may each only have a few gigabytes left of
even
> less.  Users do "df" and see lots of free space in the volume so
they continue
> writing files.  However, when GlusterFS chooses to write a file to a small 
> brick, the write fails with "device full" errors if the file
grows too large,
> which is often the case here with files typically several gigabytes in size
> for some applications.
>
> I would really like to know if there is a way to make min-free-disk a hard 
> limit.  Ideally, GlusterFS would chose a brick on which to write a file
based
> on how much free space it has left rather than choosing a brick at random
(or
> however it is done now).  That would solve the problem of non-uniform brick
> sizes without the need for a hard min-free-disk limit.
>
> Amar's comment in the May thread about QA testing being done only on
volumes
> with uniform brick sizes prompted me to start standardising on a uniform
brick
> size for each volume in my cluster.  My impression is that implementing the
> features needed for users with non-uniform brick sizes is not a priority
for
> Gluster, and that users are all expected to use uniform brick sizes.  I
really
> think this fact should be stated clearly in the GlusterFS documentation, in
> the sections on creating volumes in the Administration Guide for example.  
> That would stop other users from going down the path that I did initially, 
> which has given me a real headache because I am now having to move tens of 
> terabytes of data off bricks that are larger than the new standard size.
>
> Regards
> Dan.
>