Gluster users - Aug 2015 - cluster.min-free-disk is not working in distributed disperse volume

Hi,
  Cluster.min-free-disk controls new file creation on the bricks. If you happen
to write to the existing files on the brick and that is leading to brick getting
full, then most probably you should run a rebalance.

Regards,
Susant

----- Original Message -----
From: "Mathieu Chateau" <mathieu.chateau at lotp.fr>
To: "Mohamed Pakkeer" <mdfakkeer at gmail.com>
Cc: "gluster-users" <gluster-users at gluster.org>,
"Gluster Devel" <gluster-devel at gluster.org>
Sent: Monday, 24 August, 2015 2:47:00 PM
Subject: Re: [Gluster-users] cluster.min-free-disk is not working in distributed
disperse volume




720 brick! Respect ! 
Le 24 ao?t 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at gmail.com
> a ?crit :



Hi, 


I have a cluster of 720 bricks, all bricks are 4TB in size. I have change the
cluster.min-free-disk default value 10% to 3%. So all the disks should have 3%
minimum disk space free. But some cluster disks are getting full now. Is there
any additional configuration for keeping some percentage of disk space kept
free?





Volume Name: glustertest 
Type: Distributed-Disperse 
Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609 
Status: Started 
Number of Bricks: 72 x (8 + 2) = 720 
Transport-type: tcp 





Options Reconfigured: 
features.default-soft-limit: 95% 
cluster.min-free-disk: 3% 
performance.readdir-ahead: on 


df -h of one node 



/dev/sdb1 3.7T 3.6T 132G 97% /media/disk1 
/dev/sdc1 3.7T 3.2T 479G 88% /media/disk2 
/dev/sdd1 3.7T 3.6T 109G 98% /media/disk3 


Any help will be greatly appreciated. 




Regards 
Backer 







_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Dear Susant,

Am Mo, 24.08.2015, 11:27 schrieb Susant Palai:
>   Cluster.min-free-disk controls new file creation on the bricks. If you
> happen to write to the existing files on the brick and that is leading
> to brick getting full, then most probably you should run a rebalance.
I've had a similar issue on our setup a few months ago.

It seems that "gluster.min-free-disk" was only triggering warnings to
be
written in the logs that the disks are getting full.
The disks were written so full, that XFS ran out of space of its metadata,
causing the filesystem on the bricks to become unmountable even :(


That's with Gluster v3.4. Maybe in newer versions it prevents data from
being written?

Here's a link to the old thread, if you're interested:
http://www.gluster.org/pipermail/gluster-users/2015-March/021341.html


Regards,
Pb

Hi Susant,

   Thanks for your quick reply. We are not updating any files. Actually we
are archiving video files on this cluster. I think there is a bug in
cluster.min-free-disk.

Also i would like to know about rebalance the cluster. Currently we have 20
nodes and 10 nodes hard disks are almost full . So we need to rebalance the
data. If i run the rebalancer, it starts on first node(node1)  and starts
the migration process. The first node cpu usage is always high during
rebalance compare with rest of the cluster nodes.To reduce the cpu usage of
rebalancer  datanode( node1), i peer a new node( without disk) for
rebalance and start the rebalancer. It starts again the rebalancer on same
node1. How can we run a rebalancer on a dedicated node?

Also we are facing memory leaks in fixlayout and heal full operations.

Regards
Backer

On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai <spalai at redhat.com>
wrote:
> Hi,
>   Cluster.min-free-disk controls new file creation on the bricks. If you
> happen to write to the existing files on the brick and that is leading to
> brick getting full, then most probably you should run a rebalance.
>
> Regards,
> Susant
>
> ----- Original Message -----
> From: "Mathieu Chateau" <mathieu.chateau at lotp.fr>
> To: "Mohamed Pakkeer" <mdfakkeer at gmail.com>
> Cc: "gluster-users" <gluster-users at gluster.org>,
"Gluster Devel" <
> gluster-devel at gluster.org>
> Sent: Monday, 24 August, 2015 2:47:00 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
>
>
> 720 brick! Respect !
> Le 24 ao?t 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at
gmail.com > a ?crit :
>
>
>
> Hi,
>
>
> I have a cluster of 720 bricks, all bricks are 4TB in size. I have change
> the cluster.min-free-disk default value 10% to 3%. So all the disks should
> have 3% minimum disk space free. But some cluster disks are getting full
> now. Is there any additional configuration for keeping some percentage of
> disk space kept free?
>
>
>
>
>
> Volume Name: glustertest
> Type: Distributed-Disperse
> Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609
> Status: Started
> Number of Bricks: 72 x (8 + 2) = 720
> Transport-type: tcp
>
>
>
>
>
> Options Reconfigured:
> features.default-soft-limit: 95%
> cluster.min-free-disk: 3%
> performance.readdir-ahead: on
>
>
> df -h of one node
>
>
>
> /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1
> /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2
> /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3
>
>
> Any help will be greatly appreciated.
>
>
>
>
> Regards
> Backer
>
>
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


-- 
Thanks & Regards
K.Mohamed Pakkeer
Mobile- 0091-8754410114
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150824/ce12a83b/attachment.html>