Mohamed Pakkeer
2015-Aug-24 10:58 UTC
[Gluster-users] cluster.min-free-disk is not working in distributed disperse volume
Hi Susant, Thanks for your quick reply. We are not updating any files. Actually we are archiving video files on this cluster. I think there is a bug in cluster.min-free-disk. Also i would like to know about rebalance the cluster. Currently we have 20 nodes and 10 nodes hard disks are almost full . So we need to rebalance the data. If i run the rebalancer, it starts on first node(node1) and starts the migration process. The first node cpu usage is always high during rebalance compare with rest of the cluster nodes.To reduce the cpu usage of rebalancer datanode( node1), i peer a new node( without disk) for rebalance and start the rebalancer. It starts again the rebalancer on same node1. How can we run a rebalancer on a dedicated node? Also we are facing memory leaks in fixlayout and heal full operations. Regards Backer On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai <spalai at redhat.com> wrote:> Hi, > Cluster.min-free-disk controls new file creation on the bricks. If you > happen to write to the existing files on the brick and that is leading to > brick getting full, then most probably you should run a rebalance. > > Regards, > Susant > > ----- Original Message ----- > From: "Mathieu Chateau" <mathieu.chateau at lotp.fr> > To: "Mohamed Pakkeer" <mdfakkeer at gmail.com> > Cc: "gluster-users" <gluster-users at gluster.org>, "Gluster Devel" < > gluster-devel at gluster.org> > Sent: Monday, 24 August, 2015 2:47:00 PM > Subject: Re: [Gluster-users] cluster.min-free-disk is not working in > distributed disperse volume > > > > > 720 brick! Respect ! > Le 24 ao?t 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at gmail.com > a ?crit : > > > > Hi, > > > I have a cluster of 720 bricks, all bricks are 4TB in size. I have change > the cluster.min-free-disk default value 10% to 3%. So all the disks should > have 3% minimum disk space free. But some cluster disks are getting full > now. Is there any additional configuration for keeping some percentage of > disk space kept free? > > > > > > Volume Name: glustertest > Type: Distributed-Disperse > Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609 > Status: Started > Number of Bricks: 72 x (8 + 2) = 720 > Transport-type: tcp > > > > > > Options Reconfigured: > features.default-soft-limit: 95% > cluster.min-free-disk: 3% > performance.readdir-ahead: on > > > df -h of one node > > > > /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1 > /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2 > /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3 > > > Any help will be greatly appreciated. > > > > > Regards > Backer > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-- Thanks & Regards K.Mohamed Pakkeer Mobile- 0091-8754410114 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150824/ce12a83b/attachment.html>
Mohamed Pakkeer
2015-Aug-25 04:10 UTC
[Gluster-users] cluster.min-free-disk is not working in distributed disperse volume
Hi Sasant, We have created the disperse volume across nodes. We stopped all the upload operations and started the rebalance last night.After overnight re-balance, some harddisk is occupied 100% and some disks have 13% disk space. disk1 belongs to disperse-set-0 ..... disk36 belongs to disperse-set-35 df -h result of one data node */dev/sdb1 3.7T 3.7T 545M 100% /media/disk1* /dev/sdc1 3.7T 3.2T 496G *87% /media/disk2* /dev/sdd1 3.7T 3.7T 30G 100% /media/disk3 /dev/sde1 3.7T 3.5T 173G 96% /media/disk4 /dev/sdf1 3.7T 3.2T 458G 88% /media/disk5 /dev/sdg1 3.7T 3.5T 143G 97% /media/disk6 /dev/sdh1 3.7T 3.5T 220G 95% /media/disk7 /dev/sdi1 3.7T 3.3T 415G 89% /media/disk8 /dev/sdj1 3.7T 3.6T 72G 99% /media/disk9 /dev/sdk1 3.7T 3.5T 186G 96% /media/disk10 /dev/sdl1 3.7T 3.6T 65G 99% /media/disk11 /dev/sdm1 3.7T 3.5T 195G 95% /media/disk12 /dev/sdn1 3.7T 3.5T 199G 95% /media/disk13 /dev/sdo1 3.7T 3.6T 78G 98% /media/disk14 /dev/sdp1 3.7T 3.5T 200G 95% /media/disk15 /dev/sdq1 3.7T 3.6T 119G 97% /media/disk16 /dev/sdr1 3.7T 3.5T 206G 95% /media/disk17 /dev/sds1 3.7T 3.5T 193G 95% /media/disk18 /dev/sdt1 3.7T 3.6T 131G 97% /media/disk19 /dev/sdu1 3.7T 3.5T 141G 97% /media/disk20 /dev/sdv1 3.7T 3.5T 243G 94% /media/disk21 /dev/sdw1 3.7T 3.4T 299G 92% /media/disk22 /dev/sdx1 3.7T 3.5T 163G 96% /media/disk23 /dev/sdy1 3.7T 3.5T 168G 96% /media/disk24 /dev/sdz1 3.7T 3.5T 219G 95% /media/disk25 */dev/sdaa1 3.7T 3.7T 37G 100% /media/disk26* /dev/sdab1 3.7T 3.5T 172G 96% /media/disk27 /dev/sdac1 3.7T 3.4T 276G 93% /media/disk28 /dev/sdad1 3.7T 3.6T 108G 98% /media/disk29 /dev/sdae1 3.7T 3.3T 399G 90% /media/disk30 /dev/sdaf1 3.7T 3.5T 240G 94% /media/disk31 /dev/sdag1 3.7T 3.6T 122G 97% /media/disk32 /dev/sdah1 3.7T 3.5T 147G 97% /media/disk33 /dev/sdai1 3.7T 3.4T 342G 91% /media/disk34 /dev/sdaj1 3.7T 3.4T 288G 93% /media/disk35 /dev/sdak1 3.7T 3.4T 342G 91% /media/disk36 *disk1 belongs to disperse-set-0.* Rebalancer logs shows, still rebalancer is trying to fill the disperse-set-0 after filling to 100% [2015-08-24 19:52:53.036622] E [MSGID: 109023] [dht-rebalance.c:672:__dht_check_free_space] 0-glustertest-dht: data movement attempted from node (glustertest-disperse-7) to node *(glustertest-disperse-0) which does not have required free space* for (/Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMI-NEBO-C_R3_AUDIO_190511.mxf) [2015-08-24 19:52:53.042026] I [dht-rebalance.c:1002:dht_migrate_file] 0-glustertest-dht: /Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMINEBO-CHALLANGE_BEN_R1-2-3-4-5-6_MPEG_200511-reel-5-mpeg2.mxf: attempting to move from glustertest-disperse-13 to glustertest-disperse-0 I think, cluster.weighted-rebalance and cluster.min-free-disk have bugs for re-balancing the data, based on weight and disk free space. Thanks Backer On Mon, Aug 24, 2015 at 4:28 PM, Mohamed Pakkeer <mdfakkeer at gmail.com> wrote:> Hi Susant, > > Thanks for your quick reply. We are not updating any files. Actually we > are archiving video files on this cluster. I think there is a bug in > cluster.min-free-disk. > > Also i would like to know about rebalance the cluster. Currently we have > 20 nodes and 10 nodes hard disks are almost full . So we need to rebalance > the data. If i run the rebalancer, it starts on first node(node1) and > starts the migration process. The first node cpu usage is always high > during rebalance compare with rest of the cluster nodes.To reduce the cpu > usage of rebalancer datanode( node1), i peer a new node( without disk) for > rebalance and start the rebalancer. It starts again the rebalancer on same > node1. How can we run a rebalancer on a dedicated node? > > Also we are facing memory leaks in fixlayout and heal full operations. > > Regards > Backer > > On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai <spalai at redhat.com> wrote: > >> Hi, >> Cluster.min-free-disk controls new file creation on the bricks. If you >> happen to write to the existing files on the brick and that is leading to >> brick getting full, then most probably you should run a rebalance. >> >> Regards, >> Susant >> >> ----- Original Message ----- >> From: "Mathieu Chateau" <mathieu.chateau at lotp.fr> >> To: "Mohamed Pakkeer" <mdfakkeer at gmail.com> >> Cc: "gluster-users" <gluster-users at gluster.org>, "Gluster Devel" < >> gluster-devel at gluster.org> >> Sent: Monday, 24 August, 2015 2:47:00 PM >> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in >> distributed disperse volume >> >> >> >> >> 720 brick! Respect ! >> Le 24 ao?t 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at gmail.com > a ?crit >> : >> >> >> >> Hi, >> >> >> I have a cluster of 720 bricks, all bricks are 4TB in size. I have change >> the cluster.min-free-disk default value 10% to 3%. So all the disks should >> have 3% minimum disk space free. But some cluster disks are getting full >> now. Is there any additional configuration for keeping some percentage of >> disk space kept free? >> >> >> >> >> >> Volume Name: glustertest >> Type: Distributed-Disperse >> Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609 >> Status: Started >> Number of Bricks: 72 x (8 + 2) = 720 >> Transport-type: tcp >> >> >> >> >> >> Options Reconfigured: >> features.default-soft-limit: 95% >> cluster.min-free-disk: 3% >> performance.readdir-ahead: on >> >> >> df -h of one node >> >> >> >> /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1 >> /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2 >> /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3 >> >> >> Any help will be greatly appreciated. >> >> >> >> >> Regards >> Backer >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150825/394a6d8a/attachment.html>