Pranith Kumar Karampuri
2017-May-05 11:54 UTC
[Gluster-users] disperse volume brick counts limits in RHES
On Fri, May 5, 2017 at 5:19 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 2:38 PM, Serkan ?oban <cobanserkan at gmail.com> > wrote: > >> It is the over all time, 8TB data disk healed 2x faster in 8+2 >> configuration. >> > > Wow, that is counter intuitive for me. I will need to explore about this > to find out why that could be. Thanks a lot for this feedback! >>From memory I remember you said you have a lot of small files hosted on thevolume, right? It could be because of the bug https://review.gluster.org/17151 is fixing. That is the only reason I could guess right now. We will try to test this kind of case if you could give us a bit more details about average file-size/depth of directories etc to simulate similar looking directory structure.> > >> >> On Fri, May 5, 2017 at 10:00 AM, Pranith Kumar Karampuri >> <pkarampu at redhat.com> wrote: >> > >> > >> > On Fri, May 5, 2017 at 11:42 AM, Serkan ?oban <cobanserkan at gmail.com> >> wrote: >> >> >> >> Healing gets slower as you increase m in m+n configuration. >> >> We are using 16+4 configuration without any problems other then heal >> >> speed. >> >> I tested heal speed with 8+2 and 16+4 on 3.9.0 and see that heals on >> >> 8+2 is faster by 2x. >> > >> > >> > As you increase number of nodes that are participating in an EC set >> number >> > of parallel heals increase. Is the heal speed you saw improved per file >> or >> > the over all time it took to heal the data? >> > >> >> >> >> >> >> >> >> On Fri, May 5, 2017 at 9:04 AM, Ashish Pandey <aspandey at redhat.com> >> wrote: >> >> > >> >> > 8+2 and 8+3 configurations are not the limitation but just >> suggestions. >> >> > You can create 16+3 volume without any issue. >> >> > >> >> > Ashish >> >> > >> >> > ________________________________ >> >> > From: "Alastair Neil" <ajneil.tech at gmail.com> >> >> > To: "gluster-users" <gluster-users at gluster.org> >> >> > Sent: Friday, May 5, 2017 2:23:32 AM >> >> > Subject: [Gluster-users] disperse volume brick counts limits in RHES >> >> > >> >> > >> >> > Hi >> >> > >> >> > we are deploying a large (24node/45brick) cluster and noted that the >> >> > RHES >> >> > guidelines limit the number of data bricks in a disperse set to 8. >> Is >> >> > there >> >> > any reason for this. I am aware that you want this to be a power of >> 2, >> >> > but >> >> > as we have a large number of nodes we were planning on going with >> 16+3. >> >> > Dropping to 8+2 or 8+3 will be a real waste for us. >> >> > >> >> > Thanks, >> >> > >> >> > >> >> > Alastair >> >> > >> >> > >> >> > _______________________________________________ >> >> > Gluster-users mailing list >> >> > Gluster-users at gluster.org >> >> > http://lists.gluster.org/mailman/listinfo/gluster-users >> >> > >> >> > >> >> > _______________________________________________ >> >> > Gluster-users mailing list >> >> > Gluster-users at gluster.org >> >> > http://lists.gluster.org/mailman/listinfo/gluster-users >> >> _______________________________________________ >> >> Gluster-users mailing list >> >> Gluster-users at gluster.org >> >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >> > >> > >> > >> > -- >> > Pranith >> > > > > -- > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/cf6b61d8/attachment.html>
Serkan Çoban
2017-May-05 13:51 UTC
[Gluster-users] disperse volume brick counts limits in RHES
In our use case every node has 26 bricks. I am using 60 nodes, one 9PB volume with 16+4 EC configuration, each brick in a sub-volume is on different host. We put 15-20k 2GB files every day into 10-15 folders. So it is 1500K files/folder. Our gluster version is 3.7.11. Heal speed in this environment is 8-10MB/sec/brick. I did some tests for parallel self heal feature with version 3.9, two servers 26 bricks each, 8+2 and 16+4 EC configuration. This was a small test environment and the results are as I said 8+2 is 2x faster then 16+4 with parallel self heal threads set to 2/4. In 1-2 months our new servers arriving, I will do detailed tests for heal performance for 8+2 and 16+4 and inform you the results. On Fri, May 5, 2017 at 2:54 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 5:19 PM, Pranith Kumar Karampuri > <pkarampu at redhat.com> wrote: >> >> >> >> On Fri, May 5, 2017 at 2:38 PM, Serkan ?oban <cobanserkan at gmail.com> >> wrote: >>> >>> It is the over all time, 8TB data disk healed 2x faster in 8+2 >>> configuration. >> >> >> Wow, that is counter intuitive for me. I will need to explore about this >> to find out why that could be. Thanks a lot for this feedback! > > > From memory I remember you said you have a lot of small files hosted on the > volume, right? It could be because of the bug > https://review.gluster.org/17151 is fixing. That is the only reason I could > guess right now. We will try to test this kind of case if you could give us > a bit more details about average file-size/depth of directories etc to > simulate similar looking directory structure. > >> >> >>> >>> >>> On Fri, May 5, 2017 at 10:00 AM, Pranith Kumar Karampuri >>> <pkarampu at redhat.com> wrote: >>> > >>> > >>> > On Fri, May 5, 2017 at 11:42 AM, Serkan ?oban <cobanserkan at gmail.com> >>> > wrote: >>> >> >>> >> Healing gets slower as you increase m in m+n configuration. >>> >> We are using 16+4 configuration without any problems other then heal >>> >> speed. >>> >> I tested heal speed with 8+2 and 16+4 on 3.9.0 and see that heals on >>> >> 8+2 is faster by 2x. >>> > >>> > >>> > As you increase number of nodes that are participating in an EC set >>> > number >>> > of parallel heals increase. Is the heal speed you saw improved per file >>> > or >>> > the over all time it took to heal the data? >>> > >>> >> >>> >> >>> >> >>> >> On Fri, May 5, 2017 at 9:04 AM, Ashish Pandey <aspandey at redhat.com> >>> >> wrote: >>> >> > >>> >> > 8+2 and 8+3 configurations are not the limitation but just >>> >> > suggestions. >>> >> > You can create 16+3 volume without any issue. >>> >> > >>> >> > Ashish >>> >> > >>> >> > ________________________________ >>> >> > From: "Alastair Neil" <ajneil.tech at gmail.com> >>> >> > To: "gluster-users" <gluster-users at gluster.org> >>> >> > Sent: Friday, May 5, 2017 2:23:32 AM >>> >> > Subject: [Gluster-users] disperse volume brick counts limits in RHES >>> >> > >>> >> > >>> >> > Hi >>> >> > >>> >> > we are deploying a large (24node/45brick) cluster and noted that the >>> >> > RHES >>> >> > guidelines limit the number of data bricks in a disperse set to 8. >>> >> > Is >>> >> > there >>> >> > any reason for this. I am aware that you want this to be a power of >>> >> > 2, >>> >> > but >>> >> > as we have a large number of nodes we were planning on going with >>> >> > 16+3. >>> >> > Dropping to 8+2 or 8+3 will be a real waste for us. >>> >> > >>> >> > Thanks, >>> >> > >>> >> > >>> >> > Alastair >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > Gluster-users mailing list >>> >> > Gluster-users at gluster.org >>> >> > http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > Gluster-users mailing list >>> >> > Gluster-users at gluster.org >>> >> > http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> _______________________________________________ >>> >> Gluster-users mailing list >>> >> Gluster-users at gluster.org >>> >> http://lists.gluster.org/mailman/listinfo/gluster-users >>> > >>> > >>> > >>> > >>> > -- >>> > Pranith >> >> >> >> >> -- >> Pranith > > > > > -- > Pranith