Gluster users - Nov 2017 - Gluster Scale Limitations

Hi all,
Are there any scale limitations in terms of how many nodes can be in a single
Gluster Cluster or how much storage capacity can be managed in a single cluster?
What are some of the large deployments out there that you know of?

Thanks,
Mayur


***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for
the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171030/94d4fc37/attachment.html>

Hi, After ~2500 bricks it takes too much time bricks getting online
after a reboot. So I think ~2500 bricks is an upper limit per cluster.
I have two 40 nodes/19PiB clusters. They have only one big EC volume
and used for backup/archive purpose.

On Tue, Oct 31, 2017 at 12:51 AM, Mayur Dewaikar
<mdewaikar at commvault.com> wrote:
> Hi all,
>
> Are there any scale limitations in terms of how many nodes can be in a
> single Gluster Cluster or how much storage capacity can be managed in a
> single cluster? What are some of the large deployments out there that you
> know of?
>
>
>
> Thanks,
>
> Mayur
>
>
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
for the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank
you."
> **********************************************************************
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

On Tue, 31 Oct 2017 at 03:32, Mayur Dewaikar <mdewaikar at commvault.com>
wrote:
> Hi all,
>
> Are there any scale limitations in terms of how many nodes can be in a
> single Gluster Cluster or how much storage capacity can be managed in a
> single cluster? What are some of the large deployments out there that you
> know of?
>
>
The current design of GlusterD is not capable of handling too many nodes in
the cluster specially on the node restart/reboot condition. We have heard
about deployments with ~100-150 nodes where things are stable but in node
reboot scenario some special tweaking of parameters like
network.listen-backlog is required to ensure TCP packets don?t get
overflowed resulting into connection between brick to glusterd fail.
GlusterD2 project will definitely address this aspect of the problems.

Also since all the directory layouts are replicated on all the bricks of a
volume, mkdir, unlink or any other directory operations are costly and with
more number of bricks this impacts the latency. We?re also working on a
project called RIO to address this issue.

>
> Thanks,
>
> Mayur
>
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank
you."
> **********************************************************************
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171102/09bc9e52/attachment.html>

Hi All?
Thanks for the responses. I am mainly curious about performance impact for
read/write workloads associated with metadata updates as the number of nodes
increase. Any commentary on performance impact specific to various read/write
random/sequential IO scenario as the scale increases? Not particularly worried
about restart/reboot condition as that is an edge use case for us.


Thanks,
Mayur



From: Atin Mukherjee [mailto:amukherj at redhat.com]
Sent: Wednesday, November 1, 2017 8:53 PM
To: Mayur Dewaikar <mdewaikar at commvault.com>; gluster-users at
gluster.org
Subject: Re: [Gluster-users] Gluster Scale Limitations


On Tue, 31 Oct 2017 at 03:32, Mayur Dewaikar <mdewaikar at
commvault.com<mailto:mdewaikar at commvault.com>> wrote:
Hi all,
Are there any scale limitations in terms of how many nodes can be in a single
Gluster Cluster or how much storage capacity can be managed in a single cluster?
What are some of the large deployments out there that you know of?

The current design of GlusterD is not capable of handling too many nodes in the
cluster specially on the node restart/reboot condition. We have heard about
deployments with ~100-150 nodes where things are stable but in node reboot
scenario some special tweaking of parameters like network.listen-backlog is
required to ensure TCP packets don?t get overflowed resulting into connection
between brick to glusterd fail. GlusterD2 project will definitely address this
aspect of the problems.

Also since all the directory layouts are replicated on all the bricks of a
volume, mkdir, unlink or any other directory operations are costly and with more
number of bricks this impacts the latency. We?re also working on a project
called RIO to address this issue.


Thanks,
Mayur


***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for
the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users
--
- Atin (atinm)
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for
the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171102/651f22d8/attachment.html>