Gluster users - Sep 2014 - Proposal for GlusterD-2.0

GlusterD performs the following functions as the management daemon for
GlusterFS:
- Peer membership management
- Maintains consistency of configuration data across nodes (distributed
configuration store)
- Distributed command execution (orchestration)
- Service management (manage GlusterFS daemons)
- Portmap service for GlusterFS daemons


This proposal aims to delegate the above functions to technologies that
solve these problems well. We aim to do this in a phased manner.
The technology alternatives we would be looking for should have the
following properties,
- Open source
- Vibrant community
- Good documentation
- Easy to deploy/manage

This would allow GlusterD's architecture to be more modular. We also aim to
make GlusterD's architecture as transparent and observable as possible.
Separating out these functions would allow us to do that.

Bulk of current GlusterD code deals with keeping the configuration of the
cluster and the volumes in it consistent and available across the nodes.
The current algorithm is not scalable  (N^2 in no. of nodes) and doesn't
prevent split-brain of configuration. This is the problem area we are
targeting for the first phase.

As part of the first phase, we aim to delegate the distributed
configuration store. We are exploring consul [1] as a replacement for the
existing distributed configuration store (sum total of /var/lib/glusterd/*
across all nodes). Consul provides distributed configuration store which is
consistent and partition tolerant. By moving all Gluster related
configuration information into consul we could avoid split-brain situations.

All development efforts towards this proposal would happen in parallel to
the existing GlusterD code base. The existing code base would be actively
maintained until GlusterD-2.0 is production-ready.

This is in alignment with the GlusterFS Quattro proposals on making
GlusterFS scalable and easy to deploy. This is the first phase ground work
towards that goal.

Questions and suggestions are welcome.

~kaushal

[1] : http://www.consul.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140905/2b568313/attachment.html>

On 05/09/2014, at 11:21 AM, Kaushal M wrote:
<snip>
> As part of the first phase, we aim to delegate the distributed
configuration store. We are exploring consul [1]
Does this mean we'll need to learn Go as well as C and Python?

If so, that doesn't sound completely optimal. :/

That being said, a lot of distributed/networked computing
projects seem to be written in it these days.  Is Go specifically
a good language for our kind of challenges, or is it more a case
of "the new shiny"?

+ Justin

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

On 5 Sep 2014, at 12:21, Kaushal M <kshlmster at gmail.com> wrote:
> - Peer membership management
> - Maintains consistency of configuration data across nodes (distributed
configuration store)
> - Distributed command execution (orchestration)
> - Service management (manage GlusterFS daemons)
> - Portmap service for GlusterFS daemons

Isn't some of this covered by crm/corosync/pacemaker/heartbeat?

Marcus
-- 
Marcus Bointon
Technical Director, Synchromedia Limited

Creators of http://www.smartmessages.net/
UK 1CRM solutions http://www.syniah.com/
marcus at synchromedia.co.uk | http://www.synchromedia.co.uk/

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140905/04e553ef/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140905/04e553ef/attachment.sig>

> As part of the first phase, we aim to delegate the distributed
configuration
> store. We are exploring consul [1] as a replacement for the existing
> distributed configuration store (sum total of /var/lib/glusterd/* across
all
> nodes). Consul provides distributed configuration store which is consistent
> and partition tolerant. By moving all Gluster related configuration
> information into consul we could avoid split-brain situations.
Overall, I like the idea.  But I think you knew that.  ;)

Is the idea to run consul on all nodes as we do with glusterd, or to run
it only on a few nodes (similar to Ceph's mon cluster) and then use them
to coordinate membership etc. for the rest?

On 09/05/2014 03:51 PM, Kaushal M wrote:
> GlusterD performs the following functions as the management daemon for 
> GlusterFS:
> - Peer membership management
> - Maintains consistency of configuration data across nodes 
> (distributed configuration store)
> - Distributed command execution (orchestration)
> - Service management (manage GlusterFS daemons)
> - Portmap service for GlusterFS daemons
>
>
> This proposal aims to delegate the above functions to technologies 
> that solve these problems well. We aim to do this in a phased manner.
> The technology alternatives we would be looking for should have the 
> following properties,
> - Open source
> - Vibrant community
> - Good documentation
> - Easy to deploy/manage
>
> This would allow GlusterD's architecture to be more modular. We also 
> aim to make GlusterD's architecture as transparent and observable as 
> possible. Separating out these functions would allow us to do that.
>
> Bulk of current GlusterD code deals with keeping the configuration of 
> the cluster and the volumes in it consistent and available across the 
> nodes. The current algorithm is not scalable  (N^2 in no. of nodes) 
> and doesn't prevent split-brain of configuration. This is the problem 
> area we are targeting for the first phase.
>
> As part of the first phase, we aim to delegate the distributed 
> configuration store. We are exploring consul [1] as a replacement for 
> the existing distributed configuration store (sum total of 
> /var/lib/glusterd/* across all nodes). Consul provides distributed 
> configuration store which is consistent and partition tolerant. By 
> moving all Gluster related configuration information into consul we 
> could avoid split-brain situations.
Did you get a chance to go over the following questions while making the 
decision? If yes could you please share the info.
What are the consistency guarantees for changing the configuration in 
case of network partitions?
      specifically when there are 2 nodes and 1 of them is not reachable?
      consistency guarantees when there are more than 2 nodes?
What are the consistency guarantees for reading configuration in case of 
network partitions?

Pranith
>
> All development efforts towards this proposal would happen in parallel 
> to the existing GlusterD code base. The existing code base would be 
> actively maintained until GlusterD-2.0 is production-ready.
>
> This is in alignment with the GlusterFS Quattro proposals on making 
> GlusterFS scalable and easy to deploy. This is the first phase ground 
> work towards that goal.
>
> Questions and suggestions are welcome.
>
> ~kaushal
>
> [1] : http://www.consul.io/
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140906/43c7ac73/attachment.html>

----- Original Message -----
> From: "Kaushal M" <kshlmster at gmail.com>
> To: "Gluster Devel" <gluster-devel at gluster.org>
> Cc: gluster-users at gluster.org
> Sent: Friday, September 5, 2014 3:51:35 PM
> Subject: [Gluster-devel] Proposal for GlusterD-2.0
> 
> GlusterD performs the following functions as the management daemon for
> GlusterFS:
> - Peer membership management
> - Maintains consistency of configuration data across nodes (distributed
> configuration store)
> - Distributed command execution (orchestration)
> - Service management (manage GlusterFS daemons)
> - Portmap service for GlusterFS daemons
> 
> 
> This proposal aims to delegate the above functions to technologies that
solve
> these problems well. We aim to do this in a phased manner.
> The technology alternatives we would be looking for should have the
following
> properties,
> - Open source
> - Vibrant community
> - Good documentation
> - Easy to deploy/manage
> 

I did a small PoC on this front in SaltStack[1] which already facilitates this
infrastructure.  The PoC does

1. Adding peers to existing/form cluster.
This comes as default with Salt infra where nodes are managed.  I have nothing
written for gluster peers.

2. Create gluster volume.
This is merely a configuration management.  I am able to solve brick/volume
configuration in matter defining few lines how brick/volume configuration looks
like for plain distribute type volume.

3. Start gluster volume.
This is service management.  This is done by just defining state data what to
do.

WRT glusterd problem, I see Salt already resolves most of them at infrastructure
level.  Its worth considering it.


> This would allow GlusterD's architecture to be more modular. We also
aim to
> make GlusterD's architecture as transparent and observable as possible.
> Separating out these functions would allow us to do that.
> 
> Bulk of current GlusterD code deals with keeping the configuration of the
> cluster and the volumes in it consistent and available across the nodes.
The
> current algorithm is not scalable (N^2 in no. of nodes) and doesn't
prevent
> split-brain of configuration. This is the problem area we are targeting for
> the first phase.
> 
> As part of the first phase, we aim to delegate the distributed
configuration
> store. We are exploring consul [1] as a replacement for the existing
> distributed configuration store (sum total of /var/lib/glusterd/* across
all
> nodes). Consul provides distributed configuration store which is consistent
> and partition tolerant. By moving all Gluster related configuration
> information into consul we could avoid split-brain situations.
> 
> All development efforts towards this proposal would happen in parallel to
the
> existing GlusterD code base. The existing code base would be actively
> maintained until GlusterD-2.0 is production-ready.
> 
> This is in alignment with the GlusterFS Quattro proposals on making
GlusterFS
> scalable and easy to deploy. This is the first phase ground work towards
> that goal.
> 
> Questions and suggestions are welcome.
> 
Regards,
Bala

[1] http://www.saltstack.com/