Gluster users - Mar 2015 - Configure separate network for inter-node communication

Thank you for the detailed explanation. Due to the fact that right now it does
not make much difference to split the traffic I will refrain from doing that and
simply wait for the new style replication. This looks like a very promising
feature and I am looking forward to it. My other concern here would be that I
need to implement a GlusterFS cluster right now with 3.5 so I hope it will
possible to upgrade without much pain from AFR to NSR.


For now, and if I understand everything correclty, it means that if I want
better performance I will need to have all my gluster nodes as well as clients
on a 10 Gigabit network and not just the gluster servers themselves, right? My
main issue here is that I am serving a lot of small web files from my gluster
servers using fuse on the clients so it's not really the amount of data but
the latency which matters.

I would be very interested to read your blog post as soon as its out and I guess
many others too. Please do post the link to this list as soon as its online.


On Thursday, March 5, 2015 2:23 PM, Jeff Darcy <jdarcy at redhat.com>
wrote:
> I have two gluster nodes in a replicated setup and have connected the two
> nodes together directly through a 10 Gbit/s crossover cable. Now I would
> like to tell gluster to use this seperate private network for any
> communications between the two nodes. Does that make sense? Will this bring
> me any performance gain? and if yes how do I configure that?
It is possible, but it's not likely to improve performance much (yet).

The easiest way to do this is to use a custom /etc/hosts on the servers,
so that *on a server* every other server's name resolves to its private
back-end address.  Meanwhile, clients resolve that same name to the
server's front-end address.  You can get a similar effect with explicit
host routes or iptables rules on the servers.

The reason this won't have much effect on performance is that the
servers do not (currently) replicate to one another.  Instead, clients
send data directly to every replica themselves.  The only time time a
private network would see much traffic would be when the "clients" are
actually the servers performing administrative operations - self heal,
rebalance, and so on.

In 4.0, both parts of this answer would be different.  First, we expect
to have better handling of multiple networks and multi-homed hosts,
including user specification of which networks to use for which
traffic[1].  Second, 4.0 will have a new form of replication which
*does* replicate directly between servers[2].  Parts of this second
feature are in fact likely to appear well before the rest of 4.0, using
the server-to-server data flow but retaining our current methods of
tracking changes and re-syncing servers after a failure.  In fact I'm
writing a blog post right now about this, including some performance
measurements.  I'll respond again here when it's done.


[1]
http://www.gluster.org/community/documentation/index.php/Features/SplitNetwork
[2]
http://www.gluster.org/community/documentation/index.php/Features/new-style-replication

> I would be very interested to read your blog post as soon as its out and I
> guess many others too. Please do post the link to this list as soon as its
> online.
Sorry, forgot to do this earlier.  It's here:

http://pl.atyp.us/2015-03-life-on-the-server-side.html