Hi there, I'm planning to setup a 3-Node Cluster for oVirt and would like to use 56 GBe (RoCe) exclusively for GlusterFS. Since 56 GBe switches are far too expensive and? it's not planned to add more nodes and furthermore this would add a SPOF I'd like to cross connect the nodes as shown in the diagram below: Node 1?? Node 2??? Node3 ? ||_______||________|| ? |___________________| This way there's a dedicated 56 Gbit connection to/from each member node. Is is possible to do this with GlusterFS? My first thought was to have different IPs on each node's /etc/host mapped to the node hostnames but I'm unsure if I can force GlusterFS to hostnames instead of IPs. Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150522/d2d7ee7e/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3851 bytes Desc: Electronic Signature S/MIME URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150522/d2d7ee7e/attachment.p7s>
On 22 May 2015 at 12:11, Patrick Ernst <patrick.ernst at bullites.com> wrote:> Hi there, > > I'm planning to setup a 3-Node Cluster for oVirt and would like to use 56 > GBe (RoCe) > exclusively for GlusterFS. Since 56 GBe switches are far too expensive > and it's not > planned to add more nodes and furthermore this would add a SPOF I'd like > to > cross connect the nodes as shown in the diagram below: > > Node 1 Node 2 Node3 > ||_______||________|| > |___________________| > > This way there's a dedicated 56 Gbit connection to/from each member node. > > Is is possible to do this with GlusterFS? >IIUC, you have a 56Gbe dedicated network among glusterfs servers. Right? If that is correct, it's not going to help much. In glusterfs clustering (distribution/replication) is done by the client. So having a faster connection from client to servers would be helpful. Only glusterds (and glustershd as well???) talk to each other among servers. Maybe be someone can elaborate more. My first thought was to have different IPs on each node's /etc/host mapped> to the node > hostnames but I'm unsure if I can force GlusterFS to hostnames instead of > IPs. >Yes, you can use any properly resolvable hostname with gluster. HTH Best Regards, Vishwanath> > Patrick > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150522/35688418/attachment.html>
----- Original Message -----> Hi there,> I'm planning to setup a 3-Node Cluster for oVirt and would like to use 56 GBe > (RoCe) > exclusively for GlusterFS. Since 56 GBe switches are far too expensive and > it's not > planned to add more nodes and furthermore this would add a SPOF I'd like to > cross connect the nodes as shown in the diagram below:> Node 1 Node 2 Node3 > ||_______||________|| > |___________________|> This way there's a dedicated 56 Gbit connection to/from each member node.> Is is possible to do this with GlusterFS? > My first thought was to have different IPs on each node's /etc/host mapped to > the node > hostnames but I'm unsure if I can force GlusterFS to hostnames instead of > IPs.There are two ways you can do this. Both involve asymmetric configurations. Imagine that you have three subnets, one per wire: 192.168.1.1 and 192.168.1.2 between Node1 and Node2 192.168.2.1 and 192.168.2.2 between Node2 and Node3 192.168.3.1 and 192.168.3.2 between Node1 and Node3 So, /etc/hosts on Node1 would look like this: 192.168.1.2 node2 192.168.3.2 node3 On Node2 you'd have this: 192.168.1.1 node1 192.168.2.2 node3 And so on. Note that these are all different than the clients, which would have entries (probably in DNS rather than /etc/hosts) for the servers' slower external addresses. The other way to do the same thing is with explicit host routes or iptables rules. In that kind of setup, you put each server into its own subnet, then add routes on the others to go through the interfaces you want. For example: node1 is 172.30.16.1 node2 is 172.30.17.1 node3 is 172.30.18.1 Therefore, on node1 (using the interface addresses above): route add -host node2 gw 192.168.1.2 route add -host node3 gw 192.168.3.2 On node2: route add -host node1 gw 192.168.1.1 route add -host node3 gw 192.168.2.2 And so on, again. Don't forget to turn on IP forwarding. Also, this still requires that the servers have a different /etc/hosts than clients, but at least it can be the same across all servers. Alternatively, you could use the same /etc/hosts (or DNS) everywhere, if you can add routes on the clients as well. All that said, the benefit of such a configuration is rather limited. Using FUSE or GFAPI, replication will still occur over the slow client network because it's being driven by the clients (this is likely to change in 4.0). On the other hand, self-heal and rebalance traffic will use the faster internal network. SMB and NFS will use both, so they might see some benefit in *aggregate* but not per-client throughput. Depending on your usage pattern, the extra complexity of setting up this kind of routing might not be worth the effort.