Claudio Soprano
2021-Apr-15 09:40 UTC
[Gluster-users] Gluster using multiple VLANs for traffic separation clients <-> servers
Hi to all,
i'm sorry for the long post, but just to explain all we tried to reach
our goal.
We are new to gluster and we are trying to create a new gluster
infrastructure, we are using gluster v9.1 on ubuntu servers.
Our mail goal is to separate/balance traffic from clients <-> servers
(by volume or subvolumes if possible or using different instances of
gluster daemon) using several VLANs (with different IP address).
Just to clear about it, we have:
1) For each gluster node, 5 ip address on 2 interface 10GB in bonding mode:
first IP on VLAN 1 default, "public ip" on subnet 192.168.30.0/24,
gluster{01..12} (name of the servers), used only for management via SSH
second IP on VLAN 260 server, "private ip on subnet 172.26.0.0/24",
glustersrv{01..12} (name of the servers), used for server <-> server
traffic only
third IP on VLAN 261 web, "private ip on subnet 172.26.1.0/24",
glusterweb{01..12} (name of the servers), used for servers <-> clients
traffic only for specific volumes/subvolumes
fourth IP on VLAN 262 sys, "private ip on subnet 172.26.2.0/24",
glustersys{01..12} (name of the servers), used for servers <-> clients
traffic only for specific volumes/subvolumes
fifth IP on VLAN 263 share, "private ip on subnet 172.26.3.0/24" ,
glustershare{01..12} (name of the servers), used for servers <-> clients
traffic only for specific volumes/subvolumes
2) VLAN 260,261,262 and 263 are not routed on the switch, so only switch
ports mapped to those VLAN can talk with other "nodes" on the same
VLAN
3) For each client, 2 ip address on 2 interface:
first IP on VLAN 1 default, "public ip", glustercli{01..04} (name of
the
clients), used only for management via SSH
second IP on VLAN 261/262/263 web/sys/share,
glustercliweb/sys/share{01..04} (name of the clients), used for servers
<-> clients traffic only for specific volumes/subvolumes
4) We created the cluster adding peer using names glustersrv{01..12} to
ensure each server is talking to eachother using the IP VLAN server
root at gluster01:~# netstat -ant | grep -i list
tcp??????? 0????? 0 0.0.0.0:24007 0.0.0.0:*?????????????? LISTEN
tcp??????? 0????? 0 127.0.0.1:6010 0.0.0.0:*?????????????? LISTEN
tcp??????? 0????? 0 0.0.0.0:49152 0.0.0.0:*?????????????? LISTEN
All the gluster services are listening to 0.0.0.0:49152 (so here i don't
see any problem connecting to them using several IP/VLANs).
5) We created a volume web01 using names glustersrv{01..12} also because
if we tried to use glusterweb{01..12} we get
root at gluster01:~# gluster volume create scratch3 disperse-data 8
redundancy 4 transport tcp glusterweb{01..12}:/mnt/hdd4a/area1/web01
volume create: web01: failed: Host glusterweb03 (this name changes each
time we try to use this command) is not in 'Peer in Cluster' state
and if we try to add glusterweb03 like a peer we get
root at gluster01:~# gluster peer probe glusterweb03
peer probe: Host glusterweb03 port 24007 already in peer list (i suppose
because the daemon is listening on each address)
6) Finally we tried to mount that volume from several clients who are
attached to the VLAN 261 web, using
root at glustercli01:~# mount.glusterfs glusterweb01:/scratch /mnt
Mounting glusterfs on /mnt failed.???? (after about 1 minute it failed)
so we tried instead
root at glustercli01:~# mount.glusterfs glustersrv01:/scratch /mnt
Mounting glusterfs on /mnt failed.???? (it fails instantly but i
understand this because glustersrv01 is on another subnet/VLAN 260 that
the clients can't reach anyway)
7) To understand why it was not working using IP VLAN 261 names
glusterweb{01..12} we enabled IP routing on the VLAN 260 and magically
the command
root at glustercli01:~# mount.glusterfs glustersrv01:/scratch /mnt
was successfull instantly
So we suppose that the client reach the gluster nodes, but the gluster
node that received the request from the client sends to the client the
servernames/IPs using glustersrv{01..12} using the gluster peer names we
used in the creation of the cluster Gluster and the client (that without
routing can't reach the servers) doesn't receive information back.
We tried to search some informations about Gluster with multihomed
servers, or Gluster with multiple instances on the same servers but we
found only old documentation and also some proposal for v4.0 of gluster
multihomed but nothing more, this is a link for multiple instances
https://bugzilla.redhat.com/show_bug.cgi?id=913555
Instead we found some webpages talking about using glusterd.vol file to
make listening on different binding address but no examples for doing it
and i don't know if that will work for our use case, these are some
links for glusterd.vol configuration
https://serverfault.com/questions/864634/how-can-i-bind-glusterfsd-process-to-an-interface-ip
http://netkiller.sourceforge.net/linux/storage/gluster.html
I tried to replicate my scenario but glusterd doesn't start when i add
the volume server section (i tried only modifying glusterd.vol on a
server could be that the problem ?).
I know we could use Gluster-NFS to trying to do that but we would like
to use gluster native client directly, now my questions are:
a) is there a way to add Peer using different IPs for each servers ? in
this way will we get the features we are especting clients <->
communication on different VLANs ?
b) if a is not possible, is there a way to implent what we would like to
have with gluster, client <-> server communication using different
interfaces/ip/vlan ? If yes how ?
c) if a and b are not possible, is there a way to share a volume on a
specific IP address/network ? if yes how ?
d) if a, b and c are not possible, is there a way to have multiple
instances of gluster running on the same server so we can create
different clusters using in the peer probe different names for each
cluster ?
e) if a,b,c and e are not possible, glusterd.vol is a possible solution
? if yes where we can find some examples or documentation about it ? we
tried but we can't find any docs
Thanks anyway for each hints or for an answer.
Claudio
--
/ | / _____/ / | / _____/ | /
/ / | / / / / | / / / | /
/ / | / ___/ _____/ / / | / ___/ / | /
/ / | / / / / | / / / | /
______/ _/ __/ _/ _/ _/ __/ _/ _/ __/
Claudio Soprano phone: (+39)-06-9403.2349/2355
Computing Service fax: (+39)-06-9403.2649
LNF-INFN e-mail: Claudio.Soprano at lnf.infn.it
Via Enrico Fermi, 40 www: http://www.lnf.infn.it/
I-00044 Frascati, Italy