Hi Peter and other Libvirt-Users,
Thank you, I greatly appreciate your response. The diagram I provided may
be a little misleading, so I have attached a new diagram (also available
here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide
some more detail about my scenario here:
These physical nodes are provided to me by "bare-metal cloud"
providers
like Scaleway / OVH / Packet.net / etc.
I have opted to use Ubuntu on each bare-metal server, but I am also
well-versed in RHEL/CentOS environments, so if you want to help me out by
telling me CentOS-specific details, I'm sure that I can
"translate" that to
an Ubuntu/Debian setup. Considering my scenario with bare-metal providers,
I do *not* have the flexibility to add/remove physical NICs as I need them,
I am sort of "locked in" to a single-IP/single-NIC configuration for
each
physical node.
But, I **do** have the flexibility of adding more physical nodes as I need
them, and I would like to leverage this option when I want to expand my
cluster. Its easy for me to buy a bare-metal cloud server (takes a few
minutes), and setting up a 1U/2U server by myself on a "lab" network
could
take me a few hours or so (not to mention, I dont have the resources to
host my own network for this). Also, I need to account for the fact that
hardware *will* fail somehow eventually, so adding/removing nodes would
need to be "easy" in the way that I shouldnt have to configure new
DHCP
scopes on all the other nodes when I decide to add one more node to the
cluster. Also, as a side note, I do not intend to have any central or
shared volume pool between the nodes, all storage will be direct on a local
LVM volume, and the ability to perform live migrations of guests between
hosts is not a high priority for me, I intend for each guest to be
*"stationary"
to each host*.
Each node's *eth0* is connected via IPSec (unicast and transport mode) to
the other. This is to encrypt the communication between all nodes because
it travels over the internet. Additionally, each node currently has a
*vxlan0* interface in the *172.20.0.0/24 <http://172.20.0.0/24>* network
which allows them to be all on the same "LAN". I have attached a
simple
diagram illustrating this setup, also available here:
https://i.imgur.com/jc7bc6b.png . I would *prefer* to use the DHCP that
comes "built-in" to livbirt, but I am open to running DHCP on each
host, or
as a guest inside each guest network.
So, I hope this sheds a little more light on my scenario.
If you look at the diagram attached, you can see some *red* lines and some
question marks. These red lines are the only part of the environment that
needs to be configured, everything else seems to work fine; KVM can
provision guests, IPSec tunnel is working, vxlan seems to route/function as
I think I would need it, guests can reach the internet, and guests can ping
everything else inside the environment, *except* guests1 cannot ping
guests2.
Which makes me question the configuration I have for each guest network.
Each guest network is currently configured with NAT Forwarding, but because
KVMHost1 cannot ping guests on KVMHost2, I am concluding that a NAT'd guest
network is most likely something that I do not want. I have some choices on
how to configure each guest network, it could be an "open
<https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html>"
or
"routed
<https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode>"
network, but I would still need to configure some static routes.
I will take a stab at doing an open or routed network today, but in the
meantime, your thoughts and comments about my setup are still welcome and
encouraged.
Thank you!
-Cobin
On Wed, May 30, 2018 at 10:44 PM Peter Crowther
<peter.crowther@melandra.com>
wrote:
> I have to say that I wouldn't do the networking that way - in fact, in
the
> clusters I manage, we haven't done the networking that way :-). Rather
> than layer 3 routing between VMs, we've chosen to use layer 2 virtual
> switching (yes, using openvswitch). We have the luxury of multiple 10G
> NICs between our hosts, so we've separated out the management network
from
> the guest network, simply to ensure that we retain administrative access to
> the hosts via ssh. If you want to live a little more dangerously, you
> could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an
> extra network card on each host for the peace of mind!
>
> For the project that's been live for two years: we presently run four
> hosts on the lab's production network (another two on its
acceptance-test
> network, and another one as a "kickaround" host for playing about
with
> configs). Guest VMs on all four production hosts share 192.168.59.0/24
> (why "59" is a story for another day), on an OVS virtual switch
on each
> host named br-guest, with the guest-specific NIC also set as a port on the
> virtual switch. Guest traffic is therefore sent transparently between the
> hosts where needed, and we can live-migrate a guest from one host to
> another with no need to change the guest's IP address. Because we
share a
> common guest network and IP range between all hosts, it's trivial to
add
> (or remove) hosts - no host needs to know anything about routing to another
> host, and in fact only our management layer cares how many hosts we have.
>
> We happen to have "controller" nodes that run redundant DHCP
servers with
> non-overlapping scopes, but the exact location is not a requirement of this
> setup. We could equally well set up a DHCP service on the guest network on
> each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host,
> .101 to .200 on another host. Guests will typically receive offers from
> each DHCP server and can choose, which is fine as they're all on the
same
> network. This provides redundancy in case of a full or failed DHCP server,
> which your routed network approach wouldn't without some careful DHCP
> forwarding work.
>
> We happen to base our hosts on CentOS 7, but I manage other Debian-derived
> systems and can probably remember enough about its network setup to help
> with Ubuntu. Certainly I can help with OVS weirdnesses; it took some time
> to get my head round exactly how it works. That said, I've never set
up a
> kvm host on Debian.
>
> Good luck; happy to provide further pointers if useful.
>
> Cheers,
>
> - Peter
>
> On 30 May 2018 at 15:32, Cobin Bluth <cbluth@gmail.com> wrote:
>
>> Hello Libvirt Users,
>>
>> I would like to setup a two node bare-metal cluster. I need to guidance
>> on the network configuration. I have attached a small diagram, the same
>> diagram can be seen here: https://i.imgur.com/SOk6a6G.png
>>
>> I would like to configure the following details:
>> - Each node has a DHCP enabled guest network where VMs will run. (eg,
*192.168.1.0/24
>> <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24
>> <http://192.168.2.0/24>* for Host2)
>> - Any guest in Host1 should be able to ping guests in Host2, and vice
>> versa.
>> - All guests have routes to reach the open internet (so that '*yum
>> update*' will work "out-of-the-box")
>> - Each node will be able to operate fully if the other physical node
>> fails. (no central DHCP server, etc)
>> - I would like to *add more physical nodes later* when I need the
>> resources.
>>
>> This is what I have done so far:
>> - Installed latest Ubuntu 18.04, with latest version of libvirt and
>> supporting software from ubuntu's apt repo.
>> - Each node can reach the other via its own eth0.
>> - Each node has a working vxlan0, which can ping the other via its
>> vxlan0, so it looks like the vxlan config is working. (I used *ip link
>> add vxlan0 type vxlan...*)
>> - Configured route on Host1 like so: *ip route add 192.168.2.0/24
>> <http://192.168.2.0/24> via 172.20.0.1*
>> - Configured route on Host2 also: *ip route add 192.168.1.0/24
>> <http://192.168.1.0/24> via 172.20.0.2*
>> - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2,
and
>> vice versa, yay.
>> - Guests on Host1 *cannot* ping guests on Host2, I suspect because the
>> the default NAT config of the libvirt network.
>>
>> So, at this point I started to search for tutorials or more
>> information/documentation, but I am a little overwhelmed by the sheer
>> amount of information, as well as a lot of "stale"
information on blogs etc.
>> I have learned that I can *virsh net-edit default*, and then change it
>> to an "open" network:* <forward mode='open'/>*
>> After doing this, the guests cannot reach outside their own network,
nor
>> reach the internet, so I assume that I would need to add some routes,
or
>> something else to get the network functioning like I want it. There is
also *<forward
>> mode="route"/>*, but I dont fully understand the scenarios
where one
>> would need an *open* or a *route* forward mode. I have also shied away
>> from using openvswitch, and have opted for ifupdown2.
>> (I have taken most of my inspiration from this blog post:
>>
https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/
>> )
>>
>> Some questions that I have for the mailing list, any help would be
>> greatly appreciated:
>> - Is my target configuration of a KVM cluster uncommon? Do you see
>> drawbacks of this setup, or does it go against "typical
convention"?
>> - Would my scenario be better suited for an "*open*" network
or a "
>> *route*" network?
>> - What would be the approach to complete this setup?
>>
>>
>>
>>
>> _______________________________________________
>> libvirt-users mailing list
>> libvirt-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>
>
>