libvirt users - Oct 2021 - Iptables, et al best practices for protecting KVM host sharing "hostdev" (ixgbe-vf) interfaces with guests

Hi,

I'm a developer and I use VM's for testing patches before submitting
them (for various linux-* projects, as well as downstream things like OpenWrt,
EPEL, etc).

I have a trunked 10Gb/s Ethernet connection coming into my KVM servers (all of
which are Supermicro x86_servers running CentOS 8) that have igb and ixgbe
NIC's, so SR-IOV capable.

Here's the situation:

* the host is on a secured internal network, called "sandbox", which
is VLAN 4.

* the "external" network (i.e. public facing on my ISP) is
"dirty", and is VLAN 66.

* the main subnet that laptops, WAPs, desktops, etc. sit on is "main",
and is VLAN 1.

The KVM hosts sit wholly inside "sandbox".

They host, however, several guests some of which sit on "main", and
some of which sit on "main" and "external" (for instance, if
I'm testing an OpenWrt firewall configuration with patches and it needs to
be publicly accessible so that my testers can peer with it via IPsec).  Oh, and
one production guest which hosts "git" and sits only on
"sandbox".

Guests can be CentOS 8S, Windows 11, Ubuntu 20, Debian whatever, Fedora 33 or
34, OpenWrt, etc.

My question is this: what's best practices for making sure that a switch
VLAN misconfiguration issue, a cabling to the wrong port, etc. doesn't
compromise the KVM server itself?

How do I allow my KVM server to *not* be on "external", but some of
its guests to be, without compromising security?

Being the paranoid sort, I'd like to use defense-in-depth so I'm not
counting on any single step to protect me.

For instance, I might use ebtables or iptables rules to block VLAN traffic on
hosts/guests that I don't want to see certain VLAN's under any
circumstances.

But I might also add interfaces to the list of things that NetworkManager
ignores as /etc/NetworkManager/NetworkManager.conf lines like:

[keyfile]
unmanaged-devices=interface-name:eno*,except:interface-name:eno1
unmanaged-devices=interface-name:enp*

Thinking about that, I'm wondering what the notation is for telling
NetworkManager to ignore all interfaces except those on VLAN 3, for instance. 
Would that be:

unmanaged-devices=interface-name:eno*.*,except:interface-name:eno*.3

Or something else?

And what else can I do to further secure my KVM host?

I thought about using one untagged (non VLAN'd) Ethernet for the host, and
another tagged Ethernet for the guests, but that just seemed like a false sense
of security...

Apologies for the cross posting... these questions seems to touch on iptables,
libvirt, and NetworkManager all at the same time...

Thanks,

-Philip

Am 29.10.21 um 02:31 schrieb Philip Prindeville:
> Hi,
> 
> [?]
> 
> 
> Here's the situation:
> 
> * the host is on a secured internal network, called "sandbox",
which is VLAN 4.
> 
> * the "external" network (i.e. public facing on my ISP) is
"dirty", and is VLAN 66.
> 
> * the main subnet that laptops, WAPs, desktops, etc. sit on is
"main", and is VLAN 1.
> 
> The KVM hosts sit wholly inside "sandbox".
> 
> They host, however, several guests some of which sit on "main",
and some of which sit on "main" and "external" (for
instance, if I'm testing an OpenWrt firewall configuration with patches and
it needs to be publicly accessible so that my testers can peer with it via
IPsec).  Oh, and one production guest which hosts "git" and sits only
on "sandbox".
> 
> Guests can be CentOS 8S, Windows 11, Ubuntu 20, Debian whatever, Fedora 33
or 34, OpenWrt, etc.
> 
> My question is this: what's best practices for making sure that a
switch VLAN misconfiguration issue, a cabling to the wrong port, etc.
doesn't compromise the KVM server itself?
> 
> How do I allow my KVM server to *not* be on "external", but some
of its guests to be, without compromising security?
> 
How is this supposed to work? How does the server present network
connectivity to the guests? If the guests would directly bind to the
hardware jumping over the VM server what's the use of the VM then? (If
any of the systems in this stack does not bind services to the
particular network device it is not reachable through this network. As
soon as for example sshd is configured to bind to all network devices it
is accessible through any of them available to the VM.)

Regarding the switch: think of port duplication on the switch routing
all the traffic to a masked hoody wearing laptop owner. How can the
guest or host find out?

Wrong patching of the cables running virtual LANs (VLAN, multiple
networks over the same cable) leaves them intact (from your VMs point of
view)? I assume VLAN-configuration is per physical port? Adapting this
to VLAN misconfiguration puts your VLAN onto a different port that is
not yours? If it is instead of per port a tagging variant and link
aggregation is possible than how should you detect ? except through
bandwidth measurement/ service monitoring (connection reset/ congestion
control) ? that 4 planned links were reduced to 2?

It is very likely I got your scenario wrong. But unless you cannot trace
the hops and latency between origin of traffic and endpoint in your VMs
you cannot detect configuration changes/ wrong configuration. (If it is
the latter no connection at all is very likely.)
> I have a trunked 10Gb/s Ethernet connection coming into my KVM servers (all
of which are Supermicro x86_servers running CentOS 8) that have igb and ixgbe
NIC's, so SR-IOV capable.
> [?]
> Being the paranoid sort, ?
Just as a side and purely my personal opinion: CentOS seems not the best
choice to cure paranoia of the IT type. I also read NetworkManager as
your tool of choice. I am concerned with systems favoring an end user
targeted abstraction layer over hardened configuration. BSD flavors rely
on text files. From this basic layer you stack additional configuration
on each layer to connect VMs, workloads and so on being aware of the
necessary and possible routes. (I sometimes include dead-man-switches
when changing configs so if my ssh session is blocked by
misconfiguration it reverts to the previous state after a certain time
period.)

> 
> My question is this: what's best practices for making sure that a
switch
> VLAN misconfiguration issue, a cabling to the wrong port, etc. doesn't
> compromise the KVM server itself?
Not sure about best practice. But what about using a macvtap. That by default
does not allow host communication and only allows the guests connected to the
same master to communicate with each other.
> How do I allow my KVM server to *not* be on "external", but some
of its
> guests to be, without compromising security?
Do not configure the interface with an ip address on the host, and make sure you
do not have daemons binding to 0.0.0.0 on the host.