libvirt users - Sep 2020 - network config not working on newer libvirt

Greetings,

up until a year ago, I was running a server with Debian 10 (stable) on it with
the latest versions of libvirt, qemu and kernel 4.19.x Debian 10 had to offer
(both libvirt and qemu versions were really old).

the network config was simple, one of the vm acted as a router and provided the
ip for both the host and the vm.
I've recently switched distro and now I'm running latest stable libvirt,
qemu and kernel 5.4.x
I've tried to reinstate the network config on the new distro but I cannot
get ip via dhcp for the second vm.
if I assign manual ip and gateway, I have access to the outside world.

here are the relevant dumps:
network on the router vm:
    <interface type='network'>
      <mac address='52:54:00:53:1c:6b'/>
      <source network='default'/>
      <target dev='virtsw0-vm1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x21'
slot='0x01' function='0x0'/>
    </interface>

the other vm
    <interface type='network'>
      <mac address='52:54:00:5a:4c:8c'/>
      <source network='default'/>
      <target dev='virtsw0-vm2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
    </interface>

and finally:
<network connections='2'>
  <name>default</name>
  <uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid>
  <bridge name='virtsw0' stp='on' delay='0'/>
  <mac address='52:54:00:6b:1b:92'/>
</network>

as it is possible I'm missing a kernel config, here is the output of lsmod:
vfio_pci               49152  6
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       32768  2
vfio                   28672  16 vfio_iommu_type1,vfio_pci
ip6table_nat           16384  1
iptable_nat            16384  1
ebtables               24576  0
bridge                143360  0
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
cfg80211              647168  0
x86_pkg_temp_thermal    20480  0
kvm_intel             237568  8
vhost_net              24576  3
vhost                  36864  1 vhost_net
tap                    24576  1 vhost_net
kvm                   663552  1 kvm_intel
tun                    53248  8 vhost_net
r8152                  73728  0
nct6775                57344  0
mei_me                 32768  0
hwmon_vid              20480  1 nct6775
irqbypass              16384  22 vfio_pci,kvm
mii                    16384  1 r8152
mei                    77824  1 mei_me
coretemp               16384  0
efivarfs               16384  1

and the .config at https://dpaste.com/9ZUCBDE9R

any ideas how to fix it?

Thanks,

Dagg.

On 9/4/20 12:38 AM, daggs wrote:
> Greetings,
> 
> up until a year ago, I was running a server with Debian 10 (stable) on it
with the latest versions of libvirt, qemu and kernel 4.19.x Debian 10 had to
offer (both libvirt and qemu versions were really old).
> 
> the network config was simple, one of the vm acted as a router and provided
the ip for both the host and the vm.
> I've recently switched distro and now I'm running latest stable
libvirt, qemu and kernel 5.4.x
You haven't said which distro, nor what is the libvirt exact libvirt 
version (probably won't matter in this case, but in general "libvirt 
x.y.z" is more useful than "latest stable libvirt").

If you have full connectivity once you've manually assigned IP 
addresses, then you don't have any routing problems, so that can be 
counted out. (Anyway, DHCP packets never go beyond the local network).

In that case, you've most likely either got a firewall problem on host 
or guest, or a problem with your dhcp server.

> I've tried to reinstate the network config on the new distro but I
cannot get ip via dhcp for the second vm.
> if I assign manual ip and gateway, I have access to the outside world.
 From where? The host? or the guests?
> 
> here are the relevant dumps:
> network on the router vm:
>      <interface type='network'>
>        <mac address='52:54:00:53:1c:6b'/>
>        <source network='default'/>
>        <target dev='virtsw0-vm1'/>
>        <model type='virtio'/>
>        <address type='pci' domain='0x0000'
bus='0x21' slot='0x01' function='0x0'/>
>      </interface>
> 
> the other vm
>      <interface type='network'>
>        <mac address='52:54:00:5a:4c:8c'/>
>        <source network='default'/>
>        <target dev='virtsw0-vm2'/>
>        <model type='virtio'/>
>        <address type='pci' domain='0x0000'
bus='0x00' slot='0x03' function='0x0'/>
>      </interface>
> 
> and finally:
> <network connections='2'>
>    <name>default</name>
>    <uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid>
>    <bridge name='virtsw0' stp='on'
delay='0'/>
>    <mac address='52:54:00:6b:1b:92'/>
> </network>
Your config is for a bridge that's created by libvirt, but with no 
iptables rules, no dnsmasq instance, and no IP address on the host. So 
any DHCP server config is outside libvirt's realm, as are any iptables 
or nftables rules, so in this case there is nothing to look at in the 
libvirt config for either of these issues.

I would start troubleshooting by making sure that the dhcp server is 
running, and that you can communicate between the machine with DHCP 
server and the guest once a manual IP is assigned. Then use tcpdump or 
wireshark at different places on the path between those two to see how 
far the DHCP request is getting out, whether a response is being sent by 
the server, and if so how far the response is getting back (i.e. on the 
host, run tcpdump on the guest's tap device; if you see the DHCP request 
there, then run tcpdump on the bridge, if you see it there, run it on 
the tap device for the guest, if you see it there, then run tcpdump 
inside the guest; then check the dhcp server logs to see if it's 
receiving requests. While you're doing all of this, you can also be 
noticing whether or not a DHCP response is arriving at each step (and if 
you see the response, you can skip looking further ahead in the packet 
path, since you know by inference that it made it all the way to the 
DHCP server). Once you find the point that the packet is blocked, you'll 
be better able to determine why.

> 
> as it is possible I'm missing a kernel config, here is the output of
lsmod:
> vfio_pci               49152  6
> vfio_virqfd            16384  1 vfio_pci
> vfio_iommu_type1       32768  2
> vfio                   28672  16 vfio_iommu_type1,vfio_pci
> ip6table_nat           16384  1
> iptable_nat            16384  1
> ebtables               24576  0
> bridge                143360  0
> stp                    16384  1 bridge
> llc                    16384  2 bridge,stp
> cfg80211              647168  0
> x86_pkg_temp_thermal    20480  0
> kvm_intel             237568  8
> vhost_net              24576  3
> vhost                  36864  1 vhost_net
> tap                    24576  1 vhost_net
> kvm                   663552  1 kvm_intel
> tun                    53248  8 vhost_net
> r8152                  73728  0
> nct6775                57344  0
> mei_me                 32768  0
> hwmon_vid              20480  1 nct6775
> irqbypass              16384  22 vfio_pci,kvm
> mii                    16384  1 r8152
> mei                    77824  1 mei_me
> coretemp               16384  0
> efivarfs               16384  1
> 
> and the .config at https://dpaste.com/9ZUCBDE9R
> 
> any ideas how to fix it?
> 
> Thanks,
> 
> Dagg.
>

Greetings Laine,
> You haven't said which distro, nor what is the libvirt exact libvirt
> version (probably won't matter in this case, but in general
"libvirt
> x.y.z" is more useful than "latest stable libvirt").
you are correct, the previous os was debian 10 with libvirt 3, the new os is
gentoo with libvirt 6.2.0
>
> If you have full connectivity once you've manually assigned IP
> addresses, then you don't have any routing problems, so that can be
> counted out. (Anyway, DHCP packets never go beyond the local network).
>
> In that case, you've most likely either got a firewall problem on host
> or guest, or a problem with your dhcp server.
>
iptables is installed on the host (required by libvirt because of the virt
network features, from what I can see, it isn't running.
the guest is libreelec, somehow I don't think it has iptables installed or
configured.

on the dhcp server (the other vm) I see this:
Sat Sep  5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPDISCOVER(br-lan)
52:54:00:5a:4c:8c
Sat Sep  5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPOFFER(br-lan)
10.0.0.40 52:54:00:5a:4c:8c

multiple times, it means that the server accepted the request and offers the
correct ip to it but doesn't seem to get there.
>  From where? The host? or the guests?
I can ssh from one vm to another, without the manual ip, I cannot do it
>
> I would start troubleshooting by making sure that the dhcp server is
> running, and that you can communicate between the machine with DHCP
> server and the guest once a manual IP is assigned. Then use tcpdump or
> wireshark at different places on the path between those two to see how
> far the DHCP request is getting out, whether a response is being sent by
> the server, and if so how far the response is getting back (i.e. on the
> host, run tcpdump on the guest's tap device; if you see the DHCP
request
> there, then run tcpdump on the bridge, if you see it there, run it on
> the tap device for the guest, if you see it there, then run tcpdump
> inside the guest; then check the dhcp server logs to see if it's
> receiving requests. While you're doing all of this, you can also be
> noticing whether or not a DHCP response is arriving at each step (and if
> you see the response, you can skip looking further ahead in the packet
> path, since you know by inference that it made it all the way to the
> DHCP server). Once you find the point that the packet is blocked,
you'll
> be better able to determine why.
>
>
alright, I'll try that, thanks.

Dagg.