Greetings,
up until a year ago, I was running a server with Debian 10 (stable) on it with
the latest versions of libvirt, qemu and kernel 4.19.x Debian 10 had to offer
(both libvirt and qemu versions were really old).
the network config was simple, one of the vm acted as a router and provided the
ip for both the host and the vm.
I've recently switched distro and now I'm running latest stable libvirt,
qemu and kernel 5.4.x
I've tried to reinstate the network config on the new distro but I cannot
get ip via dhcp for the second vm.
if I assign manual ip and gateway, I have access to the outside world.
here are the relevant dumps:
network on the router vm:
<interface type='network'>
<mac address='52:54:00:53:1c:6b'/>
<source network='default'/>
<target dev='virtsw0-vm1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x21'
slot='0x01' function='0x0'/>
</interface>
the other vm
<interface type='network'>
<mac address='52:54:00:5a:4c:8c'/>
<source network='default'/>
<target dev='virtsw0-vm2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
and finally:
<network connections='2'>
<name>default</name>
<uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid>
<bridge name='virtsw0' stp='on' delay='0'/>
<mac address='52:54:00:6b:1b:92'/>
</network>
as it is possible I'm missing a kernel config, here is the output of lsmod:
vfio_pci 49152 6
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 32768 2
vfio 28672 16 vfio_iommu_type1,vfio_pci
ip6table_nat 16384 1
iptable_nat 16384 1
ebtables 24576 0
bridge 143360 0
stp 16384 1 bridge
llc 16384 2 bridge,stp
cfg80211 647168 0
x86_pkg_temp_thermal 20480 0
kvm_intel 237568 8
vhost_net 24576 3
vhost 36864 1 vhost_net
tap 24576 1 vhost_net
kvm 663552 1 kvm_intel
tun 53248 8 vhost_net
r8152 73728 0
nct6775 57344 0
mei_me 32768 0
hwmon_vid 20480 1 nct6775
irqbypass 16384 22 vfio_pci,kvm
mii 16384 1 r8152
mei 77824 1 mei_me
coretemp 16384 0
efivarfs 16384 1
and the .config at https://dpaste.com/9ZUCBDE9R
any ideas how to fix it?
Thanks,
Dagg.
On 9/4/20 12:38 AM, daggs wrote:> Greetings, > > up until a year ago, I was running a server with Debian 10 (stable) on it with the latest versions of libvirt, qemu and kernel 4.19.x Debian 10 had to offer (both libvirt and qemu versions were really old). > > the network config was simple, one of the vm acted as a router and provided the ip for both the host and the vm. > I've recently switched distro and now I'm running latest stable libvirt, qemu and kernel 5.4.xYou haven't said which distro, nor what is the libvirt exact libvirt version (probably won't matter in this case, but in general "libvirt x.y.z" is more useful than "latest stable libvirt"). If you have full connectivity once you've manually assigned IP addresses, then you don't have any routing problems, so that can be counted out. (Anyway, DHCP packets never go beyond the local network). In that case, you've most likely either got a firewall problem on host or guest, or a problem with your dhcp server.> I've tried to reinstate the network config on the new distro but I cannot get ip via dhcp for the second vm. > if I assign manual ip and gateway, I have access to the outside world.From where? The host? or the guests?> > here are the relevant dumps: > network on the router vm: > <interface type='network'> > <mac address='52:54:00:53:1c:6b'/> > <source network='default'/> > <target dev='virtsw0-vm1'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x21' slot='0x01' function='0x0'/> > </interface> > > the other vm > <interface type='network'> > <mac address='52:54:00:5a:4c:8c'/> > <source network='default'/> > <target dev='virtsw0-vm2'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> > </interface> > > and finally: > <network connections='2'> > <name>default</name> > <uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid> > <bridge name='virtsw0' stp='on' delay='0'/> > <mac address='52:54:00:6b:1b:92'/> > </network>Your config is for a bridge that's created by libvirt, but with no iptables rules, no dnsmasq instance, and no IP address on the host. So any DHCP server config is outside libvirt's realm, as are any iptables or nftables rules, so in this case there is nothing to look at in the libvirt config for either of these issues. I would start troubleshooting by making sure that the dhcp server is running, and that you can communicate between the machine with DHCP server and the guest once a manual IP is assigned. Then use tcpdump or wireshark at different places on the path between those two to see how far the DHCP request is getting out, whether a response is being sent by the server, and if so how far the response is getting back (i.e. on the host, run tcpdump on the guest's tap device; if you see the DHCP request there, then run tcpdump on the bridge, if you see it there, run it on the tap device for the guest, if you see it there, then run tcpdump inside the guest; then check the dhcp server logs to see if it's receiving requests. While you're doing all of this, you can also be noticing whether or not a DHCP response is arriving at each step (and if you see the response, you can skip looking further ahead in the packet path, since you know by inference that it made it all the way to the DHCP server). Once you find the point that the packet is blocked, you'll be better able to determine why.> > as it is possible I'm missing a kernel config, here is the output of lsmod: > vfio_pci 49152 6 > vfio_virqfd 16384 1 vfio_pci > vfio_iommu_type1 32768 2 > vfio 28672 16 vfio_iommu_type1,vfio_pci > ip6table_nat 16384 1 > iptable_nat 16384 1 > ebtables 24576 0 > bridge 143360 0 > stp 16384 1 bridge > llc 16384 2 bridge,stp > cfg80211 647168 0 > x86_pkg_temp_thermal 20480 0 > kvm_intel 237568 8 > vhost_net 24576 3 > vhost 36864 1 vhost_net > tap 24576 1 vhost_net > kvm 663552 1 kvm_intel > tun 53248 8 vhost_net > r8152 73728 0 > nct6775 57344 0 > mei_me 32768 0 > hwmon_vid 20480 1 nct6775 > irqbypass 16384 22 vfio_pci,kvm > mii 16384 1 r8152 > mei 77824 1 mei_me > coretemp 16384 0 > efivarfs 16384 1 > > and the .config at https://dpaste.com/9ZUCBDE9R > > any ideas how to fix it? > > Thanks, > > Dagg. >
Greetings Laine,> You haven't said which distro, nor what is the libvirt exact libvirt > version (probably won't matter in this case, but in general "libvirt > x.y.z" is more useful than "latest stable libvirt").you are correct, the previous os was debian 10 with libvirt 3, the new os is gentoo with libvirt 6.2.0> > If you have full connectivity once you've manually assigned IP > addresses, then you don't have any routing problems, so that can be > counted out. (Anyway, DHCP packets never go beyond the local network). > > In that case, you've most likely either got a firewall problem on host > or guest, or a problem with your dhcp server. >iptables is installed on the host (required by libvirt because of the virt network features, from what I can see, it isn't running. the guest is libreelec, somehow I don't think it has iptables installed or configured. on the dhcp server (the other vm) I see this: Sat Sep 5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPDISCOVER(br-lan) 52:54:00:5a:4c:8c Sat Sep 5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPOFFER(br-lan) 10.0.0.40 52:54:00:5a:4c:8c multiple times, it means that the server accepted the request and offers the correct ip to it but doesn't seem to get there.> From where? The host? or the guests?I can ssh from one vm to another, without the manual ip, I cannot do it> > I would start troubleshooting by making sure that the dhcp server is > running, and that you can communicate between the machine with DHCP > server and the guest once a manual IP is assigned. Then use tcpdump or > wireshark at different places on the path between those two to see how > far the DHCP request is getting out, whether a response is being sent by > the server, and if so how far the response is getting back (i.e. on the > host, run tcpdump on the guest's tap device; if you see the DHCP request > there, then run tcpdump on the bridge, if you see it there, run it on > the tap device for the guest, if you see it there, then run tcpdump > inside the guest; then check the dhcp server logs to see if it's > receiving requests. While you're doing all of this, you can also be > noticing whether or not a DHCP response is arriving at each step (and if > you see the response, you can skip looking further ahead in the packet > path, since you know by inference that it made it all the way to the > DHCP server). Once you find the point that the packet is blocked, you'll > be better able to determine why. > >alright, I'll try that, thanks. Dagg.