libvirt users - Jul 2020 - Re: host and vm on isolated network, there is ip (via dhcp) but not ping

Greetings,

I've setup an vm with openwrt in it, defined a isolated lan between the vm
and the host and booted the vm up.
I see the vm is up, made sure the vnic is visible in both the host and guest and
added it to the br in the guest.
I've issued an dhcpd call on the vnic (labeled vnic0) in the host and got an
ip, see:
dagg@NCC-5001D ~ $ dhcpcd vnet0
DUID 00:01:00:01:23:dd:d8:5b:e0:d5:5e:d9:f2:e2
vnet0: IAID 00:10:20:bf
vnet0: rebinding lease of 192.168.1.130
vnet0: probing address 192.168.1.130/24
vnet0: soliciting an IPv6 router
vnet0: leased 192.168.1.130 for 43200 seconds
vnet0: adding route to 192.168.1.0/24
vnet0: adding default route via 192.168.1.1
forked to background, child pid 26279
dagg@NCC-5001D ~ $ ifconfig
virtsw0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 52:54:00:3e:3f:88  txqueuelen 1000  (Ethernet)
        RX packets 123098  bytes 16327962 (15.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6  bytes 252 (252.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.130  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::fc54:ff:fe10:20bf  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:10:20:bf  txqueuelen 1000  (Ethernet)
        RX packets 45  bytes 8002 (7.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39  bytes 2676 (2.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

dagg@NCC-5001D ~ $ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1018ms

the vm's xml can be found at https://pastebin.com/1gXBGcPb
virtsw0 is defined as follows:
<network connections='1'>
  <name>virtsw0</name>
  <uuid>c8eb15a3-cc5c-4bd6-8f3b-5790792ddccc</uuid>
  <bridge name='virtsw0' stp='on' delay='0'/>
  <mac address='52:54:00:3e:3f:88'/>
</network>

the os is gentoo, the versions are libvirt-6.2.0 qemu-5.0.0.
I have another server running debian 10 with the same virtsw0 definition, there
the connection is working.
/var/lib/libvirt/dnsmasq/virtsw0.macs has only [] in it, can that be the issue?

Thanks,

Dagg.

On 7/20/20 12:38 PM, daggs wrote:
> Greetings,
> 
> I've setup an vm with openwrt in it, defined a isolated lan between the
vm and the host and booted the vm up.
> I see the vm is up, made sure the vnic is visible in both the host and
guest and added it to the br in the guest.
> I've issued an dhcpd call on the vnic (labeled vnic0) in the host and
got an ip, see:
> dagg@NCC-5001D ~ $ dhcpcd vnet0
You didn't run "dhcpd" (which is a dhcp server) on the host, you
ran
"dhcpcd", which is a dhcp *client*. So you've ended up assigning
an IP
address to the tap device on the host. I guess the dhcp server that's 
issuing this IP address is part of openwrt in the guest?

A tap device on the host that is attached to a bridge is merely a 
conduit between the guest's emulated NIC and  the bridge device on the 
host, and should not have its own IP address (although it may work in 
certain cases, yours apparently being one of them, since you say the 
same setup works on a debian 10 host; hmm - maybe in the debian host you 
had been running dhcpcd on the bridge device rather than the tap?). In 
general when there is a bridged connection on the host, the IP address 
for the guest should be on the emulated network device *in the guest*, 
and the IP address for the host side of that connection should be on the 
bridge device in the host, *not* the tap device.

Now if the openwrt guest and the host are the only two entities 
communicating on this connection, then you could put an IP address on 
the tap device directly, but in that case you wouldn't want the tap to 
be attached to a bridge anyway. If that's the case, just define the 
interface in the guest as something like this:

    <interface type='ethernet'>
       <mac address='52:54:00:10:20:bf'/>
       <source>
         <ip address='192.168.1.130' prefix='24'/>
       </source>
       <model type='virtio'/>
     </interface>

The IP address inside <source> will set the IP of the *host* side of the 
tap device. You can also add routes to the host's routing table inside 
<source>. See https://libvirt.org/formatdomain.html#ipconfig for details 
(it is very important to remember that the <ip>/<route> *inside the 
<source> element* is used to set the IP address of the host side of the 
tap. An <ip>/<route> as a toplevel subelement of <interface>
is intended
to set those properties *in the guest*, and won't work at all in the 
case of qemu, since the hypervisor in that case has no visibility into 
the guest's IP network configuration).
> DUID 00:01:00:01:23:dd:d8:5b:e0:d5:5e:d9:f2:e2
> vnet0: IAID 00:10:20:bf
> vnet0: rebinding lease of 192.168.1.130
> vnet0: probing address 192.168.1.130/24
> vnet0: soliciting an IPv6 router
> vnet0: leased 192.168.1.130 for 43200 seconds
> vnet0: adding route to 192.168.1.0/24
> vnet0: adding default route via 192.168.1.1
> forked to background, child pid 26279
> dagg@NCC-5001D ~ $ ifconfig
> virtsw0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>          ether 52:54:00:3e:3f:88  txqueuelen 1000  (Ethernet)
>          RX packets 123098  bytes 16327962 (15.5 MiB)
>          RX errors 0  dropped 0  overruns 0  frame 0
>          TX packets 6  bytes 252 (252.0 B)
>          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>          inet 192.168.1.130  netmask 255.255.255.0  broadcast 192.168.1.255
>          inet6 fe80::fc54:ff:fe10:20bf  prefixlen 64  scopeid
0x20<link>
>          ether fe:54:00:10:20:bf  txqueuelen 1000  (Ethernet)
>          RX packets 45  bytes 8002 (7.8 KiB)
>          RX errors 0  dropped 0  overruns 0  frame 0
>          TX packets 39  bytes 2676 (2.6 KiB)
>          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> dagg@NCC-5001D ~ $ ping 192.168.1.1
> PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> ^C
> --- 192.168.1.1 ping statistics ---
> 2 packets transmitted, 0 received, 100% packet loss, time 1018ms
> 
> the vm's xml can be found at https://pastebin.com/1gXBGcPb
> virtsw0 is defined as follows:
> <network connections='1'>
>    <name>virtsw0</name>
>    <uuid>c8eb15a3-cc5c-4bd6-8f3b-5790792ddccc</uuid>
>    <bridge name='virtsw0' stp='on'
delay='0'/>
>    <mac address='52:54:00:3e:3f:88'/>
> </network>
> 
> the os is gentoo, the versions are libvirt-6.2.0 qemu-5.0.0.
> I have another server running debian 10 with the same virtsw0 definition,
there the connection is working.

Check the iptables rules on both hosts and both guests to see if there 
are any differences.
> /var/lib/libvirt/dnsmasq/virtsw0.macs has only [] in it, can that be the
issue?
Since in your case the host is a dhcp *client*, that is irrelevant. I'm 
actually surprised that the file exists at all, since you have no <dhcp> 
section in your network definition, so dnsmasq should even be run.
> 
> Thanks,
> 
> Dagg.
> 
>

Greetings Laine,
> Sent: Tuesday, July 21, 2020 at 9:16 PM
> From: "Laine Stump" <laine@redhat.com>
> To: "libvirt-users@redhat.com" <libvirt-users@redhat.com>
> Cc: "daggs" <daggs@gmx.com>
> Subject: Re: host and vm on isolated network, there is ip (via dhcp) but
not ping
>
> On 7/20/20 12:38 PM, daggs wrote:
> > Greetings,
> >
> > I've setup an vm with openwrt in it, defined a isolated lan
between the vm and the host and booted the vm up.
> > I see the vm is up, made sure the vnic is visible in both the host and
guest and added it to the br in the guest.
> > I've issued an dhcpd call on the vnic (labeled vnic0) in the host
and got an ip, see:
> > dagg@NCC-5001D ~ $ dhcpcd vnet0
>
> You didn't run "dhcpd" (which is a dhcp server) on the host,
you ran
> "dhcpcd", which is a dhcp *client*. So you've ended up
assigning an IP
> address to the tap device on the host. I guess the dhcp server that's
> issuing this IP address is part of openwrt in the guest?
that is correct, I assumed dhcod is the dhcp client will query the dhcp server
on the vm for an ip.
I'll make sure what does this command supposes to do.
>
> A tap device on the host that is attached to a bridge is merely a
> conduit between the guest's emulated NIC and  the bridge device on the
> host, and should not have its own IP address (although it may work in
> certain cases, yours apparently being one of them, since you say the
> same setup works on a debian 10 host; hmm - maybe in the debian host you
> had been running dhcpcd on the bridge device rather than the tap?). In
> general when there is a bridged connection on the host, the IP address
> for the guest should be on the emulated network device *in the guest*,
> and the IP address for the host side of that connection should be on the
> bridge device in the host, *not* the tap device.
the configuration that works on the production env was given to me here in this
very ml.
I don't think the bridge has an internal dhcp server because the ip given is
part of the range the server provides and I see each action regarding the
connection in the router logs
which resides inside the vm.
>
> Now if the openwrt guest and the host are the only two entities
> communicating on this connection, then you could put an IP address on
> the tap device directly, but in that case you wouldn't want the tap to
> be attached to a bridge anyway. If that's the case, just define the
> interface in the guest as something like this:
>
>     <interface type='ethernet'>
>        <mac address='52:54:00:10:20:bf'/>
>        <source>
>          <ip address='192.168.1.130' prefix='24'/>
>        </source>
>        <model type='virtio'/>
>      </interface>
>
> The IP address inside <source> will set the IP of the *host* side of
the
> tap device. You can also add routes to the host's routing table inside
> <source>. See https://libvirt.org/formatdomain.html#ipconfig for
details
> (it is very important to remember that the <ip>/<route> *inside
the
> <source> element* is used to set the IP address of the host side of
the
> tap. An <ip>/<route> as a toplevel subelement of
<interface> is intended
> to set those properties *in the guest*, and won't work at all in the
> case of qemu, since the hypervisor in that case has no visibility into
> the guest's IP network configuration).
there are expect4ed to be 3 other machines on the network, the host, an lan one
(via usb pass-through)
and a wireless one (via usb pass-through).
if I setup virtsw0 to provides the ip, there is no reason to have a router
inside a vm to begin with.
>
> > DUID 00:01:00:01:23:dd:d8:5b:e0:d5:5e:d9:f2:e2
> > vnet0: IAID 00:10:20:bf
> > vnet0: rebinding lease of 192.168.1.130
> > vnet0: probing address 192.168.1.130/24
> > vnet0: soliciting an IPv6 router
> > vnet0: leased 192.168.1.130 for 43200 seconds
> > vnet0: adding route to 192.168.1.0/24
> > vnet0: adding default route via 192.168.1.1
> > forked to background, child pid 26279
> > dagg@NCC-5001D ~ $ ifconfig
> > virtsw0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >          ether 52:54:00:3e:3f:88  txqueuelen 1000  (Ethernet)
> >          RX packets 123098  bytes 16327962 (15.5 MiB)
> >          RX errors 0  dropped 0  overruns 0  frame 0
> >          TX packets 6  bytes 252 (252.0 B)
> >          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> > vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >          inet 192.168.1.130  netmask 255.255.255.0  broadcast
192.168.1.255
> >          inet6 fe80::fc54:ff:fe10:20bf  prefixlen 64  scopeid
0x20<link>
> >          ether fe:54:00:10:20:bf  txqueuelen 1000  (Ethernet)
> >          RX packets 45  bytes 8002 (7.8 KiB)
> >          RX errors 0  dropped 0  overruns 0  frame 0
> >          TX packets 39  bytes 2676 (2.6 KiB)
> >          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> > dagg@NCC-5001D ~ $ ping 192.168.1.1
> > PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
> > ^C
> > --- 192.168.1.1 ping statistics ---
> > 2 packets transmitted, 0 received, 100% packet loss, time 1018ms
> >
> > the vm's xml can be found at https://pastebin.com/1gXBGcPb
> > virtsw0 is defined as follows:
> > <network connections='1'>
> >    <name>virtsw0</name>
> >    <uuid>c8eb15a3-cc5c-4bd6-8f3b-5790792ddccc</uuid>
> >    <bridge name='virtsw0' stp='on'
delay='0'/>
> >    <mac address='52:54:00:3e:3f:88'/>
> > </network>
> >
> > the os is gentoo, the versions are libvirt-6.2.0 qemu-5.0.0.
> > I have another server running debian 10 with the same virtsw0
definition, there the connection is working.
>
>
> Check the iptables rules on both hosts and both guests to see if there
> are any differences.
>
that was one of the first things I looked at, iptables isn't running.
> > /var/lib/libvirt/dnsmasq/virtsw0.macs has only [] in it, can that be
the issue?
>
> Since in your case the host is a dhcp *client*, that is irrelevant. I'm
> actually surprised that the file exists at all, since you have no
<dhcp>
> section in your network definition, so dnsmasq should even be run.
>
the reason I felt it is important to check that file is because when I tried to
start the vm for the first time, startup filed with an error on
/var/lib/libvirt/dnsmasq/ as it didn't existed.
if I don't need it, why starting a vm up require the existent of that folder
unconditionally?