libvirt users - Nov 2019 - Re: What's the best way to make use of VLAN interfaces with VMs?

Hello

I have a problem with attaching VMs to a VLAN interface.

Here is my setup: I have several physical hosts connected by a physical 
switch.  Each host has two NICs leading to the switch, which have been 
combined into a team, team0. Each host a has a bridge br1, which has 
team0 as a slave. So communication between hosts is based on the IP 
address of bridge br1 on each host.

Up until recently, using libvirt and KVM, I was creating VMs which had 
one interface attached the default virtual network and one interface 
attached to the bridge:

virt-install ... --network network=default --network bridge=br1 ...

I would then statically assign an IP address to the bridge interface on 
the guest when installing the OS.

A few days ago, a VLAN was introduced to split up the network. I created 
a new VLAN interface br1.600 on each of the hosts. My initial attempt 
was to do try this:

virt-install ... --network network=default --network bridge=br1.600 ...

which did not work. It then dawned on me that a VLAN interface and a 
bridge aren't treated the same. So I started to look for ways to allow 
my VMs to bind to this new interface.

This would seem to be a common situation. What is the best way to work 
around this?

Both the host bridge and the host VLAN interface already have their 
assigned IP addresses and appear like this in libvirt:

[root@clusterdev01 ]# ifconfig
br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet 192.168.0.110  netmask 255.255.255.0  broadcast 192.168.0.255
         inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid 0x20<link>
         ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
         RX packets 833772  bytes 2976958254 (2.7 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 331237  bytes 23335124 (22.2 MiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br1.600: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet 192.168.1.110  netmask 255.255.255.0  broadcast 192.168.1.255
         inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid 0x20<link>
         ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
         RX packets 189315  bytes 9465744 (9.0 MiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 302  bytes 30522 (29.8 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@clusterdev01]# virsh iface-list --all
  Name                 State      MAC Address
---------------------------------------------------
  br1                  active     1c:98:ec:1b:27:6d
  br1.600           active     1c:98:ec:1b:27:6d

[root@clusterdev01 sysadmin]# virsh iface-dumpxml br1.600
<interface type='vlan' name='br1.600'>
   <protocol family='ipv4'>
     <ip address='192.168.1.110' prefix='24'/>
   </protocol>
   <protocol family='ipv6'>
     <ip address='fe80::1e98:ecff:fe1b:276d' prefix='64'/>
   </protocol>
   <link state='up'/>
   <vlan tag='600'>
     <interface name='br1'/>
   </vlan>
</interface>

I tried following some suggestions which wrapped the vlan interface in a 
bridge interface, but in ended up trashing the br1.600 interface which 
was originally defined on the host.

Is there a failsafe way to deal with such a situation? Am I doing 
something completely wrong here? In would like br1.600 to behave like 
br1 .....

Any suggestions or advice greatly appreciated.

Richard

On 11/26/19 11:07 PM, Richard Achmatowicz wrote:
> Hello
> 
> I have a problem with attaching VMs to a VLAN interface.
> 
> Here is my setup: I have several physical hosts connected by a physical 
> switch.  Each host has two NICs leading to the switch, which have been 
> combined into a team, team0. Each host a has a bridge br1, which has 
> team0 as a slave. So communication between hosts is based on the IP 
> address of bridge br1 on each host.
> 
> Up until recently, using libvirt and KVM, I was creating VMs which had 
> one interface attached the default virtual network and one interface 
> attached to the bridge:
> 
> virt-install ... --network network=default --network bridge=br1 ...
> 
> I would then statically assign an IP address to the bridge interface on 
> the guest when installing the OS.
> 
> A few days ago, a VLAN was introduced to split up the network. I created 
> a new VLAN interface br1.600 on each of the hosts. My initial attempt 
> was to do try this:
> 
> virt-install ... --network network=default --network bridge=br1.600 ...
> 
> which did not work. It then dawned on me that a VLAN interface and a 
> bridge aren't treated the same. So I started to look for ways to allow 
> my VMs to bind to this new interface.
> 
> This would seem to be a common situation. What is the best way to work 
> around this?
> 
> Both the host bridge and the host VLAN interface already have their 
> assigned IP addresses and appear like this in libvirt:
> 
> [root@clusterdev01 ]# ifconfig
> br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>          inet 192.168.0.110  netmask 255.255.255.0  broadcast 192.168.0.255
>          inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid
0x20<link>
>          ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
>          RX packets 833772  bytes 2976958254 (2.7 GiB)
>          RX errors 0  dropped 0  overruns 0  frame 0
>          TX packets 331237  bytes 23335124 (22.2 MiB)
>          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> br1.600: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>          inet 192.168.1.110  netmask 255.255.255.0  broadcast 192.168.1.255
>          inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid
0x20<link>
>          ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
>          RX packets 189315  bytes 9465744 (9.0 MiB)
>          RX errors 0  dropped 0  overruns 0  frame 0
>          TX packets 302  bytes 30522 (29.8 KiB)
>          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> [root@clusterdev01]# virsh iface-list --all
>   Name                 State      MAC Address
> ---------------------------------------------------
>   br1                  active     1c:98:ec:1b:27:6d
>   br1.600           active     1c:98:ec:1b:27:6d
> 
> [root@clusterdev01 sysadmin]# virsh iface-dumpxml br1.600
> <interface type='vlan' name='br1.600'>
>    <protocol family='ipv4'>
>      <ip address='192.168.1.110' prefix='24'/>
>    </protocol>
>    <protocol family='ipv6'>
>      <ip address='fe80::1e98:ecff:fe1b:276d'
prefix='64'/>
>    </protocol>
>    <link state='up'/>
>    <vlan tag='600'>
>      <interface name='br1'/>
>    </vlan>
> </interface>
> 
> I tried following some suggestions which wrapped the vlan interface in a 
> bridge interface, but in ended up trashing the br1.600 interface which 
> was originally defined on the host.
> 
> Is there a failsafe way to deal with such a situation? Am I doing 
> something completely wrong here? In would like br1.600 to behave like 
> br1 .....
> 
> Any suggestions or advice greatly appreciated.

I guess what you need is for all the traffic from your guests to go out 
on the physical network tagged with vlan id 600, and you want that to be 
transparent to the guests, right?

The simplest way to handle this is to create a vlan interface off of the 
ethernet that you have attached to br1 (not br1 itself), so it would be 
named something like "eth0.600", and then create a new bridge (call
it,
say "br600") and attach eth0.600 to br600. Then your guests would be 
created with "--network bridge=br600"

(Note that Linux host bridges do now support vlan tagging (and maybe 
even trunking) at the port level, but libvirt hasn't added support for 
it. (in other words, "Patches Welcome!" :-))

Hi Laine

What you have suggested sounds eminently reasonable. Thanks for your 
advice. I'm going to give it a shot and report back.

Richard

On 11/27/19 1:38 PM, Laine Stump wrote:
> On 11/26/19 11:07 PM, Richard Achmatowicz wrote:
>> Hello
>>
>> I have a problem with attaching VMs to a VLAN interface.
>>
>> Here is my setup: I have several physical hosts connected by a 
>> physical switch.  Each host has two NICs leading to the switch, which 
>> have been combined into a team, team0. Each host a has a bridge br1, 
>> which has team0 as a slave. So communication between hosts is based 
>> on the IP address of bridge br1 on each host.
>>
>> Up until recently, using libvirt and KVM, I was creating VMs which 
>> had one interface attached the default virtual network and one 
>> interface attached to the bridge:
>>
>> virt-install ... --network network=default --network bridge=br1 ...
>>
>> I would then statically assign an IP address to the bridge interface 
>> on the guest when installing the OS.
>>
>> A few days ago, a VLAN was introduced to split up the network. I 
>> created a new VLAN interface br1.600 on each of the hosts. My initial 
>> attempt was to do try this:
>>
>> virt-install ... --network network=default --network bridge=br1.600 ...
>>
>> which did not work. It then dawned on me that a VLAN interface and a 
>> bridge aren't treated the same. So I started to look for ways to 
>> allow my VMs to bind to this new interface.
>>
>> This would seem to be a common situation. What is the best way to 
>> work around this?
>>
>> Both the host bridge and the host VLAN interface already have their 
>> assigned IP addresses and appear like this in libvirt:
>>
>> [root@clusterdev01 ]# ifconfig
>> br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>          inet 192.168.0.110  netmask 255.255.255.0  broadcast 
>> 192.168.0.255
>>          inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid 
>> 0x20<link>
>>          ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
>>          RX packets 833772  bytes 2976958254 (2.7 GiB)
>>          RX errors 0  dropped 0  overruns 0  frame 0
>>          TX packets 331237  bytes 23335124 (22.2 MiB)
>>          TX errors 0  dropped 0 overruns 0  carrier 0 collisions 0
>>
>> br1.600: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>          inet 192.168.1.110  netmask 255.255.255.0  broadcast 
>> 192.168.1.255
>>          inet6 fe80::1e98:ecff:fe1b:276d  prefixlen 64  scopeid 
>> 0x20<link>
>>          ether 1c:98:ec:1b:27:6d  txqueuelen 1000  (Ethernet)
>>          RX packets 189315  bytes 9465744 (9.0 MiB)
>>          RX errors 0  dropped 0  overruns 0  frame 0
>>          TX packets 302  bytes 30522 (29.8 KiB)
>>          TX errors 0  dropped 0 overruns 0  carrier 0 collisions 0
>>
>> [root@clusterdev01]# virsh iface-list --all
>>   Name                 State      MAC Address
>> ---------------------------------------------------
>>   br1                  active     1c:98:ec:1b:27:6d
>>   br1.600           active     1c:98:ec:1b:27:6d
>>
>> [root@clusterdev01 sysadmin]# virsh iface-dumpxml br1.600
>> <interface type='vlan' name='br1.600'>
>>    <protocol family='ipv4'>
>>      <ip address='192.168.1.110' prefix='24'/>
>>    </protocol>
>>    <protocol family='ipv6'>
>>      <ip address='fe80::1e98:ecff:fe1b:276d'
prefix='64'/>
>>    </protocol>
>>    <link state='up'/>
>>    <vlan tag='600'>
>>      <interface name='br1'/>
>>    </vlan>
>> </interface>
>>
>> I tried following some suggestions which wrapped the vlan interface 
>> in a bridge interface, but in ended up trashing the br1.600 interface 
>> which was originally defined on the host.
>>
>> Is there a failsafe way to deal with such a situation? Am I doing 
>> something completely wrong here? In would like br1.600 to behave like 
>> br1 .....
>>
>> Any suggestions or advice greatly appreciated.
>
>
> I guess what you need is for all the traffic from your guests to go 
> out on the physical network tagged with vlan id 600, and you want that 
> to be transparent to the guests, right?
>
> The simplest way to handle this is to create a vlan interface off of 
> the ethernet that you have attached to br1 (not br1 itself), so it 
> would be named something like "eth0.600", and then create a new
bridge
> (call it, say "br600") and attach eth0.600 to br600. Then your
guests
> would be created with "--network bridge=br600"
>
> (Note that Linux host bridges do now support vlan tagging (and maybe 
> even trunking) at the port level, but libvirt hasn't added support for 
> it. (in other words, "Patches Welcome!" :-))
>