libvirt users - Apr 2015 - Re: Centos 7.1.1503 + libvirt 1.2.14 = broken direct network mode

Hi all.

I use LXC on Centos 7 x86-64, with libvirt version 1.2.6 and 1.2.12
My container has bridged network:
# virsh dumpxml test1
<domain type='lxc'>
  <name>test1</name>
  <uuid>518539ab-7491-45ab-bb1d-3d7f11bfb0b1</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>1048576</hard_limit>
    <soft_limit unit='KiB'>1048576</soft_limit>
    <swap_hard_limit unit='KiB'>2097152</swap_hard_limit>
  </memtune>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='block' accessmode='passthrough'>
      <source dev='/dev/data/test1'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='ram' accessmode='passthrough'>
      <source usage='524288' units='KiB'/>
      <target dir='/dev/shm'/>
    </filesystem>
    <interface type='direct'>
      <mac address='02:00:00:1a:a9:54'/>
      <source dev='br0' mode='bridge'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

And all was fine, before I accidentally upgraded Centos to  7.1.1503
After upgrade LXC can't start with diagnostic:
[root@node14 ~]# virsh start test1
error: Failed to start domain test1
error: internal error: guest failed to start: internal error: Child
process (ip link set macvlan0 netns 25263) unexpected exit status 2:
RTNETLINK answers: Invalid argument

I reinstall libvirt 1.2.14, but catch  same error.
If someone will experience in solving this problem - it will be
interesting to hear about.

b.r.
 Maxim Kozin

> And all was fine, before I accidentally upgraded Centos to  7.1.1503
> After upgrade LXC can't start with diagnostic:
> [root@node14 ~]# virsh start test1
> error: Failed to start domain test1
> error: internal error: guest failed to start: internal error: Child
> process (ip link set macvlan0 netns 25263) unexpected exit status 2:
> RTNETLINK answers: Invalid argument
>
> I reinstall libvirt 1.2.14, but catch  same error.
> If someone will experience in solving this problem - it will be
> interesting to hear about.
If I correct understand libvirt logic, libvirt make next operations
with "ip" utility (depend from lxc config):
1) # ip link add  link br0 name macvlan0 address  00:11:11:11:11:11
type macvlan mode bridge
2) # ip link set macvlan0 netns 1234
where 1234 - pid of lxc initd process
3) inside lxc - #ip link set dev  macvlan0 name eth0

If manualy run this sequence of command, we failed on step 2).
And it's work on Centos 7.0.1406
>
> b.r.
>  Maxim Kozin
>

Hi all

http://bugs.centos.org/view.php?id=8417

In short - it's not Centos bug, it's probably bug of
iproute2<->netlink interactions on
kernel 3.10.0-229
As workaround you can fallback to prev kernel and command
# ip link set macvlanX namens PID
would work.

b.r.
 Maxim Kozin

On 04/08/2015 09:38 AM, mxs kolo wrote:
> Hi all.
>
> I use LXC on Centos 7 x86-64, with libvirt version 1.2.6 and 1.2.12
> My container has bridged network:
> # virsh dumpxml test1
> <domain type='lxc'>
>   <name>test1</name>
>   <uuid>518539ab-7491-45ab-bb1d-3d7f11bfb0b1</uuid>
>   <memory unit='KiB'>1048576</memory>
>   <currentMemory unit='KiB'>1048576</currentMemory>
>   <memtune>
>     <hard_limit unit='KiB'>1048576</hard_limit>
>     <soft_limit unit='KiB'>1048576</soft_limit>
>     <swap_hard_limit
unit='KiB'>2097152</swap_hard_limit>
>   </memtune>
>   <vcpu placement='static'>2</vcpu>
>   <os>
>     <type arch='x86_64'>exe</type>
>     <init>/sbin/init</init>
>   </os>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/libexec/libvirt_lxc</emulator>
>     <filesystem type='block'
accessmode='passthrough'>
>       <source dev='/dev/data/test1'/>
>       <target dir='/'/>
>     </filesystem>
>     <filesystem type='ram' accessmode='passthrough'>
>       <source usage='524288' units='KiB'/>
>       <target dir='/dev/shm'/>
>     </filesystem>
>     <interface type='direct'>
>       <mac address='02:00:00:1a:a9:54'/>
>       <source dev='br0' mode='bridge'/>
>     </interface>
Why are you using macvtap ("type='direct') to connect to a bridge
device? You should either use macvtap to connect directly to a physical
ethernet (e.g. eth0) *or* you should use a standard tap device
("type='bridge'") to connect to the bridge. Although it might
work to
connect to a bridge using macvtap, I can't think of any reason you would
want to do it that way (and that may be the source of the error you're
seeing - since macvtap isn't intended to be used that way, that scenario
probably doesn't get much, if any, testing).

Hi all
> Why are you using macvtap ("type='direct') to connect to a
bridge
> device? You should either use macvtap to connect directly to a physical
> ethernet (e.g. eth0) *or* you should use a standard tap device
> ("type='bridge'") to connect to the bridge. Although it
might work to
> connect to a bridge using macvtap, I can't think of any reason you
would
> want to do it that way (and that may be the source of the error you're
> seeing - since macvtap isn't intended to be used that way, that
scenario
> probably doesn't get much, if any, testing).
Yes, You are right.
I already think about it and can't remember why started using bridge
over iface and macvlan over bridge, may be because used common vlan:
enp2s0->enp2s0.2->br2->lxc
enp2s0->enp2s0.502->br502->lxc

Probably I need test new schema  - it's would be fatser and  work with
new kernel:
enp2s0->enp2s0.502->lxc:
<interface type='direct'>
      <mac address='02:00:00:25:a7:56'/>
      <source dev='enp2s0.2' mode='bridge'/>
</interface>

thanks for the advice !

b.r.
 Maxim Kozin