libvirt users - Sep 2013 - Re: libvirt-1.2.2-r1 (Gentoo) fails to start LXC containers

I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available).  I
have not used LXC containers for a few weeks, so I don't recall what
version of libvirt I was using when my container last booted successfully.

Unfortunately, Gentoo's portage tree does not offer any previous versions
of libvirt that I could downgrade to.

TL;DR: My container is configured to use "br0" for its networking. 
"br0"
exists totally inside my linux server - it is NOT bound to any physical
NIC.  "br0" is used for most of my QEMU and LXC VMs.  libvirt is
reporting
that it cannot find device "veth1".  All of my Gentoo packages are
up-to-date.

Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last
successfully booted this LXC container on 2013-07-22, with libvirt
reporting version "1.1.0".

Thoughts?




ostara lxc # equery l libvirt
 * Searching for libvirt ...
[IP-] [  ] app-emulation/libvirt-1.1.2-r1:0


ostara lxc # uname -a
Linux ostara 3.10.7-gentoo #2 SMP PREEMPT Sat Aug 24 16:03:57 CDT 2013
x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux


ostara ~ # emerge -pvuND world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB


ostara lxc # virsh -c lxc:/// start dwj-hfax-dev
error: Failed to start domain dwj-hfax-dev
error: internal error: guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error


ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
2013-09-10 13:56:22.767+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
--handshake 23 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error


ostara ~ # tail /var/log/libvirt/libvirtd.log
2013-09-10 13:56:22.998+0000: 12948: info : libvirt version: 1.1.2
2013-09-10 13:56:22.998+0000: 12948: error : virLXCControllerRun:2186 :
error receiving signal from container: Input/output error
2013-09-10 13:56:23.053+0000: 12948: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"

2013-09-10 13:56:23.097+0000: 10451: error : virNetSocketReadWire:1369 :
Cannot recv data: Connection reset by peer
2013-09-10 13:56:23.153+0000: 10453: error : virLXCProcessStart:1234 :
internal error: guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error

2013-09-10 13:56:23.343+0000: 10453: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"


(first occurrence of "veth1" in my libvirtd logs)
ostara ~ # grep -a veth1 /var/log/libvirt/libvirtd.log  | head -1
2013-09-10 13:51:55.333+0000: 9742: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"




ostara ~ # which ip
/bin/ip


ostara ~ # equery b /bin/ip
 * Searching for /bin/ip ...
sys-apps/iproute2-3.8.0 (/bin/ip)


ostara ~ # ls -l /bin/ip
-rwxr-xr-x 1 root root 282616 May  4 04:08 /bin/ip



ostara ~ # ifconfig -a | grep "^[a-z]"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST>  mtu 1500
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
enp6s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500


ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
<domain type='lxc'>
  <name>dwj-hfax-dev</name>
  <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/vm/lxc/dwj-hfax-dev'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/usr/portage'/>
      <target dir='/usr/portage'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/usr/src'/>
      <target dir='/usr/src'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/home'/>
      <target dir='/home'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='82:00:00:00:01:01'/>
      <source bridge='br0'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

On Tue, Sep 10, 2013 at 9:09 AM, Dennis Jenkins
<dennis.jenkins.75@gmail.com> wrote:
> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available).  I
> have not used LXC containers for a few weeks, so I don't recall what
version
> of libvirt I was using when my container last booted successfully.
>
> Unfortunately, Gentoo's portage tree does not offer any previous
versions of
> libvirt that I could downgrade to.
Yeah our security people got a bit over zealous. That's being rectified.
>
> TL;DR: My container is configured to use "br0" for its
networking.  "br0"
> exists totally inside my linux server - it is NOT bound to any physical
NIC.
> "br0" is used for most of my QEMU and LXC VMs.  libvirt is
reporting that it
> cannot find device "veth1".  All of my Gentoo packages are
up-to-date.
>
> Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last
> successfully booted this LXC container on 2013-07-22, with libvirt
reporting
> version "1.1.0".
>
> Thoughts?
You really need to look at /var/log/libvirt/lxc/dwj-hfax-dev.log

A suggestion would be to make sure that you have all the necessary
kernel options enabled. You can check with: ebuild
/usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean

-- 
Doug Goldstein

On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins
wrote:
> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available).  I
> have not used LXC containers for a few weeks, so I don't recall what
> version of libvirt I was using when my container last booted successfully.
> 
> ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
> 2013-09-10 13:56:22.767+0000: starting up
>
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
> LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
> /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
> --handshake 23 --background --veth veth1
> PATH=/bin:/sbin TERM=linux container=lxc-libvirt
> container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
> LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
> LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
> error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this
file.

1. Stop the 'libvirtd' daemon

2. Edit /etc/libvirt/lxc.conf and set  log_with_libvirtd=1

3. Run 

   $  LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd

4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log

5. Try to start your guest with virsh


Now send us the much more verbose logfile that was created


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
> On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
>> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available).  I
>> have not used LXC containers for a few weeks, so I don't recall
what
>> version of libvirt I was using when my container last booted
successfully.
>>
>> ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
>> 2013-09-10 13:56:22.767+0000: starting up
>>
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
>> LIBVIRT_DEBUG=3
LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
>> /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20
--security=none
>> --handshake 23 --background --veth veth1
>> PATH=/bin:/sbin TERM=linux container=lxc-libvirt
>> container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
>> LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
>> LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
>> error receiving signal from container: Input/output error
> 
> We'll need you to increase the log level to get more detail in this
> file.
> 
> 1. Stop the 'libvirtd' daemon
> 
> 2. Edit /etc/libvirt/lxc.conf and set  log_with_libvirtd=1
> 
> 3. Run 
> 
>    $  LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr
/usr/sbin/libvirtd
> 
> 4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
> 
> 5. Try to start your guest with virsh
> 
> 
> Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment,
his kernel doesn't support securityfs. libvirt should handle this problem.