libvirt users - Sep 2013 - Re: libvirt-1.1.2-r1 (Gentoo) fails to start LXC containers (subject line minor edit, was libvirt-1.2.2-r1)

On Tue, Sep 10, 2013 at 1:34 PM, Doug Goldstein <cardoe@gentoo.org> wrote:
> On Tue, Sep 10, 2013 at 9:09 AM, Dennis Jenkins
> <dennis.jenkins.75@gmail.com> wrote:
>
> Yeah our security people got a bit over zealous. That's being
rectified.
>
> :)
> >
> > TL;DR: My container is configured to use "br0" for its
networking.  "br0"
> > exists totally inside my linux server - it is NOT bound to any
physical
> NIC.
> > "br0" is used for most of my QEMU and LXC VMs.  libvirt is
reporting
> that it
> > cannot find device "veth1".  All of my Gentoo packages are
up-to-date.
> >
> > Digging through my logs (/var/log/libvirt/libvirt.log), I see that I
last
> > successfully booted this LXC container on 2013-07-22, with libvirt
> reporting
> > version "1.1.0".
> >
> > Thoughts?
>
> You really need to look at /var/log/libvirt/lxc/dwj-hfax-dev.log
>
>
I did.  I thought that I posted the relevant bits.  I'll check/repost.

It just lots of this (one set from each test that I did):

2013-09-10 13:56:22.767+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
--handshake 23 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error


I see the "--veth veth1", which is the same of the NIC that cannot be
found.  But that NIC is not listed in my domain config XML.

> A suggestion would be to make sure that you have all the necessary
> kernel options enabled. You can check with: ebuild
> /usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
>
>
No errors.

ostara ~ # ebuild
/usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
 * libvirt-1.1.2.tar.gz SHA256 SHA512 WHIRLPOOL size ;-)
...                  [ ok ]
 * libvirt-1.1.2-e89bdf01.tar.xz SHA256 SHA512 WHIRLPOOL size ;-)
...         [ ok ]
 * checking ebuild checksums ;-)
...                                          [ ok ]
 * checking auxfile checksums ;-)
...                                         [ ok ]
 * checking miscfile checksums ;-)
...                                        [ ok ]
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found kernel object directory:
 *     /lib/modules/3.10.7-gentoo/build
 * Found sources for kernel version:
 *     3.10.7-gentoo
 * Checking for suitable kernel configuration
options...                      [ ok ]


16 days ago, I updated from kernel 3.8.13 to 3.10.7.  I could take a brief
outage and down-grade my kernel if you think that is a worthy test.

I just realized that when I composed my original email/bug report, that I
misquoted the version of libvirt that I am running.  I am indeed running
"1.1.2-r1", but posted as "1.2.2-r1".

ostara ~ # virsh version --daemon
Compiled against library: libvirt 1.1.2
Using library: libvirt 1.1.2
Using API: QEMU 1.1.2
Running hypervisor: QEMU 1.5.2
Running against daemon: 1.1.2

On Tue, Sep 10, 2013 at 2:20 PM, Dennis Jenkins
<dennis.jenkins.75@gmail.com
> wrote:
> On Tue, Sep 10, 2013 at 1:34 PM, Doug Goldstein
<cardoe@gentoo.org>wrote:
>
>> A suggestion would be to make sure that you have all the necessary
>> kernel options enabled. You can check with: ebuild
>> /usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
>>
>>
>
I just ran an "emerge --sync" and noticed that the ebuild file for
libvirt
changed.  Now I get a new result.  "Unexpected problems, indeed".  I
will
update my kernel and report back.

ostara ~ # ebuild
/usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
 * libvirt-1.1.2.tar.gz SHA256 SHA512 WHIRLPOOL size ;-)
...                                                       [ ok ]
 * libvirt-1.1.2-e89bdf01.tar.xz SHA256 SHA512 WHIRLPOOL size ;-)
...                                              [ ok ]
 * checking ebuild checksums ;-)
...
[ ok ]
 * checking auxfile checksums ;-)
...
[ ok ]
 * checking miscfile checksums ;-)
...
[ ok ]
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found kernel object directory:
 *     /lib/modules/3.10.7-gentoo/build
 * Found sources for kernel version:
 *     3.10.7-gentoo
 * Checking for suitable kernel configuration options...
 *   CONFIG_SECURITYFS:  is not set when it should be.
 * Please check to make sure these options are set correctly.
 * Failure to do so may cause unexpected problems.

> ostara ~ # ebuild
> /usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
>  * libvirt-1.1.2.tar.gz SHA256 SHA512 WHIRLPOOL size ;-)
> ...                                                       [ ok ]
>  * libvirt-1.1.2-e89bdf01.tar.xz SHA256 SHA512 WHIRLPOOL size ;-)
> ...                                              [ ok ]
>  * checking ebuild checksums ;-)
> ...
> [ ok ]
>  * checking auxfile checksums ;-)
> ...
> [ ok ]
>  * checking miscfile checksums ;-)
> ...
> [ ok ]
>  * Determining the location of the kernel source code
>  * Found kernel source directory:
>  *     /usr/src/linux
>  * Found kernel object directory:
>  *     /lib/modules/3.10.7-gentoo/build
>  * Found sources for kernel version:
>  *     3.10.7-gentoo
>  * Checking for suitable kernel configuration options...
>  *   CONFIG_SECURITYFS:  is not set when it should be.
>  * Please check to make sure these options are set correctly.
>  * Failure to do so may cause unexpected problems.
>
>
Doug, thank you for your assistance.  My issue is resolved:

ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
Domain dwj-hfax-dev started

ostara ~ # zgrep "SECURITYFS" /proc/config.gz
CONFIG_SECURITYFS=y


Suggestion: libvirtd, or its start script, should sanity check the local
kernel and issue a warning if its not kosher.


[off topic].  Timing!  My last reboot was 16 days ago.  Between reboots my
EXT3 file-system crossed the 180-days without a FSCK mark...  My reboot
took a looooooonnnnggg time.  My server is in my basement.  When it did not
come back up quickly post-reboot, I got nervous and had to go check on it.
:)  2.4 million files makes FSCK slow.  Too bad QEMU performs poorly when
ran on top of btrfs, or I would convert.  (yeah, I know about disabling the
COW flag per file/directory)

ostara ~ # df -i /
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/md3       60632496 2443621 58188875    5% /

ostara ~ # df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md3        909G  447G  417G  52% /