Dennis Jenkins
2013-Feb-08 18:38 UTC
[libvirt-users] libvirt v1.0.2 fails to boot LXC container, but v1.0.0 works
Hello.
tl;dr = v1.0.0 can boot my LXC containers, v.1.0.1 and v.1.0.2 fails.
Paraphrased error message: "lxcContainerMountProcFuse:616 : Failed to
mount ..../meminfo"
I'd like to know if my host is misconfigured, or my domains, or
... why 1.0.2 and 1.0.1 are not working for me.
I've been using libvirt for a while to manage QEMU instances. I
have experimented with lxc. Back in October of last year, I had some
working LXC containers. I don't recall what version of libvirt I was
using at the time.
I recently attempted to boot my containers, and they failed
(libvirt v1.0.2). I then reverted to v1.0.1 and tried again. Failed
with the same result (same error text, just different line numbers).
I then reverted to 1.0.0 and my containers boot up just fine.
My host runs Gentoo Linux on an Inter core-i5 with 16G ram. I
regularly (weekly) apply all available Gentoo updates.
I strongly perfer to install all third-party software (including
libvirt) from Gentoo portage, not manually from source or git. v1.0.2
is the most recent version available.
First, items common to all test cases:
*** GCC version
ostara ~ # gcc --version
gcc (Gentoo 4.6.3 p1.11, pie-0.5.2) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
*** Linux kernel
ostara ~ # uname -a
Linux ostara 3.6.11-gentoo #4 SMP PREEMPT Sat Jan 26 10:27:55 CST 2013
x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
ostara ~ # zcat /proc/config.gz | egrep
"CONFIG_(CGROUP|.*_NS|NAMESPACES)"
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_NS87410 is not set
*** Gento "USE flags" when building Libvirt:
ostara ~ # equery u libvirt | xargs echo
-audit -avahi +caps -firewalld -iscsi +libvirtd +lvm +lxc +macvtap
+nfs +nls -numa -openvz -parted +pcap -phyp -policykit -python +qemu
-rbd +sasl +udev +uml +vepa +virt-network -virtualbox -xen
*** libvirt config over-rides:
ostara ~ # egrep "^\w" < /etc/libvirt/libvirtd.conf
host_uuid = "ab8c50b8-2337-4a02-9274-00923fe8f476"
ostara ~ # egrep "^\w" < /etc/libvirt/libvirt.conf
ostara ~ # egrep "^\w" < /etc/conf.d/libvirtd
rc_need="net"
LIBVIRTD_OPTS="--listen"
LIBVIRTD_KVM_SHUTDOWN="managedsave"
*** Network when no domains (qemu or libvirt) are running:
ostara ~ # ifconfig -a | grep -e "^[a-z]"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436
*** Network with libvirt running (libvirt manages "virbr0" for my qemu
instances. It attaches to "br0").
ostara ~ # /etc/init.d/libvirtd start
* Starting libvirtd ...
[ ok ]
ostara ~ # ifconfig -a | grep -e "^[a-z]"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
*** Mounts (cgroup is mounted)
ostara ~ # mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs
(rw,nosuid,relatime,size=10240k,nr_inodes=2050547,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/md3 on / type ext3 (rw,noatime,errors=continue,barrier=1,data=writeback)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs
(rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
debug on /sys/fs/cgroup/debug type cgroup
(rw,nosuid,nodev,noexec,relatime,debug)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct)
memory on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
blkio on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
perf_event on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
/chroot/ssh-jails/jamiel/home/jamiel on /home/jamiel type none (rw,bind)
libvirt on /run/libvirt/lxc/dwj-lnx-dev type fuse (rw,nosuid,nodev)
*** My LXC containers:
ostara ~ # virsh -c lxc:// list --all
Id Name State
----------------------------------------------------
- dwj-lnx-dev shut off
- vm1 shut off
*** Config from one container (itself is Gentoo Linux, sharing some
file-systems)
ostara ~ # virsh -c lxc:// dumpxml dwj-lnx-dev
<domain type='lxc'>
<name>dwj-lnx-dev</name>
<uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
<memory unit='KiB'>500000</memory>
<currentMemory unit='KiB'>500000</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/vm/lxc/dwj-lnx-dev'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/portage'/>
<target dir='/usr/portage'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/src'/>
<target dir='/usr/src'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/home'/>
<target dir='/home'/>
</filesystem>
<interface type='bridge'>
<mac address='82:00:00:00:01:00'/>
<source bridge='br0'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
</domain>
ostara ~ # ls -l /vm/lxc/dwj-lnx-dev
total 108
drwxr-xr-x 2 root root 4096 Oct 18 09:38 bin
drwxr-xr-x 2 root root 4096 Apr 27 2011 boot
drwxr-xr-x 10 root root 45056 Jan 26 10:52 dev
drwxr-xr-x 53 root root 4096 Feb 8 11:23 etc
drwxr-xr-x 2 root root 4096 Apr 27 2011 home
lrwxrwxrwx 1 root root 5 Oct 18 00:46 lib -> lib64
drwxr-xr-x 2 root root 4096 Oct 18 00:51 lib32
drwxr-xr-x 10 root root 4096 Oct 18 09:11 lib64
drwxr-xr-x 4 root root 4096 Apr 27 2011 mnt
drwxr-xr-x 4 root root 4096 May 22 2012 opt
drwxr-xr-x 2 root root 4096 Apr 27 2011 proc
drwx------ 3 root root 4096 Oct 18 08:13 root
drwxr-xr-x 2 root root 4096 Oct 18 00:57 run
drwxr-xr-x 2 root root 4096 Oct 18 09:11 sbin
drwxr-xr-x 2 root root 4096 Apr 27 2011 sys
drwxrwxrwt 4 root root 4096 Feb 8 11:23 tmp
drwxr-xr-x 13 root root 4096 May 4 2011 usr
drwxr-xr-x 14 root root 4096 May 22 2012 var
************ v 1.0.0 works great!
ostara ~ # virsh --version
1.0.0
ostara ~ # virsh -c lxc:// list --all
Id Name State
----------------------------------------------------
- dwj-lnx-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:// start dwj-lnx-dev
Domain dwj-lnx-dev started
ostara ~ # virsh -c lxc:// list --all
Id Name State
----------------------------------------------------
21364 dwj-lnx-dev running
- vm1 shut off
************* v 1.0.2 fails:
ostara ~ # /etc/init.d/libvirtd start
* Caching service dependencies ...
[ ok ]
* Starting libvirtd ...
[ ok ]
ostara ~ # virsh --version
1.0.2
ostara ~ # ls -l /var/lib/libvirt/lxc/
total 0
ostara ~ # virsh -c lxc:// start dwj-lnx-dev
error: Failed to start domain dwj-lnx-dev
error: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init
2013-02-08 18:09:28.402+0000: 1: info : libvirt version: 1.0.2
2013-02-08 18:09:28.402+0000: 1: error : lxcContainerMountProcFuse:616
: Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev/meminfo
on /proc/meminfo: No such file or directory
2013-02-08 18:09:28.402+0000: 23867: info : libvirt version: 1.0.2
2013-02-08 18:09:28.402+0000: 23867: error : virLXCControllerRun:1468
: error receiving signal from container: Input/output error
2013-02-08 18:09:28.814+0000: 23867: error : virCommandWait:2287 :
internal error Child process (ip link del veth1) unexpected exit
status 1: Cannot find device "veth1"
ostara ~ # ls -l /var/lib/libvirt/lxc/
total 0
ostara ~ # mount | grep libvirt
libvirt on /run/libvirt/lxc/dwj-lnx-dev type fuse (rw,nosuid,nodev)
ostara ~ # ls -l /run/libvirt/lxc
total 0
drwxr-xr-x 2 root root 40 Feb 8 12:09 dwj-lnx-dev
srwx------ 1 root root 0 Feb 8 12:09 dwj-lnx-dev.sock
srwx------ 1 root root 0 Feb 8 11:22 vm1.sock
ostara ~ # ls -l /run/libvirt/lxc/dwj-lnx-dev
total 0
*** "veth1" (regarding the rror "Cannot find device")
*** "veth1" is still no created. But I do know that when
*** using libvirt v1.0.0, and the container is running, the device
DOES exist (not shown above)
ostara ~ # ifconfig -a | egrep "^\w"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
*** The last entry in "/var/log/libvirt/lxc/dwj-lnx-dev.log"
2013-02-08 18:09:27.771+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.2
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc
--name dwj-lnx-dev --console 20 --security=none --handshake 23
--background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init
2013-02-08 18:09:28.402+0000: 1: info : libvirt version: 1.0.2
2013-02-08 18:09:28.402+0000: 1: error : lxcContainerMountProcFuse:616
: Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev/meminfo
on /proc/meminfo: No such file or directory
2013-02-08 18:09:28.402+0000: 23867: info : libvirt version: 1.0.2
2013-02-08 18:09:28.402+0000: 23867: error : virLXCControllerRun:1468
: error receiving signal from container: Input/output error
2013-02-08 18:09:28.814+0000: 23867: error : virCommandWait:2287 :
internal error Child process (ip link del veth1) unexpected exit
status 1: Cannot find device "veth1"
Dennis Jenkins
2013-Feb-08 20:46 UTC
[libvirt-users] libvirt v1.0.2 fails to boot LXC container, but v1.0.0 works
On Fri, Feb 8, 2013 at 12:38 PM, Dennis Jenkins <dennis.jenkins.75 at gmail.com> wrote:> Hello. > > tl;dr = v1.0.0 can boot my LXC containers, v.1.0.1 and v.1.0.2 fails. > Paraphrased error message: "lxcContainerMountProcFuse:616 : Failed to > mount ..../meminfo" > > I'd like to know if my host is misconfigured, or my domains, or > ... why 1.0.2 and 1.0.1 are not working for me.http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=ae9874e471fc61d9d654247c2fd24d9baa455afa It looks like I have been bitten by the above bug. What version of libvirt will these patches appear in? eg, is there a way to map a git commit to a future release?
Possibly Parallel Threads
- Unable to start LXC on Gentoo w/ libvirt 1.0.4 or 1.0.5. 1.0.3 works
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- libvirt, lxc, "/dev/log" ENOENT