Dennis Jenkins
2013-May-14 13:28 UTC
[libvirt-users] Unable to start LXC on Gentoo w/ libvirt 1.0.4 or 1.0.5. 1.0.3 works
Hello.
I use libvirtd on my Gentoo development system to manage both QEMU and
LXC. When 1.0.3 came out, I updated to it from 1.0.3-r2, but 1.0.4 failed
to start my LXC containers. I did not research the issue at the time, so I
revert to 1.0.3-r2. Today I updated to 1.0.5 and my LXC containers still
fail to start. I have not changed my domain XML at all.
I am looking for suggestions on what to try to resolve the problem.
Maybe I'm missing a package or have something mus-configured for
libvirt-1.0.5. btw, A month ago I updates to the most recent "udev"
(udev-200). This required me to rename all of my network interfaces. My
QEMU and LXC containers continued to work. All of my VMs use bridged
networking on "br0" or "br1". Those virtual interfaces are
not bridged to
any physical NIC. They exist only inside my host's kernel. I have
extensive iptables rules to filter traffic between my network segments
(physical + virtual).
TL;DR: this was burried in a log file: " error :
lxcContainerMountFSDevPTS:808 : Failed to mount
/.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file
or directory". But that directory DOES exist in the container:
ostara ~ # ls -ld /vm/lxc/dwj-lnx-dev/dev/pts/
drwxr-xr-x 2 root root 4096 Apr 27 2011 /vm/lxc/dwj-lnx-dev/dev/pts/
If I revert to libvirt-1.0.3-r2, I can boot my LXC containers again.
But if possible, I'd like to keep my system current. I've ran the
usual
Gentoo tools to verify that all package dependencies are installed,
revdep-rebuild reports that all ELFs link against the proper shared objects.
ostara ~ # virsh --version
1.0.5
ostara ~ # equery l libvirt
* Searching for libvirt ...
[IP-] [ ] app-emulation/libvirt-1.0.5:0
ostara ~ # equery -q u libvirt | xargs echo
-audit -avahi +caps -firewalld -fuse -iscsi +libvirtd +lvm +lxc +macvtap
+nfs +nls -numa -openvz -parted +pcap -phyp -policykit -python +qemu -rbd
+sasl +udev +uml +vepa +virt-network -virtualbox -xen
ostara ~ # virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
- dwj-lnx-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:/// start dwj-lnx-dev
error: Failed to start domain dwj-lnx-dev
error: internal error guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init
error receiving signal from container: Input/output error
ostara ~ # tail -n 10 /var/log/libvirt/libvirtd.log
2013-05-14 13:03:47.549+0000: 22352: info : libvirt version: 1.0.5
2013-05-14 13:03:47.549+0000: 22352: error : virDBusGetSystemBus:86 :
internal error Unable to get DBus system bus connection: Failed to connect
to socket /var/run/dbus/system_bus_socket: No such file or directory
2013-05-14 13:03:47.549+0000: 22352: warning :
ebiptablesDriverInitCLITools:4233 : Could not find 'ip6tables'
executable
2013-05-14 13:04:21.629+0000: 22341: error : virNetSocketReadWire:1362 :
Cannot recv data: Connection reset by peer
2013-05-14 13:04:21.729+0000: 22345: error : virLXCProcessStart:1285 :
internal error guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init
error receiving signal from container: Input/output error
2013-05-14 13:04:21.731+0000: 22345: error : virNetDevSetOnline:544 :
Cannot get interface flags on 'veth0': No such device
2013-05-14 13:04:21.778+0000: 22345: error : virCommandWait:2353 : internal
error Child process (ip link del veth0) unexpected exit status 1: Cannot
find device "veth0"
2013-05-14 13:04:21.917+0000: 22345: error : virCommandWait:2353 : internal
error Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"
ostara ~ # tail -n 10 /var/log/libvirt/lxc/dwj-lnx-dev.log
2013-05-14 13:04:20.894+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.2
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc
--name dwj-lnx-dev --console 22 --security=none --handshake 25 --background
--veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f
LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init
2013-05-14 13:04:21.562+0000: 1: info : libvirt version: 1.0.5
2013-05-14 13:04:21.562+0000: 1: error : lxcContainerMountFSDevPTS:808 :
Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on
/dev/pts: No such file or directory
2013-05-14 13:04:21.562+0000: 22562: info : libvirt version: 1.0.5
2013-05-14 13:04:21.562+0000: 22562: error : virLXCControllerRun:1481 :
error receiving signal from container: Input/output error
error receiving signal from container: Input/output error
2013-05-14 13:04:21.628+0000: 22562: error : virCommandWait:2353 : internal
error Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"
ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev
<domain type='lxc'>
<name>dwj-lnx-dev</name>
<uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
<memory unit='KiB'>500000</memory>
<currentMemory unit='KiB'>500000</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/vm/lxc/dwj-lnx-dev'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/portage'/>
<target dir='/usr/portage'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/src'/>
<target dir='/usr/src'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/home'/>
<target dir='/home'/>
</filesystem>
<interface type='bridge'>
<mac address='82:00:00:00:01:00'/>
<source bridge='br0'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
</domain>
ostara ~ # ifconfig -a | egrep -v "(RX|TX) (packets|errors)"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
inet 192.168.2.1 netmask 255.255.255.0 broadcast 192.168.2.255
ether 00:00:00:00:00:00 txqueuelen 0 (Ethernet)
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet 192.168.3.1 netmask 255.255.255.0 broadcast 192.168.3.255
ether 06:9b:23:fb:14:5f txqueuelen 0 (Ethernet)
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255
ether bc:ae:c5:16:6f:3b txqueuelen 1000 (Ethernet)
device interrupt 18 memory 0xfade0000-fae00000
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet xx.xx.xx.xx netmask 255.255.252.0 broadcast xx.xx.xx.xx
ether bc:ae:c5:16:70:3f txqueuelen 1000 (Ethernet)
device interrupt 19 memory 0xfaee0000-faf00000
enp6s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.4.1 netmask 255.255.255.0 broadcast 192.168.4.255
ether 00:03:6d:14:78:30 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0 (Local Loopback)
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether c2:ff:56:b7:a7:cd txqueuelen 0 (Ethernet)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20130514/fad74c5c/attachment.htm>
Daniel P. Berrange
2013-May-14 13:55 UTC
[libvirt-users] Unable to start LXC on Gentoo w/ libvirt 1.0.4 or 1.0.5. 1.0.3 works
On Tue, May 14, 2013 at 08:28:48AM -0500, Dennis Jenkins wrote:> I use libvirtd on my Gentoo development system to manage both QEMU and > LXC. When 1.0.3 came out, I updated to it from 1.0.3-r2, but 1.0.4 failed > to start my LXC containers. I did not research the issue at the time, so I > revert to 1.0.3-r2. Today I updated to 1.0.5 and my LXC containers still > fail to start. I have not changed my domain XML at all. > > TL;DR: this was burried in a log file: " error : > lxcContainerMountFSDevPTS:808 : Failed to mount > /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file > or directory". But that directory DOES exist in the container: > > ostara ~ # ls -ld /vm/lxc/dwj-lnx-dev/dev/pts/ > drwxr-xr-x 2 root root 4096 Apr 27 2011 /vm/lxc/dwj-lnx-dev/dev/pts/Actually libvirt will mount a brand new tmpfs at /dev inside the container, so whatever your filesystem has at /dev is not used. Libvirt should of course also be creating the /dev/pts mount point. This makes me think the error message could actually refer to the "/.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts" path.> > ostara ~ # tail -n 10 /var/log/libvirt/lxc/dwj-lnx-dev.log > 2013-05-14 13:04:20.894+0000: starting up > PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.2 > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc > --name dwj-lnx-dev --console 22 --security=none --handshake 25 --background > --veth veth1 > PATH=/bin:/sbin TERM=linux container=lxc-libvirt > container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f > LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f > LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init > 2013-05-14 13:04:21.562+0000: 1: info : libvirt version: 1.0.5 > 2013-05-14 13:04:21.562+0000: 1: error : lxcContainerMountFSDevPTS:808 : > Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on > /dev/pts: No such file or directory > 2013-05-14 13:04:21.562+0000: 22562: info : libvirt version: 1.0.5 > 2013-05-14 13:04:21.562+0000: 22562: error : virLXCControllerRun:1481 : > error receiving signal from container: Input/output error > error receiving signal from container: Input/output error > 2013-05-14 13:04:21.628+0000: 22562: error : virCommandWait:2353 : internal > error Child process (ip link del veth1) unexpected exit status 1: Cannot > find device "veth1"We need to get more debug info from this file. Stop libvirtd and edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1 and then run libvirtd with LIBVIRT_LOG_OUTPUTS=1:stderr LIBVIRT_LOG_FILTERS="3:event 3:json 3:object 1:util 1:lxc" /usr/sbin/libvirtd and then try to start a container again. The container log file above should now get much more verbose data (i don't care about the log output from libvirtd itself - just the container log file). Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Possibly Parallel Threads
- libvirt v1.0.2 fails to boot LXC container, but v1.0.0 works
- libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- libvirt, lxc, "/dev/log" ENOENT