libvirt users - Nov 2012 - OpenStack+libvirt+lxc: lxcContainerGetSubtree:1199 : Failed to read /proc/mounts

Hi,

I'm running OpenStack on CentOS 6.3 to manage lxc instances. And running
into series of problem relating libvirt and lxc interaction.

For example, libvirt_lxc segfault (
https://bugzilla.redhat.com/show_bug.cgi?id=874549) which has an upstream
fix. And another bugs such as fail to start when SELinux disabled.

Finally, I decides to adopt libvirt 0.10.2, self compiled
from libvirt-0.10.2-1.fc17.src.rpm. And now coming across following message
when starting lxc container.

lxcContainerGetSubtree:1199 : Failed to read /proc/mounts: No such file or
directory


Could it be something related to my container template setting? Or host
setting? Or simply just another bug? I'm not a libvirt guy, so please, do
shed me some lights on how to fix it. Thanks very much.

libvirtd log with LIBVIRT_DEBUG=1
--
2012-11-08 12:41:10.106+0000: 28362: info : libvirt version: 0.10.2,
package: 1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:41:10.106+0000: 28362: error : virLXCControllerRun:1418 :
error receiving signal from container: Input/output error

2012-11-08 12:41:10.222+0000: 24641: error : virNetDevSetOnline:536 :
Cannot get interface flags on 'veth0': No such device
2012-11-08 12:41:10.227+0000: 24641: error : virNWFilterDHCPSnoopEnd:2131 :
internal error ifname "veth0" not in key map
2012-11-08 12:41:10.229+0000: 24641: error : virNetDevGetIndex:653 : Unable
to get index for interface veth0: No such device
2012-11-08 12:41:10.370+0000: 24640: error : virLXCProcessStop:701 :
internal error Invalid PID -1 for container
2012-11-08 12:41:10.370+0000: 24640: error : virLXCProcessStop:701 :
internal error Invalid PID -1 for container
2012-11-08 12:48:26.136+0000: 30706: info : libvirt version: 0.10.2,
package: 1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:48:26.136+0000: 30706: error : virDomainObjParseNode:10094 :
XML error: unexpected root element <domain>, expecting <domstatus>



2012-11-08 12:48:50.878+0000: 30695: error : virNetSocketReadWire:1176 :
Cannot recv data: Connection reset by peer
2012-11-08 12:48:50.977+0000: 30698: error : virLXCProcessStart:1145 :
internal error guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=ecc871cb-58be-4ae4-b216-a8a62b894a7f
LIBVIRT_LXC_UUID=ecc871cb-58be-4ae4-b216-a8a62b894a7f
LIBVIRT_LXC_NAME=instance-00000077 LIBVIRT_LXC_CMDLINE=console=ttyS0
/sbin/init
2012-11-08 12:48:50.863+0000: 1: info : libvirt version: 0.10.2, package:
1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:48:50.863+0000: 1: error : lxcContainerGetSubtree:1199 :
Failed to read /proc/mounts: No such file or directory
2012-11-08 12:48:50.863+0000: 31202: info : libvirt version: 0.10.2,
package: 1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:48:50.863+0000: 31202: error : virLXCControllerRun:1418 :
error receiving signal from container: Input/output error

2012-11-08 12:48:50.979+0000: 30698: error : virNetDevSetOnline:536 :
Cannot get interface flags on 'veth0': No such device
2012-11-08 12:48:50.984+0000: 30698: error : virNWFilterDHCPSnoopEnd:2131 :
internal error ifname "veth0" not in key map
2012-11-08 12:48:50.986+0000: 30698: error : virNetDevGetIndex:653 : Unable
to get index for interface veth0: No such device
2012-11-08 12:48:51.127+0000: 30695: error : virLXCProcessStop:701 :
internal error Invalid PID -1 for container
2012-11-08 12:48:51.128+0000: 30695: error : virLXCProcessStop:701 :
internal error Invalid PID -1 for container
2012-11-08 12:48:54.709+0000: 30695: error : virNetSocketReadWire:1184 :
End of file while reading data: Input/output error
2012-11-08 12:48:54.710+0000: 30695: error : virNetSocketReadWire:1184 :
End of file while reading data: Input/output error


lxc log
--
 tail -f lxc/instance-00000077.log
2012-11-08 12:48:50.723+0000: starting up
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc
--name instance-00000077 --console 21 --security=none --handshake 24
--background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=ecc871cb-58be-4ae4-b216-a8a62b894a7f
LIBVIRT_LXC_UUID=ecc871cb-58be-4ae4-b216-a8a62b894a7f
LIBVIRT_LXC_NAME=instance-00000077 LIBVIRT_LXC_CMDLINE=console=ttyS0
/sbin/init
2012-11-08 12:48:50.863+0000: 1: info : libvirt version: 0.10.2, package:
1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:48:50.863+0000: 1: error : lxcContainerGetSubtree:1199 :
Failed to read /proc/mounts: No such file or directory
2012-11-08 12:48:50.863+0000: 31202: info : libvirt version: 0.10.2,
package: 1.el6 (Unknown, 2012-11-08-20:20:52, localhost)
2012-11-08 12:48:50.863+0000: 31202: error : virLXCControllerRun:1418 :
error receiving signal from container: Input/output error


libvirt xml
--
<domain type='lxc'>
  <name>instance-00000077</name>
  <uuid>ecc871cb-58be-4ae4-b216-a8a62b894a7f</uuid>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
    <cmdline>console=ttyS0</cmdline>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source
dir='/home/stack/nova_state/instances/instance-00000077/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='fa:16:3e:40:ec:11'/>
      <source bridge='br100'/>
      <filterref
filter='nova-instance-instance-00000077-fa163e40ec11'>
        <parameter name='DHCPSERVER' value='10.48.253.1'/>
        <parameter name='IP' value='10.48.253.2'/>
        <parameter name='PROJMASK' value='255.255.255.0'/>
        <parameter name='PROJNET' value='10.48.253.0'/>
      </filterref>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

cat /proc/mounts
--
[root at localhost libvirt]# cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs
rw,nosuid,relatime,size=32967000k,nr_inodes=8241750,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
/dev/sda2 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/mapper/xenvg-home /home ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
none /noah/tmp tmpfs rw,relatime,mode=755 0 0
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blk_io cgroup rw,relatime,blkio 0 0
cgroup /cgroup/ns cgroup rw,relatime,ns 0 0
/dev/nbd15 /home/stack/nova_state/instances/instance-00000070/rootfs ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/nbd14 /home/stack/nova_state/instances/instance-00000071/rootfs ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/nbd13 /home/stack/nova_state/instances/instance-00000073/rootfs ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/nbd12 /home/stack/nova_state/instances/instance-00000075/rootfs ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/nbd11 /home/stack/nova_state/instances/instance-00000076/rootfs ext3
rw,relatime,errors=continue,barrier=1,data=ordered 0 0
/dev/nbd10 /home/stack/nova_state/instances/instance-00000077/rootfs ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
--
Qiu Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121108/b0dd0989/attachment.htm>

On Thu, Nov 08, 2012 at 09:11:51PM +0800, unicell wrote:
> Hi,
> 
> I'm running OpenStack on CentOS 6.3 to manage lxc instances. And
running
> into series of problem relating libvirt and lxc interaction.
> 
> For example, libvirt_lxc segfault (
> https://bugzilla.redhat.com/show_bug.cgi?id=874549) which has an upstream
> fix. And another bugs such as fail to start when SELinux disabled.
> 
> Finally, I decides to adopt libvirt 0.10.2, self compiled
> from libvirt-0.10.2-1.fc17.src.rpm. And now coming across following message
> when starting lxc container.
> 
> lxcContainerGetSubtree:1199 : Failed to read /proc/mounts: No such file or
> directory
Try using libvirt 1.0.0, the version you built has a bug in its LXC
support causing the error you see.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|