Dennis Jenkins
2013-Jul-12 16:40 UTC
[libvirt-users] libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
Hello all,
I have two issues:
1) I am unable to start a seemingly correct LXC domain (I cloned it from a
working domain).
2) I am able to crash "libvirtd" by attempting to start the cloned
domain,
but starting the original works just fine.
I humbly submit that item #2 is a bug - the "libvirtd" daemon
should
never crash due to anything the "libvirt" client throws at it. As for
item #1, I'm not sure where I went wrong. A full walk-through is below
(ending with a DIFF of the XML from the two domains).
I created by original domain ("dwj-lnx-dev") a long time ago.
Today I
created the new domain ("dwj-hfax-dev") as follows:
1) Shutdown "dwj-lnx-dev"
2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev
dwj-hfax-dev" (2.5GB, ~5 min)
3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml"
4) ${EDITOR} a.xml
a) changed MAC address, name, memory, source directory for "/"
5) "libvirt -c lxc:/// define a.xml"
6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on my
host.
It does not matter is "dwj-lnx-dev" is running or not. Any
attempt to
start "dwj-hfax-dev" will crash libvirtd.
In the past I was asked to turn on some debugging and capture a
detailed log (
https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html). I
will do this soon and post my results as a follow up.
ostara ~ # uname -a
Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013
x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
ostara ~ # equery l libvirt
* Searching for libvirt ...
[IP-] [ ] app-emulation/libvirt-1.1.0-r1:0
ostara ~ # virsh -c lxc:/// version
Compiled against library: libvirt 1.1.0
Using library: libvirt 1.1.0
Using API: LXC 1.1.0
Running hypervisor: LXC 3.8.13
ostara ~ # /etc/init.d/libvirtd restart
* Caching service dependencies
...
[ ok ]
* Stopping libvirtd ...
* Shutting down network(s):
*
default
[ ok ]
* Starting libvirtd
...
[ ok ]
ostara ~ # virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
- dwj-hfax-dev shut off
- dwj-lnx-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:/// start dwj-lnx-dev
Domain dwj-lnx-dev started
ostara ~ # virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
9441 dwj-lnx-dev running
- dwj-hfax-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
error: Failed to start domain dwj-hfax-dev
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the
hypervisor
error: Failed to reconnect to the hypervisor
ostara ~ # virsh -c lxc:/// list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
Connection refused
ostara ~ # ls -l /var/run/libvirt/libvirt-sock
srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock
ostara ~ # ps axfw | grep libvirt
9997 pts/2 S+ 0:00 \_ grep --colour=auto
libvirt
8446 ? S 0:00 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf
9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
--console 19 --security=none --handshake 23 --background --veth veth1
ostara ~ # /etc/init.d/libvirtd restart
* Stopping libvirtd ...
* start-stop-daemon: no matching processes
found
[ ok ]
* Starting libvirtd
...
[ ok ]
ostara ~ # ps axfw | grep libvirt
10130 pts/2 S+ 0:00 \_ grep --colour=auto
libvirt
8446 ? S 0:00 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf
9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
--console 19 --security=none --handshake 23 --background --veth veth1
10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen
ostara ~ # virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
9441 dwj-lnx-dev running
- dwj-hfax-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
<domain type='lxc'>
<name>dwj-hfax-dev</name>
<uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>4</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/vm/lxc/dwj-hfax-dev'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/portage'/>
<target dir='/usr/portage'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/src'/>
<target dir='/usr/src'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/home'/>
<target dir='/home'/>
</filesystem>
<interface type='bridge'>
<mac address='82:00:00:00:01:01'/>
<source bridge='br0'/>
<target dev='veth0'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
<seclabel type='none'/>
</domain>
ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev
<domain type='lxc' id='9441'>
<name>dwj-lnx-dev</name>
<uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
<memory unit='KiB'>500000</memory>
<currentMemory unit='KiB'>500000</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/vm/lxc/dwj-lnx-dev'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/portage'/>
<target dir='/usr/portage'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/src'/>
<target dir='/usr/src'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/home'/>
<target dir='/home'/>
</filesystem>
<interface type='bridge'>
<mac address='82:00:00:00:01:00'/>
<source bridge='br0'/>
<target dev='veth0'/>
</interface>
<console type='pty' tty='/dev/pts/3'>
<source path='/dev/pts/3'/>
<target type='lxc' port='0'/>
<alias name='console0'/>
</console>
</devices>
<seclabel type='none'/>
</domain>
ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml
ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml
ostara ~ # diff lnx.xml hfax.xml
1,6c1,6
< <domain type='lxc' id='9441'>
< <name>dwj-lnx-dev</name>
< <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
< <memory unit='KiB'>500000</memory>
< <currentMemory unit='KiB'>500000</currentMemory>
< <vcpu placement='static'>2</vcpu>
---> <domain type='lxc'>
> <name>dwj-hfax-dev</name>
> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>4</vcpu>
21c21
< <source dir='/vm/lxc/dwj-lnx-dev'/>
---> <source dir='/vm/lxc/dwj-hfax-dev'/>
37c37
< <mac address='82:00:00:00:01:00'/>
---> <mac address='82:00:00:00:01:01'/>
41,42c41
< <console type='pty' tty='/dev/pts/3'>
< <source path='/dev/pts/3'/>
---> <console type='pty'>
44d42
< <alias name='console0'/>
(After reseting everything, and attempting to boot hfax with dev offline,
libvirtd still crashes)
ostara ~ # virsh -c lxc:/// list --all
Id Name State
----------------------------------------------------
- dwj-hfax-dev shut off
- dwj-lnx-dev shut off
- vm1 shut off
ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
error: Failed to start domain dwj-hfax-dev
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the
hypervisor
error: Failed to reconnect to the hypervisor
Dennis Jenkins
2013-Jul-12 16:44 UTC
Re: [libvirt-users] libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
The debug log ends with this:
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:708 : Make
group /machine/dwj-hfax-dev.libvirt-lxc
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/cpu/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/cpuacct/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/cpuset/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/memory/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/devices/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/freezer/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/blkio/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:715 :
Skipping unmounted controller net_cls
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
controller /sys/fs/cgroup/perf_event/machine/dwj-hfax-dev.libvirt-lxc/
2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:779 : Done
making controllers for group
2013-07-12 16:43:31.740+0000: 21365: debug : virFileMakePathHelper:1995 :
path=/var/log/libvirt/lxc mode=0777
2013-07-12 16:43:31.740+0000: 21365: debug : virLXCProcessStart:1096 :
Setting current domain def as transient
2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1121 :
Preparing host devices
2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1139 :
Generating domain security label (if required)
====== end of log ====
Segmentation fault (core dumped)
(gdb) bt
#0 0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6
#1 0x00007fe47578ad31 in virSecurityManagerGenLabel () from
/usr/lib64/libvirt.so.0
#2 0x00007fe46aa92979 in virLXCProcessStart () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
#3 0x00007fe46aa9736e in lxcDomainCreateWithFlags () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
#4 0x00007fe47569c067 in virDomainCreate () from /usr/lib64/libvirt.so.0
#5 0x00007fe4760d5578 in remoteDispatchDomainCreateHelper ()
#6 0x00007fe4756fcd78 in virNetServerProgramDispatch () from
/usr/lib64/libvirt.so.0
#7 0x00007fe4756f7302 in virNetServerProcessMsg () from
/usr/lib64/libvirt.so.0
#8 0x00007fe4756f7a93 in virNetServerHandleJob () from
/usr/lib64/libvirt.so.0
#9 0x00007fe47560f95e in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#10 0x00007fe47560efc6 in virThreadHelper () from /usr/lib64/libvirt.so.0
#11 0x00007fe475354da6 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fe47508d99d in clone () from /lib64/libc.so.6
(gdb)
On Fri, Jul 12, 2013 at 11:40 AM, Dennis Jenkins <
dennis.jenkins.75@gmail.com> wrote:
> Hello all,
>
> I have two issues:
>
> 1) I am unable to start a seemingly correct LXC domain (I cloned it from a
> working domain).
>
> 2) I am able to crash "libvirtd" by attempting to start the
cloned domain,
> but starting the original works just fine.
>
> I humbly submit that item #2 is a bug - the "libvirtd" daemon
should
> never crash due to anything the "libvirt" client throws at it.
As for
> item #1, I'm not sure where I went wrong. A full walk-through is
below
> (ending with a DIFF of the XML from the two domains).
>
> I created by original domain ("dwj-lnx-dev") a long time ago.
Today I
> created the new domain ("dwj-hfax-dev") as follows:
>
> 1) Shutdown "dwj-lnx-dev"
> 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev
> dwj-hfax-dev" (2.5GB, ~5 min)
> 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml"
> 4) ${EDITOR} a.xml
> a) changed MAC address, name, memory, source directory for "/"
> 5) "libvirt -c lxc:/// define a.xml"
> 6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on
my host.
>
>
> It does not matter is "dwj-lnx-dev" is running or not. Any
attempt to
> start "dwj-hfax-dev" will crash libvirtd.
>
>
> In the past I was asked to turn on some debugging and capture a
> detailed log (
> https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html). I
> will do this soon and post my results as a follow up.
>
>
>
> ostara ~ # uname -a
> Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013
> x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
>
>
> ostara ~ # equery l libvirt
> * Searching for libvirt ...
> [IP-] [ ] app-emulation/libvirt-1.1.0-r1:0
>
>
> ostara ~ # virsh -c lxc:/// version
> Compiled against library: libvirt 1.1.0
> Using library: libvirt 1.1.0
> Using API: LXC 1.1.0
> Running hypervisor: LXC 3.8.13
>
>
> ostara ~ # /etc/init.d/libvirtd restart
> * Caching service dependencies
> ...
> [ ok ]
> * Stopping libvirtd ...
> * Shutting down network(s):
> *
> default
> [ ok ]
> * Starting libvirtd
> ...
> [ ok ]
>
>
> ostara ~ # virsh -c lxc:/// list --all
> Id Name State
> ----------------------------------------------------
> - dwj-hfax-dev shut off
> - dwj-lnx-dev shut off
> - vm1 shut off
>
> ostara ~ # virsh -c lxc:/// start dwj-lnx-dev
> Domain dwj-lnx-dev started
>
>
> ostara ~ # virsh -c lxc:/// list --all
> Id Name State
> ----------------------------------------------------
> 9441 dwj-lnx-dev running
> - dwj-hfax-dev shut off
> - vm1 shut off
>
>
> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
> error: Failed to start domain dwj-hfax-dev
> error: End of file while reading data: Input/output error
> error: One or more references were leaked after disconnect from the
> hypervisor
> error: Failed to reconnect to the hypervisor
>
>
> ostara ~ # virsh -c lxc:/// list --all
> error: failed to connect to the hypervisor
> error: no valid connection
> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
> Connection refused
>
>
> ostara ~ # ls -l /var/run/libvirt/libvirt-sock
> srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock
>
>
> ostara ~ # ps axfw | grep libvirt
> 9997 pts/2 S+ 0:00 \_ grep --colour=auto
> libvirt
> 8446 ? S 0:00 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
> --console 19 --security=none --handshake 23 --background --veth veth1
>
>
> ostara ~ # /etc/init.d/libvirtd restart
> * Stopping libvirtd ...
> * start-stop-daemon: no matching processes
> found
> [ ok ]
> * Starting libvirtd
> ...
> [ ok ]
>
>
> ostara ~ # ps axfw | grep libvirt
> 10130 pts/2 S+ 0:00 \_ grep --colour=auto
> libvirt
> 8446 ? S 0:00 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
> --console 19 --security=none --handshake 23 --background --veth veth1
> 10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen
>
>
> ostara ~ # virsh -c lxc:/// list --all
> Id Name State
> ----------------------------------------------------
> 9441 dwj-lnx-dev running
> - dwj-hfax-dev shut off
> - vm1 shut off
>
>
> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
> <domain type='lxc'>
> <name>dwj-hfax-dev</name>
> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>4</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='x86_64'>exe</type>
> <init>/sbin/init</init>
> </os>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/libexec/libvirt_lxc</emulator>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/vm/lxc/dwj-hfax-dev'/>
> <target dir='/'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/usr/portage'/>
> <target dir='/usr/portage'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/usr/src'/>
> <target dir='/usr/src'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/home'/>
> <target dir='/home'/>
> </filesystem>
> <interface type='bridge'>
> <mac address='82:00:00:00:01:01'/>
> <source bridge='br0'/>
> <target dev='veth0'/>
> </interface>
> <console type='pty'>
> <target type='lxc' port='0'/>
> </console>
> </devices>
> <seclabel type='none'/>
> </domain>
>
>
> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev
> <domain type='lxc' id='9441'>
> <name>dwj-lnx-dev</name>
> <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
> <memory unit='KiB'>500000</memory>
> <currentMemory unit='KiB'>500000</currentMemory>
> <vcpu placement='static'>2</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='x86_64'>exe</type>
> <init>/sbin/init</init>
> </os>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/libexec/libvirt_lxc</emulator>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/vm/lxc/dwj-lnx-dev'/>
> <target dir='/'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/usr/portage'/>
> <target dir='/usr/portage'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/usr/src'/>
> <target dir='/usr/src'/>
> </filesystem>
> <filesystem type='mount'
accessmode='passthrough'>
> <source dir='/home'/>
> <target dir='/home'/>
> </filesystem>
> <interface type='bridge'>
> <mac address='82:00:00:00:01:00'/>
> <source bridge='br0'/>
> <target dev='veth0'/>
> </interface>
> <console type='pty' tty='/dev/pts/3'>
> <source path='/dev/pts/3'/>
> <target type='lxc' port='0'/>
> <alias name='console0'/>
> </console>
> </devices>
> <seclabel type='none'/>
> </domain>
>
>
> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml
>
>
> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml
>
>
> ostara ~ # diff lnx.xml hfax.xml
> 1,6c1,6
> < <domain type='lxc' id='9441'>
> < <name>dwj-lnx-dev</name>
> < <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
> < <memory unit='KiB'>500000</memory>
> < <currentMemory unit='KiB'>500000</currentMemory>
> < <vcpu placement='static'>2</vcpu>
> ---
> > <domain type='lxc'>
> > <name>dwj-hfax-dev</name>
> > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
> > <memory unit='KiB'>4194304</memory>
> > <currentMemory
unit='KiB'>4194304</currentMemory>
> > <vcpu placement='static'>4</vcpu>
> 21c21
> < <source dir='/vm/lxc/dwj-lnx-dev'/>
> ---
> > <source dir='/vm/lxc/dwj-hfax-dev'/>
> 37c37
> < <mac address='82:00:00:00:01:00'/>
> ---
> > <mac address='82:00:00:00:01:01'/>
> 41,42c41
> < <console type='pty' tty='/dev/pts/3'>
> < <source path='/dev/pts/3'/>
> ---
> > <console type='pty'>
> 44d42
> < <alias name='console0'/>
>
>
> (After reseting everything, and attempting to boot hfax with dev offline,
> libvirtd still crashes)
>
> ostara ~ # virsh -c lxc:/// list --all
> Id Name State
> ----------------------------------------------------
> - dwj-hfax-dev shut off
> - dwj-lnx-dev shut off
> - vm1 shut off
>
>
>
> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
> error: Failed to start domain dwj-hfax-dev
> error: End of file while reading data: Input/output error
> error: One or more references were leaked after disconnect from the
> hypervisor
> error: Failed to reconnect to the hypervisor
>
>
Dennis Jenkins
2013-Jul-12 18:32 UTC
Re: [libvirt-users] libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
Update: I am able to edit the XML in "dwj-hfax-dev" such that libvirtd no longer crashes, and edit the XML for "dwj-lnx-dev" such that it will crash. The presents of "<seclabel type='none'/>" near the bottom causes libvirtd to crash. I do not recall ever manually adding that to my domain. In any event, libvirtd should probably not crash due to the XML element (which seems valid - or at least "virsh edit" allows it). On Fri, Jul 12, 2013 at 11:44 AM, Dennis Jenkins < dennis.jenkins.75@gmail.com> wrote:> The debug log ends with this: > > > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:708 : Make > group /machine/dwj-hfax-dev.libvirt-lxc > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/cpu/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/cpuacct/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/cpuset/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/memory/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/devices/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/freezer/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/blkio/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:715 : > Skipping unmounted controller net_cls > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make > controller /sys/fs/cgroup/perf_event/machine/dwj-hfax-dev.libvirt-lxc/ > 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:779 : Done > making controllers for group > 2013-07-12 16:43:31.740+0000: 21365: debug : virFileMakePathHelper:1995 : > path=/var/log/libvirt/lxc mode=0777 > 2013-07-12 16:43:31.740+0000: 21365: debug : virLXCProcessStart:1096 : > Setting current domain def as transient > 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1121 : > Preparing host devices > 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1139 : > Generating domain security label (if required) > > > ====== end of log ====> > Segmentation fault (core dumped) > > > > (gdb) bt > #0 0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6 > #1 0x00007fe47578ad31 in virSecurityManagerGenLabel () from > /usr/lib64/libvirt.so.0 > #2 0x00007fe46aa92979 in virLXCProcessStart () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so > #3 0x00007fe46aa9736e in lxcDomainCreateWithFlags () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so > #4 0x00007fe47569c067 in virDomainCreate () from /usr/lib64/libvirt.so.0 > #5 0x00007fe4760d5578 in remoteDispatchDomainCreateHelper () > #6 0x00007fe4756fcd78 in virNetServerProgramDispatch () from > /usr/lib64/libvirt.so.0 > #7 0x00007fe4756f7302 in virNetServerProcessMsg () from > /usr/lib64/libvirt.so.0 > #8 0x00007fe4756f7a93 in virNetServerHandleJob () from > /usr/lib64/libvirt.so.0 > #9 0x00007fe47560f95e in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #10 0x00007fe47560efc6 in virThreadHelper () from /usr/lib64/libvirt.so.0 > #11 0x00007fe475354da6 in start_thread () from /lib64/libpthread.so.0 > #12 0x00007fe47508d99d in clone () from /lib64/libc.so.6 > (gdb) > > > > On Fri, Jul 12, 2013 at 11:40 AM, Dennis Jenkins < > dennis.jenkins.75@gmail.com> wrote: > >> Hello all, >> >> I have two issues: >> >> 1) I am unable to start a seemingly correct LXC domain (I cloned it from >> a working domain). >> >> 2) I am able to crash "libvirtd" by attempting to start the cloned >> domain, but starting the original works just fine. >> >> I humbly submit that item #2 is a bug - the "libvirtd" daemon should >> never crash due to anything the "libvirt" client throws at it. As for >> item #1, I'm not sure where I went wrong. A full walk-through is below >> (ending with a DIFF of the XML from the two domains). >> >> I created by original domain ("dwj-lnx-dev") a long time ago. Today >> I created the new domain ("dwj-hfax-dev") as follows: >> >> 1) Shutdown "dwj-lnx-dev" >> 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev >> dwj-hfax-dev" (2.5GB, ~5 min) >> 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml" >> 4) ${EDITOR} a.xml >> a) changed MAC address, name, memory, source directory for "/" >> 5) "libvirt -c lxc:/// define a.xml" >> 6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on my host. >> >> >> It does not matter is "dwj-lnx-dev" is running or not. Any attempt >> to start "dwj-hfax-dev" will crash libvirtd. >> >> >> In the past I was asked to turn on some debugging and capture a >> detailed log ( >> https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html). >> I will do this soon and post my results as a follow up. >> >> >> >> ostara ~ # uname -a >> Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013 >> x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux >> >> >> ostara ~ # equery l libvirt >> * Searching for libvirt ... >> [IP-] [ ] app-emulation/libvirt-1.1.0-r1:0 >> >> >> ostara ~ # virsh -c lxc:/// version >> Compiled against library: libvirt 1.1.0 >> Using library: libvirt 1.1.0 >> Using API: LXC 1.1.0 >> Running hypervisor: LXC 3.8.13 >> >> >> ostara ~ # /etc/init.d/libvirtd restart >> * Caching service dependencies >> ... >> [ ok ] >> * Stopping libvirtd ... >> * Shutting down network(s): >> * >> default >> [ ok ] >> * Starting libvirtd >> ... >> [ ok ] >> >> >> ostara ~ # virsh -c lxc:/// list --all >> Id Name State >> ---------------------------------------------------- >> - dwj-hfax-dev shut off >> - dwj-lnx-dev shut off >> - vm1 shut off >> >> ostara ~ # virsh -c lxc:/// start dwj-lnx-dev >> Domain dwj-lnx-dev started >> >> >> ostara ~ # virsh -c lxc:/// list --all >> Id Name State >> ---------------------------------------------------- >> 9441 dwj-lnx-dev running >> - dwj-hfax-dev shut off >> - vm1 shut off >> >> >> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev >> error: Failed to start domain dwj-hfax-dev >> error: End of file while reading data: Input/output error >> error: One or more references were leaked after disconnect from the >> hypervisor >> error: Failed to reconnect to the hypervisor >> >> >> ostara ~ # virsh -c lxc:/// list --all >> error: failed to connect to the hypervisor >> error: no valid connection >> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': >> Connection refused >> >> >> ostara ~ # ls -l /var/run/libvirt/libvirt-sock >> srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock >> >> >> ostara ~ # ps axfw | grep libvirt >> 9997 pts/2 S+ 0:00 \_ grep --colour=auto >> libvirt >> 8446 ? S 0:00 /usr/sbin/dnsmasq >> --conf-file=/var/lib/libvirt/dnsmasq/default.conf >> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev >> --console 19 --security=none --handshake 23 --background --veth veth1 >> >> >> ostara ~ # /etc/init.d/libvirtd restart >> * Stopping libvirtd ... >> * start-stop-daemon: no matching processes >> found >> [ ok ] >> * Starting libvirtd >> ... >> [ ok ] >> >> >> ostara ~ # ps axfw | grep libvirt >> 10130 pts/2 S+ 0:00 \_ grep --colour=auto >> libvirt >> 8446 ? S 0:00 /usr/sbin/dnsmasq >> --conf-file=/var/lib/libvirt/dnsmasq/default.conf >> 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev >> --console 19 --security=none --handshake 23 --background --veth veth1 >> 10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen >> >> >> ostara ~ # virsh -c lxc:/// list --all >> Id Name State >> ---------------------------------------------------- >> 9441 dwj-lnx-dev running >> - dwj-hfax-dev shut off >> - vm1 shut off >> >> >> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev >> <domain type='lxc'> >> <name>dwj-hfax-dev</name> >> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> >> <memory unit='KiB'>4194304</memory> >> <currentMemory unit='KiB'>4194304</currentMemory> >> <vcpu placement='static'>4</vcpu> >> <resource> >> <partition>/machine</partition> >> </resource> >> <os> >> <type arch='x86_64'>exe</type> >> <init>/sbin/init</init> >> </os> >> <clock offset='utc'/> >> <on_poweroff>destroy</on_poweroff> >> <on_reboot>restart</on_reboot> >> <on_crash>destroy</on_crash> >> <devices> >> <emulator>/usr/libexec/libvirt_lxc</emulator> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/vm/lxc/dwj-hfax-dev'/> >> <target dir='/'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/usr/portage'/> >> <target dir='/usr/portage'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/usr/src'/> >> <target dir='/usr/src'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/home'/> >> <target dir='/home'/> >> </filesystem> >> <interface type='bridge'> >> <mac address='82:00:00:00:01:01'/> >> <source bridge='br0'/> >> <target dev='veth0'/> >> </interface> >> <console type='pty'> >> <target type='lxc' port='0'/> >> </console> >> </devices> >> <seclabel type='none'/> >> </domain> >> >> >> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev >> <domain type='lxc' id='9441'> >> <name>dwj-lnx-dev</name> >> <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> >> <memory unit='KiB'>500000</memory> >> <currentMemory unit='KiB'>500000</currentMemory> >> <vcpu placement='static'>2</vcpu> >> <resource> >> <partition>/machine</partition> >> </resource> >> <os> >> <type arch='x86_64'>exe</type> >> <init>/sbin/init</init> >> </os> >> <clock offset='utc'/> >> <on_poweroff>destroy</on_poweroff> >> <on_reboot>restart</on_reboot> >> <on_crash>destroy</on_crash> >> <devices> >> <emulator>/usr/libexec/libvirt_lxc</emulator> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/vm/lxc/dwj-lnx-dev'/> >> <target dir='/'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/usr/portage'/> >> <target dir='/usr/portage'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/usr/src'/> >> <target dir='/usr/src'/> >> </filesystem> >> <filesystem type='mount' accessmode='passthrough'> >> <source dir='/home'/> >> <target dir='/home'/> >> </filesystem> >> <interface type='bridge'> >> <mac address='82:00:00:00:01:00'/> >> <source bridge='br0'/> >> <target dev='veth0'/> >> </interface> >> <console type='pty' tty='/dev/pts/3'> >> <source path='/dev/pts/3'/> >> <target type='lxc' port='0'/> >> <alias name='console0'/> >> </console> >> </devices> >> <seclabel type='none'/> >> </domain> >> >> >> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml >> >> >> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml >> >> >> ostara ~ # diff lnx.xml hfax.xml >> 1,6c1,6 >> < <domain type='lxc' id='9441'> >> < <name>dwj-lnx-dev</name> >> < <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> >> < <memory unit='KiB'>500000</memory> >> < <currentMemory unit='KiB'>500000</currentMemory> >> < <vcpu placement='static'>2</vcpu> >> --- >> > <domain type='lxc'> >> > <name>dwj-hfax-dev</name> >> > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> >> > <memory unit='KiB'>4194304</memory> >> > <currentMemory unit='KiB'>4194304</currentMemory> >> > <vcpu placement='static'>4</vcpu> >> 21c21 >> < <source dir='/vm/lxc/dwj-lnx-dev'/> >> --- >> > <source dir='/vm/lxc/dwj-hfax-dev'/> >> 37c37 >> < <mac address='82:00:00:00:01:00'/> >> --- >> > <mac address='82:00:00:00:01:01'/> >> 41,42c41 >> < <console type='pty' tty='/dev/pts/3'> >> < <source path='/dev/pts/3'/> >> --- >> > <console type='pty'> >> 44d42 >> < <alias name='console0'/> >> >> >> (After reseting everything, and attempting to boot hfax with dev offline, >> libvirtd still crashes) >> >> ostara ~ # virsh -c lxc:/// list --all >> Id Name State >> ---------------------------------------------------- >> - dwj-hfax-dev shut off >> - dwj-lnx-dev shut off >> - vm1 shut off >> >> >> >> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev >> error: Failed to start domain dwj-hfax-dev >> error: End of file while reading data: Input/output error >> error: One or more references were leaked after disconnect from the >> hypervisor >> error: Failed to reconnect to the hypervisor >> >> >
Apparently Analagous Threads
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers
- libvirt-1.2.2-r1 (Gentoo) fails to start LXC containers
- Re: libvirt-1.2.2-r1 (Gentoo) fails to start LXC containers
- Unable to start LXC on Gentoo w/ libvirt 1.0.4 or 1.0.5. 1.0.3 works