libvirt users - Jul 2013 - Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers

Update:  I am able to edit the XML in "dwj-hfax-dev" such that
libvirtd no
longer crashes, and edit the XML for "dwj-lnx-dev" such that it will
crash.

The presents of "<seclabel type='none'/>" near the
bottom causes libvirtd
to crash.

I do not recall ever manually adding that to my domain.

In any event, libvirtd should probably not crash due to the XML element
(which seems valid - or at least "virsh edit" allows it).



On Fri, Jul 12, 2013 at 11:44 AM, Dennis Jenkins <
dennis.jenkins.75@gmail.com> wrote:
> The debug log ends with this:
>
>
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:708 : Make
> group /machine/dwj-hfax-dev.libvirt-lxc
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpu/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpuacct/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/cpuset/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/memory/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/devices/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/freezer/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/blkio/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:715 :
> Skipping unmounted controller net_cls
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make
> controller /sys/fs/cgroup/perf_event/machine/dwj-hfax-dev.libvirt-lxc/
> 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:779 : Done
> making controllers for group
> 2013-07-12 16:43:31.740+0000: 21365: debug : virFileMakePathHelper:1995 :
> path=/var/log/libvirt/lxc mode=0777
> 2013-07-12 16:43:31.740+0000: 21365: debug : virLXCProcessStart:1096 :
> Setting current domain def as transient
> 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1121 :
> Preparing host devices
> 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1139 :
> Generating domain security label (if required)
>
>
>      ====== end of log ====>
> Segmentation fault (core dumped)
>
>
>
> (gdb) bt
> #0  0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6
> #1  0x00007fe47578ad31 in virSecurityManagerGenLabel () from
> /usr/lib64/libvirt.so.0
> #2  0x00007fe46aa92979 in virLXCProcessStart () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
> #3  0x00007fe46aa9736e in lxcDomainCreateWithFlags () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
> #4  0x00007fe47569c067 in virDomainCreate () from /usr/lib64/libvirt.so.0
> #5  0x00007fe4760d5578 in remoteDispatchDomainCreateHelper ()
> #6  0x00007fe4756fcd78 in virNetServerProgramDispatch () from
> /usr/lib64/libvirt.so.0
> #7  0x00007fe4756f7302 in virNetServerProcessMsg () from
> /usr/lib64/libvirt.so.0
> #8  0x00007fe4756f7a93 in virNetServerHandleJob () from
> /usr/lib64/libvirt.so.0
> #9  0x00007fe47560f95e in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #10 0x00007fe47560efc6 in virThreadHelper () from /usr/lib64/libvirt.so.0
> #11 0x00007fe475354da6 in start_thread () from /lib64/libpthread.so.0
> #12 0x00007fe47508d99d in clone () from /lib64/libc.so.6
> (gdb)
>
>
>
> On Fri, Jul 12, 2013 at 11:40 AM, Dennis Jenkins <
> dennis.jenkins.75@gmail.com> wrote:
>
>> Hello all,
>>
>>     I have two issues:
>>
>> 1) I am unable to start a seemingly correct LXC domain (I cloned it
from
>> a working domain).
>>
>> 2) I am able to crash "libvirtd" by attempting to start the
cloned
>> domain, but starting the original works just fine.
>>
>>     I humbly submit that item #2 is a bug - the "libvirtd"
daemon should
>> never crash due to anything the "libvirt" client throws at
it.  As for
>> item  #1, I'm not sure where I went wrong.  A full walk-through is
below
>> (ending with a DIFF of the XML from the two domains).
>>
>>     I created by original domain ("dwj-lnx-dev") a long time
ago.  Today
>> I created the new domain ("dwj-hfax-dev") as follows:
>>
>> 1) Shutdown "dwj-lnx-dev"
>> 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev
>> dwj-hfax-dev"  (2.5GB, ~5 min)
>> 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml"
>> 4) ${EDITOR} a.xml
>>   a) changed MAC address, name, memory, source directory for
"/"
>> 5) "libvirt -c lxc:/// define a.xml"
>> 6) Edit "/etc/bind/pri/*" and
"/etc/dhcp/dhcpd.conf" on my host.
>>
>>
>>     It does not matter is "dwj-lnx-dev" is running or not. 
Any attempt
>> to start "dwj-hfax-dev" will crash libvirtd.
>>
>>
>>     In the past I was asked to turn on some debugging and capture a
>> detailed log (
>> https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html).
>> I will do this soon and post my results as a follow up.
>>
>>
>>
>> ostara ~ # uname -a
>> Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013
>> x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
>>
>>
>> ostara ~ # equery l libvirt
>>  * Searching for libvirt ...
>> [IP-] [  ] app-emulation/libvirt-1.1.0-r1:0
>>
>>
>> ostara ~ # virsh -c lxc:/// version
>> Compiled against library: libvirt 1.1.0
>> Using library: libvirt 1.1.0
>> Using API: LXC 1.1.0
>> Running hypervisor: LXC 3.8.13
>>
>>
>> ostara ~ # /etc/init.d/libvirtd restart
>>  * Caching service dependencies
>> ...
>> [ ok ]
>>  * Stopping libvirtd ...
>>  *  Shutting down network(s):
>>  *
>> default
>> [ ok ]
>>  * Starting libvirtd
>> ...
>> [ ok ]
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>>  Id    Name                           State
>> ----------------------------------------------------
>>  -     dwj-hfax-dev                   shut off
>>  -     dwj-lnx-dev                    shut off
>>  -     vm1                            shut off
>>
>> ostara ~ # virsh -c lxc:/// start dwj-lnx-dev
>> Domain dwj-lnx-dev started
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>>  Id    Name                           State
>> ----------------------------------------------------
>>  9441  dwj-lnx-dev                    running
>>  -     dwj-hfax-dev                   shut off
>>  -     vm1                            shut off
>>
>>
>> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
>> error: Failed to start domain dwj-hfax-dev
>> error: End of file while reading data: Input/output error
>> error: One or more references were leaked after disconnect from the
>> hypervisor
>> error: Failed to reconnect to the hypervisor
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>> error: failed to connect to the hypervisor
>> error: no valid connection
>> error: Failed to connect socket to
'/var/run/libvirt/libvirt-sock':
>> Connection refused
>>
>>
>> ostara ~ # ls -l /var/run/libvirt/libvirt-sock
>> srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock
>>
>>
>> ostara ~ # ps axfw | grep libvirt
>>  9997 pts/2    S+     0:00                      \_ grep --colour=auto
>> libvirt
>>  8446 ?        S      0:00 /usr/sbin/dnsmasq
>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
>>  9441 ?        Ss     0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
>> --console 19 --security=none --handshake 23 --background --veth veth1
>>
>>
>> ostara ~ # /etc/init.d/libvirtd restart
>>  * Stopping libvirtd ...
>>  * start-stop-daemon: no matching processes
>> found
>> [ ok ]
>>  * Starting libvirtd
>> ...
>> [ ok ]
>>
>>
>> ostara ~ # ps axfw | grep libvirt
>> 10130 pts/2    S+     0:00                      \_ grep --colour=auto
>> libvirt
>>  8446 ?        S      0:00 /usr/sbin/dnsmasq
>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
>>  9441 ?        Ss     0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev
>> --console 19 --security=none --handshake 23 --background --veth veth1
>> 10033 ?        Sl     0:00 /usr/sbin/libvirtd -d --listen
>>
>>
>> ostara ~ # virsh -c lxc:/// list --all
>>  Id    Name                           State
>> ----------------------------------------------------
>>  9441  dwj-lnx-dev                    running
>>  -     dwj-hfax-dev                   shut off
>>  -     vm1                            shut off
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
>> <domain type='lxc'>
>>   <name>dwj-hfax-dev</name>
>>   <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
>>   <memory unit='KiB'>4194304</memory>
>>   <currentMemory unit='KiB'>4194304</currentMemory>
>>   <vcpu placement='static'>4</vcpu>
>>   <resource>
>>     <partition>/machine</partition>
>>   </resource>
>>   <os>
>>     <type arch='x86_64'>exe</type>
>>     <init>/sbin/init</init>
>>   </os>
>>   <clock offset='utc'/>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>destroy</on_crash>
>>   <devices>
>>     <emulator>/usr/libexec/libvirt_lxc</emulator>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/vm/lxc/dwj-hfax-dev'/>
>>       <target dir='/'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/usr/portage'/>
>>       <target dir='/usr/portage'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/usr/src'/>
>>       <target dir='/usr/src'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/home'/>
>>       <target dir='/home'/>
>>     </filesystem>
>>     <interface type='bridge'>
>>       <mac address='82:00:00:00:01:01'/>
>>       <source bridge='br0'/>
>>       <target dev='veth0'/>
>>     </interface>
>>     <console type='pty'>
>>       <target type='lxc' port='0'/>
>>     </console>
>>   </devices>
>>   <seclabel type='none'/>
>> </domain>
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev
>> <domain type='lxc' id='9441'>
>>   <name>dwj-lnx-dev</name>
>>   <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
>>   <memory unit='KiB'>500000</memory>
>>   <currentMemory unit='KiB'>500000</currentMemory>
>>   <vcpu placement='static'>2</vcpu>
>>   <resource>
>>     <partition>/machine</partition>
>>   </resource>
>>   <os>
>>     <type arch='x86_64'>exe</type>
>>     <init>/sbin/init</init>
>>   </os>
>>   <clock offset='utc'/>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>destroy</on_crash>
>>   <devices>
>>     <emulator>/usr/libexec/libvirt_lxc</emulator>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/vm/lxc/dwj-lnx-dev'/>
>>       <target dir='/'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/usr/portage'/>
>>       <target dir='/usr/portage'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/usr/src'/>
>>       <target dir='/usr/src'/>
>>     </filesystem>
>>     <filesystem type='mount'
accessmode='passthrough'>
>>       <source dir='/home'/>
>>       <target dir='/home'/>
>>     </filesystem>
>>     <interface type='bridge'>
>>       <mac address='82:00:00:00:01:00'/>
>>       <source bridge='br0'/>
>>       <target dev='veth0'/>
>>     </interface>
>>     <console type='pty' tty='/dev/pts/3'>
>>       <source path='/dev/pts/3'/>
>>       <target type='lxc' port='0'/>
>>       <alias name='console0'/>
>>     </console>
>>   </devices>
>>   <seclabel type='none'/>
>> </domain>
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml
>>
>>
>> ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml
>>
>>
>> ostara ~ # diff lnx.xml hfax.xml
>> 1,6c1,6
>> < <domain type='lxc' id='9441'>
>> <   <name>dwj-lnx-dev</name>
>> <   <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid>
>> <   <memory unit='KiB'>500000</memory>
>> <   <currentMemory
unit='KiB'>500000</currentMemory>
>> <   <vcpu placement='static'>2</vcpu>
>> ---
>> > <domain type='lxc'>
>> >   <name>dwj-hfax-dev</name>
>> >   <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
>> >   <memory unit='KiB'>4194304</memory>
>> >   <currentMemory
unit='KiB'>4194304</currentMemory>
>> >   <vcpu placement='static'>4</vcpu>
>> 21c21
>> <       <source dir='/vm/lxc/dwj-lnx-dev'/>
>> ---
>> >       <source dir='/vm/lxc/dwj-hfax-dev'/>
>> 37c37
>> <       <mac address='82:00:00:00:01:00'/>
>> ---
>> >       <mac address='82:00:00:00:01:01'/>
>> 41,42c41
>> <     <console type='pty' tty='/dev/pts/3'>
>> <       <source path='/dev/pts/3'/>
>> ---
>> >     <console type='pty'>
>> 44d42
>> <       <alias name='console0'/>
>>
>>
>> (After reseting everything, and attempting to boot hfax with dev
offline,
>> libvirtd still crashes)
>>
>> ostara ~ # virsh -c lxc:/// list --all
>>  Id    Name                           State
>> ----------------------------------------------------
>>  -     dwj-hfax-dev                   shut off
>>  -     dwj-lnx-dev                    shut off
>>  -     vm1                            shut off
>>
>>
>>
>> ostara ~ # virsh -c lxc:/// start dwj-hfax-dev
>> error: Failed to start domain dwj-hfax-dev
>> error: End of file while reading data: Input/output error
>> error: One or more references were leaked after disconnect from the
>> hypervisor
>> error: Failed to reconnect to the hypervisor
>>
>>
>

On 12.07.2013 20:32, Dennis Jenkins wrote:
> Update:  I am able to edit the XML in "dwj-hfax-dev" such that
libvirtd
> no longer crashes, and edit the XML for "dwj-lnx-dev" such that
it will
> crash.
> 
> The presents of "<seclabel type='none'/>" near the
bottom causes
> libvirtd to crash.
> 
> I do not recall ever manually adding that to my domain.
> 
> In any event, libvirtd should probably not crash due to the XML element
> (which seems valid - or at least "virsh edit" allows it).
> 
Interesting. If you are still able to reproduce the crash, can you try to get
the line number within virSecurityManagerGenLabel where the crash happened? I
think it's the STREQ line (440 linenr). Question is whether model or name is
NULL.


src/security/security_manager.c-438-    for (i = 0; i < vm->nseclabels;
i++) {
src/security/security_manager.c-439-        for (j = 0; sec_managers[j]; j++)
src/security/security_manager.c-440-            if
(STREQ(vm->seclabels[i]->model, sec_managers[j]->drv->name))
src/security/security_manager.c-441-                break;
src/security/security_manager.c-442-
src/security/security_manager.c-443-        if (!sec_managers[j]) {
src/security/security_manager.c-444-           
virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
src/security/security_manager.c-445-                           _("Unable to
find security driver for label %s"),
src/security/security_manager.c-446-                          
vm->seclabels[i]->model);
src/security/security_manager.c-447-            goto cleanup;
src/security/security_manager.c-448-        }
src/security/security_manager.c-449-    }


Michal

On Mon, Jul 15, 2013 at 3:18 AM, Michal Privoznik
<mprivozn@redhat.com>wrote:
>
> Interesting. If you are still able to reproduce the crash, can you try to
> get the line number within virSecurityManagerGenLabel where the crash
> happened? I think it's the STREQ line (440 linenr). Question is whether
> model or name is NULL.
>
>
I'll try.

I'm not sure why GDB failed to list line numbers in the backtrace.  I will
recompile libvirt with "-O0 -g3" and try again.

I'm running libvirt on my Gentoo development server, built from portage.
Instead of tinkering with portage and rebuilding libvirt, I thought that I
would just try the latest pull from git.  "./configure" fails, unable
to
find an input file.  I'll try again, using the same source tarball as
listed in Gentoo's ebuild.

ostara libvirt # CFLAGS="-O0 -g3" ./configure --with-lxc

config.status: creating libvirt.pc
config.status: creating libvirt.spec
config.status: error: cannot find input file: `mingw32-libvirt.spec.in'