libvirt users - Jul 2013 - Re: libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers

On Mon, Jul 15, 2013 at 3:18 AM, Michal Privoznik
<mprivozn@redhat.com>wrote:
>
> Interesting. If you are still able to reproduce the crash, can you try to
> get the line number within virSecurityManagerGenLabel where the crash
> happened? I think it's the STREQ line (440 linenr). Question is whether
> model or name is NULL.
>
>
I'll try.

I'm not sure why GDB failed to list line numbers in the backtrace.  I will
recompile libvirt with "-O0 -g3" and try again.

I'm running libvirt on my Gentoo development server, built from portage.
Instead of tinkering with portage and rebuilding libvirt, I thought that I
would just try the latest pull from git.  "./configure" fails, unable
to
find an input file.  I'll try again, using the same source tarball as
listed in Gentoo's ebuild.

ostara libvirt # CFLAGS="-O0 -g3" ./configure --with-lxc

config.status: creating libvirt.pc
config.status: creating libvirt.spec
config.status: error: cannot find input file: `mingw32-libvirt.spec.in'

Ah ha!  I just learned about "gdb bt full".  The existing core dump
might
have what you need: Line #442.  However, the line numbers for the source
code in the source tree that my Gentoo system is building from does not
match exactly what you listed.

Line #442 for me is the one containing the "STREQ" macro:

    virObjectLock(mgr);
    for (i = 0; i < vm->nseclabels; i++) {
        for (j = 0; sec_managers[j]; j++)
            if (STREQ(vm->seclabels[i]->model,
sec_managers[j]->drv->name))
                break;


I can rebuild with "-O0" and try again.  If I can still trigger the
crash,
the backtrace might have useful values for the optimized variables.  I'll
post again in a few minutes.



(gdb) bt full
#0  0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fe47578ad31 in virSecurityManagerGenLabel (mgr=0x7fe4640acfa0,
vm=0x7fe4640c5e40) at security/security_manager.c:442
        ret = -1
        i = <optimized out>
        j = <optimized out>
        sec_managers = 0x7fe458001880
        seclabel = <optimized out>
        generated = false
        __FUNCTION__ = "virSecurityManagerGenLabel"
        __func__ = "virSecurityManagerGenLabel"
#2  0x00007fe46aa92979 in virLXCProcessStart (conn=0x7fe460000a80,
driver=0x7fe4640bd340, vm=0x7fe4640c1610, autoDestroy=false,
    reason=VIR_DOMAIN_RUNNING_BOOTED) at lxc/lxc_process.c:1144
        rc = -1
        r = <optimized out>
        nttyFDs = 1
        ttyFDs = 0x7fe458001790
        i = <optimized out>
        logfile = 0x7fe458000ad0
"/var/log/libvirt/lxc/dwj-hfax-dev.log"
        logfd = -1
        nveths = 0
        veths = 0x0
        handshakefds = {-1, -1}
        pos = -1
        ebuf
"\000\000\000\000\344\177\000\000\020\000\000\000\000\000\000\000\376\377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\377\377\377\377",
'\000' <repeats 12 times>"\364,
\377\377\377\377\377\377\377\220Y|u\344\177\000\000\034\313\376t\344\177\000\000\000\000\000\000\020\000\000\000\217\b~u\344\177\000\000\000\000\000\000\000\000\000\000\250\246\327o\344\177\000\000t\b~u\344\177\000\000\000\000\000\000\000\000\000\000\020",
'\000' <repeats 15 times>, "P\251\327o", '\000'
<repeats 12 times>,
"\006\247\327o\344\177\000\000\034\313\376t\344\177\000\000\000\000\000\000\344\177\000\000C\342{u\344\177\000\000x\247\327o\344\177\000\000\b\247\327o2739\000\342{u\344\177\000\000\000\000\000\000\000\000\000\000x",
'\000' <repeats 15 times>"\247,
Z\016v\000\000\000\000(\000\000\000\060\000\000\000\200\246\327o\344\177\000\000\300\245\327o\344\177\000\000\n",
'\000' <repeats 31 times>"\227"...
        timestamp = <optimized out>
        cmd = 0x0
        priv = 0x7fe464022500
        err = 0x0
        __FUNCTION__ = "virLXCProcessStart"
        __func__ = "virLXCProcessStart"
---Type <return> to continue, or q <return> to quit---
#3  0x00007fe46aa9736e in lxcDomainCreateWithFlags (dom=0x7fe4580008c0,
flags=<optimized out>) at lxc/lxc_driver.c:1054
        driver = 0x7fe4640bd340
        vm = 0x7fe4640c1610
        event = 0x0
        ret = -1
        __FUNCTION__ = "lxcDomainCreateWithFlags"



On Mon, Jul 15, 2013 at 7:06 AM, Dennis Jenkins
<dennis.jenkins.75@gmail.com
> wrote:
>
> On Mon, Jul 15, 2013 at 3:18 AM, Michal Privoznik
<mprivozn@redhat.com>wrote:
>
>>
>> Interesting. If you are still able to reproduce the crash, can you try
to
>> get the line number within virSecurityManagerGenLabel where the crash
>> happened? I think it's the STREQ line (440 linenr). Question is
whether
>> model or name is NULL.
>>
>>
>
> I'll try.
>
> I'm not sure why GDB failed to list line numbers in the backtrace.  I
will
> recompile libvirt with "-O0 -g3" and try again.
>
> I'm running libvirt on my Gentoo development server, built from
portage.
> Instead of tinkering with portage and rebuilding libvirt, I thought that I
> would just try the latest pull from git.  "./configure" fails,
unable to
> find an input file.  I'll try again, using the same source tarball as
> listed in Gentoo's ebuild.
>
> ostara libvirt # CFLAGS="-O0 -g3" ./configure --with-lxc
>
> config.status: creating libvirt.pc
> config.status: creating libvirt.spec
> config.status: error: cannot find input file: `mingw32-libvirt.spec.in'
>
>
>

On Mon, Jul 15, 2013 at 7:37 AM, Dennis Jenkins
<dennis.jenkins.75@gmail.com
> wrote:
> Ah ha!  I just learned about "gdb bt full".  The existing core
dump might
> have what you need: Line #442.  However, the line numbers for the source
> code in the source tree that my Gentoo system is building from does not
> match exactly what you listed.
>
> Line #442 for me is the one containing the "STREQ" macro:
>
>     virObjectLock(mgr);
>
>     for (i = 0; i < vm->nseclabels; i++) {
>         for (j = 0; sec_managers[j]; j++)
>             if (STREQ(vm->seclabels[i]->model,
sec_managers[j]->drv->name))
>                 break;
>
>
> I can rebuild with "-O0" and try again.  If I can still trigger
the crash,
> the backtrace might have useful values for the optimized variables. 
I'll
> post again in a few minutes.
>
> On Mon, Jul 15, 2013 at 7:06 AM, Dennis Jenkins <
> dennis.jenkins.75@gmail.com> wrote:
>
>>
>> On Mon, Jul 15, 2013 at 3:18 AM, Michal Privoznik
<mprivozn@redhat.com>wrote:
>>
>>>
>>> Interesting. If you are still able to reproduce the crash, can you
try
>>> to get the line number within virSecurityManagerGenLabel where the
crash
>>> happened? I think it's the STREQ line (440 linenr). Question is
whether
>>> model or name is NULL.
>>>
>>>
>>
>> I'll try.
>>
>> I'm not sure why GDB failed to list line numbers in the backtrace. 
I
>> will recompile libvirt with "-O0 -g3" and try again.
>>
>>
TL;DR: "vm->seclabels[i]->model" is NULL.


(gdb) bt full
#0  0x00007ffff6fc5d76 in __strcmp_sse42 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff76e2ae2 in virSecurityManagerGenLabel (mgr=0x7fffe40bef20,
vm=0x7fffd00097a0)
    at security/security_manager.c:442
        ret = -1
        i = 0
        j = 0
        sec_managers = 0x7fffd8003e60
        seclabel = 0x7ffff74c4911 <virAllocN+54>
        generated = false
        __FUNCTION__ = "virSecurityManagerGenLabel"
        __func__ = "virSecurityManagerGenLabel"



(gdb) list 436,450
436         if ((sec_managers = virSecurityManagerGetNested(mgr)) == NULL)
437             return ret;
438
439         virObjectLock(mgr);
440         for (i = 0; i < vm->nseclabels; i++) {
441             for (j = 0; sec_managers[j]; j++)
442                 if (STREQ(vm->seclabels[i]->model,
sec_managers[j]->drv->name))
443                     break;
444
445             if (!sec_managers[j]) {
446                 virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
447                                _("Unable to find security driver for
label %s"),
448                                vm->seclabels[i]->model);
449                 goto cleanup;
450             }

(gdb) frame 1
#1  0x00007ffff76e2ae2 in virSecurityManagerGenLabel (mgr=0x7fffe40bef20,
vm=0x7fffd00097a0)
    at security/security_manager.c:442
442                 if (STREQ(vm->seclabels[i]->model,
sec_managers[j]->drv->name))

(gdb) print vm->nseclabels
$1 = 1

(gdb) print sec_managers
$2 = (virSecurityManagerPtr *) 0x7fffd8003e60

(gdb) print sec_managers[0]
$3 = (virSecurityManagerPtr) 0x7fffe40bef20

(gdb) print vm->seclabels[i]->model
$4 = 0x0

(gdb) print sec_managers[j]->drv->name
$5 = 0x7ffff77729c0 "none"

On 15.07.2013 14:37, Dennis Jenkins wrote:
> Ah ha!  I just learned about "gdb bt full".  The existing core
dump
> might have what you need: Line #442.  However, the line numbers for the
> source code in the source tree that my Gentoo system is building from
> does not match exactly what you listed.
> 
> Line #442 for me is the one containing the "STREQ" macro:
> 
>     virObjectLock(mgr);
>     for (i = 0; i < vm->nseclabels; i++) {
>         for (j = 0; sec_managers[j]; j++)
>             if (STREQ(vm->seclabels[i]->model,
sec_managers[j]->drv->name))
>                 break;
> 
> 
> I can rebuild with "-O0" and try again.  If I can still trigger
the
> crash, the backtrace might have useful values for the optimized
> variables.  I'll post again in a few minutes.
> 
> 
So I've managed to reproduce this and found the root cause for this
problem. Will post patch shortly.

Michal