libvirt users - Apr 2015 - Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.

On 04/20/2015 05:42 PM, Laine Stump wrote:
> Well, this is when the qemu process is killed (I'm pretty clever, eh?
:-)
>
> I have 3 questions:
>
> 1) what version of libvirt are you running and on what distro?
>
> 2) can you reproduce this reliably?
>
> 3) If the answer to 2 is "yes", do you have the libvirt-debuginfo
> package installed, and can you stop libvirtd, then run it under gdb with
> a breakpoint at qemuProcessKill? Once you hit that breakpoint, you could
> get a backtrace with the "bt" command and that might give us some
useful
> info about where it is coming from.
>
> Oh, by the way, are these perhaps transient domains started with
> virDomainCreate*? Or are they standard persistent domains defined with
> virDomainDefineXML?
>
> (Also, I'll just mention that qemuProcessKill() is called from
> qemuProcessStop(), and I notice that is called in a couple places
> related to snapshot create and revert which you mentioned in the
> original message. Since this isn't normal behavior, and the snapshot
> stuff is under active development, I wonder if this is related...)
Laine,

First, thanks for your help, I appreciate it.  Your questions:

1.  # libvirtd -V
libvirtd (libvirt) 1.2.14

2.  I can reproduce this every time.

3.  I compile it myself, so I'll work on getting a libvirt-debug build
going.

So this is compiled on Slackware64-14.1 as I prefer a more bsd linux and
like the simplicity of a traditional init, as well as the ability to
cook this down into a very simple usb based distro not unlike slax.

Earlier I posted a bug
(https://bugzilla.redhat.com/show_bug.cgi?id=1210903) and got a reply
fro Shanzhi that said that he was unable to reproduce using
libvirt-1.2.14-1.el7.x86_64 and qemu-kvm-rhev-2.2.0-8.el7.x86_64, but I
don't see his comment on the ticket anymore, so I wonder if he was able
to reproduce and removed his comment or perhaps his comment got
removed.  If it does indeed work on the Redhat packages then that seems
to indicate that there is something that doesn't work when libvirt is
built on non-redhat systems which may be related.

Let me do some more digging and I'll back back to you tomorrow.

Thanks!
schu

----- Original Message -----
> From: "Matthew Schumacher" <matt.s@aptalaska.com>
> To: "Laine Stump" <laine@laine.org>,
libvirt-users@redhat.com
> Sent: Wednesday, April 22, 2015 5:53:34 AM
> Subject: Re: [libvirt-users] QemuDomainObjEndJob called when libvirtd is
> started and libvirt insists qemu is using the wrong disk source.
> On 04/20/2015 05:42 PM, Laine Stump wrote:
> > Well, this is when the qemu process is killed (I'm pretty clever,
eh? :-)
> >
> > I have 3 questions:
> >
> > 1) what version of libvirt are you running and on what distro?
> >
> > 2) can you reproduce this reliably?
> >
> > 3) If the answer to 2 is "yes", do you have the
libvirt-debuginfo
> > package installed, and can you stop libvirtd, then run it under gdb
with
> > a breakpoint at qemuProcessKill? Once you hit that breakpoint, you
could
> > get a backtrace with the "bt" command and that might give us
some useful
> > info about where it is coming from.
> >
> > Oh, by the way, are these perhaps transient domains started with
> > virDomainCreate*? Or are they standard persistent domains defined with
> > virDomainDefineXML?
> >
> > (Also, I'll just mention that qemuProcessKill() is called from
> > qemuProcessStop(), and I notice that is called in a couple places
> > related to snapshot create and revert which you mentioned in the
> > original message. Since this isn't normal behavior, and the
snapshot
> > stuff is under active development, I wonder if this is related...)
> Laine,
> First, thanks for your help, I appreciate it. Your questions:
> 1. # libvirtd -V
> libvirtd (libvirt) 1.2.14
> 2. I can reproduce this every time.
> 3. I compile it myself, so I'll work on getting a libvirt-debug build
> going.
> So this is compiled on Slackware64-14.1 as I prefer a more bsd linux and
> like the simplicity of a traditional init, as well as the ability to
> cook this down into a very simple usb based distro not unlike slax.
> Earlier I posted a bug
> (https://bugzilla.redhat.com/show_bug.cgi?id=1210903) and got a reply
> fro Shanzhi that said that he was unable to reproduce using
> libvirt-1.2.14-1.el7.x86_64 and qemu-kvm-rhev-2.2.0-8.el7.x86_64, but I
> don't see his comment on the ticket anymore, so I wonder if he was able
> to reproduce and removed his comment or perhaps his comment got
> removed. If it does indeed work on the Redhat packages then that seems
> to indicate that there is something that doesn't work when libvirt is
> built on non-redhat systems which may be related.
I can reply this question, :) 
I didn't delete the comment, and you're unable delete your comment in
bugzilla.
I just set it to Private as I mention some downstream info, so,.. I do can't
reproduce it even
with libvirt build from latest libvirt.git on my Fedora21. 
For your second problem in that bug, you can refer to bug 1197592 
> Let me do some more digging and I'll back back to you tomorrow.
> Thanks!
> schu
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
-- 
Regards 
shyu

On 04/21/2015 05:53 PM, Matthew Schumacher wrote:
> On 04/20/2015 05:42 PM, Laine Stump wrote:
>> Well, this is when the qemu process is killed (I'm pretty clever,
eh? :-)
>>
>> I have 3 questions:
>>
>> 1) what version of libvirt are you running and on what distro?
>>
>> 2) can you reproduce this reliably?
>>
>> 3) If the answer to 2 is "yes", do you have the
libvirt-debuginfo
>> package installed, and can you stop libvirtd, then run it under gdb
with
>> a breakpoint at qemuProcessKill? Once you hit that breakpoint, you
could
>> get a backtrace with the "bt" command and that might give us
some useful
>> info about where it is coming from.
>>
>> Oh, by the way, are these perhaps transient domains started with
>> virDomainCreate*? Or are they standard persistent domains defined with
>> virDomainDefineXML?
>>
>> (Also, I'll just mention that qemuProcessKill() is called from
>> qemuProcessStop(), and I notice that is called in a couple places
>> related to snapshot create and revert which you mentioned in the
>> original message. Since this isn't normal behavior, and the
snapshot
>> stuff is under active development, I wonder if this is related...)
> Laine,
>
> First, thanks for your help, I appreciate it.  Your questions:
>
> 1.  # libvirtd -V
> libvirtd (libvirt) 1.2.14
>
> 2.  I can reproduce this every time.
>
> 3.  I compile it myself, so I'll work on getting a libvirt-debug build
> going.

If you're compiling yourself, then you should be all set to run under
gdb. libvirt-debuginfo is just a separate subpackage that contains all
the symbol and line number info from the build so that backtraces in gdb
make sense. Try attaching gdb to the libvirtd process and do something
like "thread apply all bt" - if you see function/argument/file names
and
line numbers, then you already have the symbols available.
>
> So this is compiled on Slackware64-14.1 as I prefer a more bsd linux and
> like the simplicity of a traditional init, as well as the ability to
> cook this down into a very simple usb based distro not unlike slax.
Occasionally there are cases where libvirtd terminates guests on restart
due to some failure to parse the guest's status XML, inability to
connect to the guest's qemu monitor, or some other thing that would
render the guest inaccessible. We of course want to eliminate as many of
those situations as possible, so we would love to see the backtrace you
can get from gdb.

On 04/22/2015 08:19 AM, Laine Stump wrote:
> If you're compiling yourself, then you should be all set to run under
> gdb. libvirt-debuginfo is just a separate subpackage that contains all
> the symbol and line number info from the build so that backtraces in
> gdb make sense. Try attaching gdb to the libvirtd process and do
> something like "thread apply all bt" - if you see
> function/argument/file names and line numbers, then you already have
> the symbols available.
>> So this is compiled on Slackware64-14.1 as I prefer a more bsd linux
and
>> like the simplicity of a traditional init, as well as the ability to
>> cook this down into a very simple usb based distro not unlike slax.
> Occasionally there are cases where libvirtd terminates guests on restart
> due to some failure to parse the guest's status XML, inability to
> connect to the guest's qemu monitor, or some other thing that would
> render the guest inaccessible. We of course want to eliminate as many of
> those situations as possible, so we would love to see the backtrace you
> can get from gdb.
>
>
Laine,

My apologies on the delay getting back to you, I've been pretty slammed
lately.  Anyway I have the bracktrace you requested.

I simply started libvirtd, started my vm, killed libvirtd, then started
it again with gdb in foreground mode with the breakpoint set.

I got a ton of debug info that I posted here:

http://www.aptalaska.net/~matt.s/libvirt.txt

but the backtrace is simply:

Breakpoint 1, 0x00007fffc72a0240 in qemuProcessKill () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
(gdb) bt
#0  0x00007fffc72a0240 in qemuProcessKill () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#1  0x00007fffc72a19cd in qemuProcessStop () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#2  0x00007fffc72a27d6 in ?? () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#3  0x00007ffff74b6882 in ?? () from /usr/lib64/libvirt.so.0
#4  0x00007ffff5722ce2 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff31d38cd in clone () from /lib64/libc.so.6
(gdb)


Thanks!
schu