Matthew Schumacher
2015-Apr-28 22:03 UTC
Re: [libvirt-users] QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
On 04/22/2015 08:19 AM, Laine Stump wrote:> If you're compiling yourself, then you should be all set to run under > gdb. libvirt-debuginfo is just a separate subpackage that contains all > the symbol and line number info from the build so that backtraces in > gdb make sense. Try attaching gdb to the libvirtd process and do > something like "thread apply all bt" - if you see > function/argument/file names and line numbers, then you already have > the symbols available. >> So this is compiled on Slackware64-14.1 as I prefer a more bsd linux and >> like the simplicity of a traditional init, as well as the ability to >> cook this down into a very simple usb based distro not unlike slax. > Occasionally there are cases where libvirtd terminates guests on restart > due to some failure to parse the guest's status XML, inability to > connect to the guest's qemu monitor, or some other thing that would > render the guest inaccessible. We of course want to eliminate as many of > those situations as possible, so we would love to see the backtrace you > can get from gdb. > >Laine, My apologies on the delay getting back to you, I've been pretty slammed lately. Anyway I have the bracktrace you requested. I simply started libvirtd, started my vm, killed libvirtd, then started it again with gdb in foreground mode with the breakpoint set. I got a ton of debug info that I posted here: http://www.aptalaska.net/~matt.s/libvirt.txt but the backtrace is simply: Breakpoint 1, 0x00007fffc72a0240 in qemuProcessKill () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so (gdb) bt #0 0x00007fffc72a0240 in qemuProcessKill () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #1 0x00007fffc72a19cd in qemuProcessStop () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #2 0x00007fffc72a27d6 in ?? () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007ffff74b6882 in ?? () from /usr/lib64/libvirt.so.0 #4 0x00007ffff5722ce2 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffff31d38cd in clone () from /lib64/libc.so.6 (gdb) Thanks! schu
Laine Stump
2015-Apr-29 01:13 UTC
Re: [libvirt-users] QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
On 04/28/2015 06:03 PM, Matthew Schumacher wrote:> > On 04/22/2015 08:19 AM, Laine Stump wrote: >> If you're compiling yourself, then you should be all set to run under >> gdb. libvirt-debuginfo is just a separate subpackage that contains all >> the symbol and line number info from the build so that backtraces in >> gdb make sense. Try attaching gdb to the libvirtd process and do >> something like "thread apply all bt" - if you see >> function/argument/file names and line numbers, then you already have >> the symbols available. >>> So this is compiled on Slackware64-14.1 as I prefer a more bsd linux and >>> like the simplicity of a traditional init, as well as the ability to >>> cook this down into a very simple usb based distro not unlike slax. >> Occasionally there are cases where libvirtd terminates guests on restart >> due to some failure to parse the guest's status XML, inability to >> connect to the guest's qemu monitor, or some other thing that would >> render the guest inaccessible. We of course want to eliminate as many of >> those situations as possible, so we would love to see the backtrace you >> can get from gdb. >> >> > Laine, > > My apologies on the delay getting back to you, I've been pretty slammed > lately. Anyway I have the bracktrace you requested. > > I simply started libvirtd, started my vm, killed libvirtd, then started > it again with gdb in foreground mode with the breakpoint set. > > I got a ton of debug info that I posted here: > > http://www.aptalaska.net/~matt.s/libvirt.txt > > but the backtrace is simply: > > Breakpoint 1, 0x00007fffc72a0240 in qemuProcessKill () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > (gdb) bt > #0 0x00007fffc72a0240 in qemuProcessKill () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > #1 0x00007fffc72a19cd in qemuProcessStop () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > #2 0x00007fffc72a27d6 in ?? () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > #3 0x00007ffff74b6882 in ?? () from /usr/lib64/libvirt.so.0 > #4 0x00007ffff5722ce2 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007ffff31d38cd in clone () from /lib64/libc.so.6 > (gdb)Bah. Your debug symbols aren't enough to give a full stack trace - what I was really looking for was those things marked as ??. However, your debug output shows several things leading up tothe qemuProcessStop that might have been the reason for it being called: 1) virStorageFileBackendFileReadHeader() complains that there is no medium in /dev/sr0 (i.e. no disk in the CD drive) (this was actually in your much more abbreviated log in an earlier message, but for some reason I skipped over it :-/) 2) libvirt sends the following command to qemu: {"execute":"block-commit","arguments":{"device":"bogus"},"id":"libvirt-4"} and gets back this error: {"id": "libvirt-4", "error": {"class": "DeviceNotFound", "desc": "Device 'bogus' not found"}} Since (1) actually generates a libvirt error log, that is what I would look into first.
Matthew Schumacher
2015-Apr-29 01:21 UTC
Re: [libvirt-users] QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
On 04/28/2015 05:13 PM, Laine Stump wrote:> Bah. Your debug symbols aren't enough to give a full stack trace - what > I was really looking for was those things marked as ??. > > However, your debug output shows several things leading up tothe > qemuProcessStop that might have been the reason for it being called: > > 1) virStorageFileBackendFileReadHeader() complains that there is no > medium in /dev/sr0 (i.e. no disk in the CD drive) (this was actually in > your much more abbreviated log in an earlier message, but for some > reason I skipped over it :-/) > > 2) libvirt sends the following command to qemu: > > > {"execute":"block-commit","arguments":{"device":"bogus"},"id":"libvirt-4"} > > and gets back this error: > > {"id": "libvirt-4", > "error": {"class": "DeviceNotFound", "desc": "Device 'bogus' not found"}} > > Since (1) actually generates a libvirt error log, that is what I would > look into first. >Thanks Laine, Confirmed, if I: virsh attach-disk test '' hdc --type cdrom Then I can kick libvirtd all I want and it doesn't take out my domain. So now that that is fixed, I'm back to trying to figure out out why blockcopy --wait won't actually wait, as it's hard to migrate domains from one storage to another while needing to continually call blockjob to see if we are 100% mirroring. I posted plenty of debug here: https://bugzilla.redhat.com/attachment.cgi?id=1019911 I wonder if the issue isn't obvious to you since you know a bit more about how libvirt interacts with qemu than I do. Also, is there a link so some documentation that describes all of the json going between libvirt and qemu? Thanks for your help! schu
Reasonably Related Threads
- Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
- Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
- Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
- Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.
- Re: QemuDomainObjEndJob called when libvirtd is started and libvirt insists qemu is using the wrong disk source.