Marcel Juffermans
2020-Oct-27 23:35 UTC
Re: RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
Thanks Jim, Looking at the logs for the working and non-working setups, the command line for QEMU is identical and the qmp commands almost the same: they both do "query-chardev" and "query-vnc" and the working setup does an additional "qmp_capabilities" which is likely not relevant. So I guess it must be in QEMU - I'll head over to the bug tracker. Thanks for your help! Marcel On 28/10/20 9:26 am, Jim Fehlig wrote:> On 10/22/20 6:48 PM, Marcel Juffermans wrote: >> Hi there, >> >> Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the >> virtual guests don't get their RBD disks made available to them. On >> openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The >> XML is as follows: >> >> <domain type='xen' id='7'> >> <name>mytwotel-a</name> >> <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid> >> <description>mytwotel-a</description> >> <memory unit='KiB'>1048576</memory> >> <currentMemory unit='KiB'>1048576</currentMemory> >> <vcpu placement='static'>1</vcpu> >> <cputune> >> <vcpupin vcpu='0' cpuset='8-23'/> >> </cputune> >> <os> >> <type arch='x86_64' machine='xenpv'>linux</type> >> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel> >> </os> >> <clock offset='utc'/> >> <on_poweroff>destroy</on_poweroff> >> <on_reboot>restart</on_reboot> >> <on_crash>restart</on_crash> >> <devices> >> <disk type='network' device='disk'> >> <driver name='qemu' type='raw' cache='writethrough'/> >> <source protocol='rbd' name='guests/mytwotel-a'> >> <auth username='libvirt'> >> <secret type='ceph' >> uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/> >> </auth> >> </source> >> <backingStore/> >> <target dev='xvda' bus='xen'/> >> </disk> >> <controller type='xenbus' index='0'/> >> <interface type='bridge'> >> <mac address='00:16:3e:a3:ba:9f'/> >> <source bridge='br0'/> >> <target dev='vif7.0'/> >> </interface> >> <console type='pty' tty='/dev/pts/0'> >> <source path='/dev/pts/0'/> >> <target type='xen' port='0'/> >> </console> >> <input type='mouse' bus='xen'/> >> <input type='keyboard' bus='xen'/> >> <memballoon model='xen'/> >> </devices> >> </domain> >> >> The virtual guest starts, but then sits in the Grub 2 boot prompt >> because the disk is not available. The qemu log shows: >> >> qemu-system-i386: failed to create 'qdisk' device '51712': failed to >> create drive: Could not open >> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': >> No such file or directory >> qemu-system-i386: failed to create 'qdisk' device '51712': failed to >> create drive: Could not open >> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': >> No such file or directory >> qemu-system-i386: failed to create 'qdisk' device '51712': failed to >> create drive: Could not open >> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': >> No such file or directory >> qemu-system-i386: failed to create 'qdisk' device '51712': failed to >> create drive: Could not open >> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': >> No such file or directory >> ... >> >> I tried to strace libvirtd. The results are as follows: >> >> On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this: >> >> 1682 openat(AT_FDCWD, >> "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", >> O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) >> 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 >> 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 >> 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> >> 1682 <... mprotect resumed>) = 0 >> 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM >> IO], 8) = 0 >> 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 >> 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> >> 1682 <... mmap resumed>) = 0x7f538adfc000 >> 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> >> 1682 <... mprotect resumed>) = 0 >> 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> >> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 >> 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished >> ...> >> ... >> >> On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we >> see this: >> >> 16267 openat(AT_FDCWD, >> "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", >> O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) >> 16267 >> stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", >> 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) >> 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 >> 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, >> st_size=27448, ...}) = 0 >> 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", >> O_RDONLY|O_CLOEXEC) = 60 >> 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 >> &\0\0\0\0\0\0"..., 832) = 832 >> 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 >> 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, >> MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 >> 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 >> 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 >> 16267 close(60) = 0 >> ... >> >> Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That >> library *does* exist on openSUSE 15.2 but it doesn't seem to be used. > > It smells like a bug in the Leap 15.2 qemu package. Or perhaps in > xen's libxl library, which controls the qemu processes used by VMs. > I'd suggest entering a bug at bugzilla.opensuse.org. > >> I've tried to update libvirt to a newer version using the Open Build >> Service repos, but then ran into so many conflicting versions that I >> gave up. >> >> At this point I'm stuck. Does anyone have an idea I can try? > > We might be able to eliminate libxl by comparing the interaction with > qemu between the working and non-working setups. E.g. enable debug in > /etc/libvirt/libvirtd.conf, restart libvirtd, start the offending VM, > then look in /var/log/libvirt/libxl/libxl-driver.log for the qemu > command line created by libxl and qmp commands sent to qemu. If libxl > prepares qemu similarly between the two, then the issue is likely in > the qemu package. > > Regards, > Jim >
Seemingly Similar Threads
- RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
- Re: RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
- Re: RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
- libvirt 0.10 and cephx
- Using libvirt to access Ceph RBDs with Xen