Libguestfs - Nov 2018 - Re: virt-df stuck at supermin: [...] waiting for /sys/block/sdb/dev to appear

This problem occurs on certain machines but I can't tell why and how
exactly to fix it.

The message suggests a problem with kernel modules, but they all seem to
load successfully, including many virtio modules.

Best Regards,
Peter

On Tue, Nov 20, 2018 at 03:15:54PM +0200, Peter Dimitrov
wrote:
> This problem occurs on certain machines but I can't tell why and how
> exactly to fix it.
> 
> The message suggests a problem with kernel modules, but they all seem to
> load successfully, including many virtio modules.
These are quite old versions of kernel/qemu/libguestfs/supermin so I
can only really offer general advice.  You are right that the problem
is caused because the appliance kernel does not see any virtio-scsi
disks.  That can be for a variety of reasons such as:

 - The virtio drivers in the appliance are not loaded or not working
   for some reason.

 - qemu isn't presenting virtio devices to the guest.
>From the trace:
libguestfs: libvirt XML:\n [...] <devices>
    <controller type="scsi" index="0"
model="virtio-scsi"/>
    <disk device="disk" type="file">
      <source file="/tmp/libguestfsdTUUkY/overlay1">
        <seclabel model="selinux" relabel="no"/>
      </source>
      <target dev="sda" bus="scsi"/>
      <driver name="qemu" type="qcow2"
cache="unsafe"/>
      <address type="drive" controller="0"
bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="disk">
      <source file="/tmp/libguestfsdTUUkY/overlay2"/>
      <target dev="sdb" bus="scsi"/>
      <driver name="qemu" type="qcow2"
cache="unsafe"/>
      <address type="drive" controller="0"
bus="0" target="1" unit="0"/>
      <shareable/>
    </disk>

This ought to cause libvirt to tell qemu to present virtio-scsi
devices.  You can check if they are actually present on the qemu
command line by looking for /var/log/libvirt/qemu/guestfs-XXX.log

supermin: internal insmod crc32c-intel.ko
supermin: internal insmod virtio.ko
supermin: internal insmod virtio_ring.ko
supermin: internal insmod virtio_blk.ko
supermin: internal insmod virtio_console.ko
supermin: internal insmod virtio_net.ko
supermin: internal insmod virtio_scsi.ko

The Virtio SCSI driver is getting loaded.

supermin: internal insmod virtio_balloon.ko
supermin: internal insmod virtio_input.ko
supermin: internal insmod virtio_mmio.ko
supermin: internal insmod virtio_pci.ko
[    0.289514] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A ->
IRQ 10
[    0.289888] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[    0.290355] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A ->
IRQ 11
[    0.290768] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    0.291008] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A ->
IRQ 11
[    0.291392] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver

This indicates that the Virtio PCI transport module is being loaded.
The "virtio_pci: leaving for legacy driver" message is worrying.

I think what's happening here is your appliance kernel only has
drivers for virtio 1.0 ("virtio modern") but your qemu is presenting
virtio legacy drivers.  There's a kernel config option to enable
virtio legacy (VIRTIO_PCI_LEGACY) which you should probably try first.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Seems like the devices are present:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/bin/qemu-kvm -name
guestfs-f9uckq9gdkacg5hn,debug-threads=on -S -machine
pc-i440fx-2.6,accel=kvm,usb=off -cpu host -m 500 -realtime mlock=off -smp
1,sockets=1,cores=1,threads=1 -uuid 4e89287f-6f82-4f54-9808-43527e9d3c91
-nographic -no-user-config -nodefaults -device sga -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-guestfs-f9uckq9gdkac/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
-no-reboot -no-acpi -boot strict=on -kernel
/var/tmp/.guestfs-0/appliance.d/kernel -initrd
/var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0
udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off
printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
guestfs_verbose=1 TERM=xterm-256color' -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive
file*/tmp/libguestfspS25nG/overlay1*,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-drive
file=*/tmp/libguestfspS25nG/overlay2*,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
-chardev socket,id=charserial0,path=/tmp/libguestfspS25nG/console.sock
-device isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/tmp/libguestfspS25nG/guestfsd.sock -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: custom-argv
Domain id=3 is tainted: host-cpu
2018-11-20T15:17:14.956988Z qemu-system-x86_64: terminating on signal 15
from pid 6001

2018-11-20 15:17:15.158+0000: shutting down



Compiling kernel with VIRTIO_PCI_LEGACY=m fixes it the issue!


Best Regards,
Peter


On Tue, Nov 20, 2018 at 5:24 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Tue, Nov 20, 2018 at 03:15:54PM +0200, Peter Dimitrov wrote:
> > This problem occurs on certain machines but I can't tell why and
how
> > exactly to fix it.
> >
> > The message suggests a problem with kernel modules, but they all seem
to
> > load successfully, including many virtio modules.
>
> These are quite old versions of kernel/qemu/libguestfs/supermin so I
> can only really offer general advice.  You are right that the problem
> is caused because the appliance kernel does not see any virtio-scsi
> disks.  That can be for a variety of reasons such as:
>
>  - The virtio drivers in the appliance are not loaded or not working
>    for some reason.
>
>  - qemu isn't presenting virtio devices to the guest.
>
> From the trace:
>
> libguestfs: libvirt XML:\n [...] <devices>
>     <controller type="scsi" index="0"
model="virtio-scsi"/>
>     <disk device="disk" type="file">
>       <source file="/tmp/libguestfsdTUUkY/overlay1">
>         <seclabel model="selinux" relabel="no"/>
>       </source>
>       <target dev="sda" bus="scsi"/>
>       <driver name="qemu" type="qcow2"
cache="unsafe"/>
>       <address type="drive" controller="0"
bus="0" target="0" unit="0"/>
>     </disk>
>     <disk type="file" device="disk">
>       <source file="/tmp/libguestfsdTUUkY/overlay2"/>
>       <target dev="sdb" bus="scsi"/>
>       <driver name="qemu" type="qcow2"
cache="unsafe"/>
>       <address type="drive" controller="0"
bus="0" target="1" unit="0"/>
>       <shareable/>
>     </disk>
>
> This ought to cause libvirt to tell qemu to present virtio-scsi
> devices.  You can check if they are actually present on the qemu
> command line by looking for /var/log/libvirt/qemu/guestfs-XXX.log
>
> supermin: internal insmod crc32c-intel.ko
> supermin: internal insmod virtio.ko
> supermin: internal insmod virtio_ring.ko
> supermin: internal insmod virtio_blk.ko
> supermin: internal insmod virtio_console.ko
> supermin: internal insmod virtio_net.ko
> supermin: internal insmod virtio_scsi.ko
>
> The Virtio SCSI driver is getting loaded.
>
> supermin: internal insmod virtio_balloon.ko
> supermin: internal insmod virtio_input.ko
> supermin: internal insmod virtio_mmio.ko
> supermin: internal insmod virtio_pci.ko
> [    0.289514] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A
->
> IRQ 10
> [    0.289888] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy
> driver
> [    0.290355] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A
->
> IRQ 11
> [    0.290768] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy
> driver
> [    0.291008] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A
->
> IRQ 11
> [    0.291392] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy
> driver
>
> This indicates that the Virtio PCI transport module is being loaded.
> The "virtio_pci: leaving for legacy driver" message is worrying.
>
> I think what's happening here is your appliance kernel only has
> drivers for virtio 1.0 ("virtio modern") but your qemu is
presenting
> virtio legacy drivers.  There's a kernel config option to enable
> virtio legacy (VIRTIO_PCI_LEGACY) which you should probably try first.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines.  Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>