CentOS - Oct 2018 - Fw: Re: [SPDK] VM boot failed sometimes if using vhost-user-blk with spdk

Forwarded to centos mailing list

-----Original Messages-----
From: wuzhouhui <wuzhouhui14 at mails.ucas.ac.cn>
Sent Time: 2018-10-30 14:06:00 (Tuesday)
To: "storage performance development kit" <spdk at lists.01.org>
Cc: centos at centos.org, qemu-discuss at nongnu.org
Subject: Re: [SPDK] VM boot failed sometimes if using vhost-user-blk with spdk

I enable debug of systemd-udevd by changing guest's /etc/udev/udev.conf to
this:
  udev_log="debug"
and regenerate initramfs. Then, we can get more message about this issue. I
select some useful messages from /run/initramfs/rdsosreport.txt:
  systemd-udevd[199] probe /dev/vdb raid offset=8
  systemd-udevd[199] no db file to read /run/udev/data/+virtio:virtio1: No such 
  systemd-udevd[199] handling device node '/dev/vdb', devnum=b253:16,
mode=0660,
  systemd-udevd[199] set permissions /dev/vdb, 060660, uid=0, gid=6
  systemd-udevd[199] creating symlink '/dev/block/253:16' to
'../vdb'
  systemd-udevd[199] creating link '/dev/disk' to '/dev/vdb'
  systemd-udevd[199] creating symlink '/dev/disk' to 'vdb'
We can see that the symlink /dev/disk was created by system-udevd, and cause
boot failed.
I also noticed that `blkid /dev/vdb' print nothing, but `blkid
/dev/vda*' is normal.
The /dev/vdb is correspond to the malloc bdev that spdk created.

Is this a bug for system-udevd?

(Add centos at centos.org and qemu-discuss at nongun.org in CC list)
> -----Original Messages-----
> From: wuzhouhui <wuzhouhui14 at mails.ucas.ac.cn>
> Sent Time: 2018-10-30 11:31:15 (Tuesday)
> To: qemu-discuss at nongnu.org, spdk at lists.01.org
> Cc: 
> Subject: [CASS SPAM][SPDK] VM boot failed sometimes if using vhost-user-blk
with spdk
> 
> I'm using following command line to start VM (The /var/tmp/vhost.0
connected to a
> malloc bdev (16 MB ) created by SPDK):
>   /home/wuzhouhui/qemu-2.12.1/x86_64-softmmu/qemu-system-x86_64 \
>           -name guest=wzh,debug-threads=on \
>           -machine pc-i440fx-2.12,accel=kvm,usb=off \
>           -cpu host \
>           -m 4096 \
>           -object
memory-backend-file,id=mem0,size=4096M,mem-path=/dev/hugepages,share=on \
>           -numa node,memdev=mem0 \
>           -realtime mlock=off \
>           -smp 2,sockets=2,cores=1,threads=1 \
>           -uuid a84e96e6-2c53-408d-986b-c709bc6a0e51 \
>           -no-user-config \
>           -nodefaults \
>           -rtc base=utc,driftfix=slew \
>           -global kvm-pit.lost_tick_policy=delay \
>           -no-shutdown \
>           -boot strict=on \
>           -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
>           -device ahci,id=sata0,bus=pci.0,addr=0x4 \
>           -drive
file=/home/wuzhouhui/wzh.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none
\
>           -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0
\
>           -device usb-tablet,id=input0,bus=usb.0,port=1 \
>           -k en-us \
>           -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
>           -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg
timestamp=on \
>           -vnc :9 \
>           -chardev socket,id=char0,path=/var/tmp/vhost.0 \
>           -device vhost-user-blk-pci,id=blk0,chardev=char0,num-queues=4 \
> 
> But most of the time, VM boot failed with following message in vnc screen:
>   Warning: /dev/disk/by-uuid/e0dcaf0c-bc23-4df6-b2cd-d40aa1bbb0b5 does not
exist
> 
>   Generating "/run/initramfs/rdsosreport.txt"
> 
>   Entering emergency mode. Exit the shell to continue.
>   Type "journalctl" to view system logs.
>   You might want to save "/run/initramfs/rdsosreport.txt" to a
USB stick or /boot
>   after mounting them and attach it to a bug report.
> 
> There is some message from /run/initramfs/rdsosreport.txt:
>   ...
>   system-udevd[196]: symlink '../../vdb'
'/dev/disk/by-id/virtio-Malloc.tmp-b253:16' failed: Not a directory
>   ...
> 
> I checked /dev/disk, it should be a directory, but it is a symlink now:
>   :/# ls -l /dev/disk
>   lrwxrwxrwx 1 root 0 3 Oct 30 03:08 /dev/disk -> vdb
> 
> If I just remove:
>   -chardev socket,id=char0,path=/var/tmp/vhost.0 \
>   -device vhost-user-blk-pci,id=blk0,chardev=char0,num-queues=4 \
> The VM will boot normally.
> 
> Does anyone have encountered similar issue like this?
> 
> Host OS:  CentOS 7.3, with kernel 3.10.0-862.11.6.el7.x86_64
> Guest OS: CentOS 7.5
> Qemu: 2.12.1
> SPDK: f0cb7b871e in master
> _______________________________________________
> SPDK mailing list
> SPDK at lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK at lists.01.org
https://lists.01.org/mailman/listinfo/spdk