Libguestfs - Jul 2018 - Re: Is there any plan to support SPDK disks?

Considering that the technic of SPDK + QEMU is making progress toward
maturity.

Personally I'd like to do the integration work. Not sure somebody would
mind to give me some clue on that? Because I'm not familiar with libguestfs
code structures.


Thanks,
Bob

In Intel's official documents, SPDK is used as the VM's data disk to
accelerate IO performance for various applications such as database and
key-value store.

Unlike the data disk that you can log in the VM and make partitions inside,
there is no way to do the same thing for a bootable rootfs disk. Because
SPDK has its own filesystem, and the host wouldn't recognize the disk even
if you export it via NBD.

QEMU works on above SPDK by simple block layer operations, so the only way
to deploy SPDK disk is to do in inside, i.e., by using libguestfs.

P.S. I managed to boot up a guest VM with SPDK rootfs disk, and luckily
this disk is able to be exported to a file. I'm willing to share this file,
so maybe someone in the community could easily setup a environment and test
libguestfs integration.

Look forward to your help. Thanks
Bob



Bob Chen <a175818323@gmail.com> 于2018年7月18日周三 下午12:10写道：
> Considering that the technic of SPDK + QEMU is making progress toward
> maturity.
>
> Personally I'd like to do the integration work. Not sure somebody would
> mind to give me some clue on that? Because I'm not familiar with
libguestfs
> code structures.
>
>
> Thanks,
> Bob
>

On Wed, Jul 18, 2018 at 12:10:30PM +0800, Bob Chen
wrote:
> Considering that the technic of SPDK + QEMU is making progress toward
> maturity.
> 
> Personally I'd like to do the integration work. Not sure somebody would
> mind to give me some clue on that? Because I'm not familiar with
libguestfs
> code structures.
If qemu supports SPDK then we should get it "for free".  At most we
might need to add support for the qemu command line like we do for
other non-file stuff (eg. NBD, iSCSI).

What does a qemu command line accessing SPDK look like?

What sort of resources do we need to set up an SPDK test environment?
(For example, are NVMe drives absolutely required?)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

​
 CentOS-7.3-x86_64-SPDK-NVMe.img.tar.bz2
<https://drive.google.com/file/d/1AL8yXbcCa6LAJYcykE2OF31GkRUTP2um/view?usp=drive_web>
​
Theoretically SPDK should be able to support multiple storage backends,
including NVMe drive, Linux AIO and malloc ramdisk.

But after trying the latter two methods, I found the VM just couldn't boot
up. Even if I had setup a virtual CDROM drive and reinstalled the guest OS
inside. Don't know the reason yet, will consult SPDK community later.

So it looks like you do need to have a NVMe drive...

Below are the instructions of how to setup the test environment.

--------

Host Environment: CentOS 7

1. Compile SPDK
$ git clone https://github.com/spdk/spdk
$ cd spdk
$ git submodule update --init    (clone submodule DPDK)
$ sudo ./scripts/pkgdep.sh (install dependencies, ignore unnecessary
packages)
$ ./configure
$ make -j8


2. Install QEMU 2.12.0
$ wget https://download.qemu.org/qemu-2.12.0.tar.xz
$ tar xvf qemu-2.12.0.tar.xz && cd qemu-2.12.0
$ ./configure --target-list=x86_64-softmmu
$ make -j8 && sudo make install


3. Setup Hugepages
Edit /etc/default/grub, add
intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16    (reserve
16GB hugepages for testing)
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg


4. Auto load vfio module
Edit /etc/modules-load.d/vfio.conf, add
vfio_pci


5. Reboot
$ sudo reboot


6. Setup SPDK
$ cd spdk && sudo su
# ulimit -l unlimited
# HUGEMEM=16384 ./scripts/setup.sh
# ./app/vhost/vhost -S /var/tmp -s 1024 -m 0x1 &        (start vhost daemon)
# ./scripts/rpc.py construct_nvme_bdev -b nvme0 -t pcie -a 0000:03:00.0
    (NVMe drive's pci address)
# ./scripts/rpc.py construct_lvol_store nvme0n1 lvs.nvme0n1
# ./scripts/rpc.py construct_lvol_bdev -l lvs.nvme0n1 lvol.0 20480 (create
a 20GB logical volume)
# ./scripts/rpc.py construct_vhost_scsi_controller vhost.0
# ./scripts/rpc.py add_vhost_scsi_lun vhost.0 0 lvs.nvme0n1/lvol.0 (should
then have a /var/tmp/vhost.0 socket)


7. Copy rootfs image into logical volume
# ./scripts/rpc.py start_nbd_disk lvs.nvme0n1/lvol.0 /dev/nbd0
# dd if=CentOS-7.3-x86_64-SPDK-NVMe.img of=/dev/nbd0 oflag=direct bs=1M (Copy
the 20GB image into the logical volume, see attached file)
# ./scripts/rpc.py stop_nbd_disk /dev/nbd0


8. Download UEFI bootloader (
https://github.com/tianocore/tianocore.github.io/wiki/OVMF)
# wget
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180612.196.gb420d98502.noarch.rpm
rpm install ...


9. Start QEMU
# /usr/local/bin/qemu-system-x86_64 \
-enable-kvm -cpu host -smp 4 \
-m 4G -object
memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on -numa
node,memdev=mem0 \
-drive
file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,format=raw,if=pflash
\
-chardev socket,id=spdk_vhost_scsi0,path=/var/tmp/vhost.0 \
-device
vhost-user-scsi-pci,chardev=spdk_vhost_scsi0,num_queues=4,bootindex=0 \
...


Richard W.M. Jones <rjones@redhat.com> 于2018年7月18日周三 下午9:44写道：
> On Wed, Jul 18, 2018 at 12:10:30PM +0800, Bob Chen wrote:
> > Considering that the technic of SPDK + QEMU is making progress toward
> > maturity.
> >
> > Personally I'd like to do the integration work. Not sure somebody
would
> > mind to give me some clue on that? Because I'm not familiar with
> libguestfs
> > code structures.
>
> If qemu supports SPDK then we should get it "for free".  At most
we
> might need to add support for the qemu command line like we do for
> other non-file stuff (eg. NBD, iSCSI).
>
> What does a qemu command line accessing SPDK look like?
>
> What sort of resources do we need to set up an SPDK test environment?
> (For example, are NVMe drives absolutely required?)
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>