Considering that the technic of SPDK + QEMU is making progress toward maturity. Personally I'd like to do the integration work. Not sure somebody would mind to give me some clue on that? Because I'm not familiar with libguestfs code structures. Thanks, Bob
In Intel's official documents, SPDK is used as the VM's data disk to accelerate IO performance for various applications such as database and key-value store. Unlike the data disk that you can log in the VM and make partitions inside, there is no way to do the same thing for a bootable rootfs disk. Because SPDK has its own filesystem, and the host wouldn't recognize the disk even if you export it via NBD. QEMU works on above SPDK by simple block layer operations, so the only way to deploy SPDK disk is to do in inside, i.e., by using libguestfs. P.S. I managed to boot up a guest VM with SPDK rootfs disk, and luckily this disk is able to be exported to a file. I'm willing to share this file, so maybe someone in the community could easily setup a environment and test libguestfs integration. Look forward to your help. Thanks Bob Bob Chen <a175818323@gmail.com> 于2018年7月18日周三 下午12:10写道:> Considering that the technic of SPDK + QEMU is making progress toward > maturity. > > Personally I'd like to do the integration work. Not sure somebody would > mind to give me some clue on that? Because I'm not familiar with libguestfs > code structures. > > > Thanks, > Bob >
Richard W.M. Jones
2018-Jul-18 13:44 UTC
Re: [Libguestfs] Is there any plan to support SPDK disks?
On Wed, Jul 18, 2018 at 12:10:30PM +0800, Bob Chen wrote:> Considering that the technic of SPDK + QEMU is making progress toward > maturity. > > Personally I'd like to do the integration work. Not sure somebody would > mind to give me some clue on that? Because I'm not familiar with libguestfs > code structures.If qemu supports SPDK then we should get it "for free". At most we might need to add support for the qemu command line like we do for other non-file stuff (eg. NBD, iSCSI). What does a qemu command line accessing SPDK look like? What sort of resources do we need to set up an SPDK test environment? (For example, are NVMe drives absolutely required?) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
CentOS-7.3-x86_64-SPDK-NVMe.img.tar.bz2 <https://drive.google.com/file/d/1AL8yXbcCa6LAJYcykE2OF31GkRUTP2um/view?usp=drive_web> Theoretically SPDK should be able to support multiple storage backends, including NVMe drive, Linux AIO and malloc ramdisk. But after trying the latter two methods, I found the VM just couldn't boot up. Even if I had setup a virtual CDROM drive and reinstalled the guest OS inside. Don't know the reason yet, will consult SPDK community later. So it looks like you do need to have a NVMe drive... Below are the instructions of how to setup the test environment. -------- Host Environment: CentOS 7 1. Compile SPDK $ git clone https://github.com/spdk/spdk $ cd spdk $ git submodule update --init (clone submodule DPDK) $ sudo ./scripts/pkgdep.sh (install dependencies, ignore unnecessary packages) $ ./configure $ make -j8 2. Install QEMU 2.12.0 $ wget https://download.qemu.org/qemu-2.12.0.tar.xz $ tar xvf qemu-2.12.0.tar.xz && cd qemu-2.12.0 $ ./configure --target-list=x86_64-softmmu $ make -j8 && sudo make install 3. Setup Hugepages Edit /etc/default/grub, add intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16 (reserve 16GB hugepages for testing) $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg 4. Auto load vfio module Edit /etc/modules-load.d/vfio.conf, add vfio_pci 5. Reboot $ sudo reboot 6. Setup SPDK $ cd spdk && sudo su # ulimit -l unlimited # HUGEMEM=16384 ./scripts/setup.sh # ./app/vhost/vhost -S /var/tmp -s 1024 -m 0x1 & (start vhost daemon) # ./scripts/rpc.py construct_nvme_bdev -b nvme0 -t pcie -a 0000:03:00.0 (NVMe drive's pci address) # ./scripts/rpc.py construct_lvol_store nvme0n1 lvs.nvme0n1 # ./scripts/rpc.py construct_lvol_bdev -l lvs.nvme0n1 lvol.0 20480 (create a 20GB logical volume) # ./scripts/rpc.py construct_vhost_scsi_controller vhost.0 # ./scripts/rpc.py add_vhost_scsi_lun vhost.0 0 lvs.nvme0n1/lvol.0 (should then have a /var/tmp/vhost.0 socket) 7. Copy rootfs image into logical volume # ./scripts/rpc.py start_nbd_disk lvs.nvme0n1/lvol.0 /dev/nbd0 # dd if=CentOS-7.3-x86_64-SPDK-NVMe.img of=/dev/nbd0 oflag=direct bs=1M (Copy the 20GB image into the logical volume, see attached file) # ./scripts/rpc.py stop_nbd_disk /dev/nbd0 8. Download UEFI bootloader ( https://github.com/tianocore/tianocore.github.io/wiki/OVMF) # wget https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180612.196.gb420d98502.noarch.rpm rpm install ... 9. Start QEMU # /usr/local/bin/qemu-system-x86_64 \ -enable-kvm -cpu host -smp 4 \ -m 4G -object memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0 \ -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,format=raw,if=pflash \ -chardev socket,id=spdk_vhost_scsi0,path=/var/tmp/vhost.0 \ -device vhost-user-scsi-pci,chardev=spdk_vhost_scsi0,num_queues=4,bootindex=0 \ ... Richard W.M. Jones <rjones@redhat.com> 于2018年7月18日周三 下午9:44写道:> On Wed, Jul 18, 2018 at 12:10:30PM +0800, Bob Chen wrote: > > Considering that the technic of SPDK + QEMU is making progress toward > > maturity. > > > > Personally I'd like to do the integration work. Not sure somebody would > > mind to give me some clue on that? Because I'm not familiar with > libguestfs > > code structures. > > If qemu supports SPDK then we should get it "for free". At most we > might need to add support for the qemu command line like we do for > other non-file stuff (eg. NBD, iSCSI). > > What does a qemu command line accessing SPDK look like? > > What sort of resources do we need to set up an SPDK test environment? > (For example, are NVMe drives absolutely required?) > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat > http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-builder quickly builds VMs from scratch > http://libguestfs.org/virt-builder.1.html >