Stefano Garzarella
2022-Oct-26 10:17 UTC
Issue with VDUSE (QSD vduse-blk export) and vhost-vdpa
On Wed, Oct 26, 2022 at 05:39:23PM +0800, Yongji Xie wrote:>Hi Stefano, > >On Wed, Oct 26, 2022 at 5:12 PM Stefano Garzarella <sgarzare at redhat.com> wrote: >> >> Hi Xie, >> I was testing libblkio [1] with QSD vduse-blk export and had some >> issues. >> >> In a nutshell, QSD prints me the following messages when using >> vhost-vdpa to access the device: >> >> Failed to get vq[0] iova mapping >> Failed to update vring for vq[0] >> >> This happens only with vhost-vdpa, using virtio-vdpa instead the device >> works fine. >> I'm using Linux v6.0 and QEMU master (commit >> 214a8da23651f2472b296b3293e619fd58d9e212). >> >> I haven't had much time to investigate, I hope to do it next week, but >> maybe it's much faster for you. >> >> I saw that ioctl(VDUSE_IOTLB_GET_FD) in libvduse.c returns -1 (EPERM), >> so IIUC in the kernel vduse_dev_broken() was called, and the device is >> in a broken state. >> >> >> We will use libblkio in QEMU [2] to access vDPA devices via vhost-vdpa. >> But I'm doing these tests without QEMU for now, using an example inside >> the libblkio repo: >> >> # Build libblkio and examples >> # Fedora/CentOS/RHEL >> dnf install -y git meson rust cargo python3-docutils rustfmt >> # Debian/Ubuntu >> apt-get install -y git meson rustc cargo python3-docutils >> >> git clone https://gitlab.com/libblkio/libblkio.git >> >> cd libblkio >> git checkout v1.1.0 >> >> meson setup build >> meson compile -C build >> >> >> # On terminal 1 >> modprobe vduse >> modprobe vhost-vdpa >> >> qemu-img create -f qcow2 -o preallocation=full /path/to/test.qcow2 1g >> >> qemu-storage-daemon \ >> --blockdev file,filename=/path/to/test.qcow2,cache.direct=on,aio=native,node-name=file \ >> --blockdev qcow2,file=file,node-name=qcow2 \ >> --object iothread,id=iothread0 \ >> --export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on,iothread=iothread0 >> >> >> # On terminal 2 >> vdpa dev add name vduse0 mgmtdev vduse >> >> cd libblkio/build >> >> # blkio-bench executes >> ./examples/blkio-bench virtio-blk-vhost-vdpa \ >> path=/dev/vhost-vdpa-0 --runtime=5 --readwrite=randread >> >> # after this step, QSD (running on terminal 1) prints the following messages: >> Failed to get vq[0] iova mapping >> Failed to update vring for vq[0] >> >> I don't know if I'm doing something wrong or in libblkio we have some >> issue, but using vdpa-sim-blk works correctly, so maybe there is >> something in vduse that is missing. >> >> Any help or suggestion is welcome :-) >> > >I'd like to know whether bio-bench uses the shared memory >(tmpfs/hugetlbfs) as the vdpa memory region. This is what VDUSE needs.Okay, so IIUC every memory regions should have an associated fd. The buffers in libblkio are already allocated in this way, but it is not true for the virtqueue memory, I'll change it and test. Thanks for the help :-) Stefano
Stefano Garzarella
2022-Nov-03 10:37 UTC
Issue with VDUSE (QSD vduse-blk export) and vhost-vdpa
On Wed, Oct 26, 2022 at 12:17 PM Stefano Garzarella <sgarzare at redhat.com> wrote:> > On Wed, Oct 26, 2022 at 05:39:23PM +0800, Yongji Xie wrote: > >Hi Stefano, > > > >On Wed, Oct 26, 2022 at 5:12 PM Stefano Garzarella <sgarzare at redhat.com> wrote: > >> > >> Hi Xie, > >> I was testing libblkio [1] with QSD vduse-blk export and had some > >> issues. > >> > >> In a nutshell, QSD prints me the following messages when using > >> vhost-vdpa to access the device: > >> > >> Failed to get vq[0] iova mapping > >> Failed to update vring for vq[0] > >> > >> This happens only with vhost-vdpa, using virtio-vdpa instead the device > >> works fine. > >> I'm using Linux v6.0 and QEMU master (commit > >> 214a8da23651f2472b296b3293e619fd58d9e212). > >> > >> I haven't had much time to investigate, I hope to do it next week, but > >> maybe it's much faster for you. > >> > >> I saw that ioctl(VDUSE_IOTLB_GET_FD) in libvduse.c returns -1 (EPERM), > >> so IIUC in the kernel vduse_dev_broken() was called, and the device is > >> in a broken state. > >> > >> > >> We will use libblkio in QEMU [2] to access vDPA devices via vhost-vdpa. > >> But I'm doing these tests without QEMU for now, using an example inside > >> the libblkio repo: > >> > >> # Build libblkio and examples > >> # Fedora/CentOS/RHEL > >> dnf install -y git meson rust cargo python3-docutils rustfmt > >> # Debian/Ubuntu > >> apt-get install -y git meson rustc cargo python3-docutils > >> > >> git clone https://gitlab.com/libblkio/libblkio.git > >> > >> cd libblkio > >> git checkout v1.1.0 > >> > >> meson setup build > >> meson compile -C build > >> > >> > >> # On terminal 1 > >> modprobe vduse > >> modprobe vhost-vdpa > >> > >> qemu-img create -f qcow2 -o preallocation=full /path/to/test.qcow2 1g > >> > >> qemu-storage-daemon \ > >> --blockdev file,filename=/path/to/test.qcow2,cache.direct=on,aio=native,node-name=file \ > >> --blockdev qcow2,file=file,node-name=qcow2 \ > >> --object iothread,id=iothread0 \ > >> --export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on,iothread=iothread0 > >> > >> > >> # On terminal 2 > >> vdpa dev add name vduse0 mgmtdev vduse > >> > >> cd libblkio/build > >> > >> # blkio-bench executes > >> ./examples/blkio-bench virtio-blk-vhost-vdpa \ > >> path=/dev/vhost-vdpa-0 --runtime=5 --readwrite=randread > >> > >> # after this step, QSD (running on terminal 1) prints the following messages: > >> Failed to get vq[0] iova mapping > >> Failed to update vring for vq[0] > >> > >> I don't know if I'm doing something wrong or in libblkio we have some > >> issue, but using vdpa-sim-blk works correctly, so maybe there is > >> something in vduse that is missing. > >> > >> Any help or suggestion is welcome :-) > >> > > > >I'd like to know whether bio-bench uses the shared memory > >(tmpfs/hugetlbfs) as the vdpa memory region. This is what VDUSE needs. > > Okay, so IIUC every memory regions should have an associated fd. > > The buffers in libblkio are already allocated in this way, but it is not > true for the virtqueue memory, I'll change it and test.I just changed the virtqueue memory allocation as you suggested, and it worked! MR here: https://gitlab.com/libblkio/libblkio/-/merge_requests/141 Thanks for the help, Stefano