search for: vq_meta_prefetch

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (5): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 6 +- drivers/vhost/vhost.c | 434 ++++++++++++++++++++++++++++++++++++++++++++------ drivers/vhost/vhost.h | 18 ++- 3 files changed, 407 insertions...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (5): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 6 +- drivers/vhost/vhost.c | 434 ++++++++++++++++++++++++++++++++++++++++++++------ drivers/vhost/vhost.h | 18 ++- 3 files changed, 407 insertions...

...ns(+), 7 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 36f3d0f49e60..0b4b3deab5aa 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -971,7 +971,7 @@ static void handle_tx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); @@ -1140,7 +1140,7 @@ static void handle_rx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); diff --git a/drive...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (5): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 4 +- drivers/vhost/vhost.c | 416 +++++++++++++++++++++++++++++++++++++----- drivers/vhost/vhost.h | 15 +- 3 files changed, 384 insertions(+), 51 d...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (5): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 4 +- drivers/vhost/vhost.c | 416 +++++++++++++++++++++++++++++++++++++----- drivers/vhost/vhost.h | 15 +- 3 files changed, 384 insertions(+), 51 d...

...(vq, num)) && + access_ok(avail, vhost_get_avail_size(vq, num)) && + access_ok(used, vhost_get_used_size(vq, num)); } static void vhost_vq_meta_update(struct vhost_virtqueue *vq, @@ -1311,22 +1328,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, int vq_meta_prefetch(struct vhost_virtqueue *vq) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; if (!vq->iotlb) return 1; return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, - num * sizeof(*vq->desc), VHOST_ADDR_DES...

...(vq, num)) && >> + access_ok(used, vhost_get_used_size(vq, num)); >> } >> >> static void vhost_vq_meta_update(struct vhost_virtqueue *vq, >> @@ -1311,22 +1328,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, >> >> int vq_meta_prefetch(struct vhost_virtqueue *vq) >> { >> - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; >> unsigned int num = vq->num; >> >> if (!vq->iotlb) >> return 1; >> >> return iotlb_acc...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (6): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: factor out setting vring addr and num vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 4 +- drivers/vhost/vhost.c | 850 ++++++++++++++++++++++++++++++++++++------ drivers/vhost/vhost.h |...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (6): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: factor out setting vring addr and num vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 4 +- drivers/vhost/vhost.c | 850 ++++++++++++++++++++++++++++++++++++------ drivers/vhost/vhost.h |...

...ing vq mutex. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing metadata prefetch - fix build warning on MIPS Jason Wang (6): vhost: generalize adding used elem vhost: fine grain userspace memory accessors vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() vhost: introduce helpers to get the size of metadata area vhost: factor out setting vring addr and num vhost: access vq metadata through kernel virtual address drivers/vhost/net.c | 4 +- drivers/vhost/vhost.c | 852 ++++++++++++++++++++++++++++++++++++------ drivers/vhost/vhost.h |...

...+++ 3 files changed, 297 insertions(+), 2 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index bf55f99..c276371 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -982,6 +982,7 @@ static void handle_tx(struct vhost_net *net) else handle_tx_copy(net, sock); + vq_meta_prefetch_done(vq); out: mutex_unlock(&vq->mutex); } @@ -1250,6 +1251,7 @@ static void handle_rx(struct vhost_net *net) vhost_net_enable_vq(net, vq); out: vhost_net_signal_used(nvq); + vq_meta_prefetch_done(vq); mutex_unlock(&vq->mutex); } diff --git a/drivers/vhost/vhost.c b/dri...

...access_ok(avail, vhost_get_avail_size(vq, num)) && > + access_ok(used, vhost_get_used_size(vq, num)); > } > > static void vhost_vq_meta_update(struct vhost_virtqueue *vq, > @@ -1311,22 +1328,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, > > int vq_meta_prefetch(struct vhost_virtqueue *vq) > { > - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; > unsigned int num = vq->num; > > if (!vq->iotlb) > return 1; > > return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, > - num...

...access_ok(avail, vhost_get_avail_size(vq, num)) && > + access_ok(used, vhost_get_used_size(vq, num)); > } > > static void vhost_vq_meta_update(struct vhost_virtqueue *vq, > @@ -1311,22 +1328,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, > > int vq_meta_prefetch(struct vhost_virtqueue *vq) > { > - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; > unsigned int num = vq->num; > > if (!vq->iotlb) > return 1; > > return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, > - num...

Hi: This series tries to access virtqueue metadata through kernel virtual address instead of copy_user() friends since they had too much overheads like checks, spec barriers or even hardware feature toggling. Test shows about 24% improvement on TX PPS. It should benefit other cases as well. Changes from V1: - instead of pinning pages, use MMU notifier to invalidate vmaps and remap duing

...ns(-) > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index bf55f99..c276371 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -982,6 +982,7 @@ static void handle_tx(struct vhost_net *net) > else > handle_tx_copy(net, sock); > > + vq_meta_prefetch_done(vq); > out: > mutex_unlock(&vq->mutex); > } > @@ -1250,6 +1251,7 @@ static void handle_rx(struct vhost_net *net) > vhost_net_enable_vq(net, vq); > out: > vhost_net_signal_used(nvq); > + vq_meta_prefetch_done(vq); > mutex_unlock(&vq->mutex);...

...ns(-) > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index bf55f99..c276371 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -982,6 +982,7 @@ static void handle_tx(struct vhost_net *net) > else > handle_tx_copy(net, sock); > > + vq_meta_prefetch_done(vq); > out: > mutex_unlock(&vq->mutex); > } > @@ -1250,6 +1251,7 @@ static void handle_rx(struct vhost_net *net) > vhost_net_enable_vq(net, vq); > out: > vhost_net_signal_used(nvq); > + vq_meta_prefetch_done(vq); > mutex_unlock(&vq->mutex);...

...g vhost-net, in details this patch: >>>>- exposes VIRTIO_F_ACCESS_PLATFORM feature and inits the iotlb >>>>? device if the feature is acked >>>>- implements VHOST_GET_BACKEND_FEATURES and >>>>? VHOST_SET_BACKEND_FEATURES ioctls >>>>- calls vq_meta_prefetch() before vq processing to prefetch vq >>>>? metadata address in IOTLB >>>>- provides .read_iter, .write_iter, and .poll callbacks for the >>>>? chardev; they are used by the userspace to exchange IOTLB messages >>>> >>>>This patch was test...

From: Jason Wang <jasowang at redhat.com> Date: Fri, 28 Dec 2018 15:55:37 +0800 > +static int vhost_invalidate_vmap(struct vhost_virtqueue *vq, > + struct vhost_vmap *map, > + unsigned long uaddr, > + unsigned long start, > + unsigned long end, > + bool blockable) > +{ > + if (start < uaddr && end >= uaddr) { > + if (!blockable)

...stead of pinning pages, use MMU notifier to invalidate vmaps and > remap duing metadata prefetch > - fix build warning on MIPS > > Jason Wang (5): > vhost: generalize adding used elem > vhost: fine grain userspace memory accessors > vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() > vhost: introduce helpers to get the size of metadata area > vhost: access vq metadata through kernel virtual address > > drivers/vhost/net.c | 4 +- > drivers/vhost/vhost.c | 416 +++++++++++++++++++++++++++++++++++++----- > drivers/vhost/vhost.h | 15 +- > 3 fil...

...stead of pinning pages, use MMU notifier to invalidate vmaps and > remap duing metadata prefetch > - fix build warning on MIPS > > Jason Wang (5): > vhost: generalize adding used elem > vhost: fine grain userspace memory accessors > vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() > vhost: introduce helpers to get the size of metadata area > vhost: access vq metadata through kernel virtual address > > drivers/vhost/net.c | 4 +- > drivers/vhost/vhost.c | 416 +++++++++++++++++++++++++++++++++++++----- > drivers/vhost/vhost.h | 15 +- > 3 fil...