search for: virtqueue_kick_prepar

This patch allows virtio-blk to release its block queue lock while kicking the host. This improves scalability on SMP guests who would otherwise spin on the lock while another vCPU is kicking the host. This approach can be used for other virtio devices too. It simply splits the virtqueue_kick() operation into a prepare step which requires that the lock be held and the actual notify step which

This patch allows virtio-blk to release its block queue lock while kicking the host. This improves scalability on SMP guests who would otherwise spin on the lock while another vCPU is kicking the host. This approach can be used for other virtio devices too. It simply splits the virtqueue_kick() operation into a prepare step which requires that the lock be held and the actual notify step which

...;rusty at rustcorp.com.au> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -237,10 +237,12 @@ add_head: } EXPORT_SYMBOL_GPL(virtqueue_add_buf); -void virtqueue_kick(struct virtqueue *_vq) +bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq = to_vvq(_vq); u16 new, old; + bool needs_kick; + START_USE(vq); /* Descriptors and available array need to be set before we expose the * new available array entries. */ @@ -253,13 +255,30 @@ void virtqueue_kick(struct virtqueue *_v...

...;rusty at rustcorp.com.au> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -237,10 +237,12 @@ add_head: } EXPORT_SYMBOL_GPL(virtqueue_add_buf); -void virtqueue_kick(struct virtqueue *_vq) +bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq = to_vvq(_vq); u16 new, old; + bool needs_kick; + START_USE(vq); /* Descriptors and available array need to be set before we expose the * new available array entries. */ @@ -253,13 +255,30 @@ void virtqueue_kick(struct virtqueue *_v...

Replace virtqueue_kick by virtqueue_kick_prepare, which requires serialization, and virtqueue_notify, which does not. Repurpose the return values to indicate whether the vq should be notified. This fixes a lock contention with qemu host. When the guest calls vibad rtqueue_notify, the qemu vcpu thread exits the guest and waits for the qemu iot...

Replace virtqueue_kick by virtqueue_kick_prepare, which requires serialization, and virtqueue_notify, which does not. Repurpose the return values to indicate whether the vq should be notified. This fixes a lock contention with qemu host. When the guest calls vibad rtqueue_notify, the qemu vcpu thread exits the guest and waits for the qemu iot...

Replace virtqueue_kick by virtqueue_kick_prepare, which requires serialization, and virtqueue_notify, which does not. Repurpose the return values to indicate whether the vq should be notified. This fixes a bad spinlock contention when the host is qemu. When the guest calls virtqueue_notify, the qemu vcpu thread exits the guest and waits for t...

Split virtqueue_kick() call into virtqueue_kick_prepare(), which requires serialization, and virtqueue_notify(), which does not. Move the virtqueue_notify() call out of the critical section protected by the queue lock. This avoids triggering a vmexit while holding the lock and thereby fixes a rather bad spinlock contention. Suggested-by: Chia-I Wu &...

...dispatch_work(struct work_struct *work) } else { pr_debug("virtio-fs: Could not queue FORGET: err=%d. Dropping it.\n", ret); + dec_in_flight_req(fsvq); kfree(forget); } spin_unlock(&fsvq->lock); return; } - fsvq->in_flight++; notify = virtqueue_kick_prepare(vq); spin_unlock(&fsvq->lock); @@ -472,7 +467,7 @@ static void virtio_fs_requests_done_work(struct work_struct *work) fuse_request_end(fc, req); spin_lock(&fsvq->lock); - fsvq->in_flight--; + dec_in_flight_req(fsvq); spin_unlock(&fsvq->lock); } } @@ -73...

...f scatterlists readable by other side + * @in_sgs: the number of scatterlists which are writable (after readable ones) * @data: the token identifying the buffer. * @gfp: how to do memory allocations (if necessary). * @@ -1821,7 +1821,7 @@ EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx); /** * virtqueue_kick_prepare - first half of split virtqueue_kick call. - * @vq: the struct virtqueue + * @_vq: the struct virtqueue * * Instead of virtqueue_kick(), you can do: * if (virtqueue_kick_prepare(vq)) @@ -1841,7 +1841,7 @@ EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); /** * virtqueue_notify - second half of...

...Queue is full */ + ret = 1; + } else { + pr_debug("virtio-fs: Could not queue FORGET: err=%d." + " Dropping it.\n", ret); + kfree(forget); + if (in_flight) + dec_in_flight_req(fsvq); + } + goto out; + } + + if (!in_flight) + inc_in_flight_req(fsvq); + notify = virtqueue_kick_prepare(vq); + spin_unlock(&fsvq->lock); + + if (notify) + virtqueue_notify(vq); + return ret; +out: + spin_unlock(&fsvq->lock); + return ret; +} + static void virtio_fs_hiprio_dispatch_work(struct work_struct *work) { struct virtio_fs_forget *forget; struct virtio_fs_vq *fsvq = conta...

...ize_t total_buf_space; + bool notify; vrp = kzalloc(sizeof(*vrp), GFP_KERNEL); if (!vrp) @@ -1030,8 +1031,22 @@ static int rpmsg_probe(struct virtio_device *vdev) } } + /* + * Prepare to kick but don't notify yet - we can't do this before + * device is ready. + */ + notify = virtqueue_kick_prepare(vrp->rvq); + + /* From this point on, we can notify and get callbacks. */ + virtio_device_ready(vdev); + /* tell the remote processor it can start sending messages */ - virtqueue_kick(vrp->rvq); + /* + * this might be concurrent with callbacks, but we are only + * doing notify, not a ful...

...ize_t total_buf_space; + bool notify; vrp = kzalloc(sizeof(*vrp), GFP_KERNEL); if (!vrp) @@ -1030,8 +1031,22 @@ static int rpmsg_probe(struct virtio_device *vdev) } } + /* + * Prepare to kick but don't notify yet - we can't do this before + * device is ready. + */ + notify = virtqueue_kick_prepare(vrp->rvq); + + /* From this point on, we can notify and get callbacks. */ + virtio_device_ready(vdev); + /* tell the remote processor it can start sending messages */ - virtqueue_kick(vrp->rvq); + /* + * this might be concurrent with callbacks, but we are only + * doing notify, not a ful...

...+ struct vring_desc *indirect_desc, > + void *data, > + void *ctx); > + > bool virtqueue_kick(struct virtqueue *vq); > > +bool virtqueue_kick_sync(struct virtqueue *vq); > + > +bool virtqueue_kick_async(struct virtqueue *vq, wait_queue_head_t wq); > + > bool virtqueue_kick_prepare(struct virtqueue *vq); > > bool virtqueue_notify(struct virtqueue *vq); I don't much care for this API. It does exactly what balloon needs, but at cost of e.g. transparently busy-waiting. Unlikely to be a good fit for anything else. If you don't like my original _first/_next/_la...

...+ struct vring_desc *indirect_desc, > + void *data, > + void *ctx); > + > bool virtqueue_kick(struct virtqueue *vq); > > +bool virtqueue_kick_sync(struct virtqueue *vq); > + > +bool virtqueue_kick_async(struct virtqueue *vq, wait_queue_head_t wq); > + > bool virtqueue_kick_prepare(struct virtqueue *vq); > > bool virtqueue_notify(struct virtqueue *vq); I don't much care for this API. It does exactly what balloon needs, but at cost of e.g. transparently busy-waiting. Unlikely to be a good fit for anything else. If you don't like my original _first/_next/_la...

...> + > /** > * virtqueue_add_sgs - expose buffers to other end > * @vq: the struct virtqueue we're talking about. > @@ -537,18 +812,7 @@ int virtqueue_add_inbuf_ctx(struct virtqueue *vq, > } > EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx); > > -/** > - * virtqueue_kick_prepare - first half of split virtqueue_kick call. > - * @vq: the struct virtqueue > - * > - * Instead of virtqueue_kick(), you can do: > - * if (virtqueue_kick_prepare(vq)) > - * virtqueue_notify(vq); > - * > - * This is sometimes useful because the virtqueue_kick_prepare() needs &g...

...> + > /** > * virtqueue_add_sgs - expose buffers to other end > * @vq: the struct virtqueue we're talking about. > @@ -537,18 +812,7 @@ int virtqueue_add_inbuf_ctx(struct virtqueue *vq, > } > EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx); > > -/** > - * virtqueue_kick_prepare - first half of split virtqueue_kick call. > - * @vq: the struct virtqueue > - * > - * Instead of virtqueue_kick(), you can do: > - * if (virtqueue_kick_prepare(vq)) > - * virtqueue_notify(vq); > - * > - * This is sometimes useful because the virtqueue_kick_prepare() needs &g...

...kernel at I-love.SAKURA.ne.jp> Date: Fri, 6 Feb 2015 13:28:38 +0900 Subject: [PATCH] virtio: Avoid possible kernel panic if DEBUG is enabled. The virtqueue_add() calls START_USE() upon entry. The virtqueue_kick() is called if vq->num_added == (1 << 16) - 1 before calling END_USE(). The virtqueue_kick_prepare() called via virtqueue_kick() calls START_USE() upon entry, and will call panic() if DEBUG is enabled. Move this virtqueue_kick() call to after END_USE() call. Signed-off-by: Tetsuo Handa <penguin-kernel at I-love.SAKURA.ne.jp> --- drivers/virtio/virtio_ring.c | 6 +++--- 1 file changed, 3...

...kernel at I-love.SAKURA.ne.jp> Date: Fri, 6 Feb 2015 13:28:38 +0900 Subject: [PATCH] virtio: Avoid possible kernel panic if DEBUG is enabled. The virtqueue_add() calls START_USE() upon entry. The virtqueue_kick() is called if vq->num_added == (1 << 16) - 1 before calling END_USE(). The virtqueue_kick_prepare() called via virtqueue_kick() calls START_USE() upon entry, and will call panic() if DEBUG is enabled. Move this virtqueue_kick() call to after END_USE() call. Signed-off-by: Tetsuo Handa <penguin-kernel at I-love.SAKURA.ne.jp> --- drivers/virtio/virtio_ring.c | 6 +++--- 1 file changed, 3...

...tats *stats = this_cpu_ptr(vi->stats); do { if (vi->mergeable_rx_bufs) @@ -477,13 +512,24 @@ static bool try_fill_recv(struct virtnet_info *vi, gfp_t gfp) } while (err > 0); if (unlikely(vi->num > vi->max)) vi->max = vi->num; - virtqueue_kick(vi->rvq); + if (virtqueue_kick_prepare(vi->rvq)) { + virtqueue_notify(vi->rvq); + u64_stats_update_begin(&stats->syncp); + stats->rx_kick++; + u64_stats_update_end(&stats->syncp); + } return !oom; } static void skb_recv_done(struct virtqueue *rvq) { struct virtnet_info *vi = rvq->vdev->priv; +...