thr3ads.net - Linux Virtualization - [External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi

If this information is useful, please help other people find it:
Share via:

Jason Wang

2023-Mar-24 03:41 UTC

[External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi_tx disable

On Thu, Mar 23, 2023 at 4:01?PM ?? <huangjie.albert at bytedance.com>
wrote:>
> Jason Wang <jasowang at redhat.com> ?2023?3?22??? 10:37???
> >
> > On Tue, Mar 21, 2023 at 5:00?PM Albert Huang
> > <huangjie.albert at bytedance.com> wrote:
> > >
> > > From: "huangjie.albert" <huangjie.albert at
bytedance.com>
> > >
> > > fix commit 8d622d21d248 ("virtio: fix up
virtio_disable_cb")
> > >
> > > if we disable the napi_tx. when we triger a tx interrupt, the
> >
> > typo should be "trigger"
> >
>
> OK, thanks for this. I will correct it in the next version
>
> > > vq->event_triggered will be set to true. It will no longer be
> > > set to false. Unless we explicitly call
virtqueue_enable_cb_delayed
> > > or virtqueue_enable_cb_prepare
> > >
> > > if we disable the napi_tx, It will only be called when the tx
ring
> > > buffer is relatively small:
> > > virtio_net->start_xmit:
> > >         if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> > >                 netif_stop_subqueue(dev, qnum);
> > >                 if (!use_napi &&
> > >                    
unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> > >                         /* More just got used, free them then
recheck. */
> > >                         free_old_xmit_skbs(sq, false);
> > >                         if (sq->vq->num_free >=
2+MAX_SKB_FRAGS) {
> > >                                 netif_start_subqueue(dev, qnum);
> > >                                 virtqueue_disable_cb(sq->vq);
> > >                         }
> >
> > The code example here is out of date, make sure your tree has this:
>
> also, I will correct it in the next version?this is from kernel 5.15.
>
> >
> > commit d71ebe8114b4bf622804b810f5e274069060a174
> > Author: Jason Wang <jasowang at redhat.com>
> > Date:   Tue Jan 17 11:47:07 2023 +0800
> >
> >     virtio-net: correctly enable callback during start_xmit
> >
> > >                 }
> > >         }
> > > Because event_triggered is true.Therefore,
VRING_AVAIL_F_NO_INTERRUPT or
> > > VRING_PACKED_EVENT_FLAG_DISABLE will not be set.So we update
> > > vring_used_event(&vq->split.vring) or
vq->packed.vring.driver->off_wrap
> > > every time we call virtqueue_get_buf_ctx.This will bring more
interruptions.
> >
> > Can you please post how to test with the performance numbers?
> >
>
> iperf3 tcp stream:
> vm1 -----------------> vm2
> vm2 just receive tcp data stream from vm1, and send the ack to vm1,
> there are so
> many tx interruptions  in vm2.
>
> but without event_triggered there are just a few tx interruptions.
>
> > >
> > > if event_triggered is set to true, do not update
vring_used_event(&vq->split.vring)
> > > or vq->packed.vring.driver->off_wrap
> > >
> > > Signed-off-by: huangjie.albert <huangjie.albert at
bytedance.com>
> > > ---
> > >  drivers/virtio/virtio_ring.c | 6 ++++--
> > >  1 file changed, 4 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/virtio/virtio_ring.c
b/drivers/virtio/virtio_ring.c
> > > index 307e139cb11d..f486cccadbeb 100644
> > > --- a/drivers/virtio/virtio_ring.c
> > > +++ b/drivers/virtio/virtio_ring.c
> > > @@ -795,7 +795,8 @@ static void
*virtqueue_get_buf_ctx_split(struct virtqueue *_vq,
> > >         /* If we expect an interrupt for the next entry, tell
host
> > >          * by writing event index and flush out the write before
> > >          * the read in the next get_buf call. */
> > > -       if (!(vq->split.avail_flags_shadow &
VRING_AVAIL_F_NO_INTERRUPT))
> > > +       if (!(vq->split.avail_flags_shadow &
VRING_AVAIL_F_NO_INTERRUPT)
> > > +                       && (vq->event_triggered ==
false))
> >
> > I'm not sure this can work, when event_triggered is true it means
> > we've got an interrupt, in this case if we want another interrupt
for
> > the next entry, we should update used_event otherwise we will lose
> > that interrupt?
> >
> > Thanks
>
> Normally, if we receive an interrupt, we should disable the interrupt
> in the interrupt callback handler.
So the problem is:

1) event_triggered was set to true in vring_interrupt()
2) after this nothing will happen for virtqueue_disable_cb() so
VRING_AVAIL_F_NO_INTERRUPT is not set in avail_flags_shadow
3) virtqueue_get_buf_ctx_split() will still think the cb is enabled
then it tries to publish new event

This makes me think about whether or not we really need
event_triggered. The assumption in the virtqueue_disable_cb() seems
wrong:

/* If device triggered an event already it won't trigger one again:
 * no need to disable.
 */
if (vq->event_triggered)
                return;

This is wrong if there's no event index support. And the
event_triggered is somehow duplicated with the
VRING_AVAIL_F_NO_INTERRUPT in the case of event index. The correct fix
might be:

1) remove event_triggered
2) set VRING_AVAIL_F_NO_INTERRUPT in avail_flags_shadow in
vring_interrrupt if event index is supported

?

Thanks

> But because of the introduction of event_triggered, here,
> virtqueue_get_buf_ctx_split  cannot be recognized
> that the interrupt has been turned off.
>
> if we want  another interrupt for the next entry, We should probably
> call virtqueue_enable_cb?
>
> Thanks
>
> >
> > >                 virtio_store_mb(vq->weak_barriers,
> > >                                
&vring_used_event(&vq->split.vring),
> > >                                 cpu_to_virtio16(_vq->vdev,
vq->last_used_idx));
> > > @@ -1529,7 +1530,8 @@ static void
*virtqueue_get_buf_ctx_packed(struct virtqueue *_vq,
> > >          * by writing event index and flush out the write before
> > >          * the read in the next get_buf call.
> > >          */
> > > -       if (vq->packed.event_flags_shadow ==
VRING_PACKED_EVENT_FLAG_DESC)
> > > +       if (vq->packed.event_flags_shadow ==
VRING_PACKED_EVENT_FLAG_DESC
> > > +                       && (vq->event_triggered ==
false))
> > >                 virtio_store_mb(vq->weak_barriers,
> > >                                
&vq->packed.vring.driver->off_wrap,
> > >                                
cpu_to_le16(vq->last_used_idx));
> > > --
> > > 2.31.1
> > >
> >
>

Michael S. Tsirkin

2023-Mar-24 05:59 UTC

head link

[External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi_tx disable

On Fri, Mar 24, 2023 at 11:41:12AM +0800, Jason Wang
wrote:> On Thu, Mar 23, 2023 at 4:01?PM ?? <huangjie.albert at bytedance.com>
wrote:
> >
> > Jason Wang <jasowang at redhat.com> ?2023?3?22??? 10:37???
> > >
> > > On Tue, Mar 21, 2023 at 5:00?PM Albert Huang
> > > <huangjie.albert at bytedance.com> wrote:
> > > >
> > > > From: "huangjie.albert" <huangjie.albert at
bytedance.com>
> > > >
> > > > fix commit 8d622d21d248 ("virtio: fix up
virtio_disable_cb")
> > > >
> > > > if we disable the napi_tx. when we triger a tx interrupt,
the
> > >
> > > typo should be "trigger"
> > >
> >
> > OK, thanks for this. I will correct it in the next version
> >
> > > > vq->event_triggered will be set to true. It will no
longer be
> > > > set to false. Unless we explicitly call
virtqueue_enable_cb_delayed
> > > > or virtqueue_enable_cb_prepare
> > > >
> > > > if we disable the napi_tx, It will only be called when the
tx ring
> > > > buffer is relatively small:
> > > > virtio_net->start_xmit:
> > > >         if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> > > >                 netif_stop_subqueue(dev, qnum);
> > > >                 if (!use_napi &&
> > > >                    
unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> > > >                         /* More just got used, free them
then recheck. */
> > > >                         free_old_xmit_skbs(sq, false);
> > > >                         if (sq->vq->num_free >=
2+MAX_SKB_FRAGS) {
> > > >                                 netif_start_subqueue(dev,
qnum);
> > > >                                
virtqueue_disable_cb(sq->vq);
> > > >                         }
> > >
> > > The code example here is out of date, make sure your tree has
this:
> >
> > also, I will correct it in the next version?this is from kernel 5.15.
> >
> > >
> > > commit d71ebe8114b4bf622804b810f5e274069060a174
> > > Author: Jason Wang <jasowang at redhat.com>
> > > Date:   Tue Jan 17 11:47:07 2023 +0800
> > >
> > >     virtio-net: correctly enable callback during start_xmit
> > >
> > > >                 }
> > > >         }
> > > > Because event_triggered is true.Therefore,
VRING_AVAIL_F_NO_INTERRUPT or
> > > > VRING_PACKED_EVENT_FLAG_DISABLE will not be set.So we update
> > > > vring_used_event(&vq->split.vring) or
vq->packed.vring.driver->off_wrap
> > > > every time we call virtqueue_get_buf_ctx.This will bring
more interruptions.
> > >
> > > Can you please post how to test with the performance numbers?
> > >
> >
> > iperf3 tcp stream:
> > vm1 -----------------> vm2
> > vm2 just receive tcp data stream from vm1, and send the ack to vm1,
> > there are so
> > many tx interruptions  in vm2.
> >
> > but without event_triggered there are just a few tx interruptions.
> >
> > > >
> > > > if event_triggered is set to true, do not update
vring_used_event(&vq->split.vring)
> > > > or vq->packed.vring.driver->off_wrap
> > > >
> > > > Signed-off-by: huangjie.albert <huangjie.albert at
bytedance.com>
> > > > ---
> > > >  drivers/virtio/virtio_ring.c | 6 ++++--
> > > >  1 file changed, 4 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/drivers/virtio/virtio_ring.c
b/drivers/virtio/virtio_ring.c
> > > > index 307e139cb11d..f486cccadbeb 100644
> > > > --- a/drivers/virtio/virtio_ring.c
> > > > +++ b/drivers/virtio/virtio_ring.c
> > > > @@ -795,7 +795,8 @@ static void
*virtqueue_get_buf_ctx_split(struct virtqueue *_vq,
> > > >         /* If we expect an interrupt for the next entry,
tell host
> > > >          * by writing event index and flush out the write
before
> > > >          * the read in the next get_buf call. */
> > > > -       if (!(vq->split.avail_flags_shadow &
VRING_AVAIL_F_NO_INTERRUPT))
> > > > +       if (!(vq->split.avail_flags_shadow &
VRING_AVAIL_F_NO_INTERRUPT)
> > > > +                       && (vq->event_triggered
== false))
> > >
> > > I'm not sure this can work, when event_triggered is true it
means
> > > we've got an interrupt, in this case if we want another
interrupt for
> > > the next entry, we should update used_event otherwise we will
lose
> > > that interrupt?
> > >
> > > Thanks
> >
> > Normally, if we receive an interrupt, we should disable the interrupt
> > in the interrupt callback handler.
> 
> So the problem is:
> 
> 1) event_triggered was set to true in vring_interrupt()
>
> 2) after this nothing will happen for virtqueue_disable_cb() so
> VRING_AVAIL_F_NO_INTERRUPT is not set in avail_flags_shadow
> 3) virtqueue_get_buf_ctx_split() will still think the cb is enabled
> then it tries to publish new event
Oh. Good point! I think when I wrote up
8d622d21d248 ("virtio: fix up virtio_disable_cb")
I missed this corner case.


> This makes me think about whether or not we really need
> event_triggered. The assumption in the virtqueue_disable_cb() seems
> wrong:
> 
> /* If device triggered an event already it won't trigger one again:
>  * no need to disable.
>  */
> if (vq->event_triggered)
>                 return;
> 
> This is wrong if there's no event index support.

I don't get it.  how does this get triggered?

You are talking about device without event index?
Here's code from vring_interrupt():

        /* Just a hint for performance: so it's ok that this can be racy! */
        if (vq->event)
                vq->event_triggered = true;



> And the
> event_triggered is somehow duplicated with the
> VRING_AVAIL_F_NO_INTERRUPT in the case of event index. The correct fix
> might be:
> 
> 1) remove event_triggered
> 2) set VRING_AVAIL_F_NO_INTERRUPT in avail_flags_shadow in
> vring_interrrupt if event index is supported
> 
> ?
> 
> Thanks
I am not sure all this is right and I'd rather we focused
performance/correctness and cleanups separately.



> 
> > But because of the introduction of event_triggered, here,
> > virtqueue_get_buf_ctx_split  cannot be recognized
> > that the interrupt has been turned off.
> >
> > if we want  another interrupt for the next entry, We should probably
> > call virtqueue_enable_cb?
> >
> > Thanks
> >
> > >
> > > >                 virtio_store_mb(vq->weak_barriers,
> > > >                                
&vring_used_event(&vq->split.vring),
> > > >                                
cpu_to_virtio16(_vq->vdev, vq->last_used_idx));
> > > > @@ -1529,7 +1530,8 @@ static void
*virtqueue_get_buf_ctx_packed(struct virtqueue *_vq,
> > > >          * by writing event index and flush out the write
before
> > > >          * the read in the next get_buf call.
> > > >          */
> > > > -       if (vq->packed.event_flags_shadow ==
VRING_PACKED_EVENT_FLAG_DESC)
> > > > +       if (vq->packed.event_flags_shadow ==
VRING_PACKED_EVENT_FLAG_DESC
> > > > +                       && (vq->event_triggered
== false))
> > > >                 virtio_store_mb(vq->weak_barriers,
> > > >                                
&vq->packed.vring.driver->off_wrap,
> > > >                                
cpu_to_le16(vq->last_used_idx));
> > > > --
> > > > 2.31.1
> > > >
> > >
> >

Possibly Parallel Threads

Search for more reasonably related threads

Linux Virtualization - Mar 2023 - [External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi_tx disable

[External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi_tx disable

[External] Re: [PATCH] virtio_ring: Suppress tx interrupt when napi_tx disable

Possibly Parallel Threads