Linux Virtualization - Aug 2018 - [PATCH net-next v7 3/4] net: vhost: factor out busy polling logic to vhost_net_busy_poll()

On 2018?08?03? 11:24, Tonghao Zhang wrote:
> On Fri, Aug 3, 2018 at 11:07 AM Jason Wang <jasowang at redhat.com>
wrote:
>>
>>
>> On 2018?08?03? 10:51, Tonghao Zhang wrote:
>>> On Thu, Aug 2, 2018 at 5:23 PM Jason Wang <jasowang at
redhat.com> wrote:
>>>>
>>>> On 2018?08?02? 16:41, Toshiaki Makita wrote:
>>>>> On 2018/08/02 17:18, Jason Wang wrote:
>>>>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>>>>>> +static void vhost_net_busy_poll_check(struct
vhost_net *net,
>>>>>>>> +                                   struct
vhost_virtqueue *rvq,
>>>>>>>> +                                   struct
vhost_virtqueue *tvq,
>>>>>>>> +                                   bool rx)
>>>>>>>> +{
>>>>>>>> +     struct socket *sock =
rvq->private_data;
>>>>>>>> +
>>>>>>>> +     if (rx)
>>>>>>>> +            
vhost_net_busy_poll_try_queue(net, tvq);
>>>>>>>> +     else if (sock &&
sk_has_rx_data(sock->sk))
>>>>>>>> +            
vhost_net_busy_poll_try_queue(net, rvq);
>>>>>>>> +     else {
>>>>>>>> +             /* On tx here, sock has no rx
data, so we
>>>>>>>> +              * will wait for sock wakeup for
rx, and
>>>>>>>> +              * vhost_enable_notify() is not
needed. */
>>>>>>> A possible case is we do have rx data but guest
does not refill the rx
>>>>>>> queue. In this case we may lose notifications from
guest.
>>>>>> Yes, should consider this case. thanks.
>>>>> I'm a bit confused. Isn't this covered by the
previous
>>>>> "else if (sock && sk_has_rx_data(...))"
block?
>>>> The problem is it does nothing if vhost_vq_avail_empty() is
true and
>>>> vhost_enble_notify() is false.
>>>>
>>>>>>>>> +
>>>>>>>>> +             cpu_relax();
>>>>>>>>> +     }
>>>>>>>>> +
>>>>>>>>> +     preempt_enable();
>>>>>>>>> +
>>>>>>>>> +     if (!rx)
>>>>>>>>> +             vhost_net_enable_vq(net, vq);
>>>>>>>> No need to enable rx virtqueue, if we are sure
handle_rx() will be
>>>>>>>> called soon.
>>>>>>> If we disable rx virtqueue in handle_tx and
don't send packets from
>>>>>>> guest anymore(handle_tx is not called), so we can
wake up for sock rx.
>>>>>>> so the network is broken.
>>>>>> Not sure I understand here. I mean is we schedule work
for handle_rx(),
>>>>>> there's no need to enable it since handle_rx() will
do this for us.
>>>>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
>>>>> need to enable vq since in that case we have no rx data and
handle_rx()
>>>>> is not scheduled.
>>>>>
>>>> Yes.
>>> So we will use the vhost_has_work() to check whether or not the
>>> handle_rx is scheduled ?
>>> If we use the vhost_has_work(), the work in the dev work_list may
be
>>> rx work, or tx work, right ?
>> Yes. We can add a boolean to record whether or not we've called
>> vhost_poll_queue() for rvq. And avoid calling vhost_net_enable_vq() if
>> it was true.
> so, the commit be294a51a "vhost_net: Avoid rx queue wake-ups during
busypoll"
> may not consider the case: work is tx work in the dev work list.
So two kinds of work, tx kick or tx wakeup.

For tx kick, we check vhost_vq_avail_empty() and avoid unnecessary kicks 
by not enabling kick if we found something is pending on txq. For tx 
wakeup, yes, the commit does not consider it. And that's why we want to 
disable tx wakeups during busy polling.

Thanks
>
>> So here's the needed changes:
>>
>> 1) Split the wakeup disabling to another patch
>> 2) Squash the vhost_net_busy_poll_try_queue() and
>> vhost_net_busy_poll_check() into vhost_net_busy_poll() and reduce
>> duplicated checks.
>> 3) If possible, rename the boolean rx in vhost_net_busy_poll() to
>> poll_rx, this makes code more readable.
> OK
>> Thanks
>>
>>>> Thanks

On Fri, Aug 3, 2018 at 11:43 AM Jason Wang <jasowang at redhat.com>
wrote:
>
>
>
> On 2018?08?03? 11:24, Tonghao Zhang wrote:
> > On Fri, Aug 3, 2018 at 11:07 AM Jason Wang <jasowang at
redhat.com> wrote:
> >>
> >>
> >> On 2018?08?03? 10:51, Tonghao Zhang wrote:
> >>> On Thu, Aug 2, 2018 at 5:23 PM Jason Wang <jasowang at
redhat.com> wrote:
> >>>>
> >>>> On 2018?08?02? 16:41, Toshiaki Makita wrote:
> >>>>> On 2018/08/02 17:18, Jason Wang wrote:
> >>>>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
> >>>>>>>> +static void
vhost_net_busy_poll_check(struct vhost_net *net,
> >>>>>>>> +                                   struct
vhost_virtqueue *rvq,
> >>>>>>>> +                                   struct
vhost_virtqueue *tvq,
> >>>>>>>> +                                   bool
rx)
> >>>>>>>> +{
> >>>>>>>> +     struct socket *sock =
rvq->private_data;
> >>>>>>>> +
> >>>>>>>> +     if (rx)
> >>>>>>>> +            
vhost_net_busy_poll_try_queue(net, tvq);
> >>>>>>>> +     else if (sock &&
sk_has_rx_data(sock->sk))
> >>>>>>>> +            
vhost_net_busy_poll_try_queue(net, rvq);
> >>>>>>>> +     else {
> >>>>>>>> +             /* On tx here, sock has no
rx data, so we
> >>>>>>>> +              * will wait for sock wakeup
for rx, and
> >>>>>>>> +              * vhost_enable_notify() is
not needed. */
> >>>>>>> A possible case is we do have rx data but
guest does not refill the rx
> >>>>>>> queue. In this case we may lose notifications
from guest.
> >>>>>> Yes, should consider this case. thanks.
> >>>>> I'm a bit confused. Isn't this covered by the
previous
> >>>>> "else if (sock &&
sk_has_rx_data(...))" block?
> >>>> The problem is it does nothing if vhost_vq_avail_empty()
is true and
> >>>> vhost_enble_notify() is false.
> >>>>
> >>>>>>>>> +
> >>>>>>>>> +             cpu_relax();
> >>>>>>>>> +     }
> >>>>>>>>> +
> >>>>>>>>> +     preempt_enable();
> >>>>>>>>> +
> >>>>>>>>> +     if (!rx)
> >>>>>>>>> +             vhost_net_enable_vq(net,
vq);
> >>>>>>>> No need to enable rx virtqueue, if we are
sure handle_rx() will be
> >>>>>>>> called soon.
> >>>>>>> If we disable rx virtqueue in handle_tx and
don't send packets from
> >>>>>>> guest anymore(handle_tx is not called), so we
can wake up for sock rx.
> >>>>>>> so the network is broken.
> >>>>>> Not sure I understand here. I mean is we schedule
work for handle_rx(),
> >>>>>> there's no need to enable it since handle_rx()
will do this for us.
> >>>>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
> >>>>> need to enable vq since in that case we have no rx
data and handle_rx()
> >>>>> is not scheduled.
> >>>>>
> >>>> Yes.
> >>> So we will use the vhost_has_work() to check whether or not
the
> >>> handle_rx is scheduled ?
> >>> If we use the vhost_has_work(), the work in the dev work_list
may be
> >>> rx work, or tx work, right ?
> >> Yes. We can add a boolean to record whether or not we've
called
> >> vhost_poll_queue() for rvq. And avoid calling
vhost_net_enable_vq() if
> >> it was true.
> > so, the commit be294a51a "vhost_net: Avoid rx queue wake-ups
during busypoll"
> > may not consider the case: work is tx work in the dev work list.
>
> So two kinds of work, tx kick or tx wakeup.
>
> For tx kick, we check vhost_vq_avail_empty() and avoid unnecessary kicks
> by not enabling kick if we found something is pending on txq. For tx
> wakeup, yes, the commit does not consider it. And that's why we want to
> disable tx wakeups during busy polling.
And in the handle_rx but not busy polling, the tx can wakeup anytime
and the tx work will be added to dev work list. In that case, if we
add
the rx poll to the queue, it is necessary ? the commit be294a51a may
check whether the rx work is in the dev work list.
> Thanks
>
> >
> >> So here's the needed changes:
> >>
> >> 1) Split the wakeup disabling to another patch
> >> 2) Squash the vhost_net_busy_poll_try_queue() and
> >> vhost_net_busy_poll_check() into vhost_net_busy_poll() and reduce
> >> duplicated checks.
> >> 3) If possible, rename the boolean rx in vhost_net_busy_poll() to
> >> poll_rx, this makes code more readable.
> > OK
> >> Thanks
> >>
> >>>> Thanks
>

On 2018?08?03? 12:04, Tonghao Zhang wrote:
> On Fri, Aug 3, 2018 at 11:43 AM Jason Wang<jasowang at redhat.com> 
wrote:
>>
>> On 2018?08?03? 11:24, Tonghao Zhang wrote:
>>> On Fri, Aug 3, 2018 at 11:07 AM Jason Wang<jasowang at
redhat.com>  wrote:
>>>> On 2018?08?03? 10:51, Tonghao Zhang wrote:
>>>>> On Thu, Aug 2, 2018 at 5:23 PM Jason Wang<jasowang at
redhat.com>  wrote:
>>>>>> On 2018?08?02? 16:41, Toshiaki Makita wrote:
>>>>>>> On 2018/08/02 17:18, Jason Wang wrote:
>>>>>>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>>>>>>>> +static void
vhost_net_busy_poll_check(struct vhost_net *net,
>>>>>>>>>> +                                  
struct vhost_virtqueue *rvq,
>>>>>>>>>> +                                  
struct vhost_virtqueue *tvq,
>>>>>>>>>> +                                  
bool rx)
>>>>>>>>>> +{
>>>>>>>>>> +     struct socket *sock =
rvq->private_data;
>>>>>>>>>> +
>>>>>>>>>> +     if (rx)
>>>>>>>>>> +            
vhost_net_busy_poll_try_queue(net, tvq);
>>>>>>>>>> +     else if (sock &&
sk_has_rx_data(sock->sk))
>>>>>>>>>> +            
vhost_net_busy_poll_try_queue(net, rvq);
>>>>>>>>>> +     else {
>>>>>>>>>> +             /* On tx here, sock has
no rx data, so we
>>>>>>>>>> +              * will wait for sock
wakeup for rx, and
>>>>>>>>>> +              * vhost_enable_notify()
is not needed. */
>>>>>>>>> A possible case is we do have rx data but
guest does not refill the rx
>>>>>>>>> queue. In this case we may lose
notifications from guest.
>>>>>>>> Yes, should consider this case. thanks.
>>>>>>> I'm a bit confused. Isn't this covered by
the previous
>>>>>>> "else if (sock &&
sk_has_rx_data(...))" block?
>>>>>> The problem is it does nothing if
vhost_vq_avail_empty() is true and
>>>>>> vhost_enble_notify() is false.
>>>>>>
>>>>>>>>>>> +
>>>>>>>>>>> +             cpu_relax();
>>>>>>>>>>> +     }
>>>>>>>>>>> +
>>>>>>>>>>> +     preempt_enable();
>>>>>>>>>>> +
>>>>>>>>>>> +     if (!rx)
>>>>>>>>>>> +            
vhost_net_enable_vq(net, vq);
>>>>>>>>>> No need to enable rx virtqueue, if we
are sure handle_rx() will be
>>>>>>>>>> called soon.
>>>>>>>>> If we disable rx virtqueue in handle_tx and
don't send packets from
>>>>>>>>> guest anymore(handle_tx is not called), so
we can wake up for sock rx.
>>>>>>>>> so the network is broken.
>>>>>>>> Not sure I understand here. I mean is we
schedule work for handle_rx(),
>>>>>>>> there's no need to enable it since
handle_rx() will do this for us.
>>>>>>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
>>>>>>> need to enable vq since in that case we have no rx
data and handle_rx()
>>>>>>> is not scheduled.
>>>>>>>
>>>>>> Yes.
>>>>> So we will use the vhost_has_work() to check whether or not
the
>>>>> handle_rx is scheduled ?
>>>>> If we use the vhost_has_work(), the work in the dev
work_list may be
>>>>> rx work, or tx work, right ?
>>>> Yes. We can add a boolean to record whether or not we've
called
>>>> vhost_poll_queue() for rvq. And avoid calling
vhost_net_enable_vq() if
>>>> it was true.
>>> so, the commit be294a51a "vhost_net: Avoid rx queue wake-ups
during busypoll"
>>> may not consider the case: work is tx work in the dev work list.
>> So two kinds of work, tx kick or tx wakeup.
>>
>> For tx kick, we check vhost_vq_avail_empty() and avoid unnecessary
kicks
>> by not enabling kick if we found something is pending on txq. For tx
>> wakeup, yes, the commit does not consider it. And that's why we
want to
>> disable tx wakeups during busy polling.
> And in the handle_rx but not busy polling, the tx can wakeup anytime
> and the tx work will be added to dev work list. In that case, if we
> add
> the rx poll to the queue, it is necessary ? the commit be294a51a may
> check whether the rx work is in the dev work list.
I think the point this we don't poll rx during tx at that time. So if rx 
poll is interrupted, we should reschedule handle_rx(). After we poll rx 
on handle_tx(), we can try to optimize this on top.

Thanks