Linux Virtualization - Aug 2018 - [PATCH net-next v7 3/4] net: vhost: factor out busy polling logic to vhost_net_busy_poll()

On 2018?08?02? 16:41, Toshiaki Makita wrote:
> On 2018/08/02 17:18, Jason Wang wrote:
>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>> +static void vhost_net_busy_poll_check(struct vhost_net *net,
>>>> +                                   struct vhost_virtqueue
*rvq,
>>>> +                                   struct vhost_virtqueue
*tvq,
>>>> +                                   bool rx)
>>>> +{
>>>> +     struct socket *sock = rvq->private_data;
>>>> +
>>>> +     if (rx)
>>>> +             vhost_net_busy_poll_try_queue(net, tvq);
>>>> +     else if (sock && sk_has_rx_data(sock->sk))
>>>> +             vhost_net_busy_poll_try_queue(net, rvq);
>>>> +     else {
>>>> +             /* On tx here, sock has no rx data, so we
>>>> +              * will wait for sock wakeup for rx, and
>>>> +              * vhost_enable_notify() is not needed. */
>>> A possible case is we do have rx data but guest does not refill the
rx
>>> queue. In this case we may lose notifications from guest.
>> Yes, should consider this case. thanks.
> I'm a bit confused. Isn't this covered by the previous
> "else if (sock && sk_has_rx_data(...))" block?
The problem is it does nothing if vhost_vq_avail_empty() is true and 
vhost_enble_notify() is false.
>
>>>>> +
>>>>> +???????????? cpu_relax();
>>>>> +???? }
>>>>> +
>>>>> +???? preempt_enable();
>>>>> +
>>>>> +???? if (!rx)
>>>>> +???????????? vhost_net_enable_vq(net, vq);
>>>> No need to enable rx virtqueue, if we are sure handle_rx() will
be
>>>> called soon.
>>> If we disable rx virtqueue in handle_tx and don't send packets
from
>>> guest anymore(handle_tx is not called), so we can wake up for sock
rx.
>>> so the network is broken.
>> Not sure I understand here. I mean is we schedule work for handle_rx(),
>> there's no need to enable it since handle_rx() will do this for us.
> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
> need to enable vq since in that case we have no rx data and handle_rx()
> is not scheduled.
>
Yes.

Thanks

On 2018/08/02 18:23, Jason Wang wrote:
> On 2018?08?02? 16:41, Toshiaki Makita wrote:
>> On 2018/08/02 17:18, Jason Wang wrote:
>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>>> +static void vhost_net_busy_poll_check(struct vhost_net
*net,
>>>>> +?????????????????????????????????? struct vhost_virtqueue
*rvq,
>>>>> +?????????????????????????????????? struct vhost_virtqueue
*tvq,
>>>>> +?????????????????????????????????? bool rx)
>>>>> +{
>>>>> +???? struct socket *sock = rvq->private_data;
>>>>> +
>>>>> +???? if (rx)
>>>>> +???????????? vhost_net_busy_poll_try_queue(net, tvq);
>>>>> +???? else if (sock && sk_has_rx_data(sock->sk))
>>>>> +???????????? vhost_net_busy_poll_try_queue(net, rvq);
>>>>> +???? else {
>>>>> +???????????? /* On tx here, sock has no rx data, so we
>>>>> +????????????? * will wait for sock wakeup for rx, and
>>>>> +????????????? * vhost_enable_notify() is not needed. */
>>>> A possible case is we do have rx data but guest does not refill
the rx
>>>> queue. In this case we may lose notifications from guest.
>>> Yes, should consider this case. thanks.
>> I'm a bit confused. Isn't this covered by the previous
>> "else if (sock && sk_has_rx_data(...))" block?
> 
> The problem is it does nothing if vhost_vq_avail_empty() is true and
> vhost_enble_notify() is false.
If vhost_enable_notify() is false, guest will eventually kicks vq, no?

-- 
Toshiaki Makita

On 2018?08?02? 17:57, Toshiaki Makita wrote:
> On 2018/08/02 18:23, Jason Wang wrote:
>> On 2018?08?02? 16:41, Toshiaki Makita wrote:
>>> On 2018/08/02 17:18, Jason Wang wrote:
>>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>>>> +static void vhost_net_busy_poll_check(struct vhost_net
*net,
>>>>>> +?????????????????????????????????? struct
vhost_virtqueue *rvq,
>>>>>> +?????????????????????????????????? struct
vhost_virtqueue *tvq,
>>>>>> +?????????????????????????????????? bool rx)
>>>>>> +{
>>>>>> +???? struct socket *sock = rvq->private_data;
>>>>>> +
>>>>>> +???? if (rx)
>>>>>> +???????????? vhost_net_busy_poll_try_queue(net, tvq);
>>>>>> +???? else if (sock &&
sk_has_rx_data(sock->sk))
>>>>>> +???????????? vhost_net_busy_poll_try_queue(net, rvq);
>>>>>> +???? else {
>>>>>> +???????????? /* On tx here, sock has no rx data, so we
>>>>>> +????????????? * will wait for sock wakeup for rx, and
>>>>>> +????????????? * vhost_enable_notify() is not needed.
*/
>>>>> A possible case is we do have rx data but guest does not
refill the rx
>>>>> queue. In this case we may lose notifications from guest.
>>>> Yes, should consider this case. thanks.
>>> I'm a bit confused. Isn't this covered by the previous
>>> "else if (sock && sk_has_rx_data(...))" block?
>> The problem is it does nothing if vhost_vq_avail_empty() is true and
>> vhost_enble_notify() is false.
> If vhost_enable_notify() is false, guest will eventually kicks vq, no?
>
Yes, you are right.

Thanks

On Thu, Aug 2, 2018 at 5:23 PM Jason Wang <jasowang at redhat.com>
wrote:
>
>
>
> On 2018?08?02? 16:41, Toshiaki Makita wrote:
> > On 2018/08/02 17:18, Jason Wang wrote:
> >> On 2018?08?01? 17:52, Tonghao Zhang wrote:
> >>>> +static void vhost_net_busy_poll_check(struct vhost_net
*net,
> >>>> +                                   struct vhost_virtqueue
*rvq,
> >>>> +                                   struct vhost_virtqueue
*tvq,
> >>>> +                                   bool rx)
> >>>> +{
> >>>> +     struct socket *sock = rvq->private_data;
> >>>> +
> >>>> +     if (rx)
> >>>> +             vhost_net_busy_poll_try_queue(net, tvq);
> >>>> +     else if (sock &&
sk_has_rx_data(sock->sk))
> >>>> +             vhost_net_busy_poll_try_queue(net, rvq);
> >>>> +     else {
> >>>> +             /* On tx here, sock has no rx data, so we
> >>>> +              * will wait for sock wakeup for rx, and
> >>>> +              * vhost_enable_notify() is not needed. */
> >>> A possible case is we do have rx data but guest does not
refill the rx
> >>> queue. In this case we may lose notifications from guest.
> >> Yes, should consider this case. thanks.
> > I'm a bit confused. Isn't this covered by the previous
> > "else if (sock && sk_has_rx_data(...))" block?
>
> The problem is it does nothing if vhost_vq_avail_empty() is true and
> vhost_enble_notify() is false.
>
> >
> >>>>> +
> >>>>> +             cpu_relax();
> >>>>> +     }
> >>>>> +
> >>>>> +     preempt_enable();
> >>>>> +
> >>>>> +     if (!rx)
> >>>>> +             vhost_net_enable_vq(net, vq);
> >>>> No need to enable rx virtqueue, if we are sure handle_rx()
will be
> >>>> called soon.
> >>> If we disable rx virtqueue in handle_tx and don't send
packets from
> >>> guest anymore(handle_tx is not called), so we can wake up for
sock rx.
> >>> so the network is broken.
> >> Not sure I understand here. I mean is we schedule work for
handle_rx(),
> >> there's no need to enable it since handle_rx() will do this
for us.
> > Looks like in the last "else" block in
vhost_net_busy_poll_check() we
> > need to enable vq since in that case we have no rx data and
handle_rx()
> > is not scheduled.
> >
>
> Yes.
So we will use the vhost_has_work() to check whether or not the
handle_rx is scheduled ?
If we use the vhost_has_work(), the work in the dev work_list may be
rx work, or tx work, right ?
> Thanks

On 2018?08?02? 17:23, Jason Wang wrote:
>>>>>>
>>>>> No need to enable rx virtqueue, if we are sure handle_rx()
will be
>>>>> called soon.
>>>> If we disable rx virtqueue in handle_tx and don't send
packets from
>>>> guest anymore(handle_tx is not called), so we can wake up for
sock rx.
>>>> so the network is broken.
>>> Not sure I understand here. I mean is we schedule work for
handle_rx(),
>>> there's no need to enable it since handle_rx() will do this for
us.
>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
>> need to enable vq since in that case we have no rx data and handle_rx()
>> is not scheduled.
>>
>
> Yes.
>
> Thanks 
Rethink about this, looks not. We enable rx wakeups in this case, so if 
there's pending data, handle_rx() will be schedule after 
vhost_net_enable_vq().

Thanks

On 2018?08?03? 10:51, Tonghao Zhang wrote:
> On Thu, Aug 2, 2018 at 5:23 PM Jason Wang <jasowang at redhat.com>
wrote:
>>
>>
>> On 2018?08?02? 16:41, Toshiaki Makita wrote:
>>> On 2018/08/02 17:18, Jason Wang wrote:
>>>> On 2018?08?01? 17:52, Tonghao Zhang wrote:
>>>>>> +static void vhost_net_busy_poll_check(struct vhost_net
*net,
>>>>>> +                                   struct
vhost_virtqueue *rvq,
>>>>>> +                                   struct
vhost_virtqueue *tvq,
>>>>>> +                                   bool rx)
>>>>>> +{
>>>>>> +     struct socket *sock = rvq->private_data;
>>>>>> +
>>>>>> +     if (rx)
>>>>>> +             vhost_net_busy_poll_try_queue(net, tvq);
>>>>>> +     else if (sock &&
sk_has_rx_data(sock->sk))
>>>>>> +             vhost_net_busy_poll_try_queue(net, rvq);
>>>>>> +     else {
>>>>>> +             /* On tx here, sock has no rx data, so we
>>>>>> +              * will wait for sock wakeup for rx, and
>>>>>> +              * vhost_enable_notify() is not needed.
*/
>>>>> A possible case is we do have rx data but guest does not
refill the rx
>>>>> queue. In this case we may lose notifications from guest.
>>>> Yes, should consider this case. thanks.
>>> I'm a bit confused. Isn't this covered by the previous
>>> "else if (sock && sk_has_rx_data(...))" block?
>> The problem is it does nothing if vhost_vq_avail_empty() is true and
>> vhost_enble_notify() is false.
>>
>>>>>>> +
>>>>>>> +             cpu_relax();
>>>>>>> +     }
>>>>>>> +
>>>>>>> +     preempt_enable();
>>>>>>> +
>>>>>>> +     if (!rx)
>>>>>>> +             vhost_net_enable_vq(net, vq);
>>>>>> No need to enable rx virtqueue, if we are sure
handle_rx() will be
>>>>>> called soon.
>>>>> If we disable rx virtqueue in handle_tx and don't send
packets from
>>>>> guest anymore(handle_tx is not called), so we can wake up
for sock rx.
>>>>> so the network is broken.
>>>> Not sure I understand here. I mean is we schedule work for
handle_rx(),
>>>> there's no need to enable it since handle_rx() will do this
for us.
>>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
>>> need to enable vq since in that case we have no rx data and
handle_rx()
>>> is not scheduled.
>>>
>> Yes.
> So we will use the vhost_has_work() to check whether or not the
> handle_rx is scheduled ?
> If we use the vhost_has_work(), the work in the dev work_list may be
> rx work, or tx work, right ?
Yes. We can add a boolean to record whether or not we've called 
vhost_poll_queue() for rvq. And avoid calling vhost_net_enable_vq() if 
it was true.

So here's the needed changes:

1) Split the wakeup disabling to another patch
2) Squash the vhost_net_busy_poll_try_queue() and 
vhost_net_busy_poll_check() into vhost_net_busy_poll() and reduce 
duplicated checks.
3) If possible, rename the boolean rx in vhost_net_busy_poll() to 
poll_rx, this makes code more readable.

Thanks
>> Thanks

On 2018/08/03 12:07, Jason Wang wrote:
> On 2018?08?02? 17:23, Jason Wang wrote:
>>>>>>>
>>>>>> No need to enable rx virtqueue, if we are sure
handle_rx() will be
>>>>>> called soon.
>>>>> If we disable rx virtqueue in handle_tx and don't send
packets from
>>>>> guest anymore(handle_tx is not called), so we can wake up
for sock rx.
>>>>> so the network is broken.
>>>> Not sure I understand here. I mean is we schedule work for
handle_rx(),
>>>> there's no need to enable it since handle_rx() will do this
for us.
>>> Looks like in the last "else" block in
vhost_net_busy_poll_check() we
>>> need to enable vq since in that case we have no rx data and
handle_rx()
>>> is not scheduled.
>>>
> Rethink about this, looks not. We enable rx wakeups in this case, so if
> there's pending data, handle_rx() will be schedule after
> vhost_net_enable_vq().
You are right, but what I wanted to say is vhost_net_enable_vq() should
be needed (I was talking about what would happen if
vhost_net_enable_vq() were removed). Also, I think we should move
vhost_net_enable_vq() from vhost_net_busy_poll() to this last "else"
block because this is the case where rx wakeups is required.
Anyway this part will be refactored so let's see what this code will
look like in next version.

-- 
Toshiaki Makita