Linux Virtualization - Apr 2016 - Potential hang in virtnet_send_command

Hi,
I was debugging an issue caused by a bad usage of dpdk.
That is fixed in the meantime and I'll backport that into our code as well.

But along debugging that, I found a potential hang in virtnet_send_command
that my case ran into.
The following code can become an infinite loop:
/* Spin for a response, the kick causes an ioport write, trapping
 * into the hypervisor, so the request should be handled immediately.
 */
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
       !virtqueue_is_broken(vi->cvq))
        cpu_relax();

In my case dpdk broke something - not exactly clear what - and due to that
following calls through virtnet_send_command ran into this hang.
Effectively it seems that the buffers didn't get refreshed at all anymore.

That said the dpdk issue to touch devices that belong to a kernel owned
driver is fixed, so one could leave the code as is for now.
Yet I wanted to make you aware in case you would vote for a time or retry
based upper limit on that loop to avoid hangs - who knows what else might
bring it in this broken state in a different case.

Kind Regards,
Christian Ehrhardt

P.S.
Steps to reproduce, backtraces and more data can be found in:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
A general setup howto for DPDK in KVM guests which is a prereq is at:
https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-in-guest
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20160418/56cfd37d/attachment-0001.html>

On Mon, Apr 18, 2016 at 07:07:32PM +0200, Christian Ehrhardt
wrote:
> Hi,
> I was debugging an issue caused by a bad usage of dpdk.
> That is fixed in the meantime and I'll backport that into our code as
well.
> 
> But along debugging that, I found a potential hang in virtnet_send_command
that
> my case ran into.
> The following code can become an infinite loop:
> /* Spin for a response, the kick causes an ioport write, trapping
> ?* into the hypervisor, so the request should be handled immediately.
> ?*/ ??
> while (!virtqueue_get_buf(vi->cvq, &tmp) && ??
> ? ? ? ?!virtqueue_is_broken(vi->cvq)) ??
> ? ? ? ? cpu_relax();
> 
> In my case dpdk broke something - not exactly clear what - and due to that
> following calls through?virtnet_send_command ran into this hang.
> Effectively it seems that the buffers didn't get refreshed at all
anymore.
> 
> That said the dpdk issue to touch devices that belong to a kernel owned
driver
> is fixed, so one could leave the code as is for now.
> Yet I wanted to make you aware in case you would vote for a time or retry
based
> upper limit on that loop to avoid hangs - who knows what else might bring
it in
> this broken state in a different case.
> 
> Kind Regards,
> Christian Ehrhardt
> 
> P.S.
> Steps to reproduce, backtraces and more data can be found in:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
> A general setup howto for DPDK in KVM guests which is a prereq is at:
> https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-in-guest
True. It's not that clear how to handle this cleanly though.
Ideally we'd have to transmit the next command once the
previous one completes.

-- 
MST