similar to: virtio_net: another race with virtio_net and enable_cb

There is a race in virtio_net, dealing with disabling/enabling the callback. I saw the following oops: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:218! illegal operation: 0001 [#1] SMP Modules linked in: sunrpc dm_mod CPU: 2 Not tainted 2.6.25-rc1zlive-host-10623-gd358142-dirty #99 Process swapper (pid: 0, task: 000000000f85a610, ksp: 000000000f873c60) Krnl PSW : 0404300180000000

There is a race in virtio_net, dealing with disabling/enabling the callback. I saw the following oops: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:218! illegal operation: 0001 [#1] SMP Modules linked in: sunrpc dm_mod CPU: 2 Not tainted 2.6.25-rc1zlive-host-10623-gd358142-dirty #99 Process swapper (pid: 0, task: 000000000f85a610, ksp: 000000000f873c60) Krnl PSW : 0404300180000000

Signed-off-by: Dor Laor <dor.laor at qumranet.com> --- hack-module.awk | 28 +++++++++++++++++++++++++++- 1 files changed, 27 insertions(+), 1 deletions(-) diff --git a/hack-module.awk b/hack-module.awk index 580090b..bd63fb9 100644 --- a/hack-module.awk +++ b/hack-module.awk @@ -45,8 +45,8 @@ print " return 1;"; print " }"; print ""; -

Rusty, Anthony, Dor, I need your brain power :-) On smp guests I have seen a problem with virtio (the version in curent Avi's git) which do not occur on single processor guests: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:228! illegal operation: 0001 [#1] Modules linked in: ipv6 CPU: 2 Not tainted Process swapper (pid: 0, task: 000000000f83e038, ksp: 000000000f877d70) Krnl

Rusty, Anthony, Dor, I need your brain power :-) On smp guests I have seen a problem with virtio (the version in curent Avi's git) which do not occur on single processor guests: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:228! illegal operation: 0001 [#1] Modules linked in: ipv6 CPU: 2 Not tainted Process swapper (pid: 0, task: 000000000f83e038, ksp: 000000000f877d70) Krnl

I got the following oops during interface ifup. Unfortunately its not easily reproducable so I cant say for sure that my fix fixes this problem, but I am confident and I think its correct anyway: <2>kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:234! <4>illegal operation: 0001 [#1] PREEMPT SMP <4>Modules linked in: <4>CPU: 0 Not tainted

I got the following oops during interface ifup. Unfortunately its not easily reproducable so I cant say for sure that my fix fixes this problem, but I am confident and I think its correct anyway: <2>kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:234! <4>illegal operation: 0001 [#1] PREEMPT SMP <4>Modules linked in: <4>CPU: 0 Not tainted

virtqueue ops were introduced in the hope that we'll have multiple implementations besides virtio_ring, but none have surfaced so far, and given that existing virtio ring is deployed in production we are likely stuck with it now, so this layer just adds complexity and overhead. Further, the need to pass vq twice to each call (as in dev->vq->vq_ops->kick(dev->vq) ) adds potential

virtqueue ops were introduced in the hope that we'll have multiple implementations besides virtio_ring, but none have surfaced so far, and given that existing virtio ring is deployed in production we are likely stuck with it now, so this layer just adds complexity and overhead. Further, the need to pass vq twice to each call (as in dev->vq->vq_ops->kick(dev->vq) ) adds potential

Currently, the virt_ring->enable_cb and virt_ring->disable_cb functions enforce that they were called only when callbacks were disabled and enabled respectively. However, in the current vring implementation, this isn't actually a bug. What's more, the virtio_net driver does not guard against double enabling or double disabling. All that needs to happen is for an rx notification to

Currently, the virt_ring->enable_cb and virt_ring->disable_cb functions enforce that they were called only when callbacks were disabled and enabled respectively. However, in the current vring implementation, this isn't actually a bug. What's more, the virtio_net driver does not guard against double enabling or double disabling. All that needs to happen is for an rx notification to

This helper returns 1 if a call to add_buf will not fail with -ENOSPC. This will help callers that do while(1) { alloc() if (add_buf()) { free(); break; } } This will result in one less alloc/free exercise. Signed-off-by: Amit Shah <amit.shah at redhat.com> --- v2: return true/false instead of 1/0 drivers/virtio/virtio_ring.c | 8 ++++++++ include/linux/virtio.h |

This helper returns 1 if a call to add_buf will not fail with -ENOSPC. This will help callers that do while(1) { alloc() if (add_buf()) { free(); break; } } This will result in one less alloc/free exercise. Signed-off-by: Amit Shah <amit.shah at redhat.com> --- v2: return true/false instead of 1/0 drivers/virtio/virtio_ring.c | 8 ++++++++ include/linux/virtio.h |

The 'disable_cb' callback is designed as an optimization to tell the host we don't need callbacks now. As it is not reliable, the debug check is overzealous: it can happen on two CPUs at the same time. Document this. Even if it were reliable, the virtio_net driver doesn't disable callbacks on transmit so the START_USE/END_USE debugging reentrance protection can be easily tripped

The 'disable_cb' callback is designed as an optimization to tell the host we don't need callbacks now. As it is not reliable, the debug check is overzealous: it can happen on two CPUs at the same time. Document this. Even if it were reliable, the virtio_net driver doesn't disable callbacks on transmit so the START_USE/END_USE debugging reentrance protection can be easily tripped

Rusty, the patch "virtio: imply disable_cb on callbacks" is on your tree. I'd like to figure out how it works: for example: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -69,6 +69,8 @@ static void blk_done(struct virtqueue *v /* In case queue is stopped waiting for more buffers. */

Rusty, the patch "virtio: imply disable_cb on callbacks" is on your tree. I'd like to figure out how it works: for example: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -69,6 +69,8 @@ static void blk_done(struct virtqueue *v /* In case queue is stopped waiting for more buffers. */

Herbert tells me that returning NETDEV_TX_BUSY from hard_start_xmit is seen as a poor thing to do; we should cache the packet and stop the queue. Signed-off-by: Rusty Russell <rusty at rustcorp.com.au> --- drivers/net/virtio_net.c | 57 +++++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 19 deletions(-) diff -r a936d320f6d5 drivers/net/virtio_net.c ---

Herbert tells me that returning NETDEV_TX_BUSY from hard_start_xmit is seen as a poor thing to do; we should cache the packet and stop the queue. Signed-off-by: Rusty Russell <rusty at rustcorp.com.au> --- drivers/net/virtio_net.c | 57 +++++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 19 deletions(-) diff -r a936d320f6d5 drivers/net/virtio_net.c ---

Since we want to reset the device to remove them, this is simpler (device is reset for us on driver remove). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> --- drivers/net/virtio_net.c | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff -r 7e5b3ff06f60 drivers/net/virtio_net.c --- a/drivers/net/virtio_net.c Wed Jan 23 22:53:14