similar to: [PATCH][RESEND] Relax BUG_ON()s when enabling/disabling virt_ring interrupts

Rusty, Anthony, Dor, I need your brain power :-) On smp guests I have seen a problem with virtio (the version in curent Avi's git) which do not occur on single processor guests: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:228! illegal operation: 0001 [#1] Modules linked in: ipv6 CPU: 2 Not tainted Process swapper (pid: 0, task: 000000000f83e038, ksp: 000000000f877d70) Krnl

Rusty, Anthony, Dor, I need your brain power :-) On smp guests I have seen a problem with virtio (the version in curent Avi's git) which do not occur on single processor guests: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:228! illegal operation: 0001 [#1] Modules linked in: ipv6 CPU: 2 Not tainted Process swapper (pid: 0, task: 000000000f83e038, ksp: 000000000f877d70) Krnl

The 'disable_cb' callback is designed as an optimization to tell the host we don't need callbacks now. As it is not reliable, the debug check is overzealous: it can happen on two CPUs at the same time. Document this. Even if it were reliable, the virtio_net driver doesn't disable callbacks on transmit so the START_USE/END_USE debugging reentrance protection can be easily tripped

The 'disable_cb' callback is designed as an optimization to tell the host we don't need callbacks now. As it is not reliable, the debug check is overzealous: it can happen on two CPUs at the same time. Document this. Even if it were reliable, the virtio_net driver doesn't disable callbacks on transmit so the START_USE/END_USE debugging reentrance protection can be easily tripped

Hello Rusty, seems that we still have a problem with virtio_net and the enable_cb callback. During a long running network stress tests with virtio and got the following oops: ------------[ cut here ]------------ kernel BUG at drivers/virtio/virtio_ring.c:230! illegal operation: 0001 [#1] SMP Modules linked in: CPU: 0 Not tainted 2.6.26-rc2-kvm-00436-gc94c08b-dirty #34 Process netserver (pid:

Hello Rusty, seems that we still have a problem with virtio_net and the enable_cb callback. During a long running network stress tests with virtio and got the following oops: ------------[ cut here ]------------ kernel BUG at drivers/virtio/virtio_ring.c:230! illegal operation: 0001 [#1] SMP Modules linked in: CPU: 0 Not tainted 2.6.26-rc2-kvm-00436-gc94c08b-dirty #34 Process netserver (pid:

Hi Rusty, I hit a bug in virtio_ring.c:218 when I was stressing virtio_net using kvm with -smp 4. static void vring_disable_cb(struct virtqueue *_vq) { struct vring_virtqueue *vq = to_vvq(_vq); START_USE(vq); --> BUG_ON(vq->vring.avail->flags & VRING_AVAIL_F_NO_INTERRUPT); vq->vring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;

Hi Rusty, I hit a bug in virtio_ring.c:218 when I was stressing virtio_net using kvm with -smp 4. static void vring_disable_cb(struct virtqueue *_vq) { struct vring_virtqueue *vq = to_vvq(_vq); START_USE(vq); --> BUG_ON(vq->vring.avail->flags & VRING_AVAIL_F_NO_INTERRUPT); vq->vring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;

This adds a way to check ring empty state after enable_cb outside any locks. Will be used by virtio_net. Note: there's room for more optimization: caller is likely to have a memory barrier already, which means we might be able to get rid of a barrier here. Deferring this optimization until we do some benchmarking. Signed-off-by: Michael S. Tsirkin <mst at redhat.com> ---

This adds a way to check ring empty state after enable_cb outside any locks. Will be used by virtio_net. Note: there's room for more optimization: caller is likely to have a memory barrier already, which means we might be able to get rid of a barrier here. Deferring this optimization until we do some benchmarking. Signed-off-by: Michael S. Tsirkin <mst at redhat.com> ---

virtqueue ops were introduced in the hope that we'll have multiple implementations besides virtio_ring, but none have surfaced so far, and given that existing virtio ring is deployed in production we are likely stuck with it now, so this layer just adds complexity and overhead. Further, the need to pass vq twice to each call (as in dev->vq->vq_ops->kick(dev->vq) ) adds potential

virtqueue ops were introduced in the hope that we'll have multiple implementations besides virtio_ring, but none have surfaced so far, and given that existing virtio ring is deployed in production we are likely stuck with it now, so this layer just adds complexity and overhead. Further, the need to pass vq twice to each call (as in dev->vq->vq_ops->kick(dev->vq) ) adds potential

There is a race in virtio_net, dealing with disabling/enabling the callback. I saw the following oops: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:218! illegal operation: 0001 [#1] SMP Modules linked in: sunrpc dm_mod CPU: 2 Not tainted 2.6.25-rc1zlive-host-10623-gd358142-dirty #99 Process swapper (pid: 0, task: 000000000f85a610, ksp: 000000000f873c60) Krnl PSW : 0404300180000000

There is a race in virtio_net, dealing with disabling/enabling the callback. I saw the following oops: kernel BUG at /space/kvm/drivers/virtio/virtio_ring.c:218! illegal operation: 0001 [#1] SMP Modules linked in: sunrpc dm_mod CPU: 2 Not tainted 2.6.25-rc1zlive-host-10623-gd358142-dirty #99 Process swapper (pid: 0, task: 000000000f85a610, ksp: 000000000f873c60) Krnl PSW : 0404300180000000

Rusty, the patch "virtio: imply disable_cb on callbacks" is on your tree. I'd like to figure out how it works: for example: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -69,6 +69,8 @@ static void blk_done(struct virtqueue *v /* In case queue is stopped waiting for more buffers. */

Rusty, the patch "virtio: imply disable_cb on callbacks" is on your tree. I'd like to figure out how it works: for example: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -69,6 +69,8 @@ static void blk_done(struct virtqueue *v /* In case queue is stopped waiting for more buffers. */

Hi Dave, Nice driver optimization from Shirley, but requires a new virtio hook. Do you want to take both? I have nothing else overlapping it. Cheers, Rusty.

Hi Dave, Nice driver optimization from Shirley, but requires a new virtio hook. Do you want to take both? I have nothing else overlapping it. Cheers, Rusty.

Hollis Blanchard wrote: > On Tue, 2008-03-25 at 14:45 +0100, Christian Ehrhardt wrote: >> => from one not yet defined point our guest seems to receive absolutely nothing >> => when the guest is hanging it sends nfs requests which are seen externally, but it does not seem to get the respones >> => the arp requests for the guest are repeated - maybe we can add some very

Hollis Blanchard wrote: > On Tue, 2008-03-25 at 14:45 +0100, Christian Ehrhardt wrote: >> => from one not yet defined point our guest seems to receive absolutely nothing >> => when the guest is hanging it sends nfs requests which are seen externally, but it does not seem to get the respones >> => the arp requests for the guest are repeated - maybe we can add some very