search for: blk_cleanup_queu

This patch set fixes the race when hot-unplug stressed disk. Asias He (3): virtio-blk: Call del_gendisk() before disable guest kick virtio-blk: Reset device after blk_cleanup_queue() virtio-blk: Use block layer provided spinlock drivers/block/virtio_blk.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-) -- 1.7.10.2

This patch set fixes the race when hot-unplug stressed disk. Asias He (3): virtio-blk: Call del_gendisk() before disable guest kick virtio-blk: Reset device after blk_cleanup_queue() virtio-blk: Use block layer provided spinlock drivers/block/virtio_blk.c | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-) -- 1.7.10.2

On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: > Code is added to avoid calling blk_cleanup_queue() when the surprize_removal > flag is set due to a disappeared device. It avoid hangs due to incomplete > requests (e.g. in-flight requests). Such requests must be considered as lost. Ugh. Can't we complete these immediately using detach_unused_buf? If not why? > If the current remo...

On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: > Code is added to avoid calling blk_cleanup_queue() when the surprize_removal > flag is set due to a disappeared device. It avoid hangs due to incomplete > requests (e.g. in-flight requests). Such requests must be considered as lost. Ugh. Can't we complete these immediately using detach_unused_buf? If not why? > If the current remo...

3.4-stable review patch. If anyone has any objections, please let me know. ------------------ From: Asias He <asias at redhat.com> commit 483001c765af6892b3fc3726576cb42f17d1d6b5 upstream. blk_cleanup_queue() will call blk_drian_queue() to drain all the requests before queue DEAD marking. If we reset the device before blk_cleanup_queue() the drain would fail. 1) if the queue is stopped in do_virtblk_request() because device is full, the q->request_fn() will not be called. blk_drain_queue() {...

3.4-stable review patch. If anyone has any objections, please let me know. ------------------ From: Asias He <asias at redhat.com> commit 483001c765af6892b3fc3726576cb42f17d1d6b5 upstream. blk_cleanup_queue() will call blk_drian_queue() to drain all the requests before queue DEAD marking. If we reset the device before blk_cleanup_queue() the drain would fail. 1) if the queue is stopped in do_virtblk_request() because device is full, the q->request_fn() will not be called. blk_drain_queue() {...

Code is added to avoid calling blk_cleanup_queue() when the surprize_removal flag is set due to a disappeared device. It avoid hangs due to incomplete requests (e.g. in-flight requests). Such requests must be considered as lost. If the current remove callback was triggered due to an unregister driver, and the surprize_removal is not already set...

On 27/11/13 11:47, Michael S. Tsirkin wrote: > On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: >> Code is added to avoid calling blk_cleanup_queue() when the surprize_removal >> flag is set due to a disappeared device. It avoid hangs due to incomplete >> requests (e.g. in-flight requests). Such requests must be considered as lost. > > Ugh. Can't we complete these immediately using detach_unused_buf? If not why? OK, I w...

On Wed, Nov 27, 2013 at 12:37:02PM +0100, Heinz Graalfs wrote: > On 27/11/13 11:47, Michael S. Tsirkin wrote: > >On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: > >>Code is added to avoid calling blk_cleanup_queue() when the surprize_removal > >>flag is set due to a disappeared device. It avoid hangs due to incomplete > >>requests (e.g. in-flight requests). Such requests must be considered as lost. > > > >Ugh. Can't we complete these immediately using detach_unused_buf? If...

On Wed, Nov 27, 2013 at 12:37:02PM +0100, Heinz Graalfs wrote: > On 27/11/13 11:47, Michael S. Tsirkin wrote: > >On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: > >>Code is added to avoid calling blk_cleanup_queue() when the surprize_removal > >>flag is set due to a disappeared device. It avoid hangs due to incomplete > >>requests (e.g. in-flight requests). Such requests must be considered as lost. > > > >Ugh. Can't we complete these immediately using detach_unused_buf? If...

On 27/11/13 13:49, Michael S. Tsirkin wrote: > On Wed, Nov 27, 2013 at 12:37:02PM +0100, Heinz Graalfs wrote: >> On 27/11/13 11:47, Michael S. Tsirkin wrote: >>> On Wed, Nov 27, 2013 at 11:32:39AM +0100, Heinz Graalfs wrote: >>>> Code is added to avoid calling blk_cleanup_queue() when the surprize_removal >>>> flag is set due to a disappeared device. It avoid hangs due to incomplete >>>> requests (e.g. in-flight requests). Such requests must be considered as lost. >>> >>> Ugh. Can't we complete these immediately using detach_...

If we reset the virtio-blk device before the requests already dispatched to the virtio-blk driver from the block layer are finised, we will stuck in blk_cleanup_queue() and the remove will fail. blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued before DEAD marking. However it will never success if the device is already stopped. We'll have q->in_flight[] > 0, so the drain will not finish. How to reproduce the race: 1. hot-plug...

If we reset the virtio-blk device before the requests already dispatched to the virtio-blk driver from the block layer are finised, we will stuck in blk_cleanup_queue() and the remove will fail. blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued before DEAD marking. However it will never success if the device is already stopped. We'll have q->in_flight[] > 0, so the drain will not finish. How to reproduce the race: 1. hot-plug...

...ware of any errors that occur due to the lost device. This patch-set adds code to avoid further request queueing when a lost block device is detected, resulting in appropriate error info. Additionally a potential hang situation can be avoided by not waiting for requests (e.g. in-flight requests) in blk_cleanup_queue() that will never complete. On System z there exists no handshake mechanism between host and guest when a device is hot-unplugged. The device is removed and no further I/O is possible. When an online channel device disappears on System z the kernel's CIO layer informs the driver (virtio_ccw)...

...ware of any errors that occur due to the lost device. This patch-set adds code to avoid further request queueing when a lost block device is detected, resulting in appropriate error info. Additionally a potential hang situation can be avoided by not waiting for requests (e.g. in-flight requests) in blk_cleanup_queue() that will never complete. On System z there exists no handshake mechanism between host and guest when a device is hot-unplugged. The device is removed and no further I/O is possible. When an online channel device disappears on System z the kernel's CIO layer informs the driver (virtio_ccw)...

...did some testing... >> >> Removing a disk while reading from it mostly still ends up >> in hangs as of below: > > OK, we still have the problem of in-flight requests. > > I think the correct answer is to drop all requests if the virtqueue > is broken: > > - blk_cleanup_queue(vblk->disk->queue); > + if (virtqueue_is_broken(vblk->vq)) > + /* Don't wait for completion, just drop queue. */ > + blk_abandon_queue(vblk->disk->queue); Rusty, but blk_abandon_queue() would not solve the incomplete in-flight requests,...

...did some testing... >> >> Removing a disk while reading from it mostly still ends up >> in hangs as of below: > > OK, we still have the problem of in-flight requests. > > I think the correct answer is to drop all requests if the virtqueue > is broken: > > - blk_cleanup_queue(vblk->disk->queue); > + if (virtqueue_is_broken(vblk->vq)) > + /* Don't wait for completion, just drop queue. */ > + blk_abandon_queue(vblk->disk->queue); Rusty, but blk_abandon_queue() would not solve the incomplete in-flight requests,...

If we reset the virtio-blk device before the requests already dispatched to the virtio-blk driver from the block layer are finised, we will stuck in blk_cleanup_queue() and the remove will fail. blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued before DEAD marking. However it will never success if the device is already stopped. We'll have q->in_flight[] > 0, so the drain will not finish. How to reproduce the race: 1. hot-plug...

If we reset the virtio-blk device before the requests already dispatched to the virtio-blk driver from the block layer are finised, we will stuck in blk_cleanup_queue() and the remove will fail. blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued before DEAD marking. However it will never success if the device is already stopped. We'll have q->in_flight[] > 0, so the drain will not finish. How to reproduce the race: 1. hot-plug...

Hi, this patch-set tries to solve various hang situations when virtio devices (network or block) are hot-unplugged from a KVM guest. On System z there exists no handshake mechanism between host and guest when a device is hot-unplugged. The device is removed and no further I/O is possible. The guest is notified about the hard removal with a CRW machine check. As per architecture, the host must