Virtualization - Apr 2020 - [PATCH] virtio-blk: handle block_device_operations callbacks after hot unplug

A virtio_blk block device can still be referenced after hot unplug by
userspace processes that hold the file descriptor.  In this case
virtblk_getgeo() can be invoked after virtblk_remove() was called.  For
example, a program that has /dev/vdb open can call ioctl(HDIO_GETGEO)
after hot unplug.

Fix this by clearing vblk->disk->private_data and checking that the
virtio_blk driver instance is still around in virtblk_getgeo().

Note that the virtblk_getgeo() function itself is guaranteed to remain
in memory after hot unplug because the virtio_blk module refcount is
still held while a block device reference exists.

Originally-by: Lance Digby <ldigby at redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha at redhat.com>
---
 drivers/block/virtio_blk.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 93468b7c6701..b50cdf37a6f7 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -300,6 +300,10 @@ static int virtblk_getgeo(struct block_device *bd, struct
hd_geometry *geo)
 {
 	struct virtio_blk *vblk = bd->bd_disk->private_data;
 
+	/* Driver instance has been removed */
+	if (!vblk)
+		return -ENOTTY;
+
 	/* see if the host passed in geometry config */
 	if (virtio_has_feature(vblk->vdev, VIRTIO_BLK_F_GEOMETRY)) {
 		virtio_cread(vblk->vdev, struct virtio_blk_config,
@@ -835,6 +839,7 @@ static void virtblk_remove(struct virtio_device *vdev)
 	vdev->config->reset(vdev);
 
 	refc = kref_read(&disk_to_dev(vblk->disk)->kobj.kref);
+	vblk->disk->private_data = NULL;
 	put_disk(vblk->disk);
 	vdev->config->del_vqs(vdev);
 	kfree(vblk->vqs);
-- 
2.25.1

On Thu, Apr 23, 2020 at 01:37:17PM +0100, Stefan Hajnoczi
wrote:
> A virtio_blk block device can still be referenced after hot unplug by
> userspace processes that hold the file descriptor.  In this case
> virtblk_getgeo() can be invoked after virtblk_remove() was called.  For
> example, a program that has /dev/vdb open can call ioctl(HDIO_GETGEO)
> after hot unplug.
> 
> Fix this by clearing vblk->disk->private_data and checking that the
> virtio_blk driver instance is still around in virtblk_getgeo().
> 
> Note that the virtblk_getgeo() function itself is guaranteed to remain
> in memory after hot unplug because the virtio_blk module refcount is
> still held while a block device reference exists.
> 
> Originally-by: Lance Digby <ldigby at redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha at redhat.com>
> ---
>  drivers/block/virtio_blk.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 93468b7c6701..b50cdf37a6f7 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -300,6 +300,10 @@ static int virtblk_getgeo(struct block_device *bd,
struct hd_geometry *geo)
>  {
>  	struct virtio_blk *vblk = bd->bd_disk->private_data;
>  
> +	/* Driver instance has been removed */
> +	if (!vblk)
> +		return -ENOTTY;
> +
>  	/* see if the host passed in geometry config */
>  	if (virtio_has_feature(vblk->vdev, VIRTIO_BLK_F_GEOMETRY)) {
>  		virtio_cread(vblk->vdev, struct virtio_blk_config,
Just so I understand, what serializes this access?
See below for what looks like a race condition ...
> @@ -835,6 +839,7 @@ static void virtblk_remove(struct virtio_device *vdev)
>  	vdev->config->reset(vdev);
>  
>  	refc = kref_read(&disk_to_dev(vblk->disk)->kobj.kref);
So what if private_data is tested at this time ...
> +	vblk->disk->private_data = NULL;
>  	put_disk(vblk->disk);
>  	vdev->config->del_vqs(vdev);
>  	kfree(vblk->vqs);
... and then used at this time?

What prevents this?

> -- 
> 2.25.1
>

On Thu, Apr 23, 2020 at 01:37:17PM +0100, Stefan Hajnoczi
wrote:
> A virtio_blk block device can still be referenced after hot unplug by
> userspace processes that hold the file descriptor.  In this case
> virtblk_getgeo() can be invoked after virtblk_remove() was called.  For
> example, a program that has /dev/vdb open can call ioctl(HDIO_GETGEO)
> after hot unplug.
> 
> Fix this by clearing vblk->disk->private_data and checking that the
> virtio_blk driver instance is still around in virtblk_getgeo().
> 
> Note that the virtblk_getgeo() function itself is guaranteed to remain
> in memory after hot unplug because the virtio_blk module refcount is
> still held while a block device reference exists.
> 
> Originally-by: Lance Digby <ldigby at redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha at redhat.com>
> ---
>  drivers/block/virtio_blk.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 93468b7c6701..b50cdf37a6f7 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -300,6 +300,10 @@ static int virtblk_getgeo(struct block_device *bd,
struct hd_geometry *geo)
>  {
>  	struct virtio_blk *vblk = bd->bd_disk->private_data;
>  
> +	/* Driver instance has been removed */
> +	if (!vblk)
> +		return -ENOTTY;
If this ever hits you have a nasty race condition and this is not
going to fix it, as it could be removed just after this check as well.

On Thu, Apr 23, 2020 at 01:37:17PM +0100, Stefan Hajnoczi
wrote:
> A virtio_blk block device can still be referenced after hot unplug by
> userspace processes that hold the file descriptor.  In this case
> virtblk_getgeo() can be invoked after virtblk_remove() was called.  For
> example, a program that has /dev/vdb open can call ioctl(HDIO_GETGEO)
> after hot unplug.
> 
> Fix this by clearing vblk->disk->private_data and checking that the
> virtio_blk driver instance is still around in virtblk_getgeo().
> 
> Note that the virtblk_getgeo() function itself is guaranteed to remain
> in memory after hot unplug because the virtio_blk module refcount is
> still held while a block device reference exists.
> 
> Originally-by: Lance Digby <ldigby at redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha at redhat.com>
> ---
>  drivers/block/virtio_blk.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 93468b7c6701..b50cdf37a6f7 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -300,6 +300,10 @@ static int virtblk_getgeo(struct block_device *bd,
struct hd_geometry *geo)
>  {
>  	struct virtio_blk *vblk = bd->bd_disk->private_data;
>  
> +	/* Driver instance has been removed */
> +	if (!vblk)
> +		return -ENOTTY;
> +
>  	/* see if the host passed in geometry config */
>  	if (virtio_has_feature(vblk->vdev, VIRTIO_BLK_F_GEOMETRY)) {
>  		virtio_cread(vblk->vdev, struct virtio_blk_config,
> @@ -835,6 +839,7 @@ static void virtblk_remove(struct virtio_device *vdev)
>  	vdev->config->reset(vdev);
>  
>  	refc = kref_read(&disk_to_dev(vblk->disk)->kobj.kref);
> +	vblk->disk->private_data = NULL;
>  	put_disk(vblk->disk);
>  	vdev->config->del_vqs(vdev);
>  	kfree(vblk->vqs);
As pointed out, can be a race. We had a very similar issue in
virtio-vsock, and we solved using RCU to assign and get the pointer [1],
maybe the same solution can work here.

Cheers,
Stefano

[1] 0deab087b16a vsock/virtio: use RCU to avoid use-after-free on
the_virtio_vsock

On Thu, Apr 23, 2020 at 07:13:38AM -0700, Christoph Hellwig
wrote:
> On Thu, Apr 23, 2020 at 01:37:17PM +0100, Stefan Hajnoczi wrote:
> > A virtio_blk block device can still be referenced after hot unplug by
> > userspace processes that hold the file descriptor.  In this case
> > virtblk_getgeo() can be invoked after virtblk_remove() was called. 
For
> > example, a program that has /dev/vdb open can call ioctl(HDIO_GETGEO)
> > after hot unplug.
> > 
> > Fix this by clearing vblk->disk->private_data and checking that
the
> > virtio_blk driver instance is still around in virtblk_getgeo().
> > 
> > Note that the virtblk_getgeo() function itself is guaranteed to remain
> > in memory after hot unplug because the virtio_blk module refcount is
> > still held while a block device reference exists.
> > 
> > Originally-by: Lance Digby <ldigby at redhat.com>
> > Signed-off-by: Stefan Hajnoczi <stefanha at redhat.com>
> > ---
> >  drivers/block/virtio_blk.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > index 93468b7c6701..b50cdf37a6f7 100644
> > --- a/drivers/block/virtio_blk.c
> > +++ b/drivers/block/virtio_blk.c
> > @@ -300,6 +300,10 @@ static int virtblk_getgeo(struct block_device
*bd, struct hd_geometry *geo)
> >  {
> >  	struct virtio_blk *vblk = bd->bd_disk->private_data;
> >  
> > +	/* Driver instance has been removed */
> > +	if (!vblk)
> > +		return -ENOTTY;
> 
> If this ever hits you have a nasty race condition and this is not
> going to fix it, as it could be removed just after this check as well.
Perhaps a better fix is to keep a refcount in struct virtio_blk and
implement block_device_operations->release() to decrement the refcount.

That way the struct virtio_blk stays around until all block device
references are dropped.  This is similar to nbd_put() in
drivers/block/nbd.c.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20200424/9270bf98/attachment.sig>