Lee Jones
2022-Mar-07 19:17 UTC
[PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use
vhost_vsock_handle_tx_kick() already holds the mutex during its call
to vhost_get_vq_desc(). All we have to do here is take the same lock
during virtqueue clean-up and we mitigate the reported issues.
Also WARN() as a precautionary measure. The purpose of this is to
capture possible future race conditions which may pop up over time.
Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
Cc: <stable at vger.kernel.org>
Reported-by: syzbot+adc3cb32385586bec859 at syzkaller.appspotmail.com
Signed-off-by: Lee Jones <lee.jones at linaro.org>
---
drivers/vhost/vhost.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 59edb5a1ffe28..ef7e371e3e649 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev)
int i;
for (i = 0; i < dev->nvqs; ++i) {
+ /* No workers should run here by design. However, races have
+ * previously occurred where drivers have been unable to flush
+ * all work properly prior to clean-up. Without a successful
+ * flush the guest will malfunction, but avoiding host memory
+ * corruption in those cases does seem preferable.
+ */
+ WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex));
+
+ mutex_lock(&dev->vqs[i]->mutex);
if (dev->vqs[i]->error_ctx)
eventfd_ctx_put(dev->vqs[i]->error_ctx);
if (dev->vqs[i]->kick)
@@ -700,6 +709,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev)
if (dev->vqs[i]->call_ctx.ctx)
eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx);
vhost_vq_reset(dev, dev->vqs[i]);
+ mutex_unlock(&dev->vqs[i]->mutex);
}
vhost_dev_free_iovecs(dev);
if (dev->log_ctx)
--
2.35.1.616.g0bdcbb4464-goog
Greg KH
2022-Mar-07 19:33 UTC
[PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use
On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote:> vhost_vsock_handle_tx_kick() already holds the mutex during its call > to vhost_get_vq_desc(). All we have to do here is take the same lock > during virtqueue clean-up and we mitigate the reported issues. > > Also WARN() as a precautionary measure. The purpose of this is to > capture possible future race conditions which may pop up over time. > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > Cc: <stable at vger.kernel.org> > Reported-by: syzbot+adc3cb32385586bec859 at syzkaller.appspotmail.com > Signed-off-by: Lee Jones <lee.jones at linaro.org> > --- > drivers/vhost/vhost.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > index 59edb5a1ffe28..ef7e371e3e649 100644 > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > int i; > > for (i = 0; i < dev->nvqs; ++i) { > + /* No workers should run here by design. However, races have > + * previously occurred where drivers have been unable to flush > + * all work properly prior to clean-up. Without a successful > + * flush the guest will malfunction, but avoiding host memory > + * corruption in those cases does seem preferable. > + */ > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex));So you are trading one syzbot triggered issue for another one in the future? :) If this ever can happen, handle it, but don't log it with a WARN_ON() as that will trigger the panic-on-warn boxes, as well as syzbot. Unless you want that to happen? And what happens if the mutex is locked _RIGHT_ after you checked it? You still have a race... thanks, greg k-h
Michael S. Tsirkin
2022-Mar-07 22:37 UTC
[PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use
On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote:> vhost_vsock_handle_tx_kick() already holds the mutex during its call > to vhost_get_vq_desc(). All we have to do here is take the same lock > during virtqueue clean-up and we mitigate the reported issues.Pls just basically copy the code comment here. this is just confuses.> Also WARN() as a precautionary measure. The purpose of this is to > capture possible future race conditions which may pop up over time. > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00And this is a bug we already fixed, right?> Cc: <stable at vger.kernel.org> > Reported-by: syzbot+adc3cb32385586bec859 at syzkaller.appspotmail.comnot really applicable anymore ...> Signed-off-by: Lee Jones <lee.jones at linaro.org> > --- > drivers/vhost/vhost.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > index 59edb5a1ffe28..ef7e371e3e649 100644 > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > int i; > > for (i = 0; i < dev->nvqs; ++i) { > + /* No workers should run here by design. However, races have > + * previously occurred where drivers have been unable to flush > + * all work properly prior to clean-up. Without a successful > + * flush the guest will malfunction, but avoiding host memory > + * corruption in those cases does seem preferable. > + */ > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > + > + mutex_lock(&dev->vqs[i]->mutex); > if (dev->vqs[i]->error_ctx) > eventfd_ctx_put(dev->vqs[i]->error_ctx); > if (dev->vqs[i]->kick) > @@ -700,6 +709,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > if (dev->vqs[i]->call_ctx.ctx) > eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); > vhost_vq_reset(dev, dev->vqs[i]); > + mutex_unlock(&dev->vqs[i]->mutex); > } > vhost_dev_free_iovecs(dev); > if (dev->log_ctx) > -- > 2.35.1.616.g0bdcbb4464-goog
Jason Wang
2022-Mar-08 06:15 UTC
[PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use
On Tue, Mar 8, 2022 at 3:18 AM Lee Jones <lee.jones at linaro.org> wrote:> > vhost_vsock_handle_tx_kick() already holds the mutex during its call > to vhost_get_vq_desc(). All we have to do here is take the same lock > during virtqueue clean-up and we mitigate the reported issues. > > Also WARN() as a precautionary measure. The purpose of this is to > capture possible future race conditions which may pop up over time. > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > Cc: <stable at vger.kernel.org> > Reported-by: syzbot+adc3cb32385586bec859 at syzkaller.appspotmail.com > Signed-off-by: Lee Jones <lee.jones at linaro.org> > --- > drivers/vhost/vhost.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > index 59edb5a1ffe28..ef7e371e3e649 100644 > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > int i; > > for (i = 0; i < dev->nvqs; ++i) { > + /* No workers should run here by design. However, races have > + * previously occurred where drivers have been unable to flush > + * all work properly prior to clean-up. Without a successful > + * flush the guest will malfunction, but avoiding host memory > + * corruption in those cases does seem preferable. > + */ > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > +I don't get how this can help, the mutex could be grabbed in the middle of the above and below line.> + mutex_lock(&dev->vqs[i]->mutex); > if (dev->vqs[i]->error_ctx) > eventfd_ctx_put(dev->vqs[i]->error_ctx); > if (dev->vqs[i]->kick) > @@ -700,6 +709,7 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > if (dev->vqs[i]->call_ctx.ctx) > eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx); > vhost_vq_reset(dev, dev->vqs[i]); > + mutex_unlock(&dev->vqs[i]->mutex); > }I'm not sure it's correct to assume some behaviour of a buggy device. For the device mutex, we use that to protect more than just err/call and vq. Thanks> vhost_dev_free_iovecs(dev); > if (dev->log_ctx) > -- > 2.35.1.616.g0bdcbb4464-goog >