Michael S. Tsirkin
2021-Nov-24 07:22 UTC
[PATCH V5 1/4] virtio_ring: validate used buffer length
On Wed, Nov 24, 2021 at 10:33:28AM +0800, Jason Wang wrote:> On Wed, Nov 24, 2021 at 10:26 AM Jason Wang <jasowang at redhat.com> wrote: > > > > On Wed, Nov 24, 2021 at 9:30 AM Michael Ellerman <mpe at ellerman.id.au> wrote: > > > > > > "Michael S. Tsirkin" <mst at redhat.com> writes: > > > > On Tue, Nov 23, 2021 at 10:25:20AM +0800, Jason Wang wrote: > > > >> On Tue, Nov 23, 2021 at 4:24 AM Halil Pasic <pasic at linux.ibm.com> wrote: > > > >> > > > > >> > On Mon, 22 Nov 2021 14:25:26 +0800 > > > >> > Jason Wang <jasowang at redhat.com> wrote: > > > >> > > > > >> > > I think the fixes are: > > > >> > > > > > >> > > 1) fixing the vhost vsock > > > >> > > 2) use suppress_used_validation=true to let vsock driver to validate > > > >> > > the in buffer length > > > >> > > 3) probably a new feature so the driver can only enable the validation > > > >> > > when the feature is enabled. > > > >> > > > > >> > I'm not sure, I would consider a F_DEV_Y_FIXED_BUG_X a perfectly good > > > >> > feature. Frankly the set of such bugs is device implementation > > > >> > specific and it makes little sense to specify a feature bit > > > >> > that says the device implementation claims to adhere to some > > > >> > aspect of the specification. Also what would be the semantic > > > >> > of not negotiating F_DEV_Y_FIXED_BUG_X? > > > >> > > > >> Yes, I agree. Rethink of the feature bit, it seems unnecessary, > > > >> especially considering the driver should not care about the used > > > >> length for tx. > > > >> > > > >> > > > > >> > On the other hand I see no other way to keep the validation > > > >> > permanently enabled for fixed implementations, and get around the problem > > > >> > with broken implementations. So we could have something like > > > >> > VHOST_USED_LEN_STRICT. > > > >> > > > >> It's more about a choice of the driver's knowledge. For vsock TX it > > > >> should be fine. If we introduce a parameter and disable it by default, > > > >> it won't be very useful. > > > >> > > > >> > > > > >> > Maybe, we can also think of 'warn and don't alter behavior' instead of > > > >> > 'warn' and alter behavior. Or maybe even not having such checks on in > > > >> > production, but only when testing. > > > >> > > > >> I think there's an agreement that virtio drivers need more hardening, > > > >> that's why a lot of patches were merged. Especially considering the > > > >> new requirements came from confidential computing, smart NIC and > > > >> VDUSE. For virtio drivers, enabling the validation may help to > > > >> > > > >> 1) protect the driver from the buggy and malicious device > > > >> 2) uncover the bugs of the devices (as vsock did, and probably rpmsg) > > > >> 3) force the have a smart driver that can do the validation itself > > > >> then we can finally remove the validation in the core > > > >> > > > >> So I'd like to keep it enabled. > > > >> > > > >> Thanks > > > > > > > > Let's see how far we can get. But yes, maybe we were too aggressive in > > > > breaking things by default, a warning might be a better choice for a > > > > couple of cycles. > > > > Ok, considering we saw the issues with balloons I think I can post a > > patch to use warn instead. I wonder if we need to taint the kernel in > > this case. > > Rethink this, consider we still have some time, I tend to convert the > drivers to validate the length by themselves. Does this make sense? > > ThanksThat's separate but let's stop crashing guests for people ASAP.> > > > > > > > This series appears to break the virtio_balloon driver as well. > > > > > > The symptom is soft lockup warnings, eg: > > > > > > INFO: task kworker/1:1:109 blocked for more than 614 seconds. > > > Not tainted 5.16.0-rc2-gcc-10.3.0 #21 > > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > task:kworker/1:1 state:D stack:12496 pid: 109 ppid: 2 flags:0x00000800 > > > Workqueue: events_freezable update_balloon_size_func > > > Call Trace: > > > [c000000003cef7c0] [c000000003cef820] 0xc000000003cef820 (unreliable) > > > [c000000003cef9b0] [c00000000001e238] __switch_to+0x1e8/0x2f0 > > > [c000000003cefa10] [c000000000f0a00c] __schedule+0x2cc/0xb50 > > > [c000000003cefae0] [c000000000f0a8fc] schedule+0x6c/0x140 > > > [c000000003cefb10] [c00000000095b6c4] tell_host+0xe4/0x130 > > > [c000000003cefba0] [c00000000095d234] update_balloon_size_func+0x394/0x3f0 > > > [c000000003cefc70] [c000000000178064] process_one_work+0x2c4/0x5b0 > > > [c000000003cefd10] [c0000000001783f8] worker_thread+0xa8/0x640 > > > [c000000003cefda0] [c000000000185444] kthread+0x1b4/0x1c0 > > > [c000000003cefe10] [c00000000000cee4] ret_from_kernel_thread+0x5c/0x64 > > > > > > Similar backtrace reported here by Luis: > > > > > > https://lore.kernel.org/lkml/YY2duTi0wAyAKUTJ at bombadil.infradead.org/ > > > > > > Bisect points to: > > > > > > # first bad commit: [939779f5152d161b34f612af29e7dc1ac4472fcf] virtio_ring: validate used buffer length > > > > > > Adding suppress used validation to the virtio balloon driver "fixes" it, eg. > > > > > > diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c > > > index c22ff0117b46..a14b82ceebb2 100644 > > > --- a/drivers/virtio/virtio_balloon.c > > > +++ b/drivers/virtio/virtio_balloon.c > > > @@ -1150,6 +1150,7 @@ static unsigned int features[] = { > > > }; > > > > > > static struct virtio_driver virtio_balloon_driver = { > > > + .suppress_used_validation = true, > > > .feature_table = features, > > > .feature_table_size = ARRAY_SIZE(features), > > > .driver.name = KBUILD_MODNAME, > > > > Looks good, we need a formal patch for this. > > > > And we need fix Qemu as well which advertise non zero used length for > > inflate/deflate queue: > > > > static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) > > ... > > virtqueue_push(vq, elem, offset); > > > > Thanks > > > > > > > > > > > cheers > > >
On Wed, Nov 24, 2021 at 3:22 PM Michael S. Tsirkin <mst at redhat.com> wrote:> > On Wed, Nov 24, 2021 at 10:33:28AM +0800, Jason Wang wrote: > > On Wed, Nov 24, 2021 at 10:26 AM Jason Wang <jasowang at redhat.com> wrote: > > > > > > On Wed, Nov 24, 2021 at 9:30 AM Michael Ellerman <mpe at ellerman.id.au> wrote: > > > > > > > > "Michael S. Tsirkin" <mst at redhat.com> writes: > > > > > On Tue, Nov 23, 2021 at 10:25:20AM +0800, Jason Wang wrote: > > > > >> On Tue, Nov 23, 2021 at 4:24 AM Halil Pasic <pasic at linux.ibm.com> wrote: > > > > >> > > > > > >> > On Mon, 22 Nov 2021 14:25:26 +0800 > > > > >> > Jason Wang <jasowang at redhat.com> wrote: > > > > >> > > > > > >> > > I think the fixes are: > > > > >> > > > > > > >> > > 1) fixing the vhost vsock > > > > >> > > 2) use suppress_used_validation=true to let vsock driver to validate > > > > >> > > the in buffer length > > > > >> > > 3) probably a new feature so the driver can only enable the validation > > > > >> > > when the feature is enabled. > > > > >> > > > > > >> > I'm not sure, I would consider a F_DEV_Y_FIXED_BUG_X a perfectly good > > > > >> > feature. Frankly the set of such bugs is device implementation > > > > >> > specific and it makes little sense to specify a feature bit > > > > >> > that says the device implementation claims to adhere to some > > > > >> > aspect of the specification. Also what would be the semantic > > > > >> > of not negotiating F_DEV_Y_FIXED_BUG_X? > > > > >> > > > > >> Yes, I agree. Rethink of the feature bit, it seems unnecessary, > > > > >> especially considering the driver should not care about the used > > > > >> length for tx. > > > > >> > > > > >> > > > > > >> > On the other hand I see no other way to keep the validation > > > > >> > permanently enabled for fixed implementations, and get around the problem > > > > >> > with broken implementations. So we could have something like > > > > >> > VHOST_USED_LEN_STRICT. > > > > >> > > > > >> It's more about a choice of the driver's knowledge. For vsock TX it > > > > >> should be fine. If we introduce a parameter and disable it by default, > > > > >> it won't be very useful. > > > > >> > > > > >> > > > > > >> > Maybe, we can also think of 'warn and don't alter behavior' instead of > > > > >> > 'warn' and alter behavior. Or maybe even not having such checks on in > > > > >> > production, but only when testing. > > > > >> > > > > >> I think there's an agreement that virtio drivers need more hardening, > > > > >> that's why a lot of patches were merged. Especially considering the > > > > >> new requirements came from confidential computing, smart NIC and > > > > >> VDUSE. For virtio drivers, enabling the validation may help to > > > > >> > > > > >> 1) protect the driver from the buggy and malicious device > > > > >> 2) uncover the bugs of the devices (as vsock did, and probably rpmsg) > > > > >> 3) force the have a smart driver that can do the validation itself > > > > >> then we can finally remove the validation in the core > > > > >> > > > > >> So I'd like to keep it enabled. > > > > >> > > > > >> Thanks > > > > > > > > > > Let's see how far we can get. But yes, maybe we were too aggressive in > > > > > breaking things by default, a warning might be a better choice for a > > > > > couple of cycles. > > > > > > Ok, considering we saw the issues with balloons I think I can post a > > > patch to use warn instead. I wonder if we need to taint the kernel in > > > this case. > > > > Rethink this, consider we still have some time, I tend to convert the > > drivers to validate the length by themselves. Does this make sense? > > > > Thanks > > That's separate but let's stop crashing guests for people ASAP.Ok, will post a patch soon. Thanks> > > > > > > > > > > > > This series appears to break the virtio_balloon driver as well. > > > > > > > > The symptom is soft lockup warnings, eg: > > > > > > > > INFO: task kworker/1:1:109 blocked for more than 614 seconds. > > > > Not tainted 5.16.0-rc2-gcc-10.3.0 #21 > > > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > task:kworker/1:1 state:D stack:12496 pid: 109 ppid: 2 flags:0x00000800 > > > > Workqueue: events_freezable update_balloon_size_func > > > > Call Trace: > > > > [c000000003cef7c0] [c000000003cef820] 0xc000000003cef820 (unreliable) > > > > [c000000003cef9b0] [c00000000001e238] __switch_to+0x1e8/0x2f0 > > > > [c000000003cefa10] [c000000000f0a00c] __schedule+0x2cc/0xb50 > > > > [c000000003cefae0] [c000000000f0a8fc] schedule+0x6c/0x140 > > > > [c000000003cefb10] [c00000000095b6c4] tell_host+0xe4/0x130 > > > > [c000000003cefba0] [c00000000095d234] update_balloon_size_func+0x394/0x3f0 > > > > [c000000003cefc70] [c000000000178064] process_one_work+0x2c4/0x5b0 > > > > [c000000003cefd10] [c0000000001783f8] worker_thread+0xa8/0x640 > > > > [c000000003cefda0] [c000000000185444] kthread+0x1b4/0x1c0 > > > > [c000000003cefe10] [c00000000000cee4] ret_from_kernel_thread+0x5c/0x64 > > > > > > > > Similar backtrace reported here by Luis: > > > > > > > > https://lore.kernel.org/lkml/YY2duTi0wAyAKUTJ at bombadil.infradead.org/ > > > > > > > > Bisect points to: > > > > > > > > # first bad commit: [939779f5152d161b34f612af29e7dc1ac4472fcf] virtio_ring: validate used buffer length > > > > > > > > Adding suppress used validation to the virtio balloon driver "fixes" it, eg. > > > > > > > > diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c > > > > index c22ff0117b46..a14b82ceebb2 100644 > > > > --- a/drivers/virtio/virtio_balloon.c > > > > +++ b/drivers/virtio/virtio_balloon.c > > > > @@ -1150,6 +1150,7 @@ static unsigned int features[] = { > > > > }; > > > > > > > > static struct virtio_driver virtio_balloon_driver = { > > > > + .suppress_used_validation = true, > > > > .feature_table = features, > > > > .feature_table_size = ARRAY_SIZE(features), > > > > .driver.name = KBUILD_MODNAME, > > > > > > Looks good, we need a formal patch for this. > > > > > > And we need fix Qemu as well which advertise non zero used length for > > > inflate/deflate queue: > > > > > > static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) > > > ... > > > virtqueue_push(vq, elem, offset); > > > > > > Thanks > > > > > > > > > > > > > > > cheers > > > > >