On Mon, Nov 22, 2021 at 1:49 PM Halil Pasic <pasic at linux.ibm.com> wrote:> > On Mon, 22 Nov 2021 06:35:18 +0100 > Halil Pasic <pasic at linux.ibm.com> wrote: > > > > I think it should be a common issue, looking at > > > vhost_vsock_handle_tx_kick(), it did: > > > > > > len += sizeof(pkt->hdr); > > > vhost_add_used(vq, head, len); > > > > > > which looks like a violation of the spec since it's TX. > > > > I'm not sure the lines above look like a violation of the spec. If you > > examine vhost_vsock_alloc_pkt() I believe that you will agree that: > > len == pkt->len == pkt->hdr.len > > which makes sense since according to the spec both tx and rx messages > > are hdr+payload. And I believe hdr.len is the size of the payload, > > although that does not seem to be properly documented by the spec.Sorry for being unclear, what I meant is that we probably should use zero here. TX doesn't use in buffer actually. According to the spec, 0 should be the used length: "and len the total of bytes written into the buffer."> > > > On the other hand tx messages are stated to be device read-only (in the > > spec) so if the device writes stuff, that is certainly wrong. > >Yes.> > If that is what happens. > > > > Looking at virtqueue_get_buf_ctx_split() I'm not sure that is what > > happens. My hypothesis is that we just a last descriptor is an 'in' > > type descriptor (i.e. a device writable one). For tx that assumption > > would be wrong. > > > > I will have another look at this today and send a fix patch if my > > suspicion is confirmed. > > If my suspicion is right something like: > > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c > index 00f64f2f8b72..efb57898920b 100644 > --- a/drivers/virtio/virtio_ring.c > +++ b/drivers/virtio/virtio_ring.c > @@ -764,6 +764,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > struct vring_virtqueue *vq = to_vvq(_vq); > void *ret; > unsigned int i; > + bool has_in; > u16 last_used; > > START_USE(vq); > @@ -787,6 +788,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > vq->split.vring.used->ring[last_used].id); > *len = virtio32_to_cpu(_vq->vdev, > vq->split.vring.used->ring[last_used].len); > + has_in = virtio16_to_cpu(_vq->vdev, > + vq->split.vring.used->ring[last_used].flags) > + & VRING_DESC_F_WRITE;Did you mean vring.desc actually? If yes, it's better not depend on the descriptor ring which can be modified by the device. We've stored the flags in desc_extra[].> > if (unlikely(i >= vq->split.vring.num)) { > BAD_RING(vq, "id %u out of range\n", i); > @@ -796,7 +800,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > BAD_RING(vq, "id %u is not a head!\n", i); > return NULL; > } > - if (vq->buflen && unlikely(*len > vq->buflen[i])) { > + if (has_in && q->buflen && unlikely(*len > vq->buflen[i])) { > BAD_RING(vq, "used len %d is larger than in buflen %u\n", > *len, vq->buflen[i]); > return NULL; > > would fix the problem for split. I will try that out and let you know > later.I'm not sure I get this, in virtqueue_add_split, the buflen[i] only contains the in buffer length. I think the fixes are: 1) fixing the vhost vsock 2) use suppress_used_validation=true to let vsock driver to validate the in buffer length 3) probably a new feature so the driver can only enable the validation when the feature is enabled. Thanks> > Regards, > Halil >
Stefano Garzarella
2021-Nov-22 07:55 UTC
[PATCH V5 1/4] virtio_ring: validate used buffer length
On Mon, Nov 22, 2021 at 02:25:26PM +0800, Jason Wang wrote:>On Mon, Nov 22, 2021 at 1:49 PM Halil Pasic <pasic at linux.ibm.com> wrote: >> >> On Mon, 22 Nov 2021 06:35:18 +0100 >> Halil Pasic <pasic at linux.ibm.com> wrote: >> >> > > I think it should be a common issue, looking at >> > > vhost_vsock_handle_tx_kick(), it did: >> > > >> > > len += sizeof(pkt->hdr); >> > > vhost_add_used(vq, head, len); >> > > >> > > which looks like a violation of the spec since it's TX. >> > >> > I'm not sure the lines above look like a violation of the spec. If you >> > examine vhost_vsock_alloc_pkt() I believe that you will agree that: >> > len == pkt->len == pkt->hdr.len >> > which makes sense since according to the spec both tx and rx messages >> > are hdr+payload. And I believe hdr.len is the size of the payload, >> > although that does not seem to be properly documented by the spec. > >Sorry for being unclear, what I meant is that we probably should use >zero here. TX doesn't use in buffer actually. > >According to the spec, 0 should be the used length: > >"and len the total of bytes written into the buffer." > >> > >> > On the other hand tx messages are stated to be device read-only (in the >> > spec) so if the device writes stuff, that is certainly wrong. >> > > >Yes. > >> > If that is what happens. >> > >> > Looking at virtqueue_get_buf_ctx_split() I'm not sure that is what >> > happens. My hypothesis is that we just a last descriptor is an 'in' >> > type descriptor (i.e. a device writable one). For tx that assumption >> > would be wrong. >> > >> > I will have another look at this today and send a fix patch if my >> > suspicion is confirmed. >> >> If my suspicion is right something like: >> >> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c >> index 00f64f2f8b72..efb57898920b 100644 >> --- a/drivers/virtio/virtio_ring.c >> +++ b/drivers/virtio/virtio_ring.c >> @@ -764,6 +764,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, >> struct vring_virtqueue *vq = to_vvq(_vq); >> void *ret; >> unsigned int i; >> + bool has_in; >> u16 last_used; >> >> START_USE(vq); >> @@ -787,6 +788,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, >> vq->split.vring.used->ring[last_used].id); >> *len = virtio32_to_cpu(_vq->vdev, >> vq->split.vring.used->ring[last_used].len); >> + has_in = virtio16_to_cpu(_vq->vdev, >> + vq->split.vring.used->ring[last_used].flags) >> + & VRING_DESC_F_WRITE; > >Did you mean vring.desc actually? If yes, it's better not depend on >the descriptor ring which can be modified by the device. We've stored >the flags in desc_extra[]. > >> >> if (unlikely(i >= vq->split.vring.num)) { >> BAD_RING(vq, "id %u out of range\n", i); >> @@ -796,7 +800,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, >> BAD_RING(vq, "id %u is not a head!\n", i); >> return NULL; >> } >> - if (vq->buflen && unlikely(*len > vq->buflen[i])) { >> + if (has_in && q->buflen && unlikely(*len > vq->buflen[i])) { >> BAD_RING(vq, "used len %d is larger than in buflen %u\n", >> *len, vq->buflen[i]); >> return NULL; >> >> would fix the problem for split. I will try that out and let you know >> later. > >I'm not sure I get this, in virtqueue_add_split, the buflen[i] only >contains the in buffer length. > >I think the fixes are: > >1) fixing the vhost vsockYep, in vhost_vsock_handle_tx_kick() we should have vhost_add_used(vq, head, 0) since the device doesn't write anything.>2) use suppress_used_validation=true to let vsock driver to validate >the in buffer length >3) probably a new feature so the driver can only enable the validation >when the feature is enabled.I fully agree with these steps. Thanks, Stefano
On Mon, 22 Nov 2021 14:25:26 +0800 Jason Wang <jasowang at redhat.com> wrote:> On Mon, Nov 22, 2021 at 1:49 PM Halil Pasic <pasic at linux.ibm.com> wrote: > > > > On Mon, 22 Nov 2021 06:35:18 +0100 > > Halil Pasic <pasic at linux.ibm.com> wrote: > > > > > > I think it should be a common issue, looking at > > > > vhost_vsock_handle_tx_kick(), it did: > > > > > > > > len += sizeof(pkt->hdr); > > > > vhost_add_used(vq, head, len); > > > > > > > > which looks like a violation of the spec since it's TX. > > > > > > I'm not sure the lines above look like a violation of the spec. If you > > > examine vhost_vsock_alloc_pkt() I believe that you will agree that: > > > len == pkt->len == pkt->hdr.len > > > which makes sense since according to the spec both tx and rx messages > > > are hdr+payload. And I believe hdr.len is the size of the payload, > > > although that does not seem to be properly documented by the spec. > > Sorry for being unclear, what I meant is that we probably should use > zero here. TX doesn't use in buffer actually. > > According to the spec, 0 should be the used length: > > "and len the total of bytes written into the buffer."Right, I was wrong. I somehow assumed this is the total length and not just the number of bytes written.> > > > > > > On the other hand tx messages are stated to be device read-only (in the > > > spec) so if the device writes stuff, that is certainly wrong. > > > > > Yes. > > > > If that is what happens. > > > > > > Looking at virtqueue_get_buf_ctx_split() I'm not sure that is what > > > happens. My hypothesis is that we just a last descriptor is an 'in' > > > type descriptor (i.e. a device writable one). For tx that assumption > > > would be wrong. > > > > > > I will have another look at this today and send a fix patch if my > > > suspicion is confirmed.Yeah, I didn't remember the semantic of vq->split.vring.used->ring[last_used].len correctly, and in fact also how exactly the rings work. So your objection is correct. Maybe updating some stuff would make it easier to not make this mistake. For example the spec and also the linux header says: /* le32 is used here for ids for padding reasons. */ struct virtq_used_elem { /* Index of start of used descriptor chain. */ le32 id; /* Total length of the descriptor chain which was used (written to) */ le32 len; }; I think that comment isn't as clear as it could be. I would prefer: /* The number of bytes written into the device writable portion of the buffer described by the descriptor chain. */ I believe "the descriptor chain which was used" includes both the descriptors that map the device read only and the device write only portions of the buffer described by the descriptor chain. And the total length of that descriptor chain may be defined either as a number of the descriptors that form the chain, or the length of the buffer. One has to use the descriptor chain even if the whole buffer is device read only. So "used" == "written to" does not make any sense to me. Also something like int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int bytes_written) instead of int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len) would make it easier to read the code correctly.> > > > If my suspicion is right something like: > > > > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c > > index 00f64f2f8b72..efb57898920b 100644 > > --- a/drivers/virtio/virtio_ring.c > > +++ b/drivers/virtio/virtio_ring.c > > @@ -764,6 +764,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > > struct vring_virtqueue *vq = to_vvq(_vq); > > void *ret; > > unsigned int i; > > + bool has_in; > > u16 last_used; > > > > START_USE(vq); > > @@ -787,6 +788,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > > vq->split.vring.used->ring[last_used].id); > > *len = virtio32_to_cpu(_vq->vdev, > > vq->split.vring.used->ring[last_used].len); > > + has_in = virtio16_to_cpu(_vq->vdev, > > + vq->split.vring.used->ring[last_used].flags) > > + & VRING_DESC_F_WRITE; > > Did you mean vring.desc actually? If yes, it's better not depend on > the descriptor ring which can be modified by the device. We've stored > the flags in desc_extra[]. > > > > > if (unlikely(i >= vq->split.vring.num)) { > > BAD_RING(vq, "id %u out of range\n", i); > > @@ -796,7 +800,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > > BAD_RING(vq, "id %u is not a head!\n", i); > > return NULL; > > } > > - if (vq->buflen && unlikely(*len > vq->buflen[i])) { > > + if (has_in && q->buflen && unlikely(*len > vq->buflen[i])) { > > BAD_RING(vq, "used len %d is larger than in buflen %u\n", > > *len, vq->buflen[i]); > > return NULL; > > > > would fix the problem for split. I will try that out and let you know > > later. > > I'm not sure I get this, in virtqueue_add_split, the buflen[i] only > contains the in buffer length.Sorry my diff is indeed silly.> > I think the fixes are: > > 1) fixing the vhost vsock > 2) use suppress_used_validation=true to let vsock driver to validate > the in buffer length > 3) probably a new feature so the driver can only enable the validation > when the feature is enabled. >Makes sense! Regards, Halil
On Mon, 22 Nov 2021 14:25:26 +0800 Jason Wang <jasowang at redhat.com> wrote:> I think the fixes are: > > 1) fixing the vhost vsock > 2) use suppress_used_validation=true to let vsock driver to validate > the in buffer length > 3) probably a new feature so the driver can only enable the validation > when the feature is enabled.I'm not sure, I would consider a F_DEV_Y_FIXED_BUG_X a perfectly good feature. Frankly the set of such bugs is device implementation specific and it makes little sense to specify a feature bit that says the device implementation claims to adhere to some aspect of the specification. Also what would be the semantic of not negotiating F_DEV_Y_FIXED_BUG_X? On the other hand I see no other way to keep the validation permanently enabled for fixed implementations, and get around the problem with broken implementations. So we could have something like VHOST_USED_LEN_STRICT. Maybe, we can also think of 'warn and don't alter behavior' instead of 'warn' and alter behavior. Or maybe even not having such checks on in production, but only when testing. Regards, Halil