On 10/27/2023 2:31 AM, Jason Wang wrote:> On Fri, Oct 27, 2023 at 4:07?AM Steve Sistare <steven.sistare at
oracle.com> wrote:
>>
>> mlx5_vdpa does not preserve userland's view of vring base for the
control
>> queue in the following sequence:
>>
>> ioctl VHOST_SET_VRING_BASE
>> ioctl VHOST_VDPA_SET_STATUS VIRTIO_CONFIG_S_DRIVER_OK
>> mlx5_vdpa_set_status()
>> setup_cvq_vring()
>> vringh_init_iotlb()
>> vringh_init_kern()
>> vrh->last_avail_idx = 0;
>> ioctl VHOST_GET_VRING_BASE
>>
>> To fix, restore the value of cvq->vring.last_avail_idx after calling
>> vringh_init_iotlb.
>>
>> Signed-off-by: Steve Sistare <steven.sistare at oracle.com>
>> ---
>> drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 ++++++-
>> drivers/vhost/vringh.c | 30 ++++++++++++++++++++++++++++++
>> include/linux/vringh.h | 2 ++
>> 3 files changed, 38 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
b/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> index 946488b8989f..f64758143115 100644
>> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
>> @@ -2795,13 +2795,18 @@ static int setup_cvq_vring(struct mlx5_vdpa_dev
*mvdev)
>> struct mlx5_control_vq *cvq = &mvdev->cvq;
>> int err = 0;
>>
>> - if (mvdev->actual_features &
BIT_ULL(VIRTIO_NET_F_CTRL_VQ))
>> + if (mvdev->actual_features &
BIT_ULL(VIRTIO_NET_F_CTRL_VQ)) {
>> + u16 last_avail_idx = cvq->vring.last_avail_idx;
>> +
>> err = vringh_init_iotlb(&cvq->vring,
mvdev->actual_features,
>> MLX5_CVQ_MAX_ENT, false,
>> (struct vring_desc
*)(uintptr_t)cvq->desc_addr,
>> (struct vring_avail
*)(uintptr_t)cvq->driver_addr,
>> (struct vring_used
*)(uintptr_t)cvq->device_addr);
>>
>> + if (!err)
>> + vringh_set_base_iotlb(&cvq->vring,
last_avail_idx);
>
> Btw, vringh_set_base_iotlb() deserves an independent patch and it
> seems it is not specific to IOTLB,
Agreed on both. I initially submitted the smallest change needed to show the
problem.
so we probably need an indirection> to have vringh_set_base() first.
Not sure what you mean. I have defined:
static inline int __vringh_set_base() ...
int vringh_set_base_iotlb()
return __vringh_set_base(vrh, idx, putu16_iotlb);
to which I would add:
int vringh_set_base_user()
return __vringh_set_base(vrh, idx, putu16_user);
int vringh_set_base_kern()
return __vringh_set_base(vrh, idx, putu16_kern;
all in the same patch.
The call site in mlx5_vnet.c would be a 2nd patch.
- Steve
> Or I wonder if it's better to just introduce a new parameter to
> vringh_init_iotlb()...
I considered that, but IMO the parameter list there is already large, and it
would
be strange to add a parameter for the initial value of avail, but not for used,
and
no one needs the latter.
- Steve
>> + }
>> return err;
>> }
>>
>> diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c
>> index 7b8fd977f71c..799762c83007 100644
>> --- a/drivers/vhost/vringh.c
>> +++ b/drivers/vhost/vringh.c
>> @@ -595,6 +595,24 @@ static inline void __vringh_notify_disable(struct
vringh *vrh,
>> }
>> }
>>
>> +static inline int __vringh_set_base(struct vringh *vrh, u16 idx,
>> + int (*putu16)(const struct vringh *vrh,
>> + __virtio16 *p, u16 val))
>> +{
>> + int ret;
>> +
>> + ret = putu16(vrh, &vrh->vring.avail->idx, idx);
>> + if (ret)
>> + return ret;
>> +
>> + ret = putu16(vrh, &vrh->vring.used->idx, idx);
>> + if (ret)
>> + return ret;
>> +
>> + vrh->last_avail_idx = vrh->last_used_idx = idx;
>> + return 0;
>> +}
>> +
>> /* Userspace access helpers: in this case, addresses are really
userspace. */
>> static inline int getu16_user(const struct vringh *vrh, u16 *val,
const __virtio16 *p)
>> {
>> @@ -1456,6 +1474,18 @@ void vringh_set_iotlb(struct vringh *vrh, struct
vhost_iotlb *iotlb,
>> }
>> EXPORT_SYMBOL(vringh_set_iotlb);
>>
>> +/**
>> + * vringh_set_base_iotlb - set avail_idx and used_idx
>> + * @vrh: the vring
>> + * @idx: the value to set
>> + */
>> +int vringh_set_base_iotlb(struct vringh *vrh, u16 idx)
>> +{
>> + return __vringh_set_base(vrh, idx, putu16_iotlb);
>> +}
>> +EXPORT_SYMBOL(vringh_set_base_iotlb);
>> +
>> +
>> /**
>> * vringh_getdesc_iotlb - get next available descriptor from ring with
>> * IOTLB.
>> diff --git a/include/linux/vringh.h b/include/linux/vringh.h
>> index c3a8117dabe8..e9b8af4e6a5e 100644
>> --- a/include/linux/vringh.h
>> +++ b/include/linux/vringh.h
>> @@ -306,6 +306,8 @@ int vringh_init_iotlb_va(struct vringh *vrh, u64
features,
>> struct vring_avail *avail,
>> struct vring_used *used);
>>
>> +int vringh_set_base_iotlb(struct vringh *vrh, u16 idx);
>> +
>> int vringh_getdesc_iotlb(struct vringh *vrh,
>> struct vringh_kiov *riov,
>> struct vringh_kiov *wiov,
>> --
>> 2.39.3
>>
>