Virtualization - Oct 2023 - [PATCH vhost 16/16] vdpa/mlx5: Update cvq iotlb mapping on ASID change

On Thu, 2023-10-05 at 12:41 +0200, Eugenio Perez Martin
wrote:
> On Thu, Sep 28, 2023 at 6:50?PM Dragos Tatulea <dtatulea at
nvidia.com> wrote:
> > 
> > For the following sequence:
> > - cvq group is in ASID 0
> > - .set_map(1, cvq_iotlb)
> > - .set_group_asid(cvq_group, 1)
> > 
> > ... the cvq mapping from ASID 0 will be used. This is not always
correct
> > behaviour.
> > 
> > This patch adds support for the above mentioned flow by saving the
iotlb
> > on each .set_map and updating the cvq iotlb with it on a cvq group
change.
> > 
> > Signed-off-by: Dragos Tatulea <dtatulea at nvidia.com>
> > ---
> > ?drivers/vdpa/mlx5/core/mlx5_vdpa.h |? 2 ++
> > ?drivers/vdpa/mlx5/core/mr.c??????? | 26 ++++++++++++++++++++++++++
> > ?drivers/vdpa/mlx5/net/mlx5_vnet.c? |? 9 ++++++++-
> > ?3 files changed, 36 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> > b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> > index ae09296f4270..db988ced5a5d 100644
> > --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> > +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
> > @@ -32,6 +32,8 @@ struct mlx5_vdpa_mr {
> > ??????? unsigned long num_directs;
> > ??????? unsigned long num_klms;
> > 
> > +?????? struct vhost_iotlb *iotlb;
> > +
> > ??????? bool user_mr;
> > ?};
> > 
> > diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
> > index a4135c16b5bf..403c08271489 100644
> > --- a/drivers/vdpa/mlx5/core/mr.c
> > +++ b/drivers/vdpa/mlx5/core/mr.c
> > @@ -499,6 +499,8 @@ static void _mlx5_vdpa_destroy_mr(struct
mlx5_vdpa_dev
> > *mvdev, struct mlx5_vdpa_
> > ??????????????? destroy_user_mr(mvdev, mr);
> > ??????? else
> > ??????????????? destroy_dma_mr(mvdev, mr);
> > +
> > +?????? vhost_iotlb_free(mr->iotlb);
> > ?}
> > 
> > ?void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev,
> > @@ -558,6 +560,30 @@ static int _mlx5_vdpa_create_mr(struct
mlx5_vdpa_dev
> > *mvdev,
> > ??????? else
> > ??????????????? err = create_dma_mr(mvdev, mr);
> > 
> > +?????? if (err)
> > +?????????????? return err;
> > +
> > +?????? mr->iotlb = vhost_iotlb_alloc(0, 0);
> > +?????? if (!mr->iotlb) {
> > +?????????????? err = -ENOMEM;
> > +?????????????? goto err_mr;
> > +?????? }
> > +
> > +?????? err = dup_iotlb(mr->iotlb, iotlb);
> > +?????? if (err)
> > +?????????????? goto err_iotlb;
> > +
> > +?????? return 0;
> > +
> > +err_iotlb:
> > +?????? vhost_iotlb_free(mr->iotlb);
> > +
> > +err_mr:
> > +?????? if (iotlb)
> > +?????????????? destroy_user_mr(mvdev, mr);
> > +?????? else
> > +?????????????? destroy_dma_mr(mvdev, mr);
> > +
> > ??????? return err;
> > ?}
> > 
> > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > index 46441e41892c..fc5d6b989a5a 100644
> > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > @@ -3154,12 +3154,19 @@ static int mlx5_set_group_asid(struct
vdpa_device
> > *vdev, u32 group,
> > ?????????????????????????????? unsigned int asid)
> > ?{
> > ??????? struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev);
> > +?????? int err = 0;
> > 
> > ??????? if (group >= MLX5_VDPA_NUMVQ_GROUPS || asid >=
MLX5_VDPA_NUM_AS)
> > ??????????????? return -EINVAL;
> > 
> > ??????? mvdev->group2asid[group] = asid;
> > -?????? return 0;
> > +
> > +?????? mutex_lock(&mvdev->mr_mtx);
> > +?????? if (group == MLX5_VDPA_CVQ_GROUP &&
mvdev->mr[asid])
> > +?????????????? err = mlx5_vdpa_update_cvq_iotlb(mvdev,
mvdev->mr[asid]-
> > >iotlb, asid);
> 
> Do we need to protect here in case userspace sets the same ASID twice?
> mlx5_vdpa_update_cvq_iotlb shouldn't call dup_iotlb with the same src
> and dst.
Why would src and dst be identical? Doesn't dup_iotlb create a copy of src?

That being said, there should be a check in dup_iotlb for dst == src. Not sure
if this is an error or success though...

Thanks,
Dragos
- 
> Apart from that:
> 
> Acked-by: Eugenio P?rez <eperezma at redhat.com>
> 
> > +?????? mutex_unlock(&mvdev->mr_mtx);
> > +
> > +?????? return err;
> > ?}
> > 
> > ?static const struct vdpa_config_ops mlx5_vdpa_ops = {
> > --
> > 2.41.0
> > 
>