[External] Re: [PATCH 0/7] Introduce vdpa management tool

Parav Pandit
Wed, 02 Dec 2020 11:13:09 -0000

> From: Yongji Xie <xieyongji at bytedance.com>
> Sent: Wednesday, December 2, 2020 2:52 PM
>
> On Wed, Dec 2, 2020 at 12:53 PM Parav Pandit <parav at nvidia.com>
wrote:
> >
> >
> >
> > > From: Yongji Xie <xieyongji at bytedance.com>
> > > Sent: Wednesday, December 2, 2020 9:00 AM
> > >
> > > On Tue, Dec 1, 2020 at 11:59 PM Parav Pandit <parav at
nvidia.com> wrote:
> > > >
> > > >
> > > >
> > > > > From: Yongji Xie <xieyongji at bytedance.com>
> > > > > Sent: Tuesday, December 1, 2020 7:49 PM
> > > > >
> > > > > On Tue, Dec 1, 2020 at 7:32 PM Parav Pandit <parav
at nvidia.com>
> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > > From: Yongji Xie <xieyongji at
bytedance.com>
> > > > > > > Sent: Tuesday, December 1, 2020 3:26 PM
> > > > > > >
> > > > > > > On Tue, Dec 1, 2020 at 2:25 PM Jason Wang
> > > > > > > <jasowang at redhat.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On 2020/11/30 ??3:07, Yongji Xie wrote:
> > > > > > > > >>> Thanks for adding me,
Jason!
> > > > > > > > >>>
> > > > > > > > >>> Now I'm working on a v2
patchset for VDUSE (vDPA
> > > > > > > > >>> Device in
> > > > > > > > >>> Userspace) [1]. This tool
is very useful for the vduse device.
> > > > > > > > >>> So I'm considering
integrating this into my v2 patchset.
> > > > > > > > >>> But there is one problem?
> > > > > > > > >>>
> > > > > > > > >>> In this tool, vdpa device
config action and enable
> > > > > > > > >>> action are combined into
one netlink msg:
> > > > > > > > >>> VDPA_CMD_DEV_NEW. But in
> > > > > vduse
> > > > > > > > >>> case, it needs to be
splitted because a chardev should
> > > > > > > > >>> be created and opened by a
userspace process before we
> > > > > > > > >>> enable the vdpa device
(call vdpa_register_device()).
> > > > > > > > >>>
> > > > > > > > >>> So I'd like to know
whether it's possible (or have
> > > > > > > > >>> some
> > > > > > > > >>> plans) to add two new
netlink msgs something like:
> > > > > > > > >>> VDPA_CMD_DEV_ENABLE
> > > > > > > and
> > > > > > > > >>> VDPA_CMD_DEV_DISABLE to
make the config path more
> flexible.
> > > > > > > > >>>
> > > > > > > > >> Actually, we've discussed
such intermediate step in
> > > > > > > > >> some early discussion. It looks
to me VDUSE could be
> > > > > > > > >> one of the users of
> > > this.
> > > > > > > > >>
> > > > > > > > >> Or I wonder whether we can
switch to use anonymous
> > > > > > > > >> inode(fd) for VDUSE then
fetching it via an
> > > > > > > > >> VDUSE_GET_DEVICE_FD
> > > ioctl?
> > > > > > > > >>
> > > > > > > > > Yes, we can. Actually the current
implementation in
> > > > > > > > > VDUSE is like this.  But seems like
this is still a intermediate
> step.
> > > > > > > > > The fd should be binded to a name
or something else
> > > > > > > > > which need to be configured before.
> > > > > > > >
> > > > > > > >
> > > > > > > > The name could be specified via the
netlink. It looks to
> > > > > > > > me the real issue is that until the
device is connected
> > > > > > > > with a userspace, it can't be used.
So we also need to
> > > > > > > > fail the enabling if it doesn't
> > > > > opened.
> > > > > > > >
> > > > > > >
> > > > > > > Yes, that's true. So you mean we can
firstly try to fetch
> > > > > > > the fd binded to a name/vduse_id via an
VDUSE_GET_DEVICE_FD,
> > > > > > > then use the name/vduse_id as a attribute to
create vdpa
> > > > > > > device? It looks fine to
> > > me.
> > > > > >
> > > > > > I probably do not well understand. I tried reading
patch [1]
> > > > > > and few things
> > > > > do not look correct as below.
> > > > > > Creating the vdpa device on the bus device and
destroying the
> > > > > > device from
> > > > > the workqueue seems unnecessary and racy.
> > > > > >
> > > > > > It seems vduse driver needs
> > > > > > This is something should be done as part of the
vdpa dev add
> > > > > > command,
> > > > > instead of connecting two sides separately and ensuring
race
> > > > > free access to it.
> > > > > >
> > > > > > So VDUSE_DEV_START and VDUSE_DEV_STOP should
possibly be
> avoided.
> > > > > >
> > > > >
> > > > > Yes, we can avoid these two ioctls with the help of the
management
> tool.
> > > > >
> > > > > > $ vdpa dev add parentdev vduse_mgmtdev type net
name foo2
> > > > > >
> > > > > > When above command is executed it creates
necessary vdpa
> > > > > > device
> > > > > > foo2
> > > > > on the bus.
> > > > > > When user binds foo2 device with the vduse driver,
in the
> > > > > > probe(), it
> > > > > creates respective char device to access it from user
space.
> > > > >
> > > > I see. So vduse cannot work with any existing vdpa devices
like
> > > > ifc, mlx5 or
> > > netdevsim.
> > > > It has its own implementation similar to fuse with its own
backend of
> choice.
> > > > More below.
> > > >
> > > > > But vduse driver is not a vdpa bus driver. It works
like vdpasim
> > > > > driver, but offloads the data plane and control plane
to a user space
> process.
> > > >
> > > > In that case to draw parallel lines,
> > > >
> > > > 1. netdevsim:
> > > > (a) create resources in kernel sw
> > > > (b) datapath simulates in kernel
> > > >
> > > > 2. ifc + mlx5 vdpa dev:
> > > > (a) creates resource in hw
> > > > (b) data path is in hw
> > > >
> > > > 3. vduse:
> > > > (a) creates resources in userspace sw
> > > > (b) data path is in user space.
> > > > hence creates data path resources for user space.
> > > > So char device is created, removed as result of vdpa device
creation.
> > > >
> > > > For example,
> > > > $ vdpa dev add parentdev vduse_mgmtdev type net name foo2
> > > >
> > > > Above command will create char device for user space.
> > > >
> > > > Similar command for ifc/mlx5 would have created similar
channel
> > > > for rest of
> > > the config commands in hw.
> > > > vduse channel = char device, eventfd etc.
> > > > ifc/mlx5 hw channel = bar, irq, command interface etc Netdev
sim
> > > > channel = sw direct calls
> > > >
> > > > Does it make sense?
> > >
> > > In my understanding, to make vdpa work, we need a backend
(datapath
> > > resources) and a frontend (a vdpa device attached to a vdpa bus).
In
> > > the above example, it looks like we use the command "vdpa
dev add ..."
> > >  to create a backend, so do we need another command to create a
> frontend?
> > >
> > For block device there is certainly some backend to process the IOs.
> > Sometimes backend to be setup first, before its front end is exposed.
>
> Yes, the backend need to be setup firstly, this is vendor device specific,
not
> vdpa specific.
>
> > "vdpa dev add" is the front end command who connects to the
backend
> (implicitly) for network device.
> >
> > vhost->vdpa_block_device->backend_io_processor (usr,hw,kernel).
> >
> > And it needs a way to connect to backend when explicitly specified
during
> creation time.
> > Something like,
> > $ vdpa dev add parentdev vdpa_vduse type block name foo3 handle
> <uuid>
> > In above example some vendor device specific unique handle is passed
> based on backend setup in hardware/user space.
> >
>
> Yes, we can work like this. After we setup a backend through an anonymous
> inode(fd) from /dev/vduse, we can get a unique handle. Then use it to
> create a frontend which will connect to the specific backend.

I do not fully understand the inode. But I assume this is some unique handle say
uuid or something that both sides backend and vdpa device understand.
It cannot be some kernel internal handle expose to user space.

>
> > In below 3 examples, vdpa block simulator is connecting to backend
block
> or file.
> >
> > $ vdpa dev add parentdev vdpa_blocksim type block name foo4 blockdev
> > /dev/zero
> >
> > $ vdpa dev add parentdev vdpa_blocksim type block name foo5 blockdev
> > /dev/sda2 size=100M offset=10M
> >
> > $ vdpa dev add parentdev vdpa_block filebackend_sim type block name
> > foo6 file /root/file_backend.txt
> >
> > Or may be backend connects to the created vdpa device is bound to the
> driver.
> > Can vduse attach to the created vdpa block device through the char
device
> and establish the channel to receive IOs, and to setup the block config
space?
> >
>
> How to create the vdpa block device? If we use the command "vdpa dev
> add..", the command will hang there until a vduse process attaches to
the
> vdpa block device.
I was suggesting that vdpa device is created, but it doesn?t have backend
attached to it.
It is attached to the backend when ioctl() side does enough setup. This state is
handled internally the vduse driver.

But the above method of preparing backend looks more sane.

Regardless of which method is preferred, vduse driver must need a state to
detach the vdpa bus device queues etc from the user space.
This is needed because user space process can terminate anytime resulting in
detaching dpa bus device in_use by the vhost side.