Jason Wang
2022-Oct-17 07:08 UTC
[PATCH 0/4] vDPA: dev config export via "vdpa dev show" command
Adding Sean and Daniel for more thoughts. On Sat, Oct 15, 2022 at 9:33 AM Si-Wei Liu <si-wei.liu at oracle.com> wrote:> > Live migration of vdpa would typically require re-instate vdpa > device with an idential set of configs on the destination node, > same way as how source node created the device in the first place. > > In order to allow live migration orchestration software to export the > initial set of vdpa attributes with which the device was created, it > will be useful if the vdpa tool can report the config on demand with > simple query.For live migration, I think the management layer should have this knowledge and they can communicate directly without bothering the vdpa tool on the source. If I was not wrong this is the way libvirt is doing now.> This will ease the orchestration software implementation > so that it doesn't have to keep track of vdpa config change, or have > to persist vdpa attributes across failure and recovery, in fear of > being killed due to accidental software error. > > In this series, the initial device config for vdpa creation will be > exported via the "vdpa dev show" command. > This is unlike the "vdpa > dev config show" command that usually goes with the live value in > the device config space, which is not reliable subject to the dynamics > of feature negotiation and possible change in device config space. > > Examples: > > 1) Create vDPA by default without any config attribute > > $ vdpa dev add mgmtdev pci/0000:41:04.2 name vdpa0 > $ vdpa dev show vdpa0 > vdpa0: type network mgmtdev pci/0000:41:04.2 vendor_id 5555 max_vqs 9 max_vq_size 256 > $ vdpa dev -jp show vdpa0 > { > "dev": { > "vdpa0": { > "type": "network", > "mgmtdev": "pci/0000:41:04.2", > "vendor_id": 5555, > "max_vqs": 9, > "max_vq_size": 256, > } > } > } > > 2) Create vDPA with config attribute(s) specified > > $ vdpa dev add mgmtdev pci/0000:41:04.2 name vdpa0 \ > mac e4:11:c6:d3:45:f0 max_vq_pairs 4 > $ vdpa dev show > vdpa0: type network mgmtdev pci/0000:41:04.2 vendor_id 5555 max_vqs 9 max_vq_size 256 > mac e4:11:c6:d3:45:f0 max_vq_pairs 4 > $ vdpa dev -jp show > { > "dev": { > "vdpa0": { > "type": "network", > "mgmtdev": "pci/0000:41:04.2",So "mgmtdev" looks not necessary for live migration. Thanks> "vendor_id": 5555, > "max_vqs": 9, > "max_vq_size": 256, > "mac": "e4:11:c6:d3:45:f0", > "max_vq_pairs": 4 > } > } > } > > --- > > Si-Wei Liu (4): > vdpa: save vdpa_dev_set_config in struct vdpa_device > vdpa: pass initial config to _vdpa_register_device() > vdpa: show dev config as-is in "vdpa dev show" output > vdpa: fix improper error message when adding vdpa dev > > drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- > drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- > drivers/vdpa/vdpa.c | 63 +++++++++++++++++++++++++++++++++--- > drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 2 +- > drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 2 +- > drivers/vdpa/vdpa_user/vduse_dev.c | 2 +- > drivers/vdpa/virtio_pci/vp_vdpa.c | 3 +- > include/linux/vdpa.h | 26 ++++++++------- > 8 files changed, 80 insertions(+), 22 deletions(-) > > -- > 1.8.3.1 >
Si-Wei Liu
2022-Oct-17 22:58 UTC
[PATCH 0/4] vDPA: dev config export via "vdpa dev show" command
On 10/17/2022 12:08 AM, Jason Wang wrote:> Adding Sean and Daniel for more thoughts. > > On Sat, Oct 15, 2022 at 9:33 AM Si-Wei Liu <si-wei.liu at oracle.com> wrote: >> Live migration of vdpa would typically require re-instate vdpa >> device with an idential set of configs on the destination node, >> same way as how source node created the device in the first place. >> >> In order to allow live migration orchestration software to export the >> initial set of vdpa attributes with which the device was created, it >> will be useful if the vdpa tool can report the config on demand with >> simple query. > For live migration, I think the management layer should have this > knowledge and they can communicate directly without bothering the vdpa > tool on the source. If I was not wrong this is the way libvirt is > doing now.I think this series doesn't conflict with what libvirt is doing now. For example it can still remember the supported features for the parent mgmtdev, and mtu and mac properties for vdpa creation, and use them to replicate vdpa device on? destination node. The extra benefit is - the management software (for live migration) doesn't need to care those mgmtdev specifics - such as what features the parent mgmtdev supports, whether some features are mandatory, and what are the default values for those, whether there's enough system or hardware resource available to create vdpa with requested features et al. This kind of process can be simplified by just getting a vdpa created with the exact same features and configus exposed via the 'vdpa dev show' command. Essentially this export facility just provides the layer of abstraction needed for virtio related device configuration and for the very core need of vdpa live migration. For e.g. what're exported can even be useful to facilitate live migration from vdpa to software virtio. Basically, it doesn't prevent libvirt from implementing another layer on top to manage vdpa between mgmtdev devices and vdpa creation, and on the other hand, would benefit light-weighted mgmt software implementation with device management and live migration orchestration decoupled in the upper level.>> This will ease the orchestration software implementation >> so that it doesn't have to keep track of vdpa config change, or have >> to persist vdpa attributes across failure and recovery, in fear of >> being killed due to accidental software error. >> >> In this series, the initial device config for vdpa creation will be >> exported via the "vdpa dev show" command. >> This is unlike the "vdpa >> dev config show" command that usually goes with the live value in >> the device config space, which is not reliable subject to the dynamics >> of feature negotiation and possible change in device config space. >> >> Examples: >> >> 1) Create vDPA by default without any config attribute >> >> $ vdpa dev add mgmtdev pci/0000:41:04.2 name vdpa0 >> $ vdpa dev show vdpa0 >> vdpa0: type network mgmtdev pci/0000:41:04.2 vendor_id 5555 max_vqs 9 max_vq_size 256 >> $ vdpa dev -jp show vdpa0 >> { >> "dev": { >> "vdpa0": { >> "type": "network", >> "mgmtdev": "pci/0000:41:04.2", >> "vendor_id": 5555, >> "max_vqs": 9, >> "max_vq_size": 256, >> } >> } >> } >> >> 2) Create vDPA with config attribute(s) specified >> >> $ vdpa dev add mgmtdev pci/0000:41:04.2 name vdpa0 \ >> mac e4:11:c6:d3:45:f0 max_vq_pairs 4 >> $ vdpa dev show >> vdpa0: type network mgmtdev pci/0000:41:04.2 vendor_id 5555 max_vqs 9 max_vq_size 256 >> mac e4:11:c6:d3:45:f0 max_vq_pairs 4 >> $ vdpa dev -jp show >> { >> "dev": { >> "vdpa0": { >> "type": "network", >> "mgmtdev": "pci/0000:41:04.2", > So "mgmtdev" looks not necessary for live migration.Right, so once the resulting device_features is exposed to the 'vdpa dev show' output, the mgmt software could infer the set of config options to recreate vdpa with, and filter out those unwanted attributes (or pick what it really wants). -Siwei> > Thanks > >> "vendor_id": 5555, >> "max_vqs": 9, >> "max_vq_size": 256, >> "mac": "e4:11:c6:d3:45:f0", >> "max_vq_pairs": 4 >> } >> } >> } >> >> --- >> >> Si-Wei Liu (4): >> vdpa: save vdpa_dev_set_config in struct vdpa_device >> vdpa: pass initial config to _vdpa_register_device() >> vdpa: show dev config as-is in "vdpa dev show" output >> vdpa: fix improper error message when adding vdpa dev >> >> drivers/vdpa/ifcvf/ifcvf_main.c | 2 +- >> drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +- >> drivers/vdpa/vdpa.c | 63 +++++++++++++++++++++++++++++++++--- >> drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 2 +- >> drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 2 +- >> drivers/vdpa/vdpa_user/vduse_dev.c | 2 +- >> drivers/vdpa/virtio_pci/vp_vdpa.c | 3 +- >> include/linux/vdpa.h | 26 ++++++++------- >> 8 files changed, 80 insertions(+), 22 deletions(-) >> >> -- >> 1.8.3.1 >>