Linux Virtualization - Apr 2018 - [virtio-dev] Re: [RFC PATCH 2/3] netdev: kernel-only IFF_HIDDEN netdevice

On Wed, Apr 18, 2018 at 04:33:34PM -0700, Samudrala, Sridhar
wrote:
> On 4/17/2018 5:26 PM, Siwei Liu wrote:
> > I ran this with a few folks offline and gathered some good feedbacks
> > that I'd like to share thus revive the discussion.
> > 
> > First of all, as illustrated in the reply below, cloud service
> > providers require transparent live migration. Specifically, the main
> > target of our case is to support SR-IOV live migration via kernel
> > upgrade while keeping the userspace of old distros unmodified. If
it's
> > because this use case is not appealing enough for the mainline to
> > adopt, I will shut up and not continue discussing, although
> > technically it's entirely possible (and there's precedent in
other
> > implementation) to do so to benefit any cloud service providers.
> > 
> > If it's just the implementation of hiding netdev itself needs to
be
> > improved, such as implementing it as attribute flag or adding linkdump
> > API, that's completely fine and we can look into that. However,
the
> > specific issue needs to be undestood beforehand is to make transparent
> > SR-IOV to be able to take over the name (so inherit all the configs)
> > from the lower netdev, which needs some games with uevents and name
> > space reservation. So far I don't think it's been well
discussed.
> > 
> > One thing in particular I'd like to point out is that the 3-netdev
> > model currently missed to address the core problem of live migration:
> > migration of hardware specific feature/state, for e.g. ethtool configs
> > and hardware offloading states. Only general network state (IP
> > address, gateway, for eg.) associated with the bypass interface can be
> > migrated. As a follow-up work, bypass driver can/should be enhanced to
> > save and apply those hardware specific configs before or after
> > migration as needed. The transparent 1-netdev model being proposed as
> > part of this patch series will be able to solve that problem naturally
> > by making all hardware specific configurations go through the central
> > bypass driver, such that hardware configurations can be replayed when
> > new VF or passthrough gets plugged back in. Although that
> > corresponding function hasn't been implemented today, I'd like
to
> > refresh everyone's mind that is the core problem any live
migration
> > proposal should have addressed.
> > 
> > If it would make things more clear to defer netdev hiding until all
> > functionalities regarding centralizing and replay are implemented,
> > we'd take advices like that and move on to implementing those
features
> > as follow-up patches. Once all needed features get done, we'd
resume
> > the work for hiding lower netdev at that point. Think it would be the
> > best to make everyone understand the big picture in advance before
> > going too far.
> 
> I think we should get the 3-netdev model integrated and add any additional
> ndo_ops/ethool ops that we would like to support/migrate before looking
into
> hiding the lower netdevs.
Once they are exposed, I don't think we'll be able to hide them -
they will be a kernel ABI.

Do you think everyone needs to hide the SRIOV device?
Or that only some users need this?

-- 
MST

On 4/18/2018 9:41 PM, Michael S. Tsirkin wrote:
> On Wed, Apr 18, 2018 at 04:33:34PM -0700, Samudrala, Sridhar wrote:
>> On 4/17/2018 5:26 PM, Siwei Liu wrote:
>>> I ran this with a few folks offline and gathered some good
feedbacks
>>> that I'd like to share thus revive the discussion.
>>>
>>> First of all, as illustrated in the reply below, cloud service
>>> providers require transparent live migration. Specifically, the
main
>>> target of our case is to support SR-IOV live migration via kernel
>>> upgrade while keeping the userspace of old distros unmodified. If
it's
>>> because this use case is not appealing enough for the mainline to
>>> adopt, I will shut up and not continue discussing, although
>>> technically it's entirely possible (and there's precedent
in other
>>> implementation) to do so to benefit any cloud service providers.
>>>
>>> If it's just the implementation of hiding netdev itself needs
to be
>>> improved, such as implementing it as attribute flag or adding
linkdump
>>> API, that's completely fine and we can look into that. However,
the
>>> specific issue needs to be undestood beforehand is to make
transparent
>>> SR-IOV to be able to take over the name (so inherit all the
configs)
>>> from the lower netdev, which needs some games with uevents and name
>>> space reservation. So far I don't think it's been well
discussed.
>>>
>>> One thing in particular I'd like to point out is that the
3-netdev
>>> model currently missed to address the core problem of live
migration:
>>> migration of hardware specific feature/state, for e.g. ethtool
configs
>>> and hardware offloading states. Only general network state (IP
>>> address, gateway, for eg.) associated with the bypass interface can
be
>>> migrated. As a follow-up work, bypass driver can/should be enhanced
to
>>> save and apply those hardware specific configs before or after
>>> migration as needed. The transparent 1-netdev model being proposed
as
>>> part of this patch series will be able to solve that problem
naturally
>>> by making all hardware specific configurations go through the
central
>>> bypass driver, such that hardware configurations can be replayed
when
>>> new VF or passthrough gets plugged back in. Although that
>>> corresponding function hasn't been implemented today, I'd
like to
>>> refresh everyone's mind that is the core problem any live
migration
>>> proposal should have addressed.
>>>
>>> If it would make things more clear to defer netdev hiding until all
>>> functionalities regarding centralizing and replay are implemented,
>>> we'd take advices like that and move on to implementing those
features
>>> as follow-up patches. Once all needed features get done, we'd
resume
>>> the work for hiding lower netdev at that point. Think it would be
the
>>> best to make everyone understand the big picture in advance before
>>> going too far.
>> I think we should get the 3-netdev model integrated and add any
additional
>> ndo_ops/ethool ops that we would like to support/migrate before looking
into
>> hiding the lower netdevs.
> Once they are exposed, I don't think we'll be able to hide them -
> they will be a kernel ABI.
>
> Do you think everyone needs to hide the SRIOV device?
> Or that only some users need this?
Hyper-V is currently supporting live migration without hiding the SR-IOV device.
So i don't
think it is a hard requirement. And also,  as we don't yet have a consensus
on how to hide
the lower netdevs, we could make it as another feature bit to hide lower netdevs
once
we have an acceptable solution.

On Wed, Apr 18, 2018 at 10:00:51PM -0700, Samudrala, Sridhar
wrote:
> On 4/18/2018 9:41 PM, Michael S. Tsirkin wrote:
> > On Wed, Apr 18, 2018 at 04:33:34PM -0700, Samudrala, Sridhar wrote:
> > > On 4/17/2018 5:26 PM, Siwei Liu wrote:
> > > > I ran this with a few folks offline and gathered some good
feedbacks
> > > > that I'd like to share thus revive the discussion.
> > > > 
> > > > First of all, as illustrated in the reply below, cloud
service
> > > > providers require transparent live migration. Specifically,
the main
> > > > target of our case is to support SR-IOV live migration via
kernel
> > > > upgrade while keeping the userspace of old distros
unmodified. If it's
> > > > because this use case is not appealing enough for the
mainline to
> > > > adopt, I will shut up and not continue discussing, although
> > > > technically it's entirely possible (and there's
precedent in other
> > > > implementation) to do so to benefit any cloud service
providers.
> > > > 
> > > > If it's just the implementation of hiding netdev itself
needs to be
> > > > improved, such as implementing it as attribute flag or
adding linkdump
> > > > API, that's completely fine and we can look into that.
However, the
> > > > specific issue needs to be undestood beforehand is to make
transparent
> > > > SR-IOV to be able to take over the name (so inherit all the
configs)
> > > > from the lower netdev, which needs some games with uevents
and name
> > > > space reservation. So far I don't think it's been
well discussed.
> > > > 
> > > > One thing in particular I'd like to point out is that
the 3-netdev
> > > > model currently missed to address the core problem of live
migration:
> > > > migration of hardware specific feature/state, for e.g.
ethtool configs
> > > > and hardware offloading states. Only general network state
(IP
> > > > address, gateway, for eg.) associated with the bypass
interface can be
> > > > migrated. As a follow-up work, bypass driver can/should be
enhanced to
> > > > save and apply those hardware specific configs before or
after
> > > > migration as needed. The transparent 1-netdev model being
proposed as
> > > > part of this patch series will be able to solve that problem
naturally
> > > > by making all hardware specific configurations go through
the central
> > > > bypass driver, such that hardware configurations can be
replayed when
> > > > new VF or passthrough gets plugged back in. Although that
> > > > corresponding function hasn't been implemented today,
I'd like to
> > > > refresh everyone's mind that is the core problem any
live migration
> > > > proposal should have addressed.
> > > > 
> > > > If it would make things more clear to defer netdev hiding
until all
> > > > functionalities regarding centralizing and replay are
implemented,
> > > > we'd take advices like that and move on to implementing
those features
> > > > as follow-up patches. Once all needed features get done,
we'd resume
> > > > the work for hiding lower netdev at that point. Think it
would be the
> > > > best to make everyone understand the big picture in advance
before
> > > > going too far.
> > > I think we should get the 3-netdev model integrated and add any
additional
> > > ndo_ops/ethool ops that we would like to support/migrate before
looking into
> > > hiding the lower netdevs.
> > Once they are exposed, I don't think we'll be able to hide
them -
> > they will be a kernel ABI.
> > 
> > Do you think everyone needs to hide the SRIOV device?
> > Or that only some users need this?
> 
> Hyper-V is currently supporting live migration without hiding the SR-IOV
device. So i don't
> think it is a hard requirement.
OK, fine.
> And also,  as we don't yet have a consensus on how to hide
> the lower netdevs, we could make it as another feature bit to hide lower
netdevs once
> we have an acceptable solution.
Guest/host interface isn't more flexible than the userspace/kernel
interface.  The feature bit you propose would say what exactly?
Hypervisor has no idea what guest kernel shows guest userspace.
Note that the backup flag doesn't tell guest kernel what to do,
it just tells guest that there is or will be a faster main device
connected to the same backend, so the backup should only be used
when main device is not present.

-- 
MST

On Wed, Apr 18, 2018 at 10:00 PM, Samudrala, Sridhar
<sridhar.samudrala at intel.com> wrote:
> On 4/18/2018 9:41 PM, Michael S. Tsirkin wrote:
>>
>> On Wed, Apr 18, 2018 at 04:33:34PM -0700, Samudrala, Sridhar wrote:
>>>
>>> On 4/17/2018 5:26 PM, Siwei Liu wrote:
>>>>
>>>> I ran this with a few folks offline and gathered some good
feedbacks
>>>> that I'd like to share thus revive the discussion.
>>>>
>>>> First of all, as illustrated in the reply below, cloud service
>>>> providers require transparent live migration. Specifically, the
main
>>>> target of our case is to support SR-IOV live migration via
kernel
>>>> upgrade while keeping the userspace of old distros unmodified.
If it's
>>>> because this use case is not appealing enough for the mainline
to
>>>> adopt, I will shut up and not continue discussing, although
>>>> technically it's entirely possible (and there's
precedent in other
>>>> implementation) to do so to benefit any cloud service
providers.
>>>>
>>>> If it's just the implementation of hiding netdev itself
needs to be
>>>> improved, such as implementing it as attribute flag or adding
linkdump
>>>> API, that's completely fine and we can look into that.
However, the
>>>> specific issue needs to be undestood beforehand is to make
transparent
>>>> SR-IOV to be able to take over the name (so inherit all the
configs)
>>>> from the lower netdev, which needs some games with uevents and
name
>>>> space reservation. So far I don't think it's been well
discussed.
>>>>
>>>> One thing in particular I'd like to point out is that the
3-netdev
>>>> model currently missed to address the core problem of live
migration:
>>>> migration of hardware specific feature/state, for e.g. ethtool
configs
>>>> and hardware offloading states. Only general network state (IP
>>>> address, gateway, for eg.) associated with the bypass interface
can be
>>>> migrated. As a follow-up work, bypass driver can/should be
enhanced to
>>>> save and apply those hardware specific configs before or after
>>>> migration as needed. The transparent 1-netdev model being
proposed as
>>>> part of this patch series will be able to solve that problem
naturally
>>>> by making all hardware specific configurations go through the
central
>>>> bypass driver, such that hardware configurations can be
replayed when
>>>> new VF or passthrough gets plugged back in. Although that
>>>> corresponding function hasn't been implemented today,
I'd like to
>>>> refresh everyone's mind that is the core problem any live
migration
>>>> proposal should have addressed.
>>>>
>>>> If it would make things more clear to defer netdev hiding until
all
>>>> functionalities regarding centralizing and replay are
implemented,
>>>> we'd take advices like that and move on to implementing
those features
>>>> as follow-up patches. Once all needed features get done,
we'd resume
>>>> the work for hiding lower netdev at that point. Think it would
be the
>>>> best to make everyone understand the big picture in advance
before
>>>> going too far.
>>>
>>> I think we should get the 3-netdev model integrated and add any
>>> additional
>>> ndo_ops/ethool ops that we would like to support/migrate before
looking
>>> into
>>> hiding the lower netdevs.
>>
>> Once they are exposed, I don't think we'll be able to hide them
-
>> they will be a kernel ABI.
>>
>> Do you think everyone needs to hide the SRIOV device?
>> Or that only some users need this?
>
>
> Hyper-V is currently supporting live migration without hiding the SR-IOV
> device. So i don't
> think it is a hard requirement. And also,  as we don't yet have a
consensus
This is a vague point as Hyper-V is mostly Windows oriented: the
target users don't change adapter settings in device manager much as
it's hidden too deep already. Actually it does not address the general
case for SR-IOV live migration but just a subset, why are we making
such comparison?

Note it's always the hard requirement for live migration that *all
states* should be migrated no matter what the implementation it is
going to be. The current 3-netdev model is remote to be useful for
real world scenario and it has no advantage compared to using
userspace generic bonding.

-Siwei
> on how to hide
> the lower netdevs, we could make it as another feature bit to hide lower
> netdevs once
> we have an acceptable solution.
>