Linux Virtualization - Jul 2020 - [PATCH 0/7] *** IRQ offloading for vDPA ***

On 2020/7/16 ??12:13, Zhu, Lingshan wrote:
>
>
> On 7/16/2020 12:02 PM, Jason Wang wrote:
>>
>> On 2020/7/16 ??11:59, Zhu, Lingshan wrote:
>>>
>>> On 7/16/2020 10:59 AM, Jason Wang wrote:
>>>>
>>>> On 2020/7/16 ??9:39, Zhu, Lingshan wrote:
>>>>>
>>>>>
>>>>> On 7/15/2020 9:43 PM, Jason Wang wrote:
>>>>>>
>>>>>> On 2020/7/12 ??10:52, Zhu Lingshan wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> This series intends to implement IRQ offloading for
>>>>>>> vhost_vdpa.
>>>>>>>
>>>>>>> By the feat of irq forwarding facilities like
posted
>>>>>>> interrupt on X86, irq bypass can? help deliver
>>>>>>> interrupts to vCPU directly.
>>>>>>>
>>>>>>> vDPA devices have dedicated hardware backends like
VFIO
>>>>>>> pass-throughed devices. So it would be possible to
setup
>>>>>>> irq offloading(irq bypass) for vDPA devices and
gain
>>>>>>> performance improvements.
>>>>>>>
>>>>>>> In my testing, with this feature, we can save 0.1ms
>>>>>>> in a ping between two VFs on average.
>>>>>>
>>>>>>
>>>>>> Hi Lingshan:
>>>>>>
>>>>>> During the virtio-networking meeting, Michael spots two
possible
>>>>>> issues:
>>>>>>
>>>>>> 1) do we need an new uAPI to stop the irq offloading?
>>>>>> 2) can interrupt lost during the eventfd ctx?
>>>>>>
>>>>>> For 1) I think we probably not, we can allocate an
independent
>>>>>> eventfd which does not map to MSIX. So the consumer
can't match
>>>>>> the producer and we fallback to eventfd based irq.
>>>>> Hi Jason,
>>>>>
>>>>> I wonder why we need to stop irq offloading, but if we need
to do
>>>>> so, maybe a new uAPI would be more intuitive to me,
>>>>> but why and who(user? qemu?) shall initialize this process,
based
>>>>> on what kinda of basis to make the decision?
>>>>
>>>>
>>>> The reason is we may want to fallback to software datapath for
some
>>>> reason (e.g software assisted live migration). In this case we
need
>>>> intercept device write to used ring so we can not offloading 
>>>> virtqueue interrupt in this case.
>>> so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a 
>>> VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully control 
>>> this? Or any better approaches? 
>>
>>
>> Probably not, it's as simple as allocating another eventfd (but not
>> irqfd), and pass it to vhost-vdpa. Then the offloading is disabled 
>> since it doesn't have a consumer.
> OK, sounds like QEMU work, no need to take care in this series, right?

That's my understanding.

Thanks

>
> Thanks,
> BR
> Zhu Lingshan
>>
>> Thanks
>>
>>

On Thu, Jul 16, 2020 at 12:20:09PM +0800, Jason Wang
wrote:
> 
> On 2020/7/16 ??12:13, Zhu, Lingshan wrote:
> > 
> > 
> > On 7/16/2020 12:02 PM, Jason Wang wrote:
> > > 
> > > On 2020/7/16 ??11:59, Zhu, Lingshan wrote:
> > > > 
> > > > On 7/16/2020 10:59 AM, Jason Wang wrote:
> > > > > 
> > > > > On 2020/7/16 ??9:39, Zhu, Lingshan wrote:
> > > > > > 
> > > > > > 
> > > > > > On 7/15/2020 9:43 PM, Jason Wang wrote:
> > > > > > > 
> > > > > > > On 2020/7/12 ??10:52, Zhu Lingshan wrote:
> > > > > > > > Hi All,
> > > > > > > > 
> > > > > > > > This series intends to implement IRQ
offloading for
> > > > > > > > vhost_vdpa.
> > > > > > > > 
> > > > > > > > By the feat of irq forwarding facilities
like posted
> > > > > > > > interrupt on X86, irq bypass can? help
deliver
> > > > > > > > interrupts to vCPU directly.
> > > > > > > > 
> > > > > > > > vDPA devices have dedicated hardware
backends like VFIO
> > > > > > > > pass-throughed devices. So it would be
possible to setup
> > > > > > > > irq offloading(irq bypass) for vDPA
devices and gain
> > > > > > > > performance improvements.
> > > > > > > > 
> > > > > > > > In my testing, with this feature, we can
save 0.1ms
> > > > > > > > in a ping between two VFs on average.
> > > > > > > 
> > > > > > > 
> > > > > > > Hi Lingshan:
> > > > > > > 
> > > > > > > During the virtio-networking meeting, Michael
spots
> > > > > > > two possible issues:
> > > > > > > 
> > > > > > > 1) do we need an new uAPI to stop the irq
offloading?
> > > > > > > 2) can interrupt lost during the eventfd ctx?
> > > > > > > 
> > > > > > > For 1) I think we probably not, we can
allocate an
> > > > > > > independent eventfd which does not map to
MSIX. So
> > > > > > > the consumer can't match the producer and
we
> > > > > > > fallback to eventfd based irq.
> > > > > > Hi Jason,
> > > > > > 
> > > > > > I wonder why we need to stop irq offloading, but
if we
> > > > > > need to do so, maybe a new uAPI would be more
intuitive
> > > > > > to me,
> > > > > > but why and who(user? qemu?) shall initialize this
> > > > > > process, based on what kinda of basis to make the
> > > > > > decision?
> > > > > 
> > > > > 
> > > > > The reason is we may want to fallback to software
datapath
> > > > > for some reason (e.g software assisted live migration).
In
> > > > > this case we need intercept device write to used ring
so we
> > > > > can not offloading virtqueue interrupt in this case.
> > > > so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a
> > > > VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully
> > > > control this? Or any better approaches?
> > > 
> > > 
> > > Probably not, it's as simple as allocating another eventfd
(but not
> > > irqfd), and pass it to vhost-vdpa. Then the offloading is
disabled
> > > since it doesn't have a consumer.
> > OK, sounds like QEMU work, no need to take care in this series, right?
> 
> 
> That's my understanding.
> 
> Thanks
Let's confirm a switch happens atomically so each interrupt
is sent either to eventfd to guest directly though.
> 
> > 
> > Thanks,
> > BR
> > Zhu Lingshan
> > > 
> > > Thanks
> > > 
> > >

On 2020/7/16 ??2:15, Michael S. Tsirkin wrote:
> On Thu, Jul 16, 2020 at 12:20:09PM +0800, Jason Wang wrote:
>> On 2020/7/16 ??12:13, Zhu, Lingshan wrote:
>>> On 7/16/2020 12:02 PM, Jason Wang wrote:
>>>> On 2020/7/16 ??11:59, Zhu, Lingshan wrote:
>>>>> On 7/16/2020 10:59 AM, Jason Wang wrote:
>>>>>> On 2020/7/16 ??9:39, Zhu, Lingshan wrote:
>>>>>>> On 7/15/2020 9:43 PM, Jason Wang wrote:
>>>>>>>> On 2020/7/12 ??10:52, Zhu Lingshan wrote:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> This series intends to implement IRQ
offloading for
>>>>>>>>> vhost_vdpa.
>>>>>>>>>
>>>>>>>>> By the feat of irq forwarding facilities
like posted
>>>>>>>>> interrupt on X86, irq bypass can? help
deliver
>>>>>>>>> interrupts to vCPU directly.
>>>>>>>>>
>>>>>>>>> vDPA devices have dedicated hardware
backends like VFIO
>>>>>>>>> pass-throughed devices. So it would be
possible to setup
>>>>>>>>> irq offloading(irq bypass) for vDPA devices
and gain
>>>>>>>>> performance improvements.
>>>>>>>>>
>>>>>>>>> In my testing, with this feature, we can
save 0.1ms
>>>>>>>>> in a ping between two VFs on average.
>>>>>>>> Hi Lingshan:
>>>>>>>>
>>>>>>>> During the virtio-networking meeting, Michael
spots
>>>>>>>> two possible issues:
>>>>>>>>
>>>>>>>> 1) do we need an new uAPI to stop the irq
offloading?
>>>>>>>> 2) can interrupt lost during the eventfd ctx?
>>>>>>>>
>>>>>>>> For 1) I think we probably not, we can allocate
an
>>>>>>>> independent eventfd which does not map to MSIX.
So
>>>>>>>> the consumer can't match the producer and
we
>>>>>>>> fallback to eventfd based irq.
>>>>>>> Hi Jason,
>>>>>>>
>>>>>>> I wonder why we need to stop irq offloading, but if
we
>>>>>>> need to do so, maybe a new uAPI would be more
intuitive
>>>>>>> to me,
>>>>>>> but why and who(user? qemu?) shall initialize this
>>>>>>> process, based on what kinda of basis to make the
>>>>>>> decision?
>>>>>> The reason is we may want to fallback to software
datapath
>>>>>> for some reason (e.g software assisted live migration).
In
>>>>>> this case we need intercept device write to used ring
so we
>>>>>> can not offloading virtqueue interrupt in this case.
>>>>> so add a VHOST_VDPA_STOP_IRQ_OFFLOADING? Then do we need a
>>>>> VHOST_VDPA_START_IRQ_OFFLOADING, then let userspace fully
>>>>> control this? Or any better approaches?
>>>> Probably not, it's as simple as allocating another eventfd
(but not
>>>> irqfd), and pass it to vhost-vdpa. Then the offloading is
disabled
>>>> since it doesn't have a consumer.
>>> OK, sounds like QEMU work, no need to take care in this series,
right?
>> That's my understanding.
>>
>> Thanks
> Let's confirm a switch happens atomically so each interrupt
> is sent either to eventfd to guest directly though.

I think it's safe since:

1) we don't alloc/free interrupt during the eventfd change
2) The irte is modified automatically through cmpxchg_double() in 
modify_irte(), so the interrupt is either remapping to eventfd or pi 
descriptor

Thanks

>