Linux Virtualization - Jun 2013 - [PATCH RFC] virtio-pci: new config layout: using memory BAR

Gleb Natapov <gleb at redhat.com> writes:
> On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
>> "H. Peter Anvin" <hpa at zytor.com> writes:
>> 
>> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
>> >>>
>> >>> Definitely an option.  However, we want to be able to boot
from native
>> >>> devices, too, so having an I/O BAR (which would not be
used by the OS
>> >>> driver) should still at the very least be an option.
>> >> 
>> >> What makes it so difficult to work with an MMIO bar for PCI-e?
>> >> 
>> >> With legacy PCI, tracking allocation of MMIO vs. PIO is pretty
straight
>> >> forward.  Is there something special about PCI-e here?
>> >> 
>> >
>> > It's not tracking allocation.  It is that accessing memory
above 1 MiB
>> > is incredibly painful in the BIOS environment, which basically
means
>> > MMIO is inaccessible.
>> 
>> Oh, you mean in real mode.
>> 
>> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
>> There are loads of ASSERT32FLAT()s in the code to make sure of this.
>> 
> Well, not exactly. Initialization is done in 32bit, but disk
> reads/writes are done in 16bit mode since it should work from int13
> interrupt handler. The only way I know to access MMIO bars from 16 bit
> is to use SMM which we do not have in KVM.
Ah, if it's just the dataplane operations then there's another solution.

We can introduce a virtqueue flag that asks the backend to poll for new
requests.  Then SeaBIOS can add the request to the queue and not worry
about kicking or reading the ISR.

SeaBIOS is polling for completion anyway.

Regards,

Anthony Liguori
>
> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori
wrote:
> Gleb Natapov <gleb at redhat.com> writes:
> 
> > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
> >> "H. Peter Anvin" <hpa at zytor.com> writes:
> >> 
> >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> >> >>>
> >> >>> Definitely an option.  However, we want to be able to
boot from native
> >> >>> devices, too, so having an I/O BAR (which would not
be used by the OS
> >> >>> driver) should still at the very least be an option.
> >> >> 
> >> >> What makes it so difficult to work with an MMIO bar for
PCI-e?
> >> >> 
> >> >> With legacy PCI, tracking allocation of MMIO vs. PIO is
pretty straight
> >> >> forward.  Is there something special about PCI-e here?
> >> >> 
> >> >
> >> > It's not tracking allocation.  It is that accessing
memory above 1 MiB
> >> > is incredibly painful in the BIOS environment, which
basically means
> >> > MMIO is inaccessible.
> >> 
> >> Oh, you mean in real mode.
> >> 
> >> SeaBIOS runs the virtio code in 32-bit mode with a flat memory
layout.
> >> There are loads of ASSERT32FLAT()s in the code to make sure of
this.
> >> 
> > Well, not exactly. Initialization is done in 32bit, but disk
> > reads/writes are done in 16bit mode since it should work from int13
> > interrupt handler. The only way I know to access MMIO bars from 16 bit
> > is to use SMM which we do not have in KVM.
> 
> Ah, if it's just the dataplane operations then there's another
solution.
> 
> We can introduce a virtqueue flag that asks the backend to poll for new
> requests.  Then SeaBIOS can add the request to the queue and not worry
> about kicking or reading the ISR.
> 
And when backend stops polling? When guest's driver reconfigure it? But
what if this will never happen?
> SeaBIOS is polling for completion anyway.
> 
> Regards,
> 
> Anthony Liguori
> 
> >
> > --
> > 			Gleb.
> > --
> > To unsubscribe from this list: send the line "unsubscribe
kvm" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
			Gleb.

On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori
wrote:
> Gleb Natapov <gleb at redhat.com> writes:
> 
> > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
> >> "H. Peter Anvin" <hpa at zytor.com> writes:
> >> 
> >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> >> >>>
> >> >>> Definitely an option.  However, we want to be able to
boot from native
> >> >>> devices, too, so having an I/O BAR (which would not
be used by the OS
> >> >>> driver) should still at the very least be an option.
> >> >> 
> >> >> What makes it so difficult to work with an MMIO bar for
PCI-e?
> >> >> 
> >> >> With legacy PCI, tracking allocation of MMIO vs. PIO is
pretty straight
> >> >> forward.  Is there something special about PCI-e here?
> >> >> 
> >> >
> >> > It's not tracking allocation.  It is that accessing
memory above 1 MiB
> >> > is incredibly painful in the BIOS environment, which
basically means
> >> > MMIO is inaccessible.
> >> 
> >> Oh, you mean in real mode.
> >> 
> >> SeaBIOS runs the virtio code in 32-bit mode with a flat memory
layout.
> >> There are loads of ASSERT32FLAT()s in the code to make sure of
this.
> >> 
> > Well, not exactly. Initialization is done in 32bit, but disk
> > reads/writes are done in 16bit mode since it should work from int13
> > interrupt handler. The only way I know to access MMIO bars from 16 bit
> > is to use SMM which we do not have in KVM.
> 
> Ah, if it's just the dataplane operations then there's another
solution.
> 
> We can introduce a virtqueue flag that asks the backend to poll for new
> requests.  Then SeaBIOS can add the request to the queue and not worry
> about kicking or reading the ISR.
This will pin a host CPU.
If we do something timer based it will likely
both increase host CPU utilization and slow device down.

If we didn't care about performance at all we could
do config cycles for signalling, which is much
more elegant than polling in host, but I don't think
that's the case.
> 
> SeaBIOS is polling for completion anyway.
I think that's different because a disk will normally respond
quickly. So it polls a bit, but then it stops as
there are no outstanding requests.

-- 
MST

On Tue, Jun 11, 2013 at 10:10:47AM +0300, Michael S. Tsirkin
wrote:
> On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori wrote:
> > Gleb Natapov <gleb at redhat.com> writes:
> > 
> > > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
> > >> "H. Peter Anvin" <hpa at zytor.com> writes:
> > >> 
> > >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> > >> >>>
> > >> >>> Definitely an option.  However, we want to be
able to boot from native
> > >> >>> devices, too, so having an I/O BAR (which would
not be used by the OS
> > >> >>> driver) should still at the very least be an
option.
> > >> >> 
> > >> >> What makes it so difficult to work with an MMIO bar
for PCI-e?
> > >> >> 
> > >> >> With legacy PCI, tracking allocation of MMIO vs. PIO
is pretty straight
> > >> >> forward.  Is there something special about PCI-e
here?
> > >> >> 
> > >> >
> > >> > It's not tracking allocation.  It is that accessing
memory above 1 MiB
> > >> > is incredibly painful in the BIOS environment, which
basically means
> > >> > MMIO is inaccessible.
> > >> 
> > >> Oh, you mean in real mode.
> > >> 
> > >> SeaBIOS runs the virtio code in 32-bit mode with a flat
memory layout.
> > >> There are loads of ASSERT32FLAT()s in the code to make sure
of this.
> > >> 
> > > Well, not exactly. Initialization is done in 32bit, but disk
> > > reads/writes are done in 16bit mode since it should work from
int13
> > > interrupt handler. The only way I know to access MMIO bars from
16 bit
> > > is to use SMM which we do not have in KVM.
> > 
> > Ah, if it's just the dataplane operations then there's another
solution.
> > 
> > We can introduce a virtqueue flag that asks the backend to poll for
new
> > requests.  Then SeaBIOS can add the request to the queue and not worry
> > about kicking or reading the ISR.
> 
> This will pin a host CPU.
> If we do something timer based it will likely
> both increase host CPU utilization and slow device down.
> 
> If we didn't care about performance at all we could
> do config cycles for signalling, which is much
> more elegant than polling in host, but I don't think
> that's the case.
> 
I wouldn't call BIOS int13 interface performance critical.
> > 
> > SeaBIOS is polling for completion anyway.
> 
> I think that's different because a disk will normally respond
> quickly. So it polls a bit, but then it stops as
> there are no outstanding requests.
> 
> -- 
> MST
--
			Gleb.