Linux Virtualization - Jun 2013 - [PATCH RFC] virtio-pci: new config layout: using memory BAR

On Tue, Jun 11, 2013 at 10:10:47AM +0300, Michael S. Tsirkin
wrote:
> On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori wrote:
> > Gleb Natapov <gleb at redhat.com> writes:
> > 
> > > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
> > >> "H. Peter Anvin" <hpa at zytor.com> writes:
> > >> 
> > >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> > >> >>>
> > >> >>> Definitely an option.  However, we want to be
able to boot from native
> > >> >>> devices, too, so having an I/O BAR (which would
not be used by the OS
> > >> >>> driver) should still at the very least be an
option.
> > >> >> 
> > >> >> What makes it so difficult to work with an MMIO bar
for PCI-e?
> > >> >> 
> > >> >> With legacy PCI, tracking allocation of MMIO vs. PIO
is pretty straight
> > >> >> forward.  Is there something special about PCI-e
here?
> > >> >> 
> > >> >
> > >> > It's not tracking allocation.  It is that accessing
memory above 1 MiB
> > >> > is incredibly painful in the BIOS environment, which
basically means
> > >> > MMIO is inaccessible.
> > >> 
> > >> Oh, you mean in real mode.
> > >> 
> > >> SeaBIOS runs the virtio code in 32-bit mode with a flat
memory layout.
> > >> There are loads of ASSERT32FLAT()s in the code to make sure
of this.
> > >> 
> > > Well, not exactly. Initialization is done in 32bit, but disk
> > > reads/writes are done in 16bit mode since it should work from
int13
> > > interrupt handler. The only way I know to access MMIO bars from
16 bit
> > > is to use SMM which we do not have in KVM.
> > 
> > Ah, if it's just the dataplane operations then there's another
solution.
> > 
> > We can introduce a virtqueue flag that asks the backend to poll for
new
> > requests.  Then SeaBIOS can add the request to the queue and not worry
> > about kicking or reading the ISR.
> 
> This will pin a host CPU.
> If we do something timer based it will likely
> both increase host CPU utilization and slow device down.
> 
> If we didn't care about performance at all we could
> do config cycles for signalling, which is much
> more elegant than polling in host, but I don't think
> that's the case.
> 
I wouldn't call BIOS int13 interface performance critical.
> > 
> > SeaBIOS is polling for completion anyway.
> 
> I think that's different because a disk will normally respond
> quickly. So it polls a bit, but then it stops as
> there are no outstanding requests.
> 
> -- 
> MST
--
			Gleb.

On Tue, Jun 11, 2013 at 10:53:48AM +0300, Gleb Natapov
wrote:
> On Tue, Jun 11, 2013 at 10:10:47AM +0300, Michael S. Tsirkin wrote:
> > On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori wrote:
> > > Gleb Natapov <gleb at redhat.com> writes:
> > > 
> > > > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori
wrote:
> > > >> "H. Peter Anvin" <hpa at zytor.com>
writes:
> > > >> 
> > > >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> > > >> >>>
> > > >> >>> Definitely an option.  However, we want to
be able to boot from native
> > > >> >>> devices, too, so having an I/O BAR (which
would not be used by the OS
> > > >> >>> driver) should still at the very least be
an option.
> > > >> >> 
> > > >> >> What makes it so difficult to work with an MMIO
bar for PCI-e?
> > > >> >> 
> > > >> >> With legacy PCI, tracking allocation of MMIO
vs. PIO is pretty straight
> > > >> >> forward.  Is there something special about
PCI-e here?
> > > >> >> 
> > > >> >
> > > >> > It's not tracking allocation.  It is that
accessing memory above 1 MiB
> > > >> > is incredibly painful in the BIOS environment,
which basically means
> > > >> > MMIO is inaccessible.
> > > >> 
> > > >> Oh, you mean in real mode.
> > > >> 
> > > >> SeaBIOS runs the virtio code in 32-bit mode with a flat
memory layout.
> > > >> There are loads of ASSERT32FLAT()s in the code to make
sure of this.
> > > >> 
> > > > Well, not exactly. Initialization is done in 32bit, but disk
> > > > reads/writes are done in 16bit mode since it should work
from int13
> > > > interrupt handler. The only way I know to access MMIO bars
from 16 bit
> > > > is to use SMM which we do not have in KVM.
> > > 
> > > Ah, if it's just the dataplane operations then there's
another solution.
> > > 
> > > We can introduce a virtqueue flag that asks the backend to poll
for new
> > > requests.  Then SeaBIOS can add the request to the queue and not
worry
> > > about kicking or reading the ISR.
> > 
> > This will pin a host CPU.
> > If we do something timer based it will likely
> > both increase host CPU utilization and slow device down.
> > 
> > If we didn't care about performance at all we could
> > do config cycles for signalling, which is much
> > more elegant than polling in host, but I don't think
> > that's the case.
> > 
> I wouldn't call BIOS int13 interface performance critical.
So the plan always was to
- add an MMIO BAR
- add a register for pci-config based access to devices

hpa felt performance does matter there but didn't clarify why ...
> > > 
> > > SeaBIOS is polling for completion anyway.
> > 
> > I think that's different because a disk will normally respond
> > quickly. So it polls a bit, but then it stops as
> > there are no outstanding requests.
> > 
> > -- 
> > MST
> 
> --
> 			Gleb.

On Tue, Jun 11, 2013 at 11:02:26AM +0300, Michael S. Tsirkin
wrote:
> On Tue, Jun 11, 2013 at 10:53:48AM +0300, Gleb Natapov wrote:
> > On Tue, Jun 11, 2013 at 10:10:47AM +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori wrote:
> > > > Gleb Natapov <gleb at redhat.com> writes:
> > > > 
> > > > > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony
Liguori wrote:
> > > > >> "H. Peter Anvin" <hpa at zytor.com>
writes:
> > > > >> 
> > > > >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> > > > >> >>>
> > > > >> >>> Definitely an option.  However, we
want to be able to boot from native
> > > > >> >>> devices, too, so having an I/O BAR
(which would not be used by the OS
> > > > >> >>> driver) should still at the very least
be an option.
> > > > >> >> 
> > > > >> >> What makes it so difficult to work with an
MMIO bar for PCI-e?
> > > > >> >> 
> > > > >> >> With legacy PCI, tracking allocation of
MMIO vs. PIO is pretty straight
> > > > >> >> forward.  Is there something special about
PCI-e here?
> > > > >> >> 
> > > > >> >
> > > > >> > It's not tracking allocation.  It is that
accessing memory above 1 MiB
> > > > >> > is incredibly painful in the BIOS environment,
which basically means
> > > > >> > MMIO is inaccessible.
> > > > >> 
> > > > >> Oh, you mean in real mode.
> > > > >> 
> > > > >> SeaBIOS runs the virtio code in 32-bit mode with a
flat memory layout.
> > > > >> There are loads of ASSERT32FLAT()s in the code to
make sure of this.
> > > > >> 
> > > > > Well, not exactly. Initialization is done in 32bit, but
disk
> > > > > reads/writes are done in 16bit mode since it should
work from int13
> > > > > interrupt handler. The only way I know to access MMIO
bars from 16 bit
> > > > > is to use SMM which we do not have in KVM.
> > > > 
> > > > Ah, if it's just the dataplane operations then
there's another solution.
> > > > 
> > > > We can introduce a virtqueue flag that asks the backend to
poll for new
> > > > requests.  Then SeaBIOS can add the request to the queue and
not worry
> > > > about kicking or reading the ISR.
> > > 
> > > This will pin a host CPU.
> > > If we do something timer based it will likely
> > > both increase host CPU utilization and slow device down.
> > > 
> > > If we didn't care about performance at all we could
> > > do config cycles for signalling, which is much
> > > more elegant than polling in host, but I don't think
> > > that's the case.
> > > 
> > I wouldn't call BIOS int13 interface performance critical.
> 
> So the plan always was to
> - add an MMIO BAR
> - add a register for pci-config based access to devices
> 
> hpa felt performance does matter there but didn't clarify why ...
> 
You do not what to make it too slow obviously, this is interface that is
used to load OS during boot.
> > > > 
> > > > SeaBIOS is polling for completion anyway.
> > > 
> > > I think that's different because a disk will normally respond
> > > quickly. So it polls a bit, but then it stops as
> > > there are no outstanding requests.
> > > 
> > > -- 
> > > MST
> > 
> > --
> > 			Gleb.
--
			Gleb.

On Tue, Jun 11, 2013 at 11:02:26AM +0300, Michael S. Tsirkin
wrote:
> On Tue, Jun 11, 2013 at 10:53:48AM +0300, Gleb Natapov wrote:
> > On Tue, Jun 11, 2013 at 10:10:47AM +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jun 06, 2013 at 10:02:14AM -0500, Anthony Liguori wrote:
> > > > Gleb Natapov <gleb at redhat.com> writes:
> > > > 
> > > > > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony
Liguori wrote:
> > > > >> "H. Peter Anvin" <hpa at zytor.com>
writes:
> > > > >> 
> > > > >> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> > > > >> >>>
> > > > >> >>> Definitely an option.  However, we
want to be able to boot from native
> > > > >> >>> devices, too, so having an I/O BAR
(which would not be used by the OS
> > > > >> >>> driver) should still at the very least
be an option.
> > > > >> >> 
> > > > >> >> What makes it so difficult to work with an
MMIO bar for PCI-e?
> > > > >> >> 
> > > > >> >> With legacy PCI, tracking allocation of
MMIO vs. PIO is pretty straight
> > > > >> >> forward.  Is there something special about
PCI-e here?
> > > > >> >> 
> > > > >> >
> > > > >> > It's not tracking allocation.  It is that
accessing memory above 1 MiB
> > > > >> > is incredibly painful in the BIOS environment,
which basically means
> > > > >> > MMIO is inaccessible.
> > > > >> 
> > > > >> Oh, you mean in real mode.
> > > > >> 
> > > > >> SeaBIOS runs the virtio code in 32-bit mode with a
flat memory layout.
> > > > >> There are loads of ASSERT32FLAT()s in the code to
make sure of this.
> > > > >> 
> > > > > Well, not exactly. Initialization is done in 32bit, but
disk
> > > > > reads/writes are done in 16bit mode since it should
work from int13
> > > > > interrupt handler. The only way I know to access MMIO
bars from 16 bit
> > > > > is to use SMM which we do not have in KVM.
> > > > 
> > > > Ah, if it's just the dataplane operations then
there's another solution.
> > > > 
> > > > We can introduce a virtqueue flag that asks the backend to
poll for new
> > > > requests.  Then SeaBIOS can add the request to the queue and
not worry
> > > > about kicking or reading the ISR.
> > > 
> > > This will pin a host CPU.
> > > If we do something timer based it will likely
> > > both increase host CPU utilization and slow device down.
> > > 
> > > If we didn't care about performance at all we could
> > > do config cycles for signalling, which is much
> > > more elegant than polling in host, but I don't think
> > > that's the case.
> > > 
> > I wouldn't call BIOS int13 interface performance critical.
> 
> So the plan always was to
> - add an MMIO BAR
> - add a register for pci-config based access to devices
> 
> hpa felt performance does matter there but didn't clarify why ...
Also

<gleb> mst, well the question is if it is safe to call int13 in the
middle of pci bus enumeration/configuration
<gleb> mst, and int13 predates PCI, so how knows
> > > > 
> > > > SeaBIOS is polling for completion anyway.
> > > 
> > > I think that's different because a disk will normally respond
> > > quickly. So it polls a bit, but then it stops as
> > > there are no outstanding requests.
> > > 
> > > -- 
> > > MST
> > 
> > --
> > 			Gleb.