Linux Virtualization - Jun 2013 - [PATCH RFC] virtio-pci: new config layout: using memory BAR

"H. Peter Anvin" <hpa at zytor.com> writes:
> On 06/05/2013 03:08 PM, Anthony Liguori wrote:
>>>
>>> Definitely an option.  However, we want to be able to boot from
native
>>> devices, too, so having an I/O BAR (which would not be used by the
OS
>>> driver) should still at the very least be an option.
>> 
>> What makes it so difficult to work with an MMIO bar for PCI-e?
>> 
>> With legacy PCI, tracking allocation of MMIO vs. PIO is pretty straight
>> forward.  Is there something special about PCI-e here?
>> 
>
> It's not tracking allocation.  It is that accessing memory above 1 MiB
> is incredibly painful in the BIOS environment, which basically means
> MMIO is inaccessible.
Oh, you mean in real mode.

SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
There are loads of ASSERT32FLAT()s in the code to make sure of this.

Regards,

Anthony Liguori
>
> 	-hpa
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori
wrote:
> "H. Peter Anvin" <hpa at zytor.com> writes:
> 
> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
> >>>
> >>> Definitely an option.  However, we want to be able to boot
from native
> >>> devices, too, so having an I/O BAR (which would not be used by
the OS
> >>> driver) should still at the very least be an option.
> >> 
> >> What makes it so difficult to work with an MMIO bar for PCI-e?
> >> 
> >> With legacy PCI, tracking allocation of MMIO vs. PIO is pretty
straight
> >> forward.  Is there something special about PCI-e here?
> >> 
> >
> > It's not tracking allocation.  It is that accessing memory above 1
MiB
> > is incredibly painful in the BIOS environment, which basically means
> > MMIO is inaccessible.
> 
> Oh, you mean in real mode.
> 
> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
> There are loads of ASSERT32FLAT()s in the code to make sure of this.
> 
Well, not exactly. Initialization is done in 32bit, but disk
reads/writes are done in 16bit mode since it should work from int13
interrupt handler. The only way I know to access MMIO bars from 16 bit
is to use SMM which we do not have in KVM.

--
			Gleb.

On 06/05/2013 11:34 PM, Gleb Natapov wrote:
>>
>> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
>> There are loads of ASSERT32FLAT()s in the code to make sure of this.
>>
> Well, not exactly. Initialization is done in 32bit, but disk
> reads/writes are done in 16bit mode since it should work from int13
> interrupt handler. The only way I know to access MMIO bars from 16 bit
> is to use SMM which we do not have in KVM.
> 
In some ways it is even worse for PXE, since PXE is defined to work from
16-bit protected mode... but the mechanism to request a segment mapping
for the high memory area doesn't actually work.

	-hpa

Gleb Natapov <gleb at redhat.com> writes:
> On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
>> "H. Peter Anvin" <hpa at zytor.com> writes:
>> 
>> > On 06/05/2013 03:08 PM, Anthony Liguori wrote:
>> >>>
>> >>> Definitely an option.  However, we want to be able to boot
from native
>> >>> devices, too, so having an I/O BAR (which would not be
used by the OS
>> >>> driver) should still at the very least be an option.
>> >> 
>> >> What makes it so difficult to work with an MMIO bar for PCI-e?
>> >> 
>> >> With legacy PCI, tracking allocation of MMIO vs. PIO is pretty
straight
>> >> forward.  Is there something special about PCI-e here?
>> >> 
>> >
>> > It's not tracking allocation.  It is that accessing memory
above 1 MiB
>> > is incredibly painful in the BIOS environment, which basically
means
>> > MMIO is inaccessible.
>> 
>> Oh, you mean in real mode.
>> 
>> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
>> There are loads of ASSERT32FLAT()s in the code to make sure of this.
>> 
> Well, not exactly. Initialization is done in 32bit, but disk
> reads/writes are done in 16bit mode since it should work from int13
> interrupt handler. The only way I know to access MMIO bars from 16 bit
> is to use SMM which we do not have in KVM.
Ah, if it's just the dataplane operations then there's another solution.

We can introduce a virtqueue flag that asks the backend to poll for new
requests.  Then SeaBIOS can add the request to the queue and not worry
about kicking or reading the ISR.

SeaBIOS is polling for completion anyway.

Regards,

Anthony Liguori
>
> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 06/06/13 08:34, Gleb Natapov wrote:
> On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
>>
>> Oh, you mean in real mode.
>>
>> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
>> There are loads of ASSERT32FLAT()s in the code to make sure of this.
>>
> Well, not exactly. Initialization is done in 32bit, but disk
> reads/writes are done in 16bit mode since it should work from int13
> interrupt handler.
Exactly.  It's "only" the initialization code which has
ASSERt32FLAT()
all over the place.  Which actually is the majority of the code in most
cases as all the hardware detection and initialization code is there.
But kicking I/O requests must work from 16bit mode too.
> The only way I know to access MMIO bars from 16 bit
> is to use SMM which we do not have in KVM.
For seabios itself this isn't a big issue, see pci_{readl,writel} in
src/pci.c.  When called in 16bit mode it goes into 32bit mode
temporarily, just for accessing the mmio register.  ahci driver uses it,
xhci driver (wip atm) will use that too, and virtio-{blk,scsi} drivers
in seabios can do the same.

But as hpa mentioned it will be more tricky for option roms (aka
virtio-net).

cheers,
  Gerd