Linux Virtualization - Jun 2013 - [PATCH RFC] virtio-pci: new config layout: using memory BAR

On 06/06/13 08:34, Gleb Natapov wrote:
> On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
>>
>> Oh, you mean in real mode.
>>
>> SeaBIOS runs the virtio code in 32-bit mode with a flat memory layout.
>> There are loads of ASSERT32FLAT()s in the code to make sure of this.
>>
> Well, not exactly. Initialization is done in 32bit, but disk
> reads/writes are done in 16bit mode since it should work from int13
> interrupt handler.
Exactly.  It's "only" the initialization code which has
ASSERt32FLAT()
all over the place.  Which actually is the majority of the code in most
cases as all the hardware detection and initialization code is there.
But kicking I/O requests must work from 16bit mode too.
> The only way I know to access MMIO bars from 16 bit
> is to use SMM which we do not have in KVM.
For seabios itself this isn't a big issue, see pci_{readl,writel} in
src/pci.c.  When called in 16bit mode it goes into 32bit mode
temporarily, just for accessing the mmio register.  ahci driver uses it,
xhci driver (wip atm) will use that too, and virtio-{blk,scsi} drivers
in seabios can do the same.

But as hpa mentioned it will be more tricky for option roms (aka
virtio-net).

cheers,
  Gerd

On Thu, Jun 06, 2013 at 05:06:32PM +0200, Gerd Hoffmann
wrote:
> On 06/06/13 08:34, Gleb Natapov wrote:
> > On Wed, Jun 05, 2013 at 07:41:17PM -0500, Anthony Liguori wrote:
> >>
> >> Oh, you mean in real mode.
> >>
> >> SeaBIOS runs the virtio code in 32-bit mode with a flat memory
layout.
> >> There are loads of ASSERT32FLAT()s in the code to make sure of
this.
> >>
> > Well, not exactly. Initialization is done in 32bit, but disk
> > reads/writes are done in 16bit mode since it should work from int13
> > interrupt handler.
> 
> Exactly.  It's "only" the initialization code which has
ASSERt32FLAT()
> all over the place.  Which actually is the majority of the code in most
> cases as all the hardware detection and initialization code is there.
> But kicking I/O requests must work from 16bit mode too.
> 
> > The only way I know to access MMIO bars from 16 bit
> > is to use SMM which we do not have in KVM.
> 
> For seabios itself this isn't a big issue, see pci_{readl,writel} in
> src/pci.c.  When called in 16bit mode it goes into 32bit mode
> temporarily, just for accessing the mmio register.  ahci driver uses it,
> xhci driver (wip atm) will use that too, and virtio-{blk,scsi} drivers
> in seabios can do the same.
> 
Isn't this approach broken? How can SeaBIOS be sure it restores real
mode registers to exactly same state they were before entering 32bit
mode?
> But as hpa mentioned it will be more tricky for option roms (aka
> virtio-net).
> 
> cheers,
>   Gerd
> 
--
			Gleb.

On 06/06/2013 08:10 AM, Gleb Natapov wrote:
> On Thu, Jun 06, 2013 at 05:06:32PM +0200, Gerd Hoffmann wrote:
>>
> Isn't this approach broken? How can SeaBIOS be sure it restores real
> mode registers to exactly same state they were before entering 32bit
> mode?
> 
It can't... so yes, it is broken.

	-hpa

Hi,
>> For seabios itself this isn't a big issue, see pci_{readl,writel}
in
>> src/pci.c.  When called in 16bit mode it goes into 32bit mode
>> temporarily, just for accessing the mmio register.  ahci driver uses
it,
>> xhci driver (wip atm) will use that too, and virtio-{blk,scsi} drivers
>> in seabios can do the same.
>>
> Isn't this approach broken? How can SeaBIOS be sure it restores real
> mode registers to exactly same state they were before entering 32bit
> mode?
Don't know the details of that magic.  Kevin had some concerns on the
stability of this, so maybe there is a theoretic hole.  So far I havn't
seen any issues in practice, but also didn't stress it too much.
Basically only used that with all kinds of boot loaders, could very well
be it breaks if you try to use that with more esoteric stuff such as dos
extenders, then hit unhandled corner cases ...

cheers,
  Gerd

I only recently saw this email.

On Thu, Jun 06, 2013 at 06:10:12PM +0300, Gleb Natapov
wrote:
> On Thu, Jun 06, 2013 at 05:06:32PM +0200, Gerd Hoffmann wrote:
> > For seabios itself this isn't a big issue, see pci_{readl,writel}
in
> > src/pci.c.  When called in 16bit mode it goes into 32bit mode
> > temporarily, just for accessing the mmio register.  ahci driver uses
it,
> > xhci driver (wip atm) will use that too, and virtio-{blk,scsi} drivers
> > in seabios can do the same.
> > 
> Isn't this approach broken? How can SeaBIOS be sure it restores real
> mode registers to exactly same state they were before entering 32bit
> mode?
You are correct - SeaBIOS can't fully restore the "hidden" segment
registers.  So, in a way it is broken.

In practice, it seems to work on modern bootloaders (eg, ntldr, grub,
lilo).  It definitely doesn't work with EMM386 (old DOS stuff), but
does seem to work okay with FreeDOS as long as one doesn't run EMM386.

The AHCI code uses this 32bit/16bit trampoline because it would not be
possible to support AHCI otherwise.  I haven't seen any complaints of
failures with the AHCI code - probably because people using AHCI are
using modern guests.

I explored this a bit some time back and the only way I could think of
to reliably restore the 16bit registers would be via SMM.
Unfortunately, using SMM introduces a whole host of complexity and
problems.

-Kevin