Xen devel - Jan 2011 - QEMU upstreaming: status and todo

Hi all,
I would like to spent few words to let people know the current status of
the Qemu upstreaming work and what is still missing.

Anthony Perard sent the 8th version of his "Xen device model support"
patch series to qemu-devel and he has already received an ack from
Alexander Graf. We are currently waiting for another review by Anthony
Liguori but we are positive the series is close to be accepted in its
current form.

Once the series is upstream we can start using upstream Qemu to do Xen
development, however before we can actually use it as default device
model we need to fill some gaps.
In fact some things are still missing compared to qemu-xen, the
followings in particular: 

- VGA dirty bits optimization
This is a performance optimization for the emulated VGA card, Anthony
Perard is already working on it.

- switch to SeaBios
Qemu uses SeaBios, that has several advantages on RomBios, one of them
is that SeaBios is actually maintained.
It makes sense to switch from the old Qemu and the old bios to a new
Qemu and a new bios at the same time in order to reduce the
compatibility pains to users.

- stubdom support
The "Xen device model support" patch series lacks stubdom support.
This is mainly a matter of compiling upstream Qemu against MiniOS,
the problem is that MiniOS is not exactly posix compliant so we''ll need
to make some changes.

- pci passthrough
The patch series also lacks pci passthrough support.
The current pci passthrough code in qemu-xen is rather large and
doesn''t
integrate very well with the rest of Qemu so it is non-trivial to
upstream. 

- QMP support in libxl
upstream Qemu supports an RPC protocol called QMP that is JSON based.
We need to be able to use it to issue commands to Qemu, replacing the
current mechanism based on xenstore. For example this would be needed by
pci passthrough to send the "plug" and "unplug" commands to
Qemu.


Ideally we''ll be able to complete these tasks by the end of the Xen 4.2
release cycle so that we can have the new Qemu as default device model
by then.

Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/01/2011 17:03, "Stefano Stabellini"
<Stefano.Stabellini@eu.citrix.com>
wrote:
> - switch to SeaBios
> Qemu uses SeaBios, that has several advantages on RomBios, one of them
> is that SeaBios is actually maintained.
> It makes sense to switch from the old Qemu and the old bios to a new
> Qemu and a new bios at the same time in order to reduce the
> compatibility pains to users.
Are the two really dependent on each other? Our virtual firmware seems to
work okay now, it took a while to get there, and it now rarely needs to be
fixed. I guess I don''t care about the legacy rombios/vgabios bits too
much,
but I''d be unhappy about throwing away all our ACPI stuff. That was a
pain
to actually get working well. That means you can have
tools/firmware/{rom,vga}bios if you want, but trickier to mess with anything
under tools/firmware/hvmloader.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wednesday 05 January 2011 18:03:15 Stefano Stabellini
wrote:
> Hi all,
> I would like to spent few words to let people know the current status of
> the Qemu upstreaming work and what is still missing.
> 
> Anthony Perard sent the 8th version of his "Xen device model
support"
> patch series to qemu-devel and he has already received an ack from
> Alexander Graf. We are currently waiting for another review by Anthony
> Liguori but we are positive the series is close to be accepted in its
> current form.
> 
> Once the series is upstream we can start using upstream Qemu to do Xen
> development, however before we can actually use it as default device
> model we need to fill some gaps.
> In fact some things are still missing compared to qemu-xen, the
> followings in particular:
> 
> - VGA dirty bits optimization
> This is a performance optimization for the emulated VGA card, Anthony
> Perard is already working on it.
> 
> - switch to SeaBios
> Qemu uses SeaBios, that has several advantages on RomBios, one of them
> is that SeaBios is actually maintained.
> It makes sense to switch from the old Qemu and the old bios to a new
> Qemu and a new bios at the same time in order to reduce the
> compatibility pains to users.
> 
> - stubdom support
> The "Xen device model support" patch series lacks stubdom
support.
> This is mainly a matter of compiling upstream Qemu against MiniOS,
> the problem is that MiniOS is not exactly posix compliant so we''ll
need
> to make some changes.
> 
> - pci passthrough
> The patch series also lacks pci passthrough support.
> The current pci passthrough code in qemu-xen is rather large and
doesn''t
> integrate very well with the rest of Qemu so it is non-trivial to
> upstream.
> 
> - QMP support in libxl
> upstream Qemu supports an RPC protocol called QMP that is JSON based.
> We need to be able to use it to issue commands to Qemu, replacing the
> current mechanism based on xenstore. For example this would be needed by
> pci passthrough to send the "plug" and "unplug"
commands to Qemu.
> 
> 
> Ideally we''ll be able to complete these tasks by the end of the
Xen 4.2
> release cycle so that we can have the new Qemu as default device model
> by then.
Great job, it sounds really interesting. 

Is it usable in current state? Can I test it on current xen 4.0.x?

Can You write what are the coolest things in upstream qemu?

Regards,

--
Łukasz Oleś

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 5 Jan 2011, Keir Fraser wrote:
> On 05/01/2011 17:03, "Stefano Stabellini"
<Stefano.Stabellini@eu.citrix.com>
> wrote:
> 
> > - switch to SeaBios
> > Qemu uses SeaBios, that has several advantages on RomBios, one of them
> > is that SeaBios is actually maintained.
> > It makes sense to switch from the old Qemu and the old bios to a new
> > Qemu and a new bios at the same time in order to reduce the
> > compatibility pains to users.
> 
> Are the two really dependent on each other? Our virtual firmware seems to
> work okay now, it took a while to get there, and it now rarely needs to be
> fixed. I guess I don''t care about the legacy rombios/vgabios bits
too much,
> but I''d be unhappy about throwing away all our ACPI stuff. That
was a pain
> to actually get working well. That means you can have
> tools/firmware/{rom,vga}bios if you want, but trickier to mess with
anything
> under tools/firmware/hvmloader.
 
I was proposing only to replace rombios: from a conversation we had with
the SeaBios maintainer at LPC, I think SeaBios can be made to work from
hvmloader without too much trouble. However I haven''t read the code so
this needs to be tested.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 5 Jan 2011, Łukasz Oleś wrote:
> Great job, it sounds really interesting. 
> 
> Is it usable in current state? Can I test it on current xen 4.0.x?
> 
Yes, it is usable even though without the VGA dirty bits is going to be
much slower. I suggest to wait until the VGA dirty bits are supported
before trying to use it, otherwise it only makes sense if you use
nographic=1.
It could work with xen 4.0.x but the command line options are different
and at the moment the only toolstack that supports the new format is
libxenlight in xen-unstable.

> Can You write what are the coolest things in upstream qemu?
> 
 
The codebase is much cleaner and easier to work with. This means far
less bugs and easier development in the future.
Regarding user visible features, the followings are the firsts that came
to my mind:

- multiple vnc connections at the same time;

- audio over vnc;

- sdl zoom (you can resize the guest''s window, Qemu takes care of the
  scaling);

- support for linux aio, this means much faster dma transfers from the guest;


also several very interesting features are being developed by the Qemu
Community, they are not ready yet, but they should be useful soon:

- AHCI emulation (much faster emulated disks);

- Spice;

- Q35 chipset emulation (pci express within the guest).


I am sure there is more...
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Are the two really dependent on each other? Our virtual firmware
> seems to
> > work okay now, it took a while to get there, and it now rarely needs
> to be
> > fixed. I guess I don''t care about the legacy rombios/vgabios
bits too
> much,
> > but I''d be unhappy about throwing away all our ACPI stuff.
That was a
> pain
> > to actually get working well. That means you can have
> > tools/firmware/{rom,vga}bios if you want, but trickier to mess with
> anything
> > under tools/firmware/hvmloader.
> 
> I was proposing only to replace rombios: from a conversation we had
> with
> the SeaBios maintainer at LPC, I think SeaBios can be made to work from
> hvmloader without too much trouble. However I haven''t read the
code so
> this needs to be tested.
Here is one more point to add to Stefano''s note and to clarify
Keir''s concern about throwing away our ACPI stuff - There will be some
ACPI tables setup code that is in our hvmloader that wouldn''t be needed
as SeaBIOS would do the ACPI tables setup but from what I heard, SeaBIOS
provides a way for us to feed it a list of our SSDTs and more for it to take it
from there and load them.  So, I believe we won''t end up losing any
ACPI functionality we added if that is your concern.

Kamala


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jan 05, 2011 at 07:15:14PM +0000, Stefano Stabellini
wrote:
> On Wed, 5 Jan 2011, ??ukasz Ole?? wrote:
> > Great job, it sounds really interesting. 
> > 
> > Is it usable in current state? Can I test it on current xen 4.0.x?
> > 
> 
> Yes, it is usable even though without the VGA dirty bits is going to be
> much slower. I suggest to wait until the VGA dirty bits are supported
> before trying to use it, otherwise it only makes sense if you use
> nographic=1.
> It could work with xen 4.0.x but the command line options are different
> and at the moment the only toolstack that supports the new format is
> libxenlight in xen-unstable.
> 
> 
> > Can You write what are the coolest things in upstream qemu?
> > 
>  
> The codebase is much cleaner and easier to work with. This means far
> less bugs and easier development in the future.
> Regarding user visible features, the followings are the firsts that came
> to my mind:
> 
> - multiple vnc connections at the same time;
> 
> - audio over vnc;
> 
> - sdl zoom (you can resize the guest''s window, Qemu takes care of
the
>   scaling);
> 
> - support for linux aio, this means much faster dma transfers from the
guest;
> 
> 
> also several very interesting features are being developed by the Qemu
> Community, they are not ready yet, but they should be useful soon:
> 
> - AHCI emulation (much faster emulated disks);
> 
> - Spice;
> 
> - Q35 chipset emulation (pci express within the guest).
> 
IIRC: SCSI and SAS support aswell..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/01/2011 18:57, "Stefano Stabellini"
<stefano.stabellini@eu.citrix.com>
wrote:
>> Are the two really dependent on each other? Our virtual firmware seems
to
>> work okay now, it took a while to get there, and it now rarely needs to
be
>> fixed. I guess I don''t care about the legacy rombios/vgabios
bits too much,
>> but I''d be unhappy about throwing away all our ACPI stuff.
That was a pain
>> to actually get working well. That means you can have
>> tools/firmware/{rom,vga}bios if you want, but trickier to mess with
anything
>> under tools/firmware/hvmloader.
>  
> I was proposing only to replace rombios: from a conversation we had with
> the SeaBios maintainer at LPC, I think SeaBios can be made to work from
> hvmloader without too much trouble. However I haven''t read the
code so
> this needs to be tested.
If we could get rid of vgabios as well, and get rid of our dependency on
vgabios, that would be nice. :-) Else I remain skeptical to be honest. But
open to discussing it further, I may just need some more info on the
advantages of the move.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 5 Jan 2011, Keir Fraser wrote:
> On 05/01/2011 18:57, "Stefano Stabellini"
<stefano.stabellini@eu.citrix.com>
> wrote:
> 
> >> Are the two really dependent on each other? Our virtual firmware
seems to
> >> work okay now, it took a while to get there, and it now rarely
needs to be
> >> fixed. I guess I don''t care about the legacy
rombios/vgabios bits too much,
> >> but I''d be unhappy about throwing away all our ACPI
stuff. That was a pain
> >> to actually get working well. That means you can have
> >> tools/firmware/{rom,vga}bios if you want, but trickier to mess
with anything
> >> under tools/firmware/hvmloader.
> >  
> > I was proposing only to replace rombios: from a conversation we had
with
> > the SeaBios maintainer at LPC, I think SeaBios can be made to work
from
> > hvmloader without too much trouble. However I haven''t read
the code so
> > this needs to be tested.
> 
> If we could get rid of vgabios as well, and get rid of our dependency on
> vgabios, that would be nice. :-)
completely agree, unfortunately there is no replacement for vgabios yet

> Else I remain skeptical to be honest. But
> open to discussing it further, I may just need some more info on the
> advantages of the move.
> 
The advantages are code readability, having a common platform to avoid future
compatibility issues (like the ACPI io ports problems he had a little
while ago) and being able to reuse all the development that the SeaBios and
Qemu people are doing.
For example SeaBios supports 64 bit BAR addresses and is able to boot
from virtio devices (they might work on Xen in a not too distant
future).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel