Xen devel - Jul 2007 - Prototype to use QEMU for PV guest framebuffer

As many of us are all too painfully aware we have completely different VNC 
server implementations for paravirt vs fullyvirt Xen guests. The former 
based on libvncserver, the latter integrated into QEMU. There are many new
and interesting ideas being tried out in the VNC server space in particular
wrt to virtualization and having to implement them all twice is not very
desirable.  Also libvncserver code is terrible - many bad thread race 
conditions that are near impossible to diagnose, let alone solve :-(

At the summit there were a couple of suggestions, either taking the VNC
server code from QEMU and splicing it into the xenfb daemon. Another was
to take the QEMU VNC code and put it into a library that can be used by
both QEMU & xenfb. Neither of those is a particularly appealing prospect.

Then Anthony Liguori suggested a 3rd approach, which was to add a new QEMU
machine type to the x86 target, and simply disable all the emulated hardwre
and just make use of QEMU infrastructure for the VNC server, select() event 
loop and display state.

This sounded very intriguing so I did a little experimentation today doing
exactly that. First of all I''ve copied  tools/xenfb/xenfb.c into the 
qemu/hw/xenfb.c file. Next I added a qemu/hw/xen.c file to provide the 
implementation of the QEMU machine type - called ''xenpv''. 

All the interesting code (all 100 lines of it) is in hw/xen.c in the
xen_init_pv method - which is called when using the ''-M xenpv''
command
line arg. 

The first thing this does is get the domain ID - to avoid touching
QEMU''s
command line args, I just use an environ variable XEN_DOMID to pass this
in.

The main_loop() in qemu/vl.c assumes there is always one CPU created in
the VM, but for our purposes we''re not doing any CPU emulation merely 
using the event loop/display state. So to workaround this assumption I
create a single CPUState  *  instance, and permanently disable it by 
setting ''cpu->hflags = HF_HALTED_MASK''. Seems to do the
job.

Next, up we attach to the guest PVFB frontend - just using the existing code
for this - qemu/hw/xenfb.c (formerly xen-unstable/tools/xenfb/xenfb.c).

Now we register a QEMU graphical console, a mouse event receiver, a
keyboard event receiver, and register the xenstored and event channel
file handles with QEMU''s event loop.

Finally, initialize QEMU''s display state to match the PVFB framebuffer
config (ie 800x600x32).

Pushing mouse & keyboard events through from QEMU to PVFB frontend is
trivial. The only bit I''m unhappy about is that QEMU can''t
access the
guest framebuffer directly. The DisplayState * struct has its own copy
of the framebuffer - allocated by the VNC or SDL impls in QEMU - and
so whenever the guest framebuffer changes, we have to memcpy() the data
from the guest into the QEMU framebuffer. Still, this is no worse than
what the HVM guests already do. Its probably not too hard to change the
QEMU impl of VNC / SDL to use the guest framebuffer directly if we did
a little re-factoring. I wanted to keep it simple for now & not change
any of the upstream QEMU code.

This attached patch is against the current upstream QEMU  CVS code, not
Xen''s
ioemu, since I wanted to work against pristine QEMU codebase & avoid any
potential wierd iteractions with HVM stuff added to ioemu. The diff is

$ diffstat ~/xen-qemu-pvfb-machine.patch
 Makefile.target |    3 
 b/hw/xen.c      |  113 +++++++
 b/hw/xenfb.c    |  822 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 b/hw/xenfb.h    |   35 ++
 vl.c            |    1 
 vl.h            |    3 
 6 files changed, 976 insertions(+), 1 deletion(-)

So, you can see a mere 113 lines of new code to make this work - xenfb.c 
is the existnig tools/xenfb/xenfb.c file with minor tweaks. And we get to
delete 20,000 lines of terrible libvncserver code !

Apply the patch to QEMU CVS checkout, configure to build an x86 target,
and once done you can run with 

  # XEN_DOMID=31 ./i386-softmmu/qemu -M xenpv -hda /dev/null -vnc :0

The -hda flag is just to keep QEMU''s command line parse quiet - it
wants
at least one disk.

You can also run in SDL mode

  # XEN_DOMID=31 ./i386-softmmu/qemu -M xenpv -hda /dev/null

Thoughts... ?  If there''s positive feedback on this approach
I''ll develop
the patch further & properly integrate with XenD & domain
create/teardown.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel P. Berrange wrote:
> As many of us are all too painfully aware we have completely different VNC 
> server implementations for paravirt vs fullyvirt Xen guests. The former 
> based on libvncserver, the latter integrated into QEMU. There are many new
> and interesting ideas being tried out in the VNC server space in particular
> wrt to virtualization and having to implement them all twice is not very
> desirable.  Also libvncserver code is terrible - many bad thread race 
> conditions that are near impossible to diagnose, let alone solve :-(
> 
> At the summit there were a couple of suggestions, either taking the VNC
> server code from QEMU and splicing it into the xenfb daemon. Another was
> to take the QEMU VNC code and put it into a library that can be used by
> both QEMU & xenfb. Neither of those is a particularly appealing
prospect.
> 
> Then Anthony Liguori suggested a 3rd approach, which was to add a new QEMU
> machine type to the x86 target, and simply disable all the emulated hardwre
> and just make use of QEMU infrastructure for the VNC server, select() event
> loop and display state.
> 
> This sounded very intriguing so I did a little experimentation today doing
> exactly that. First of all I''ve copied  tools/xenfb/xenfb.c into
the
> qemu/hw/xenfb.c file. Next I added a qemu/hw/xen.c file to provide the 
> implementation of the QEMU machine type - called ''xenpv''.
> 
> All the interesting code (all 100 lines of it) is in hw/xen.c in the
> xen_init_pv method - which is called when using the ''-M
xenpv'' command
> line arg. 
> 
> The first thing this does is get the domain ID - to avoid touching
QEMU''s
> command line args, I just use an environ variable XEN_DOMID to pass this
> in.
> 
> The main_loop() in qemu/vl.c assumes there is always one CPU created in
> the VM, but for our purposes we''re not doing any CPU emulation
merely
> using the event loop/display state. So to workaround this assumption I
> create a single CPUState  *  instance, and permanently disable it by 
> setting ''cpu->hflags = HF_HALTED_MASK''. Seems to do
the job.
> 
> Next, up we attach to the guest PVFB frontend - just using the existing
code
> for this - qemu/hw/xenfb.c (formerly xen-unstable/tools/xenfb/xenfb.c).
> 
> Now we register a QEMU graphical console, a mouse event receiver, a
> keyboard event receiver, and register the xenstored and event channel
> file handles with QEMU''s event loop.
> 
> Finally, initialize QEMU''s display state to match the PVFB
framebuffer
> config (ie 800x600x32).
> 
> Pushing mouse & keyboard events through from QEMU to PVFB frontend is
> trivial. The only bit I''m unhappy about is that QEMU
can''t access the
> guest framebuffer directly. The DisplayState * struct has its own copy
> of the framebuffer - allocated by the VNC or SDL impls in QEMU - and
> so whenever the guest framebuffer changes, we have to memcpy() the data
> from the guest into the QEMU framebuffer.
The DisplayState driver (VNC or SDL) chooses what the depth and format 
of this ds->data buffer should be.  The PVFB expects a certain format so 
you would have to do more than just a memcpy().  You''ll have to be able
to translate between the xenfb format and the ds->data format.

This is very cool stuff though!  Since QEMU already supports VNC and 
SDL, we don''t need to maintain any additional pvfb backends.

Regards,

Anthony Liguori
> Still, this is no worse than
> what the HVM guests already do. Its probably not too hard to change the
> QEMU impl of VNC / SDL to use the guest framebuffer directly if we did
> a little re-factoring. I wanted to keep it simple for now & not change
> any of the upstream QEMU code.
> 
> This attached patch is against the current upstream QEMU  CVS code, not
Xen''s
> ioemu, since I wanted to work against pristine QEMU codebase & avoid
any
> potential wierd iteractions with HVM stuff added to ioemu. The diff is
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Pushing mouse & keyboard events through from QEMU to PVFB frontend is
> trivial. The only bit I''m unhappy about is that QEMU
can''t access the
> guest framebuffer directly. The DisplayState * struct has its own copy
> of the framebuffer - allocated by the VNC or SDL impls in QEMU - and
> so whenever the guest framebuffer changes, we have to memcpy() the
data
> from the guest into the QEMU framebuffer. Still, this is no worse than
> what the HVM guests already do. Its probably not too hard to change
the
> QEMU impl of VNC / SDL to use the guest framebuffer directly if we did
> a little re-factoring. I wanted to keep it simple for now & not change
> any of the upstream QEMU code.
> 
> This attached patch is against the current upstream QEMU  CVS code,
not
> Xen''s ioemu, since I wanted to work against pristine QEMU codebase
&
avoid
> any potential wierd iteractions with HVM stuff added to ioemu. The
diff is

It''ll certainly be good to see the back of libvncserver.

Could you investigate whether this patch applies to qemu-dm easily
enough?

Thanks,
Ian




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, Jul 29, 2007 at 11:03:09PM +0100, Ian Pratt
wrote:
> > Pushing mouse & keyboard events through from QEMU to PVFB frontend
is
> > trivial. The only bit I''m unhappy about is that QEMU
can''t access the
> > guest framebuffer directly. The DisplayState * struct has its own copy
> > of the framebuffer - allocated by the VNC or SDL impls in QEMU - and
> > so whenever the guest framebuffer changes, we have to memcpy() the
> data
> > from the guest into the QEMU framebuffer. Still, this is no worse than
> > what the HVM guests already do. Its probably not too hard to change
> the
> > QEMU impl of VNC / SDL to use the guest framebuffer directly if we did
> > a little re-factoring. I wanted to keep it simple for now & not
change
> > any of the upstream QEMU code.
> > 
> > This attached patch is against the current upstream QEMU  CVS code,
> not
> > Xen''s ioemu, since I wanted to work against pristine QEMU
codebase &
> avoid
> > any potential wierd iteractions with HVM stuff added to ioemu. The
> diff is
> 
> It''ll certainly be good to see the back of libvncserver.
> 
> Could you investigate whether this patch applies to qemu-dm easily
> enough?
The answer was yes & no :-)  Although we''ve got a separate target
for
QEMU, there''s still a bunch of stuff in the main vl.c that is specific
to HVM guests - the memory map initialization basically.

The way I''ve approached this problem is to define two QEMU machines

  $ ./ioemu/i386-dm/qemu-dm -M ?
  Supported machines are:
  xenfv      Xen Fullyvirtualized PC (default)
  xenpv      Xen Paravirtualized PC

The little bit of HVM specific stuff from vl.c I''ve moved into the
machine
specific init function in  hw/xenfv.c

As with my first prototype the hw/xenpv.c file provides a machine for Xen
paravirt guests which merely talks the PVFB protocol. The only change from
my previous patch is that it now does pixel swizzling if the guest display
depth is not the same as the QEMU display depth.

So, in summary this does look like it''ll work fairly well for PVFB. 
The
patch attached is my latest work-in-progress

 b/tools/ioemu/hw/xenfb.c             |  822 +++++++++++++++++++++++++++++++++++
 b/tools/ioemu/hw/xenfb.h             |   35 +
 b/tools/ioemu/hw/xenfv.c             |  258 ++++++++++
 b/tools/ioemu/hw/xenpv.c             |  157 ++++++
 tools/ioemu/Makefile.target          |    6 
 tools/ioemu/target-i386-dm/helper2.c |    3 
 tools/ioemu/vl.c                     |  242 ----------
 tools/ioemu/vl.h                     |    4 
 12 files changed, 1287 insertions(+), 240 deletions(-)



Taking this idea of using QEMU for PV services a little further it occured
to me that if we could figure out a way to get the bootloaders to be run
from QEMU instead of XenD then we would be able to interact with pygrub
remotely over the graphical VNC console - currently you can only use it 
over the text serial console on the local host. It might also require that
we have QEMU handling the guest console directly instead of xenconsoled.
ie so that QEMU could make the bootloader (pygrub) available on both VNC & 
a PTY at the same time. This would also mean the serial console could take 
advantage of QEMU''s support for accessing it via UDP, or TCP, or
TCP+telnet,
as well as local PTYs.

Finally, I notice that ioemu/patches/qemu-dm disables the QEMU code which
lets you boot a linux kernel+initrd directly, instead of using the MBR
to run a bootloader. Is there any particular reason why this can''t be
made
to work for Xen HVM guests ? The ability to boot a kenrel+initrd directly
is very handy for some deployment & installation scenarios - eg to automate
an anaconda installer, passing in command line args to kickstart, without
needing to modify a CDROM iso, or use PXE.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 02, 2007 at 01:00:50AM +0100, Daniel P. Berrange
wrote:
> On Sun, Jul 29, 2007 at 11:03:09PM +0100, Ian Pratt wrote:
> > 
> > It''ll certainly be good to see the back of libvncserver.
> > 
> > Could you investigate whether this patch applies to qemu-dm easily
> > enough?
> 
> The answer was yes & no :-)  Although we''ve got a separate
target for
> QEMU, there''s still a bunch of stuff in the main vl.c that is
specific
> to HVM guests - the memory map initialization basically.
> 
> The way I''ve approached this problem is to define two QEMU
machines
> 
>   $ ./ioemu/i386-dm/qemu-dm -M ?
>   Supported machines are:
>   xenfv      Xen Fullyvirtualized PC (default)
>   xenpv      Xen Paravirtualized PC
> 
> The little bit of HVM specific stuff from vl.c I''ve moved into the
machine
> specific init function in  hw/xenfv.c
> 
> As with my first prototype the hw/xenpv.c file provides a machine for Xen
> paravirt guests which merely talks the PVFB protocol. The only change from
> my previous patch is that it now does pixel swizzling if the guest display
> depth is not the same as the QEMU display depth.
> 
> So, in summary this does look like it''ll work fairly well for
PVFB.  The
> patch attached is my latest work-in-progress
> 
>  b/tools/ioemu/hw/xenfb.c             |  822
+++++++++++++++++++++++++++++++++++
>  b/tools/ioemu/hw/xenfb.h             |   35 +
>  b/tools/ioemu/hw/xenfv.c             |  258 ++++++++++
>  b/tools/ioemu/hw/xenpv.c             |  157 ++++++
>  tools/ioemu/Makefile.target          |    6 
>  tools/ioemu/target-i386-dm/helper2.c |    3 
>  tools/ioemu/vl.c                     |  242 ----------
>  tools/ioemu/vl.h                     |    4 
>  12 files changed, 1287 insertions(+), 240 deletions(-)
> 
> 
> 
> Taking this idea of using QEMU for PV services a little further it occured
> to me that if we could figure out a way to get the bootloaders to be run
> from QEMU instead of XenD then we would be able to interact with pygrub
> remotely over the graphical VNC console - currently you can only use it 
> over the text serial console on the local host. It might also require that
> we have QEMU handling the guest console directly instead of xenconsoled.
> ie so that QEMU could make the bootloader (pygrub) available on both VNC
&
> a PTY at the same time. This would also mean the serial console could take 
> advantage of QEMU''s support for accessing it via UDP, or TCP, or
TCP+telnet,
> as well as local PTYs.
Here''s a further iteration of the patch which demonstrates how QEMU can
provide the guest console too. I launch it with this:

  qemu-dm -M xenpv -d 6 -vnc :0 -serial pty

A couple of open issues

 - QEMU doesn''t let us poll() on write event for its character device
sinks,
   so in theory we could be loosing data if the pty were to give EAGAIN.
   I think it should be possible to address this fairly easily.

 - The xenconsoled has persistent logging of guest & HV. For the latter we
   could just strip xenconsoled to merely provide HV logging, or even move
   that into XenD since its such a tiny amount of code. For guest logging
   QEMU can use ''-serial file:/path'' but then you
can''t use it via the pty
   Probably could extend QEMU command line syntax ''-serial
pty;file:/path''
   to open and PTY & log at the same time.

Those minor details aside though, this seems to work quite well - the code
is only 400 lines, compared to about 1200 lines in xenconsoled.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel