Xen devel - Nov 2007 - [RFC] hvm-stub for ia64

Hi,

this is my initial work on hvm-stub domains.
HVM-stub domains are a mix of PV and HVM: it requires VT extension but
look externally as a PV domain.

All the io emulation is done within the domain by a stub firmware.  This
layer translates raw IO accesses to PV commands, ie it can run unmodified
linux without running qemu-dm.

Currently it works only for ia64 and can boot linux (serial console).
I cross-post to xen-devel because I can be of a general interest.

I''d plan to submit patches soon but here is a preview.

Part 1: tools/ - modification to start an hvmstub domain (small)
Part 2: xen/ - IO reflection (small)
Part 3: extra/stubfw - stub firmware (large)

Change to common files are rather small and this is a good news.
The stub firmware is more a proof of concept and need serious enhancement.
It is just minimal to be able to boot linux (ie serial console and one ide
disk).

I am not sure where should go stubfw, so if anyone has suggestions...

Comments are welcome,
Tristan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/07 03:59, "tgingold@free.fr" <tgingold@free.fr> wrote:
> Part 1: tools/ - modification to start an hvmstub domain (small)
> Part 2: xen/ - IO reflection (small)
> Part 3: extra/stubfw - stub firmware (large)
> 
> Change to common files are rather small and this is a good news.
> The stub firmware is more a proof of concept and need serious enhancement.
> It is just minimal to be able to boot linux (ie serial console and one ide
> disk).
> 
> I am not sure where should go stubfw, so if anyone has suggestions...
Does the hvmstub have to look different to an hvm guest from the p.o.v. of
the tools? It''s rather a shame to introduce a third kind of domain at
that
level.

How do you think this appraoch compares with a separate stub domain per HVM
guest? That''s the approach being developed on the x86 side, but it
would be
nice to not increase divergence between x86 and ia64.

 -- Keir



_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel

Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> Does the hvmstub have to look different to an hvm guest from the p.o.v. of
> the tools? It''s rather a shame to introduce a third kind of domain
at that
> level.
There are of course differences: qemu shouldn''t be started.  And the
builder
C function is different.  I can try to merge with the hvm builder function
but I am not sure it is worth.  The amount of new code is still rather small
for tools.
> How do you think this appraoch compares with a separate stub domain per HVM
> guest? That''s the approach being developed on the x86 side, but it
would be
> nice to not increase divergence between x86 and ia64.
I think my approach is much more simpler.  Just look at the amount of new code
for hypervisor and tools: very small.

There is nothing ia64 specific in hvmstub.  For performance, it requires
cheap virtual to physical (ie switching paging on and off) transitions in the
guest, but I think this is already true for x86.

The real work is in the stub firmware which could be partially shared between
x86 and ia64.  However I''d also like to implement more realistic
machine
emulation for ia64 in order to be able to run more OSes.

Tristan.

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel

On 22/11/07 09:31, "tgingold@free.fr" <tgingold@free.fr> wrote:
>> Does the hvmstub have to look different to an hvm guest from the p.o.v.
of
>> the tools? It''s rather a shame to introduce a third kind of
domain at that
>> level.
> There are of course differences: qemu shouldn''t be started.  And
the builder
> C function is different.  I can try to merge with the hvm builder function
> but I am not sure it is worth.  The amount of new code is still rather
small
> for tools.
Presumably the old HVM builder for ia64 becomes dead code though? So really
this is the ''new'' hvm building path. Do you need to support
the old way (on
ia64) if this is checked in?
>> How do you think this appraoch compares with a separate stub domain per
HVM
>> guest? That''s the approach being developed on the x86 side,
but it would be
>> nice to not increase divergence between x86 and ia64.
> I think my approach is much more simpler.  Just look at the amount of new
code
> for hypervisor and tools: very small.
I doubt the magnitude of changes in tools/hypervisor is different between
the two approaches. It might even be smaller in the separate domain
approach.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> On 22/11/07 09:31, "tgingold@free.fr" <tgingold@free.fr>
wrote:
>
> Presumably the old HVM builder for ia64 becomes dead code though? So really
> this is the ''new'' hvm building path. Do you need to
support the old way (on
> ia64) if this is checked in?
Yes, I don''t plan to remove the old way on ia64.  I have not
implemented
a VGA in the stub firmware and I don''t plan to do it soon.  Only
Windows
requires a VGA and I don''t have Windows for ia64.
> I doubt the magnitude of changes in tools/hypervisor is different between
> the two approaches. It might even be smaller in the separate domain
> approach.
We are speaking about around 200 lines of code in the hypervisor and slightly
more for tools (300 lines mostly ia64 specific), right ?

Is the stub domain code already available ?  I''d like to have a look.

Tristan.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/07 10:00, "tgingold@free.fr" <tgingold@free.fr> wrote:
>> I doubt the magnitude of changes in tools/hypervisor is different
between
>> the two approaches. It might even be smaller in the separate domain
>> approach.
> We are speaking about around 200 lines of code in the hypervisor and
slightly
> more for tools (300 lines mostly ia64 specific), right ?
> 
> Is the stub domain code already available ?  I''d like to have a
look.
No, it''s not released yet. But same as your approach, all the
complexity is
in the stub.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

tgingold@free.fr, le Thu 22 Nov 2007 11:00:24 +0100, a écrit
:
> Quoting Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> 
> > On 22/11/07 09:31, "tgingold@free.fr"
<tgingold@free.fr> wrote:
> >
> > Presumably the old HVM builder for ia64 becomes dead code though? So
really
> > this is the ''new'' hvm building path. Do you need to
support the old way (on
> > ia64) if this is checked in?
> Yes, I don''t plan to remove the old way on ia64.  I have not
implemented
> a VGA in the stub firmware and I don''t plan to do it soon.
That''s the main difference between your approach (reimplement a
firmware) and the approach being developped here at XenSource (embed
qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
something useful, but reimplementing the hardware emulation support of
qemu looks to me like duplicate work.  I haven''t ported my stubdomain
to
ia64 yet, but that shouldn''t be very hard now that it works for x86.

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> That''s the main difference between your approach (reimplement a
> firmware) and the approach being developped here at XenSource (embed
> qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
> something useful, but reimplementing the hardware emulation support of
> qemu looks to me like duplicate work.
Of course it doesn''t reimplement the emulation.  The current biggest
emulation
is the IDE and I have copied ide.c from qemu and made almost no modification.

The advantage is my approach is no need for scheduling between two domains
for hardware emulation.  With the current hvm model, I reach the EFI prompt
in about 10sec.  With my hvmstub it took less than 2 seconds. Very pleasant
to use!

Tristan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit
:
> Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> 
> > That''s the main difference between your approach (reimplement
a
> > firmware) and the approach being developped here at XenSource (embed
> > qemu in a tiny stubdomain).  Your VMX work in Xen is most probably
> > something useful, but reimplementing the hardware emulation support of
> > qemu looks to me like duplicate work.
> 
> Of course it doesn''t reimplement the emulation.  The current
biggest emulation
> is the IDE and I have copied ide.c from qemu and made almost no
modification.
That''s still forking.  And you''ll also need to do it for vga,
network, ...
> The advantage is my approach is no need for scheduling between two domains
> for hardware emulation.
Ah, yes, on ia64 you have room to load your firmware in the domain
itself, which makes an extra benefit (with the stubdomain we avoid the
need for linux scheduling, but not the need for xen scheduling).
> With the current hvm model, I reach the EFI prompt
> in about 10sec.  With my hvmstub it took less than 2 seconds.
And I wonder how much it would take with a stub domain, probably
somewhere in between.

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit :
> > Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > Of course it doesn''t reimplement the emulation.  The current
biggest
> emulation
> > is the IDE and I have copied ide.c from qemu and made almost no
> modification.
>
> That''s still forking.  And you''ll also need to do it for
vga,
> network, ...
To be fair, I''d need to know better how stubdomain is implemented.

> > The advantage is my approach is no need for scheduling between two
domains
> > for hardware emulation.
>
> Ah, yes, on ia64 you have room to load your firmware in the domain
> itself, which makes an extra benefit (with the stubdomain we avoid the
> need for linux scheduling, but not the need for xen scheduling).
Should be the same on x86.  Firmware is loaded between 4GB-16MB and 4GB.
Enough room for stub firmware and EFI.

Tristan.


_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel

Hi,

tgingold@free.fr, le Thu 22 Nov 2007 14:04:47 +0100, a écrit
:
> Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > tgingold@free.fr, le Thu 22 Nov 2007 12:12:31 +0100, a écrit :
> > > Quoting Samuel Thibault <samuel.thibault@eu.citrix.com>:
> > > Of course it doesn''t reimplement the emulation.  The
current biggest
> > emulation
> > > is the IDE and I have copied ide.c from qemu and made almost no
> > modification.
> >
> > That''s still forking.  And you''ll also need to do it
for vga,
> > network, ...
> 
> To be fair, I''d need to know better how stubdomain is implemented.
It''s actually "just" a matter of compiling qemu against
newlib, mini-os
and lwIP and make that run as a kernel in its own domain. That way it''s
just the same code as the usual dm approach and mini-os, with some
#ifdefs.

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Nov 22, 2007 at 10:20:44PM +0000, Samuel Thibault
wrote:
> Hi,
> 
> It''s actually "just" a matter of compiling qemu against
newlib, mini-os
> and lwIP and make that run as a kernel in its own domain. That way
it''s
> just the same code as the usual dm approach and mini-os, with some
> #ifdefs.
I suppose you had to add vbd and full vif to mini-os ?
VGA will only be available with vnc ?

I suppose you also have to do a double save-and-restore ?

Tristan.

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel

Hello,

Tristan Gingold, le Fri 23 Nov 2007 04:30:53 +0100, a écrit
:
> On Thu, Nov 22, 2007 at 10:20:44PM +0000, Samuel Thibault wrote:
> > It''s actually "just" a matter of compiling qemu
against newlib, mini-os
> > and lwIP and make that run as a kernel in its own domain. That way
it''s
> > just the same code as the usual dm approach and mini-os, with some
> > #ifdefs.
> I suppose you had to add vbd and full vif to mini-os ?
Yes, and they will profit whichever project using mini-os.
> VGA will only be available with vnc ?
I only tested vnc for now, but I can''t see a reason why vfb
shouldn''t
work.
> I suppose you also have to do a double save-and-restore ?
Of the domain states when saving/restoring them you mean?

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

If you want to give it a try, a repository is available on 
http://xenbits.xensource.com/ext/xen-minios-stubdom.hg

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel