Xen users - Mar 2009 - Device Drivers in xen, looking also for a white book about how xen works exactly‏

Hello,
 
I''m reading an excellent documentation about virtualisation in embedded
system.
 (available on open kernel labs site)
 
I have read these things : (It talks about managing drivers in a virtualization
system)
 
"A straight virtualization approach can accomodate this by running the
device the device driver inside the VMM. This requires porting all drivers to
the hypervisor environnement , with no reuse of guest os driver"
 
A much better approach is to share a single driver between multiple vms, without
including it in the hypervisor. This requires that ecah participating subsytem
has a device modem for which it has a device driver. Typically, the real device
driver is contained in one of the participating system, but a better  (safer)
solution is to separate it out into in own subsystem"
 
I want to know how it works on xen.
I think that :
 
 All drivers are in the hypervisor, when a guest requires to use a device, 
guest uses drivers which has been rewriten for xen in order to contact the
hypervisor. Then, the hypervisor which has all real drivers (not modified) call
the real device ?
 
or that :
The real device driver is contained in the dom0 system, the dom0 manage driver
for all domU.
 
I have three questions :
 
1 -  Which is the good scenario ? (if one is good ) :o)
2 - If drivers for guests are rewritten to call hypervisor or dom0 instead of
the device directly, who devellops driver ?
(xen devellopers ? or developper for kernel mainline)
3- Which is the real utility of dom0 system ? (just I/O and administration of
virtual machines)
 
Last question, is someone have find a good white paper or doc which explains how
xen works ?
(and where i can find probably answers for my questions ??)
 
Thanks for all this details.
Best regards
 
Patrick Archibal
_________________________________________________________________
Inédit ! Des Emoticônes Déjantées! Installez les dans votre Messenger ! 
http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi there,
> I want to know how it works on xen.
> I think that :
>
>  All drivers are in the hypervisor, when a guest requires to use a device, 
> guest uses drivers which has been rewriten for xen in order to contact the
> hypervisor. Then, the hypervisor which has all real drivers (not modified)
> call the real device ?
That''s how Xen 1.x worked.  Porting drivers to Xen was ugly and time 
consuming, so we moved it to the other model you describe.  Let''s call
this
approach #1.
> or that :
> The real device driver is contained in the dom0 system, the dom0 manage
> driver for all domU.
This is how Xen worked from Xen 2.x onwards.  Xen 3.x uses a very similar 
architecture but moves *even more* driver stuff into dom0.  Xen knows very 
little about devices.  Lets call this approach #2.
> I have three questions :
>
> 1 -  Which is the good scenario ? (if one is good ) :o)
As usual in Operating Systems (and everything else!) the answer is probably 
"It depends" ;-)

If you were making an embedded device, where you have a small selection of 
hardware to expect then it might make sense to take approach #1 and put the 
drivers in the hypervisor.

In Xen''s case, the goal is to have a fairly small hypervisor but be
able to
run on a wide variety of PCs.  Running on a wide variety of PCs means that 
lots of drivers need to be available and the only sensible way to do this was 
to run the drivers in a Linux guest, instead of porting them all to Xen.

Another potentially benefit of approach #2 is that your device drivers are 
contained in VMs.  If you go a step further than what you described and make 
it possible to run drivers in domains *other* than dom0 then if a driver 
crashes, it can be killed and restarted.  You can''t do this when the
drivers
are in the hypervisor itself (or dom0).

For Xen''s purposes, approach #2 has been much better. 
Microsoft''s Hyper-V is
taking this approach too, I believe.

Note, however, that VMware ESX - a *very* strong and successful product - uses 
approach #1 with great success.  Arguably Linux-based approaches like KVM use 
approach #1 also but the difference there is that *Linux is the hypervisor* so 
drivers do not need to be ported to it.  For this reason, approach #1 is not a 
disadvantage to them (it''s even a strength!).
> 2 - If drivers for guests are rewritten to call hypervisor or dom0 instead
> of the device directly, who devellops driver ? (xen devellopers ? or
> developper for kernel mainline)
The Xen patches are designed so that the *real* device drivers work unmodified 
(they just need to be compiled for Xen).  To get device access to domUs, a set 
of "Virtual Device" drivers are written.  These are written by the Xen
developers and are used to give network and block (and framebuffer) access to 
domUs.
> 3- Which is the real utility of dom0 system ? (just I/O and administration
> of virtual machines)
You can run what you want in dom0 - some people run X.org and use it as their 
"desktop" OS.  For a secure deployment you want a minimal set of
drivers and
administration tools in it.
> Last question, is someone have find a good white paper or doc which
> explains how xen works ? (and where i can find probably answers for my
> questions ??)
Look at these:
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&target=Xen+Architecture_Q1+2008.pdf
http://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf
http://www.cl.cam.ac.uk/netos/papers/2004-oasis-ngio.pdf
http://www.cl.cam.ac.uk/research/srg/netos/xen/architecture.html

"Xen and the art of virtualization" describes the Xen 1.x driver
architecture
but it is a useful guide to the rest of the system.  "Safe hardware
access..."
describes the newer architecture.


> Thanks for all this details.
> Best regards
>
> Patrick Archibal
Hope this helps,
Cheers!
Mark
> Inédit ! Des Emoticônes Déjantées! Installez les dans votre Messenger !
> http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Your explanation misses to describe the difference between fully-virtualized
VM''s and Paravirtualized ones, and why the performance of the later is
much,
much better.
Please kindly continue. You are doing a great job summarizing the ongoing
discussion in the science and art of virtualization.

-----Original Message-----
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Mark Williamson
Sent: Sunday, March 01, 2009 11:39 AM
To: xen-users@lists.xensource.com
Cc: Patrick Archibal
Subject: Re: [Xen-users] Device Drivers in xen, looking also for a white
book about how xen works exactly‏

Hi there,
> I want to know how it works on xen.
> I think that :
>
>  All drivers are in the hypervisor, when a guest requires to use a device,
> guest uses drivers which has been rewriten for xen in order to contact the
> hypervisor. Then, the hypervisor which has all real drivers (not modified)
> call the real device ?
That''s how Xen 1.x worked.  Porting drivers to Xen was ugly and time 
consuming, so we moved it to the other model you describe.  Let''s call
this
approach #1.
> or that :
> The real device driver is contained in the dom0 system, the dom0 manage
> driver for all domU.
This is how Xen worked from Xen 2.x onwards.  Xen 3.x uses a very similar 
architecture but moves *even more* driver stuff into dom0.  Xen knows very 
little about devices.  Lets call this approach #2.
> I have three questions :
>
> 1 -  Which is the good scenario ? (if one is good ) :o)
As usual in Operating Systems (and everything else!) the answer is probably 
"It depends" ;-)

If you were making an embedded device, where you have a small selection of 
hardware to expect then it might make sense to take approach #1 and put the 
drivers in the hypervisor.

In Xen''s case, the goal is to have a fairly small hypervisor but be
able to
run on a wide variety of PCs.  Running on a wide variety of PCs means that 
lots of drivers need to be available and the only sensible way to do this
was 
to run the drivers in a Linux guest, instead of porting them all to Xen.

Another potentially benefit of approach #2 is that your device drivers are 
contained in VMs.  If you go a step further than what you described and make

it possible to run drivers in domains *other* than dom0 then if a driver 
crashes, it can be killed and restarted.  You can''t do this when the
drivers

are in the hypervisor itself (or dom0).

For Xen''s purposes, approach #2 has been much better. 
Microsoft''s Hyper-V
is 
taking this approach too, I believe.

Note, however, that VMware ESX - a *very* strong and successful product -
uses 
approach #1 with great success.  Arguably Linux-based approaches like KVM
use 
approach #1 also but the difference there is that *Linux is the hypervisor*
so 
drivers do not need to be ported to it.  For this reason, approach #1 is not
a 
disadvantage to them (it''s even a strength!).
> 2 - If drivers for guests are rewritten to call hypervisor or dom0 instead
> of the device directly, who devellops driver ? (xen devellopers ? or
> developper for kernel mainline)
The Xen patches are designed so that the *real* device drivers work
unmodified 
(they just need to be compiled for Xen).  To get device access to domUs, a
set 
of "Virtual Device" drivers are written.  These are written by the Xen
developers and are used to give network and block (and framebuffer) access
to 
domUs.
> 3- Which is the real utility of dom0 system ? (just I/O and administration
> of virtual machines)
You can run what you want in dom0 - some people run X.org and use it as
their 
"desktop" OS.  For a secure deployment you want a minimal set of
drivers and

administration tools in it.
> Last question, is someone have find a good white paper or doc which
> explains how xen works ? (and where i can find probably answers for my
> questions ??)
Look at these:
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&t
arget=Xen+Architecture_Q1+2008.pdf
http://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf
http://www.cl.cam.ac.uk/netos/papers/2004-oasis-ngio.pdf
http://www.cl.cam.ac.uk/research/srg/netos/xen/architecture.html

"Xen and the art of virtualization" describes the Xen 1.x driver
architecture 
but it is a useful guide to the rest of the system.  "Safe hardware
access..." 
describes the newer architecture.


> Thanks for all this details.
> Best regards
>
> Patrick Archibal
Hope this helps,
Cheers!
Mark
> Inédit ! Des Emoticônes Déjantées! Installez les dans votre Messenger !
> http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sunday 01 March 2009 17:13:16 Venefax wrote:
> Your explanation misses to describe the difference between
> fully-virtualized VM''s and Paravirtualized ones, and why the
performance of
> the later is much, much better.
This is true and the original question does touch on it since it asks about 
virtual devices.
> Please kindly continue. You are doing a great job summarizing the ongoing
> discussion in the science and art of virtualization.
OK, thanks ;-)

Well...

Paravirtualised VMs on Xen are traditionally VMs that run a fully Xen-aware 
OS.  The kernel (usually Linux but also NetBSD, sometimes Netware, sometimes 
others) has been modified deeply to understand that it is running on a 
hypervisor.  As a result it knows it shouldn''t be looking for real
physical
devices like PCI network cards, graphics cards, etc.  Instead, when a 
paravirtualised OS boots, it establishes shared memory communications with an 
already-running VM which is able to provide it with block and network IO.  
Usually this involves connecting to virtual devices provided by dom0.

Dom0 runs a kernel module called a "backend driver".  There''s
one of these for
block devices (i.e. virtual disks) and one for virtual network devices.  The 
backend driver receives requests from paravirtualised domUs and multiplexes 
them onto the real hardware.  The backend driver allows the domUs to *safely* 
share the real hardware in the machine so that they can correctly access 
storage and network when they need to.  To make sure that the guests
don''t try
to do bad things with the system, the backend driver must check (and sometimes 
alter) their requests.  The backend driver is a kind of low level
"proxy" for
devices.

On the other hand, fully virtualised VMs are ones that believe they are 
running on a real PC.  The VMM stack (Xen+dom0 in this case) must somehow 
provide the illusion of this by emulating real-world devices for disk, network 
card, graphics card, etc.  Fully virtualised VMs don''t know how to
establish
shared memory connections to dom0 because that does not exist in a real PC.  
Instead they issue MMIO accesses and port IO to try and discover what 
"hardware" they have available to them.  These accesses can be caught
by Xen
and are passed to a "device model".  The device model is a program
running in
another VM which can emulate the devices the guest OS is expecting.  It 
simulates these devices, then generates the responses the guest OS would 
expect from a real device.

Amongst the devices that are emulated by the Device Model are the disk and 
network devices that the guest will use to do IO.  When the Device Model gets 
requests to read or write these emulated devices, it issues requests
"behind
the scenes" to do *real* IO.  The guest itself does not know how this
happens,
as it believes it is talking directly to the hardware.

There are two ways that the Device Model usually runs.  Originally the Device 
Models were simply a userspace process running in domain 0.  This is separate 
from the "backend" drivers that paravirtualised guests talked to.  The
device
models did IO by accessing files and devices using normal system calls - the 
same as any other process would do.  Some special calls were made available to 
the device model so that it could communicate back to the fully virtualised 
guests.  More recently the "stub domain" or "stubdom" option
has been created.
This this model the device model is a effectively a separate virtual machine 
that is solely responsible for providing emulated devices to the fully 
virtualised VM it is paired with.  The stubdom services the IO by using normal 
paravirtualised shared memory to talk to dom0.  Think of the stubdom as a 
"proxy" that turns emulated device requests into paravirtual device
requests.

In either case, the device model is based very heavily on the PC emulation 
code from the Qemu (http://www.qemu.org) project.

Things can be *slightly* more complicated than this, however, since it is 
possible to run paravirtualised drivers inside a full virtualised VM.  This 
blurs the lines between the two classes of VM and includes some of the 
strengths (and weaknesses) of both.  If paravirtualised drivers are made to 
run within a fully virtualised OS then you can get better IO performance since 
the overheads of emulating real devices are removed.  The OS itself does not 
need to be fully modified for Xen in order take to take advantage of this, so 
it is an ideal technique to apply to Windows guests.  The GPLPV Windows 
drivers do this, for instance.

I hope this helps to summarise the differences in IO between the different kinds
of VMs Xen might run.  This description is mostly Xen-specific - there are some 
common concepts with other VMM stacks but they tend to vary greatly in exactly 
how they accomplish these goals.

Cheers,
Mark
> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Mark Williamson
> Sent: Sunday, March 01, 2009 11:39 AM
> To: xen-users@lists.xensource.com
> Cc: Patrick Archibal
> Subject: Re: [Xen-users] Device Drivers in xen, looking also for a white
> book about how xen works exactly‏
>
> Hi there,
>
> > I want to know how it works on xen.
> > I think that :
> >
> >  All drivers are in the hypervisor, when a guest requires to use a
> > device,
> >
> > guest uses drivers which has been rewriten for xen in order to contact
> > the hypervisor. Then, the hypervisor which has all real drivers (not
> > modified) call the real device ?
>
> That''s how Xen 1.x worked.  Porting drivers to Xen was ugly and
time
> consuming, so we moved it to the other model you describe.  Let''s
call this
> approach #1.
>
> > or that :
> > The real device driver is contained in the dom0 system, the dom0
manage
> > driver for all domU.
>
> This is how Xen worked from Xen 2.x onwards.  Xen 3.x uses a very similar
> architecture but moves *even more* driver stuff into dom0.  Xen knows very
> little about devices.  Lets call this approach #2.
>
> > I have three questions :
> >
> > 1 -  Which is the good scenario ? (if one is good ) :o)
>
> As usual in Operating Systems (and everything else!) the answer is probably
> "It depends" ;-)
>
> If you were making an embedded device, where you have a small selection of
> hardware to expect then it might make sense to take approach #1 and put the
> drivers in the hypervisor.
>
> In Xen''s case, the goal is to have a fairly small hypervisor but
be able to
> run on a wide variety of PCs.  Running on a wide variety of PCs means that
> lots of drivers need to be available and the only sensible way to do this
> was
> to run the drivers in a Linux guest, instead of porting them all to Xen.
>
> Another potentially benefit of approach #2 is that your device drivers are
> contained in VMs.  If you go a step further than what you described and
> make
>
> it possible to run drivers in domains *other* than dom0 then if a driver
> crashes, it can be killed and restarted.  You can''t do this when
the
> drivers
>
> are in the hypervisor itself (or dom0).
>
> For Xen''s purposes, approach #2 has been much better. 
Microsoft''s Hyper-V
> is
> taking this approach too, I believe.
>
> Note, however, that VMware ESX - a *very* strong and successful product -
> uses
> approach #1 with great success.  Arguably Linux-based approaches like KVM
> use
> approach #1 also but the difference there is that *Linux is the hypervisor*
> so
> drivers do not need to be ported to it.  For this reason, approach #1 is
> not a
> disadvantage to them (it''s even a strength!).
>
> > 2 - If drivers for guests are rewritten to call hypervisor or dom0
> > instead of the device directly, who devellops driver ? (xen
devellopers ?
> > or developper for kernel mainline)
>
> The Xen patches are designed so that the *real* device drivers work
> unmodified
> (they just need to be compiled for Xen).  To get device access to domUs, a
> set
> of "Virtual Device" drivers are written.  These are written by
the Xen
> developers and are used to give network and block (and framebuffer) access
> to
> domUs.
>
> > 3- Which is the real utility of dom0 system ? (just I/O and
> > administration of virtual machines)
>
> You can run what you want in dom0 - some people run X.org and use it as
> their
> "desktop" OS.  For a secure deployment you want a minimal set of
drivers
> and
>
> administration tools in it.
>
> > Last question, is someone have find a good white paper or doc which
> > explains how xen works ? (and where i can find probably answers for my
> > questions ??)
>
> Look at these:
>
http://wiki.xensource.com/xenwiki/XenArchitecture?action=AttachFile&do=get&
>t arget=Xen+Architecture_Q1+2008.pdf
> http://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf
> http://www.cl.cam.ac.uk/netos/papers/2004-oasis-ngio.pdf
> http://www.cl.cam.ac.uk/research/srg/netos/xen/architecture.html
>
> "Xen and the art of virtualization" describes the Xen 1.x driver
> architecture
> but it is a useful guide to the rest of the system.  "Safe hardware
> access..."
> describes the newer architecture.
>
> > Thanks for all this details.
> > Best regards
> >
> > Patrick Archibal
>
> Hope this helps,
> Cheers!
> Mark
>
> > Inédit ! Des Emoticônes Déjantées! Installez les dans votre Messenger
!
> > http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

El Dom 01 Marzo 2009, Mark Williamson escribió:
> On Sunday 01 March 2009 17:13:16 Venefax wrote:
> > Your explanation misses to describe the difference between
> > fully-virtualized VM''s and Paravirtualized ones, and why the
performance
> > of the later is much, much better.
>
> This is true and the original question does touch on it since it asks about
> virtual devices.
>
> > Please kindly continue. You are doing a great job summarizing the
ongoing
> > discussion in the science and art of virtualization.
>
> OK, thanks ;-)
>
> Well...
[ ... ]

Wow, I was catching up with old emails and found this precious gem!

Better late than never, I just wanted to thank you for posting this.
I''ll save
it for future reference for when I need to explain it to someone else.

Best regards,
-- 
Ricardo J. Barberis 
Senior SysAdmin - I+D 
Dattatec.com :: Soluciones de Web Hosting 
Su Hosting hecho Simple..! 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users