Xen devel - Sep 2013 - [RFC] XenGT - An Mediated Graphics Passthrough Solution from Intel

Hi, Xen Experts,

This email is aimed at a first time disclosure of project XenGT, which is
a Graphics virtualization solution based on Xen.
 

 
As you can see, requirements for GPU to be sharable among virtual machines
have been constantly rising. The targeted usage model might be
accelerating tasks ranging from gaming, video playback, fancy GUIs,
high-performance computing (GPU-based). This trend is observed on both
client and cloud. Efficient GPU virtualization is required to address the
increasing demands.
 

We have developed XenGT - a prototype based on a mediated passthrough
architecture. We support running a native graphics driver in multiple VMs
to achieve high performance. A specific mediator owns the scheduling and
sharing hardware resources among all the virtual machines. By saying
mediated pass-through, we actually divide graphics resource to two
categories: performance critical and others. We partition performance
critical resources for VM''s direct access like pass-through, while save
and restore others.
 

 
XenGT implements the mediator in dom0, called vgt driver. This avoids
adding complex device knowledge to Xen, and also permits a more flexible
release model. In the meantime, we want to have a unified architecture to
mediate all the VMs, including dom0. Thus, We developed a ³deprivileged²
dom0 mode, which essentially trapped Dom0''s access to selected
resources
(graphics resources in XenGT case) and forwarded to vgt driver (which is
also in Dom0) for processing.

 
Performance critical resources are:
- Part of the global virtual memory space
- VM¹s own per-process virtual memory spaces
- VM¹s own allocated command buffers (actually in graphics memory)

 
Others are:
- MMIO/PIO
- PCI configuration registers
- GTT tables
- Submission of queued GPU commands

Right now, we support 4 accelerated VMs: Dom0 + 3 HVM DomUs. We''ve
conducted verifications based on Ubuntu 12.04 and 13.04. Tests conducted
in VM include but are not limited to 3D gaming, media playbacks, 2D
accelerations.

We believe the architecture itself can be general so that different GPUs
can all use this mediated passthrough concept. However, we only developed
codes based on Intel 4th Core Processor with integrated Graphics. Note
that we don''t require VT-d to run our solution.

If you have interests in trying, you can refer to the attached setup
guide, which provides step-to-step details on building/configuring/running
XenGT.

Source code is made available at github:
Xen: https://github.com/01org/XenGT-Preview-xen.git
Linux: https://github.com/01org/XenGT-Preview-kernel.git
Qemu: https://github.com/01org/XenGT-Preview-qemu.git

Any comments are welcome!


Special note: We are making this code available to general public since we
take community''s involvement and feedbacks seriously. However, while
we''ve
tested our solution with various workloads, the code is only at pre-alpha
stage. Hangs might happen, so please don''t try it on the system that
hosting critical data for you.

Shan Haitao



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao
wrote:
> Hi, Xen Experts,
> 
> This email is aimed at a first time disclosure of project XenGT, which is
> a Graphics virtualization solution based on Xen.
>  
> 
>  
> As you can see, requirements for GPU to be sharable among virtual machines
> have been constantly rising. The targeted usage model might be
> accelerating tasks ranging from gaming, video playback, fancy GUIs,
> high-performance computing (GPU-based). This trend is observed on both
> client and cloud. Efficient GPU virtualization is required to address the
> increasing demands.
>  
> 
> We have developed XenGT - a prototype based on a mediated passthrough
> architecture. We support running a native graphics driver in multiple VMs
> to achieve high performance. A specific mediator owns the scheduling and
> sharing hardware resources among all the virtual machines. By saying
> mediated pass-through, we actually divide graphics resource to two
> categories: performance critical and others. We partition performance
> critical resources for VM''s direct access like pass-through, while
save
> and restore others.
>  
> 
>  
> XenGT implements the mediator in dom0, called vgt driver. This avoids
> adding complex device knowledge to Xen, and also permits a more flexible
> release model. In the meantime, we want to have a unified architecture to
> mediate all the VMs, including dom0. Thus, We developed a ³deprivileged²
> dom0 mode, which essentially trapped Dom0''s access to selected
resources
> (graphics resources in XenGT case) and forwarded to vgt driver (which is
> also in Dom0) for processing.
But could be in another domain itself - an device driver domain
right?
> 
>  
> Performance critical resources are:
> - Part of the global virtual memory space
> - VM¹s own per-process virtual memory spaces
> - VM¹s own allocated command buffers (actually in graphics memory)
> 
>  
> Others are:
> - MMIO/PIO
> - PCI configuration registers
> - GTT tables
> - Submission of queued GPU commands
> 
> Right now, we support 4 accelerated VMs: Dom0 + 3 HVM DomUs. We''ve
> conducted verifications based on Ubuntu 12.04 and 13.04. Tests conducted
> in VM include but are not limited to 3D gaming, media playbacks, 2D
> accelerations.
> 
> We believe the architecture itself can be general so that different GPUs
> can all use this mediated passthrough concept. However, we only developed
> codes based on Intel 4th Core Processor with integrated Graphics. Note
> that we don''t require VT-d to run our solution.
> 
> If you have interests in trying, you can refer to the attached setup
> guide, which provides step-to-step details on building/configuring/running
> XenGT.
> 
> Source code is made available at github:
> Xen: https://github.com/01org/XenGT-Preview-xen.git
> Linux: https://github.com/01org/XenGT-Preview-kernel.git
> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
Yeey! Awesome! Thank you for sharing that.
> 
> Any comments are welcome!
> 
> 
> Special note: We are making this code available to general public since we
> take community''s involvement and feedbacks seriously. However,
while we''ve
> tested our solution with various workloads, the code is only at pre-alpha
> stage. Hangs might happen, so please don''t try it on the system
that
> hosting critical data for you.
> 
> Shan Haitao
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 10, 2013 3:28 AM
> 
> On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
> > Hi, Xen Experts,
> >
> > This email is aimed at a first time disclosure of project XenGT, which
is
> > a Graphics virtualization solution based on Xen.
> >
> >
> >
> > As you can see, requirements for GPU to be sharable among virtual
machines
> > have been constantly rising. The targeted usage model might be
> > accelerating tasks ranging from gaming, video playback, fancy GUIs,
> > high-performance computing (GPU-based). This trend is observed on both
> > client and cloud. Efficient GPU virtualization is required to address
the
> > increasing demands.
> >
> >
> > We have developed XenGT - a prototype based on a mediated passthrough
> > architecture. We support running a native graphics driver in multiple
VMs
> > to achieve high performance. A specific mediator owns the scheduling
and
> > sharing hardware resources among all the virtual machines. By saying
> > mediated pass-through, we actually divide graphics resource to two
> > categories: performance critical and others. We partition performance
> > critical resources for VM''s direct access like pass-through,
while save
> > and restore others.
> >
> >
> >
> > XenGT implements the mediator in dom0, called vgt driver. This avoids
> > adding complex device knowledge to Xen, and also permits a more
flexible
> > release model. In the meantime, we want to have a unified architecture
to
> > mediate all the VMs, including dom0. Thus, We developed a
³deprivileged²
> > dom0 mode, which essentially trapped Dom0''s access to
selected resources
> > (graphics resources in XenGT case) and forwarded to vgt driver (which
is
> > also in Dom0) for processing.
> 
> But could be in another domain itself - an device driver domain right?
definitely. it''s just way how it works today. :-)

Thanks
Kevin
> >
> >
> > Performance critical resources are:
> > - Part of the global virtual memory space
> > - VM¹s own per-process virtual memory spaces
> > - VM¹s own allocated command buffers (actually in graphics memory)
> >
> >
> > Others are:
> > - MMIO/PIO
> > - PCI configuration registers
> > - GTT tables
> > - Submission of queued GPU commands
> >
> > Right now, we support 4 accelerated VMs: Dom0 + 3 HVM DomUs.
We''ve
> > conducted verifications based on Ubuntu 12.04 and 13.04. Tests
conducted
> > in VM include but are not limited to 3D gaming, media playbacks, 2D
> > accelerations.
> >
> > We believe the architecture itself can be general so that different
GPUs
> > can all use this mediated passthrough concept. However, we only
developed
> > codes based on Intel 4th Core Processor with integrated Graphics. Note
> > that we don''t require VT-d to run our solution.
> >
> > If you have interests in trying, you can refer to the attached setup
> > guide, which provides step-to-step details on
building/configuring/running
> > XenGT.
> >
> > Source code is made available at github:
> > Xen: https://github.com/01org/XenGT-Preview-xen.git
> > Linux: https://github.com/01org/XenGT-Preview-kernel.git
> > Qemu: https://github.com/01org/XenGT-Preview-qemu.git
> 
> Yeey! Awesome! Thank you for sharing that.
> 
> >
> > Any comments are welcome!
> >
> >
> > Special note: We are making this code available to general public
since we
> > take community''s involvement and feedbacks seriously.
However, while we''ve
> > tested our solution with various workloads, the code is only at
pre-alpha
> > stage. Hangs might happen, so please don''t try it on the
system that
> > hosting critical data for you.
> >
> > Shan Haitao
> >
> 
> 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao
wrote:
> Hi, Xen Experts,
> 
> This email is aimed at a first time disclosure of project XenGT, which is
> a Graphics virtualization solution based on Xen.
[...] 
> Source code is made available at github:
> Xen: https://github.com/01org/XenGT-Preview-xen.git
> Linux: https://github.com/01org/XenGT-Preview-kernel.git
> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
Hi,

These links don''t seem to work. Has the code been published yet?

--msw

Matt Wilson wrote on 2013-09-09:
> On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
>> Hi, Xen Experts,
>> 
>> This email is aimed at a first time disclosure of project XenGT, which
is
>> a Graphics virtualization solution based on Xen.
> 
> [...]
> 
>> Source code is made available at github:
>> Xen: https://github.com/01org/XenGT-Preview-xen.git
>> Linux: https://github.com/01org/XenGT-Preview-kernel.git
>> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
> 
> Hi,
> 
> These links don''t seem to work. Has the code been published yet?
> 
> --msw
Hi, looks one needs to log in github to use the browser to open the page,
or, "HTTP 404 - not found" will show in the browser.

Thanks,
-- Dexuan

On Mon, Sep 09, 2013 at 09:14:44PM +0000, Cui, Dexuan
wrote:
> Matt Wilson wrote on 2013-09-09:
> > On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
> >> Hi, Xen Experts,
> >> 
> >> This email is aimed at a first time disclosure of project XenGT,
which is
> >> a Graphics virtualization solution based on Xen.
> > 
> > [...]
> > 
> >> Source code is made available at github:
> >> Xen: https://github.com/01org/XenGT-Preview-xen.git
> >> Linux: https://github.com/01org/XenGT-Preview-kernel.git
> >> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
> > 
> > Hi,
> > 
> > These links don''t seem to work. Has the code been published
yet?
> Hi, looks one needs to log in github to use the browser to open the page,
> or, "HTTP 404 - not found" will show in the browser.
Strange! I was logged in before, but now I see the code (I''ve not done
anything in my browser).

--msw

Can you clone the code now? I thought it might be the issue that I sent out the
mail a bit earlier while one of my colleagues were still preparing the github
access.

Shan Haitao

-----Original Message-----
From: Matt Wilson [mailto:mswilson@gmail.com] On Behalf Of Matt Wilson
Sent: Tuesday, September 10, 2013 5:39 AM
To: Cui, Dexuan
Cc: Shan, Haitao; Tian, Kevin; White, Michael L; Dong, Eddie; Li, Susie;
Cowperthwaite, David J; xen-devel@lists.xen.org; Haron, Sandra
Subject: Re: [Xen-devel] [RFC] XenGT - An Mediated Graphics Passthrough Solution
from Intel

On Mon, Sep 09, 2013 at 09:14:44PM +0000, Cui, Dexuan
wrote:
> Matt Wilson wrote on 2013-09-09:
> > On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
> >> Hi, Xen Experts,
> >> 
> >> This email is aimed at a first time disclosure of project XenGT, 
> >> which is a Graphics virtualization solution based on Xen.
> > 
> > [...]
> > 
> >> Source code is made available at github:
> >> Xen: https://github.com/01org/XenGT-Preview-xen.git
> >> Linux: https://github.com/01org/XenGT-Preview-kernel.git
> >> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
> > 
> > Hi,
> > 
> > These links don''t seem to work. Has the code been published
yet?
> Hi, looks one needs to log in github to use the browser to open the 
> page, or, "HTTP 404 - not found" will show in the browser.
Strange! I was logged in before, but now I see the code (I''ve not done
anything in my browser).

--msw

> -----Original Message-----
> 
> XenGT implements the mediator in dom0, called vgt driver. This avoids
> adding complex device knowledge to Xen, and also permits a more flexible
> release model. In the meantime, we want to have a unified architecture to
> mediate all the VMs, including dom0. Thus, We developed a ³deprivileged²
> dom0 mode, which essentially trapped Dom0''s access to selected
resources
> (graphics resources in XenGT case) and forwarded to vgt driver (which is
> also in Dom0) for processing.
> 
Hi,

  Thanks for sharing the patches. From my cursory glance at the code do seem to
have put quite a lot of it into the hypervisor itself. Are you implementing your
device model there and talking directly to the vgt driver in some way, or is
trapped IO going via QEMU?

  Paul

Il 10/09/2013 01:19, Shan, Haitao ha scritto:
> Can you clone the code now? I thought it might be the issue that I sent out
the mail a bit earlier while one of my colleagues were still preparing the
github access.
>
> Shan Haitao
Now github links are working.
Thanks for this project that seems very good.
I think that is good rebase the patches for latest upstream develop 
version of xen,kernel,seabios and qemu and require the apply when will 
be possible to have more develop/testing support.
About xen patch I saw a small error, the configurazione file example 
should be done on xlexample.hvm and not xmexample.hvm.
I saw also append seabiose''s stdvgabios on hvmloader (on 
tools/firmware/hvmloader/Makefile): if I''m not wrong the vgabios are 
included automatically with seabios / upstream qemu. I have posted the 
same question long time ago and I got responded so.
About change on tools/libxl/libxl_dm.c, it is not needed to add support 
on libxl__build_device_model_args_old function which is for qemu 
traditional but only the new one for the upstream qemu.
About cpus support I see the 4th generation of core i3,i5, i7 processors 
but I didn''t found nothing about xeon, there are xeon series that 
support it?

Thanks for any reply.
>
> -----Original Message-----
> From: Matt Wilson [mailto:mswilson@gmail.com] On Behalf Of Matt Wilson
> Sent: Tuesday, September 10, 2013 5:39 AM
> To: Cui, Dexuan
> Cc: Shan, Haitao; Tian, Kevin; White, Michael L; Dong, Eddie; Li, Susie;
Cowperthwaite, David J; xen-devel@lists.xen.org; Haron, Sandra
> Subject: Re: [Xen-devel] [RFC] XenGT - An Mediated Graphics Passthrough
Solution from Intel
>
> On Mon, Sep 09, 2013 at 09:14:44PM +0000, Cui, Dexuan wrote:
>> Matt Wilson wrote on 2013-09-09:
>>> On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
>>>> Hi, Xen Experts,
>>>>
>>>> This email is aimed at a first time disclosure of project
XenGT,
>>>> which is a Graphics virtualization solution based on Xen.
>>> [...]
>>>
>>>> Source code is made available at github:
>>>> Xen: https://github.com/01org/XenGT-Preview-xen.git
>>>> Linux: https://github.com/01org/XenGT-Preview-kernel.git
>>>> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
>>> Hi,
>>>
>>> These links don''t seem to work. Has the code been
published yet?
>> Hi, looks one needs to log in github to use the browser to open the
>> page, or, "HTTP 404 - not found" will show in the browser.
> Strange! I was logged in before, but now I see the code (I''ve not
done anything in my browser).
>
> --msw
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

> From: Paul Durrant [mailto:Paul.Durrant@citrix.com]
> Sent: Tuesday, September 10, 2013 4:38 PM
> 
> > -----Original Message-----
> >
> > XenGT implements the mediator in dom0, called vgt driver. This avoids
> > adding complex device knowledge to Xen, and also permits a more
flexible
> > release model. In the meantime, we want to have a unified architecture
to
> > mediate all the VMs, including dom0. Thus, We developed a
³deprivileged²
> > dom0 mode, which essentially trapped Dom0''s access to
selected resources
> > (graphics resources in XenGT case) and forwarded to vgt driver (which
is
> > also in Dom0) for processing.
> >
> 
> Hi,
> 
>   Thanks for sharing the patches. From my cursory glance at the code do
seem
> to have put quite a lot of it into the hypervisor itself. Are you
implementing your
> device model there and talking directly to the vgt driver in some way, or
is
> trapped IO going via QEMU?
> 
device model is in vgt driver in dom0. hypervisor changes are more about:
	a) trap dom0''s mmio/pio accesses to gfx device and forward to vgt
driver for
emulation
	b) hypercall support to allow vgt driver accessing gfx device to avoid trap

Qemu is only involved for legacy VGA port emulation, i.e. legacy VGA (boot,
bsod, etc.)
is still shown in the qemu window. When gfx driver is loaded, the output will be
full screen on the physical monitor.

Thanks
Kevin

> From: Fabio Fantoni [mailto:fabio.fantoni@m2r.biz]
> Sent: Tuesday, September 10, 2013 4:47 PM
> 
> Il 10/09/2013 01:19, Shan, Haitao ha scritto:
> > Can you clone the code now? I thought it might be the issue that I
sent out
> the mail a bit earlier while one of my colleagues were still preparing the
github
> access.
> >
> > Shan Haitao
> 
> Now github links are working.
> Thanks for this project that seems very good.
> I think that is good rebase the patches for latest upstream develop
> version of xen,kernel,seabios and qemu and require the apply when will
> be possible to have more develop/testing support.
Agree. Now the repo is more to announce the project for early preview from
the community. We''ll rebase to latest upstream repos when starting
upstream
those bits.
> About xen patch I saw a small error, the configurazione file example
> should be done on xlexample.hvm and not xmexample.hvm.
> I saw also append seabiose''s stdvgabios on hvmloader (on
> tools/firmware/hvmloader/Makefile): if I''m not wrong the vgabios
are
> included automatically with seabios / upstream qemu. I have posted the
> same question long time ago and I got responded so.
> About change on tools/libxl/libxl_dm.c, it is not needed to add support
> on libxl__build_device_model_args_old function which is for qemu
> traditional but only the new one for the upstream qemu.
thanks for suggestion.
> About cpus support I see the 4th generation of core i3,i5, i7 processors
> but I didn''t found nothing about xeon, there are xeon series that
> support it?
We didn''t test Xeon so far, but you can expect the list to be extended
once we verified more platforms. :-)
> 
> Thanks for any reply.
> 
Thanks
Kevin
> >
> > -----Original Message-----
> > From: Matt Wilson [mailto:mswilson@gmail.com] On Behalf Of Matt Wilson
> > Sent: Tuesday, September 10, 2013 5:39 AM
> > To: Cui, Dexuan
> > Cc: Shan, Haitao; Tian, Kevin; White, Michael L; Dong, Eddie; Li,
Susie;
> Cowperthwaite, David J; xen-devel@lists.xen.org; Haron, Sandra
> > Subject: Re: [Xen-devel] [RFC] XenGT - An Mediated Graphics
Passthrough
> Solution from Intel
> >
> > On Mon, Sep 09, 2013 at 09:14:44PM +0000, Cui, Dexuan wrote:
> >> Matt Wilson wrote on 2013-09-09:
> >>> On Mon, Sep 09, 2013 at 12:46:23PM +0000, Shan, Haitao wrote:
> >>>> Hi, Xen Experts,
> >>>>
> >>>> This email is aimed at a first time disclosure of project
XenGT,
> >>>> which is a Graphics virtualization solution based on Xen.
> >>> [...]
> >>>
> >>>> Source code is made available at github:
> >>>> Xen: https://github.com/01org/XenGT-Preview-xen.git
> >>>> Linux: https://github.com/01org/XenGT-Preview-kernel.git
> >>>> Qemu: https://github.com/01org/XenGT-Preview-qemu.git
> >>> Hi,
> >>>
> >>> These links don''t seem to work. Has the code been
published yet?
> >> Hi, looks one needs to log in github to use the browser to open
the
> >> page, or, "HTTP 404 - not found" will show in the
browser.
> > Strange! I was logged in before, but now I see the code (I''ve
not done
> anything in my browser).
> >
> > --msw
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel