Xen devel - May 2010 - feature suggestion: DMAR table emulation for Xen

As many people on this list realize, lots of machines that have VT-d in
hardware, effectively does not support VT-d because of broken BIOSes
that usually do not expose DMAR ACPI table at all (or do it with some
bugs?).

So, the idea is to implement dmar emulation for Xen. E.g. if we pass
certain boot option, e.g. emulate_chipset4_dmar, then Xen would read
DMAR table from the image hardcoded into the Xen hypervisor.

We could provide images for series 4 and 5 chipsets. I think there are
no differences in how DMAR table looks between various specific chipsets
from the same family (e.g. Q45 would be probably using the very same
DMAR as PM45). Can somebody from Intel confirm this?

Implementing such a feature would make it possible to use Xen w/ IOMMU
support on many newer machines that otherwise do not support it.

joanna.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Joanna,

RMRR memory ranges specified memory ranges that might be used for BIOS initiated
DMA''s for USB and integrated graphics devices.  These ranges are set by
the BIOS.

Do they stay the same across different platforms?

Allen

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Joanna Rutkowska
Sent: Thursday, May 13, 2010 7:14 AM
To: xen-devel@lists.xensource.com
Subject: [Xen-devel] feature suggestion: DMAR table emulation for Xen

As many people on this list realize, lots of machines that have VT-d in
hardware, effectively does not support VT-d because of broken BIOSes that
usually do not expose DMAR ACPI table at all (or do it with some bugs?).

So, the idea is to implement dmar emulation for Xen. E.g. if we pass certain
boot option, e.g. emulate_chipset4_dmar, then Xen would read DMAR table from the
image hardcoded into the Xen hypervisor.

We could provide images for series 4 and 5 chipsets. I think there are no
differences in how DMAR table looks between various specific chipsets from the
same family (e.g. Q45 would be probably using the very same DMAR as PM45). Can
somebody from Intel confirm this?

Implementing such a feature would make it possible to use Xen w/ IOMMU support
on many newer machines that otherwise do not support it.

joanna.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 07:41 AM, Kay, Allen M wrote:
> Joanna,
> 
> RMRR memory ranges specified memory ranges that might be used for
> BIOS initiated DMA''s for USB and integrated graphics devices. 
These
> ranges are set by the BIOS.
> 
> Do they stay the same across different platforms?
> 
To be frank, those RMRR ranges have always been a bit of a mystery for
me. Do you actually know any OS drivers for USB devices that would
depend on them? Aren''t they only used by the BIOS to implement various
dirty tricks they like to implement?

The Intel VT-d spec says that: "USB controllers and UMA integrated
graphics devices are the only legacy device usages identified
that depend on DMA to reserved system memory".

I think we can safely ignore the legacy integrated graphics devices,
because I would expect any VT-d enabled platform to have a modern
graphics device and modern OS drivers that works well with VT-d (you
even provide a dedicated DMA Remapping Unit for your integrated graphics
device, so I would expect it doesn''t require any RMMR exceptions).

So USB devices seem to be the only potential problem. But USB devices
are implemented by the ICH, which in most cases is Intel ICH9, or 10.
So, if we took a DMAR from a platform, say PM45+ICH9M w/ Intel BIOS, it
should also work fine on any other laptop with this chipset (and
probably other variants too), just minus the potential
BIOS-specific-dirty-tricks...?

joanna.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/05/2010 10:22, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
wrote:
> The Intel VT-d spec says that: "USB controllers and UMA integrated
> graphics devices are the only legacy device usages identified
> that depend on DMA to reserved system memory".
> 
> I think we can safely ignore the legacy integrated graphics devices,
> because I would expect any VT-d enabled platform to have a modern
> graphics device and modern OS drivers that works well with VT-d (you
> even provide a dedicated DMA Remapping Unit for your integrated graphics
> device, so I would expect it doesn''t require any RMMR exceptions).
Yeah, actually the integrated graphics can implement all sorts of dirty
tricks between OS driver, video BIOS, and SMM. This can rely on fixed memory
areas for communication -- both for host accesses and DMA, the latter
requiring RMRR setup. Maybe the RMRRs are static per-chipset, but I
wouldn''t
be too sure of it.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

<< So, if we took a DMAR from a platform, say PM45+ICH9M w/ Intel BIOS, it
should also work fine on any other laptop with this chipset (and
probably other variants too), just minus the potential
BIOS-specific-dirty-tricks...?
>>
Bringing-up mobile variants with same base BIOS may not work, at least in my
experience, as OEMs tend to add various custom features (GPIO). These features
may to over-ride RMRR assumptions from base BIOS. This tends to be the most
prominant cause behind RMRR issues.

-Kaushik



******************************************************************************************
 This email and its attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed above.
Any use of the information contained here in any way (including, but not limited
to, total or partial disclosure, reproduction, or dissemination) by persons
other than the intended recipient(s) is prohibited. If you receive this email in
error, please notify the sender by phone or email immediately and delete it!
 *****************************************************************************************

----- Original Message -----
From: Joanna Rutkowska <joanna@invisiblethingslab.com>
Date: Friday, May 14, 2010 2:20 am
Subject: Re: [Xen-devel] feature suggestion: DMAR table emulation for Xen
To: "Kay, Allen M" <allen.m.kay@intel.com>
Cc: "Cihula, Joseph" <joseph.cihula@intel.com>,
"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
"Han, Weidong" <weidong.han@intel.com>
> On 05/14/2010 07:41 AM, Kay, Allen M wrote:
> > Joanna,
> > 
> > RMRR memory ranges specified memory ranges that might be used for
> > BIOS initiated DMA''s for USB and integrated graphics devices.
These
> > ranges are set by the BIOS.
> > 
> > Do they stay the same across different platforms?
> > 
> 
> To be frank, those RMRR ranges have always been a bit of a mystery for
> me. Do you actually know any OS drivers for USB devices that would
> depend on them? Aren''t they only used by the BIOS to implement
various
> dirty tricks they like to implement?
> 
> The Intel VT-d spec says that: "USB controllers and UMA integrated
> graphics devices are the only legacy device usages identified
> that depend on DMA to reserved system memory".
> 
> I think we can safely ignore the legacy integrated graphics devices,
> because I would expect any VT-d enabled platform to have a modern
> graphics device and modern OS drivers that works well with VT-d (you
> even provide a dedicated DMA Remapping Unit for your integrated 
> graphicsdevice, so I would expect it doesn''t require any RMMR 
> exceptions).
> So USB devices seem to be the only potential problem. But USB devices
> are implemented by the ICH, which in most cases is Intel ICH9, or 10.
> So, if we took a DMAR from a platform, say PM45+ICH9M w/ Intel 
> BIOS, it
> should also work fine on any other laptop with this chipset (and
> probably other variants too), just minus the potential
> BIOS-specific-dirty-tricks...?
> 
> joanna.
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 11:35 AM, Keir Fraser wrote:
> On 14/05/2010 10:22, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
> wrote:
> 
>> The Intel VT-d spec says that: "USB controllers and UMA integrated
>> graphics devices are the only legacy device usages identified
>> that depend on DMA to reserved system memory".
>>
>> I think we can safely ignore the legacy integrated graphics devices,
>> because I would expect any VT-d enabled platform to have a modern
>> graphics device and modern OS drivers that works well with VT-d (you
>> even provide a dedicated DMA Remapping Unit for your integrated
graphics
>> device, so I would expect it doesn''t require any RMMR
exceptions).
> 
> Yeah, actually the integrated graphics can implement all sorts of dirty
> tricks between OS driver, video BIOS, and SMM. This can rely on fixed
memory
> areas for communication -- both for host accesses and DMA, the latter
> requiring RMRR setup. Maybe the RMRRs are static per-chipset, but I
wouldn''t
> be too sure of it.
> 
Hmmm... Shouldn''t this affect only (and potentially) the text mode
display? I would expect that once Dom0 Linux takes over, it would be
using its own IGD driver that is VT-d aware and is not on the mercy of
the evil BIOS?

And we can disable the (ugly) Xen debug output, and have Dom0 linux boot
with kmodeset boot argument that would load the proper driver and switch
to a graphics mode right at the beginning.

j.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 11:41 AM, Barde Kaushik 00901718 wrote:
> << So, if we took a DMAR from a platform, say PM45+ICH9M w/ Intel
> BIOS, it should also work fine on any other laptop with this chipset
> (and probably other variants too), just minus the potential 
> BIOS-specific-dirty-tricks...?
>>> 
> 
> Bringing-up mobile variants with same base BIOS may not work, at
> least in my experience, as OEMs tend to add various custom features
> (GPIO). These features may to over-ride RMRR assumptions from base
> BIOS. This tends to be the most prominant cause behind RMRR issues.
> 
And what would happen if we ignore RMRRs on such a system?

j.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/05/2010 11:15, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
wrote:
>> Yeah, actually the integrated graphics can implement all sorts of dirty
>> tricks between OS driver, video BIOS, and SMM. This can rely on fixed
memory
>> areas for communication -- both for host accesses and DMA, the latter
>> requiring RMRR setup. Maybe the RMRRs are static per-chipset, but I
wouldn''t
>> be too sure of it.
>> 
> Hmmm... Shouldn''t this affect only (and potentially) the text mode
> display? I would expect that once Dom0 Linux takes over, it would be
> using its own IGD driver that is VT-d aware and is not on the mercy of
> the evil BIOS?
Well, if you do not pass through the IGD to a domU then the issue is moot.
Dom0 gets an all-inclusive mapping below 4GB, which should be a superset of
anything the RMRRs would specify. It''s when passing through to a domU
that
the RMRRs matter, especially if you pass through as the primary adaptor and
hence re-execute the video BIOS in domU context.

 -- Keir
> And we can disable the (ugly) Xen debug output, and have Dom0 linux boot
> with kmodeset boot argument that would load the proper driver and switch
> to a graphics mode right at the beginning.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 12:48 PM, Keir Fraser wrote:
> On 14/05/2010 11:15, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
> wrote:
> 
>>> Yeah, actually the integrated graphics can implement all sorts of
dirty
>>> tricks between OS driver, video BIOS, and SMM. This can rely on
fixed memory
>>> areas for communication -- both for host accesses and DMA, the
latter
>>> requiring RMRR setup. Maybe the RMRRs are static per-chipset, but I
wouldn''t
>>> be too sure of it.
>>>
>> Hmmm... Shouldn''t this affect only (and potentially) the text
mode
>> display? I would expect that once Dom0 Linux takes over, it would be
>> using its own IGD driver that is VT-d aware and is not on the mercy of
>> the evil BIOS?
> 
> Well, if you do not pass through the IGD to a domU then the issue is moot.
> Dom0 gets an all-inclusive mapping below 4GB, which should be a superset of
> anything the RMRRs would specify. It''s when passing through to a
domU that
> the RMRRs matter, especially if you pass through as the primary adaptor and
> hence re-execute the video BIOS in domU context.
> 
Well, we don''t do graphics passthrough in Qubes, mostly for two
reasons:

1) We believe users prefer seamless integration of all apps onto one
desktop (and that requires only one domain, e.g. Dom0, to have access to
the graphics card),

2) Giving a potentially untrusted domain full access to the graphics
device creates a potential security risk. In fact, you cannot make such
an architecture secure without using TXT (yes, TXT in addition to VT-d).

Do you do IGD passthrough in Xen Client?

joanna.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/05/2010 11:58, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
wrote:
> Well, we don''t do graphics passthrough in Qubes, mostly for two
reasons:
> 
> 1) We believe users prefer seamless integration of all apps onto one
> desktop (and that requires only one domain, e.g. Dom0, to have access to
> the graphics card),
> 
> 2) Giving a potentially untrusted domain full access to the graphics
> device creates a potential security risk. In fact, you cannot make such
> an architecture secure without using TXT (yes, TXT in addition to VT-d).
> 
> Do you do IGD passthrough in Xen Client?
Yes, particularly necessary to get decent 3D performance (e.g., Windows
Aero). There are tricks to get security and performance (doing a mix of
emulation and passthrough).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 01:29 PM, Keir Fraser wrote:
> On 14/05/2010 11:58, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
> wrote:
> 
>> Well, we don''t do graphics passthrough in Qubes, mostly for
two reasons:
>>
>> 1) We believe users prefer seamless integration of all apps onto one
>> desktop (and that requires only one domain, e.g. Dom0, to have access
to
>> the graphics card),
>>
>> 2) Giving a potentially untrusted domain full access to the graphics
>> device creates a potential security risk. In fact, you cannot make such
>> an architecture secure without using TXT (yes, TXT in addition to
VT-d).
>>
>> Do you do IGD passthrough in Xen Client?
> 
> Yes, particularly necessary to get decent 3D performance (e.g., Windows
> Aero). There are tricks to get security and performance (doing a mix of
> emulation and passthrough).
> 
It''s good to distinguish 3D acceleration for the *Window Manager* vs.
3D
acceleration for the *apps*. In Qubes we keep the Window Manager in
Dom0, so, as long as the graphics driver is not broken, the Window
Manager gets all the fancy 3D effects, just like on the native.

The only(?) advantage you gain by using graphics passthorugh is for the
apps. But is it really worth it? How many 3D-capable apps business users
use today? Google Earth? Anything else? So, in order to support Google
Earth, is it really worth to introduce potential VM escape attacks?

joanna.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Yes, particularly necessary to get decent 3D performance (e.g.,
Windows
> > Aero). There are tricks to get security and performance (doing a mix
of
> > emulation and passthrough).
> >
> 
> It''s good to distinguish 3D acceleration for the *Window Manager*
vs.
3D
> acceleration for the *apps*. In Qubes we keep the Window Manager in
> Dom0, so, as long as the graphics driver is not broken, the Window
> Manager gets all the fancy 3D effects, just like on the native.
> 
> The only(?) advantage you gain by using graphics passthorugh is for
the
> apps. But is it really worth it? How many 3D-capable apps business
users
> use today? Google Earth? Anything else? So, in order to support Google
> Earth, is it really worth to introduce potential VM escape attacks?
> 
CAD? Nothing to do with Xen in my case but it''s causing one of our
clients grief at the moment as they want to use Windows 7 but one CAD
app only works with XP and ''XP Mode''[1] under Windows 7 just
emulates a
craptastic S3 video adapter with no acceleration at all. It looks like
the solution will be 2 PC''s for a while...

In that same organisation, most workstations have a requirement to view
the 3D models at reasonable speeds.

I don''t see CAD workstations as big users of xen as they are generally
single purpose, but it''s probably a mistake to think that the only
business app with 3D visualisation requirements is Google Earth.

James

[1] for anyone who doesn''t know, ''XP Mode'' is a
product that uses
Microsoft VirtualPC to run an instance of XP, licensed as part of the
Windows 7 license. Good for a lot of things, but not CAD.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/05/2010 12:47, "Joanna Rutkowska"
<joanna@invisiblethingslab.com>
wrote:
> It''s good to distinguish 3D acceleration for the *Window Manager*
vs. 3D
> acceleration for the *apps*. In Qubes we keep the Window Manager in
> Dom0, so, as long as the graphics driver is not broken, the Window
> Manager gets all the fancy 3D effects, just like on the native.
>
> The only(?) advantage you gain by using graphics passthorugh is for the
> apps. But is it really worth it? How many 3D-capable apps business users
> use today? Google Earth? Anything else? So, in order to support Google
> Earth, is it really worth to introduce potential VM escape attacks?
Well, XenClient != Qubes. ;-) Different system architecture; different
threat model; and probably different usage scenarios as well.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sure, you can but I think, you may have issues if RMRR reports both existent and
non-existent device scopes, and if ignored, existent device may be impacted
along with non-existent one''s.

-Kaushik

******************************************************************************************
 This email and its attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed above.
Any use of the information contained here in any way (including, but not limited
to, total or partial disclosure, reproduction, or dissemination) by persons
other than the intended recipient(s) is prohibited. If you receive this email in
error, please notify the sender by phone or email immediately and delete it!
 *****************************************************************************************

----- Original Message -----
From: Joanna Rutkowska <joanna@invisiblethingslab.com>
Date: Friday, May 14, 2010 3:13 am
Subject: Re: [Xen-devel] feature suggestion: DMAR table emulation for Xen
To: Barde Kaushik 00901718 <kbarde@huawei.com>
Cc: "Kay, Allen M" <allen.m.kay@intel.com>, "Cihula,
Joseph" <joseph.cihula@intel.com>,
"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
"Han, Weidong" <weidong.han@intel.com>
> On 05/14/2010 11:41 AM, Barde Kaushik 00901718 wrote:
> > << So, if we took a DMAR from a platform, say PM45+ICH9M w/
Intel
> > BIOS, it should also work fine on any other laptop with this chipset
> > (and probably other variants too), just minus the potential 
> > BIOS-specific-dirty-tricks...?
> >>> 
> > 
> > Bringing-up mobile variants with same base BIOS may not work, at
> > least in my experience, as OEMs tend to add various custom features
> > (GPIO). These features may to over-ride RMRR assumptions from base
> > BIOS. This tends to be the most prominant cause behind RMRR issues.
> > 
> And what would happen if we ignore RMRRs on such a system?
> 
> j.
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The only(?) advantage you gain by using graphics passthorugh is for the
> apps. But is it really worth it? How many 3D-capable apps business
> users
> use today? Google Earth? Anything else? So, in order to support Google
> Earth, is it really worth to introduce potential VM escape attacks?
Are you assuming business users never play the hottest latest
games at lunch or after hours?  I suppose you could use crappy
game performance as a positive selling point for your product
as long as your target audience for your sales pitch is
the CTO and other executives and not the business users
themselves. :-)

(Note, tongue-in-cheek, no need to defend your product. :)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/14/2010 05:57 PM, Dan Magenheimer wrote:
>> The only(?) advantage you gain by using graphics passthorugh is for the
>> apps. But is it really worth it? How many 3D-capable apps business
>> users
>> use today? Google Earth? Anything else? So, in order to support Google
>> Earth, is it really worth to introduce potential VM escape attacks?
> 
> Are you assuming business users never play the hottest latest
> games at lunch or after hours?
Gaming during lunch?! Hey, do you know how unhealthy that is? :)

Gaming after hours?! Besides the standard argument -- get some social
life! -- why not get a proper gaming console? :)

j.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The only(?) advantage you gain by using graphics passthorugh is for the
apps.
> But is it really worth it? How many 3D-capable apps business users use
today?
> Google Earth? Anything else? So, in order to support Google Earth, is it
really
> worth to introduce potential VM escape attacks?
>
> joanna.
>
Video also needs HW acceleration.  Internet video now consists of most of the
internet traffic.  Large companies also use webcasts for employee communication.

Allen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Well, we don''t do graphics passthrough in Qubes, mostly for two
reasons:
> 
> 1) We believe users prefer seamless integration of all apps onto one
> desktop (and that requires only one domain, e.g. Dom0, to have access to
> the graphics card),
Not necessarily. If dom0 retains control of the display side of the GPU you can
use video overlays to merge in windows from a couple of other domains. [Though
ideally the GPU would use different PCI requestor IDs for displaying the
framebuffer or fetching overlay data vs rendering]
> 2) Giving a potentially untrusted domain full access to the graphics
> device creates a potential security risk. In fact, you cannot make such an
> architecture secure without using TXT (yes, TXT in addition to VT-d).
I agree you can''t give an untrusted domain full access, but
there''s an interesting design point where you emulate some parts of the
GPU (e.g. anything controlling what is displayed) and pass-through others. GPU
designs that enable you to do this without babysitting the command rings are
obviously preferable...

Best,
Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 05/15/2010 06:54 PM, Ian Pratt wrote:
>> Well, we don''t do graphics passthrough in Qubes, mostly for
two reasons:
>>
>> 1) We believe users prefer seamless integration of all apps onto one
>> desktop (and that requires only one domain, e.g. Dom0, to have access
to
>> the graphics card),
> 
> Not necessarily. If dom0 retains control of the display side of the
> GPU you can use video overlays to merge in windows from a couple of
> other domains. [Though ideally the GPU would use different PCI
> requestor IDs for displaying the framebuffer or fetching overlay data
> vs rendering]
> 
>> 2) Giving a potentially untrusted domain full access to the graphics
>> device creates a potential security risk. In fact, you cannot make such
an
>> architecture secure without using TXT (yes, TXT in addition to VT-d).
> 
> I agree you can''t give an untrusted domain full access, but
there''s
> an interesting design point where you emulate some parts of the GPU
> (e.g. anything controlling what is displayed) and pass-through
> others. GPU designs that enable you to do this without babysitting
> the command rings are obviously preferable...
> 
I''ve been actually more concerned about the *rich* interface to the GPU
that you still must expose to DomU. While semantically it might not be
insecure (from what you wrote it might be perhaps possible to prevent
DomU from reading the screen buffer, etc), I bet it will contain
exploitable implementation flaws. And DomU would be able to exploit
them, similarly like one can exploit many modern NIC cards today (see [1]).

Cheers,
joanna.

[1]
http://theinvisiblethings.blogspot.com/2010/04/remotely-attacking-network-cards-or-why.html



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel