For background about why I''m asking this, and some other things, you may want to optionally read this Thread: http://www.xtremesystems.org/forums/showthread.php?285408-IOMMU-virtualization Also, keep in mind that I have no Xen experience - all my virtualization has been done using Microsoft Virtual PC 6, so my practical knowledge is limited. But I want to run Xen, after I get the proper Hardware to run it like I want to. When I started to hear about virtualization around 5 or 6 years ago, one of the main issues at that time was that for the average Windows user it was mostly useless for just one reason: You couldn''t actually use the Video Card. And that means no gaming on a virtualized enviroment. Some time later, I started to hear about Xen supporting "VGA passthrough", that allowed a Virtual Machine to see and use the Video Card. If I recall correctly, it was in these videos: http://www.youtube.com/watch?v=1ia3IwG6tp4 http://www.youtube.com/watch?v=5I13E1MQbMc http://www.youtube.com/watch?v=yYg6n8yBktM Because the machine that as been used for these videos is supposed to be a Core 2 Duo E6300, it couldn''t have any type of IOMMU virtualization as that feature, Intel VT-d, was introduced one generation later, starting with Nehalem based Processors. However, everytime I google around I hear people saying that in order to do VGA passthrough, you need AMD-Vi or Intel VT-d, when according to those old videos passthrough was a different thing that didn''t requiere IOMMU virtualization support for it and that is why it worked for him. So, to resume this, can anyone explain or give a documentation that can exactly tell what is the difference, pros or cons, bewthem that old passthrough that I saw and what you can currently do with AMD-Vi/Intel VT-d? At least, if I recall correctly, the problem with that type of passthrough is that only ONE Virtual Machine can make use of the Video Card. If I virtualize it, can multiple VMs share the same GPU, like the OSes currently do with the CPU? What are the limitations? _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hello Zir, I may not have an answer to your question, because I believe Teo said his CPU did support VT-d despite it not being listed in the specs, and I am certain his motherboard did. As far as I know VT-d or AMD-Vi is required for passthrough, and nVidia cards require a certain amount of patchwork with the latest stable. VT-d allows hardware to be addressed by a virtual machine, granting you near-native performance. I wrote a guide on setup and configuration<http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial>(slightly dated), and you can find a more modern demo video of the performance here <http://www.youtube.com/watch?v=Nz2c0Up2axk&t=4m58s>. On Mon, Apr 8, 2013 at 3:53 PM, Zir Blazer <zir_blazer@hotmail.com> wrote:> For background about why I''m asking this, and some other things, you may > want to optionally read this Thread: > > http://www.xtremesystems.org/forums/showthread.php?285408-IOMMU-virtualization > Also, keep in mind that I have no Xen experience - all my virtualization > has been done using Microsoft Virtual PC 6, so my practical knowledge is > limited. But I want to run Xen, after I get the proper Hardware to run it > like I want to. > > > When I started to hear about virtualization around 5 or 6 years ago, one > of the main issues at that time was that for the average Windows user it > was mostly useless for just one reason: You couldn''t actually use the Video > Card. And that means no gaming on a virtualized enviroment. > Some time later, I started to hear about Xen supporting "VGA passthrough", > that allowed a Virtual Machine to see and use the Video Card. If I recall > correctly, it was in these videos: > <http://www.youtube.com/watch?v=1ia3IwG6tp4> > http://www.youtube.com/watch?v=1ia3IwG6tp4 > http://www.youtube.com/watch?v=5I13E1MQbMc > http://www.youtube.com/watch?v=yYg6n8yBktM > > Because the machine that as been used for these videos is supposed to be a > Core 2 Duo E6300, it couldn''t have any type of IOMMU virtualization as that > feature, Intel VT-d, was introduced one generation later, starting with > Nehalem based Processors. However, everytime I google around I hear people > saying that in order to do VGA passthrough, you need AMD-Vi or Intel VT-d, > when according to those old videos passthrough was a different thing that > didn''t requiere IOMMU virtualization support for it and that is why it > worked for him. > > So, to resume this, can anyone explain or give a documentation that can > exactly tell what is the difference, pros or cons, bewthem that old > passthrough that I saw and what you can currently do with AMD-Vi/Intel > VT-d? At least, if I recall correctly, the problem with that type of > passthrough is that only ONE Virtual Machine can make use of the Video > Card. If I virtualize it, can multiple VMs share the same GPU, like the > OSes currently do with the CPU? What are the limitations? > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Andrew Bobulsky
2013-Apr-08 22:55 UTC
Re: AMD-Vi/Intel VT-d: Passthrough or virtualization?
Hello Zir, On Apr 8, 2013, at 5:38 PM, Zir Blazer <zir_blazer@hotmail.com> wrote: <snip>> At least, if I recall correctly, the problem with that type of passthrough is that only ONE Virtual Machine can make use of the Video Card. If I virtualize it, can multiple VMs share the same GPU, like the OSes currently do with the CPU? What are the limitations?The limitations are tough to Hague because, in practice, they''ll vary from product to product, and likely driver to driver. That said, PCI Express technically supports hot plugging, so you can disconnect the device from one machine and attach it to another while they''re online, and it''ll likely work at the bus level, at least. That''s a project on my infinitely large shelf. :P Cheers, Andrew Bobulsky
On 8 April 2013 20:53, Zir Blazer <zir_blazer@hotmail.com> wrote: [...snip...]> > When I started to hear about virtualization around 5 or 6 years ago, one of > the main issues at that time was that for the average Windows user it was > mostly useless for just one reason: You couldn''t actually use the Video > Card. And that means no gaming on a virtualized enviroment. > Some time later, I started to hear about Xen supporting "VGA passthrough", > that allowed a Virtual Machine to see and use the Video Card. If I recall[..snip..]> Because the machine that as been used for these videos is supposed to be a > Core 2 Duo E6300, it couldn''t have any type of IOMMU virtualization as that > feature, Intel VT-d, was introduced one generation later, starting with > Nehalem based Processors. However, everytime I google around I hear people > saying that in order to do VGA passthrough, you need AMD-Vi or Intel VT-d, > when according to those old videos passthrough was a different thing that > didn''t requiere IOMMU virtualization support for it and that is why it > worked for him. > > So, to resume this, can anyone explain or give a documentation that can > exactly tell what is the difference, pros or cons, bewthem that old > passthrough that I saw and what you can currently do with AMD-Vi/Intel VT-d? > At least, if I recall correctly, the problem with that type of passthrough > is that only ONE Virtual Machine can make use of the Video Card. If I > virtualize it, can multiple VMs share the same GPU, like the OSes currently > do with the CPU? What are the limitations?Strictly speaking there are two types of VT-d - VT-d, and VT-d2. VT-d is the initial virtualisation for directed I/O introduced in the Conroe era of Intel processors, whilst VT-d2 arrived with the Nehalem architecture. VT-d is mostly a chipset technology in particular a memory controller feature not a processor feature. Additionally this varies on an individual motherboard basis - the X38/X48 chipsets have the potential to support VT-d for instance, but it must be supported in the BIOS (it needs various tables, it''s not just an on/off switch). With the introduction of Nehalem, the Memory Controller Hub was of course moved into the processor and for some reason certain processors (usually the K models) are not enabled for VT-d. Nehalem and the equivalent AMD chips also support Second Level Address Translation (EPT on Intel, RVI on AMD) which offers a substantial speedup for various operations, and in particular virtualised graphics cards in some instances. The latter is supposedly the reason why Windows Phone 8 development (which needs Hyper-V) also requires a SLAT capable CPU. Multiple VMs can share the same card, if they are SR IOV capable. Graphics cards are a whole extra level of pain - performing passthrough to them is still leading edge except with very specific hardware and software configurations. You should be able to to hot plug cards as mentioned. You''re better with multiple cards if you want to run multiple graphically accelerated VMs. As above, you need a processor that supports VT-d, a chipset that supports it and a BIOS that is enabled for it and actually works. For instance, I have passthrough working on my pre nehalem system with a basic Matrox card, but not yet an AMD configuration. The BIOS did not support VT-d initially until an update was applied. PK
As I don''t know how mailing list works (First time in one. Seriously, after one decade using Forums I see them much more convenient and a mailing list quite hard to use and archaic), I hope that when I replied you, it will look nested in the tree of this Thread''s tree. > Strictly speaking there are two types of VT-d - VT-d, and VT-d2. VT-d> is the initial virtualisation for directed I/O introduced in the > Conroe era of Intel processors, whilst VT-d2 arrived with the Nehalem > architecture. > > VT-d is mostly a chipset technology in particular a memory controller > feature not a processor feature. Additionally this varies on an > individual motherboard basis - the X38/X48 chipsets have the potential > to support VT-d for instance, but it must be supported in the BIOS (it > needs various tables, it''s not just an on/off switch).I did some more research and you''re right. VT-d Hardware support was actually introduced with the Wolfdale platform, specifically Chipsets Q35, Q45, X38, and X48. I just never paid attention to it before Nehalem: http://en.wikipedia.org/wiki/List_of_intel_chipsets#Core_2_chipsets The Motherboard that the guy from the videos I linked before was using is a Intel DQ45CB, so he should have VT-d support (It actually mentions it on the description of some of his other videos). This pretty much solves my original question, as in order to do the passthrough, you need to have it virtualized. So IOMMU virtualization is a must. Just for the record, according to another Wikipedia list, on the old Wolfdale platform, VT-d support ALSO requiered some degree of Processor support besides Chipset: http://en.wikipedia.org/wiki/List_of_Intel_Core_2_microprocessors#.22Wolfdale.22_.2845_nm.29 It says that Core 2 Duo E8200 (Wolfdale) didn''t had VT-d support. A check on Intel Ark shows that difference, if you compare a C2D E8200 to a E8300: http://ark.intel.com/products/33909 C2D E8200 http://ark.intel.com/products/35070 C2D E8300 However, the C2D E6300 (Conroe) of the guy that did the videos doesn''t show VT-d support as feature, neither: http://ark.intel.com/products/27248/ To be honest, considering that the Memory Controller responsible of VT-d support was at the Chipset, I suppose that Processor support was pretty much a cosmetic thing simply to enforce you to have to spend on a more expensive Processor to have that feature, a common Intel practice. But as in the C2D E6300 it isn''t, it leaves me thinking if you really needed Processor support or they simply forgot to add it as feature on the earlier Conroe models. Not that it matters at this point, as I don''t think that anyone right now would get a 6 years old system for GPU virtualization purposes.> With the introduction of Nehalem, the Memory Controller Hub was of > course moved into the processor and for some reason certain processors > (usually the K models) are not enabled for VT-d.I always took notice that the K models doesn''t got VT-d support. The exception is the Core i7 3930K (Sandy Bridge-E), that is more that twice as expensive as the Core i5 2500K or the newer Ivy Bridge equivalent. For some reason, Intel doesn''t want that people that wants to have overclocking freedom uses IOMMU virtualization. A pretty nasty market segmentation tactic, if you ask me, as a K model is more expensive due to the Unlocked Multiplier and usually slighty better Integrated GPU, yet you lose VT-d, vPro and TXT. Besides, I don''t have entirely clear if after Nehalem you still need Chipset support for VT-d, or now that it is a Processor feature, its support is Chipset independent. Considering that in order to make use of the Unlocked Multiplier of a K series Intel Processor you need to have some specific Chipsets (Usually the Z series and some P), and that also for vPro you need another different specific Chipsets series (The Q series inteded for business), I wouldn''t be surprised if VT-d would need a specific Chipset too besides Processor support, but I couldn''t find any sort of artificially imposed Hardware limitation. At least, I find weird what happens on the AMD side: If the Memory Controller was integrated to the Processor during the K8 era, I don''t get why AMD-Vi is a Chipset feature. Yet there is no mention that you need a specific Processor to use it (Or some are artificially limited like Intel does), if all them includes support for it, or if support isn''t actually related to the Processor itself and only Chipset and BIOS.> Nehalem and the > equivalent AMD chips also support Second Level Address Translation > (EPT on Intel, RVI on AMD) which offers a substantial speedup for > various operations, and in particular virtualised graphics cards in > some instances. The latter is supposedly the reason why Windows Phone > 8 development (which needs Hyper-V) also requires a SLAT capable CPU.At least from AMD side, RVI seems to be supported from pretty much everything after K10 Barcelona, so it doesn''t seem to be any type of show stopper: http://support.amd.com/us/kbarticles/Pages/GPU120AMDRVICPUsHyperVWin8.aspx It is still good to know that.> As above, you need a processor that > supports VT-d, a chipset that supports it and a BIOS that is enabled > for it and actually works. For instance, I have passthrough working on > my pre nehalem system with a basic Matrox card, but not yet an AMD > configuration. The BIOS did not support VT-d initially until an update > was applied.One of the things that I mentioned in the Thread that I linked in my original Post here, is that is pretty much unknow if you have proper, working BIOS support until you actually give the Motherboard a try, and nothing guarantees you that the Motherboard manufacturer will provide you later with a BIOS upgrade. At least on AMD Motherboards, this seems to be a huge issue, no idea on current Intel platforms. I was thinking about going to Coreboot (An open source BIOS replacement) mailing list precisely to ask about this matter, so in case that an AMD Motherboard got a supporting Chipset but the BIOS doesn''t works properly (If I recall correctly, a common issue on AMD Motherboards was that the ACPI IVRS tables were missing), I could at least have a backup alternative. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
From my experience if VT-d or IOMMU are not explicitly mentioned in the user manuals (available for download off the net before you spend a dime on the board) then it likely does not have support for it. A couple of years ago support as abysmal, and I ended up finding ASRock to be an excellent option since they had it on all their premium boards. Besides BIOS options another huge problem I encountered was PCI Bridges on boards with lots of PCI Express slots, which are an extra layer and not always compatible with IOMMU. This caused me a lot of headaches when I bought an ASRock Z68 Extreme7 Gen3, as the NF200 PCI Bridge broke all but one PCI Express slot from being IOMMU compatible. Knowing which controllers on the board map to which devices on the otherhand is not mentioned in the board manual, so for example the 5 USB controllers on my current board I had to figure out their mappings manually, and one is shared with the BlueTooth, so that''s another hardware concern. On Tue, Apr 9, 2013 at 6:20 PM, Zir Blazer <zir_blazer@hotmail.com> wrote:> As I don''t know how mailing list works (First time in one. Seriously, > after one decade using Forums I see them much more convenient and a mailing > list quite hard to use and archaic), I hope that when I replied you, it > will look nested in the tree of this Thread''s tree. > > > > > Strictly speaking there are two types of VT-d - VT-d, and VT-d2. VT-d > > is the initial virtualisation for directed I/O introduced in the > > Conroe era of Intel processors, whilst VT-d2 arrived with the Nehalem > > architecture. > > > > VT-d is mostly a chipset technology in particular a memory controller > > feature not a processor feature. Additionally this varies on an > > individual motherboard basis - the X38/X48 chipsets have the potential > > to support VT-d for instance, but it must be supported in the BIOS (it > > needs various tables, it''s not just an on/off switch). > > I did some more research and you''re right. VT-d Hardware support was > actually introduced with the Wolfdale platform, specifically Chipsets Q35, > Q45, X38, and X48. I just never paid attention to it before Nehalem: > > http://en.wikipedia.org/wiki/List_of_intel_chipsets#Core_2_chipsets > > The Motherboard that the guy from the videos I linked before was using is > a Intel DQ45CB, so he should have VT-d support (It actually mentions it on > the description of some of his other videos). This pretty much solves my > original question, as in order to do the passthrough, you need to have it > virtualized. So IOMMU virtualization is a must. > > > Just for the record, according to another Wikipedia list, on the old > Wolfdale platform, VT-d support ALSO requiered some degree of Processor > support besides Chipset: > > > http://en.wikipedia.org/wiki/List_of_Intel_Core_2_microprocessors#.22Wolfdale.22_.2845_nm.29 > > It says that Core 2 Duo E8200 (Wolfdale) didn''t had VT-d support. A check > on Intel Ark shows that difference, if you compare a C2D E8200 to a E8300: > > http://ark.intel.com/products/33909 C2D E8200 > <http://ark.intel.com/products/35070>http://ark.intel.com/products/35070C2D E8300 > > However, the C2D E6300 (Conroe) of the guy that did the videos doesn''t > show VT-d support as feature, neither: > > http://ark.intel.com/products/27248/ > > To be honest, considering that the Memory Controller responsible of VT-d > support was at the Chipset, I suppose that Processor support was pretty > much a cosmetic thing simply to enforce you to have to spend on a more > expensive Processor to have that feature, a common Intel practice. But as > in the C2D E6300 it isn''t, it leaves me thinking if you really needed > Processor support or they simply forgot to add it as feature on the earlier > Conroe models. Not that it matters at this point, as I don''t think that > anyone right now would get a 6 years old system for GPU virtualization > purposes. > > > > > With the introduction of Nehalem, the Memory Controller Hub was of > > course moved into the processor and for some reason certain processors > > (usually the K models) are not enabled for VT-d. > > I always took notice that the K models doesn''t got VT-d support. The > exception is the Core i7 3930K (Sandy Bridge-E), that is more that twice as > expensive as the Core i5 2500K or the newer Ivy Bridge equivalent. For some > reason, Intel doesn''t want that people that wants to have overclocking > freedom uses IOMMU virtualization. A pretty nasty market segmentation > tactic, if you ask me, as a K model is more expensive due to the Unlocked > Multiplier and usually slighty better Integrated GPU, yet you lose VT-d, > vPro and TXT. > > Besides, I don''t have entirely clear if after Nehalem you still need > Chipset support for VT-d, or now that it is a Processor feature, its > support is Chipset independent. Considering that in order to make use of > the Unlocked Multiplier of a K series Intel Processor you need to have some > specific Chipsets (Usually the Z series and some P), and that also for vPro > you need another different specific Chipsets series (The Q series inteded > for business), I wouldn''t be surprised if VT-d would need a specific > Chipset too besides Processor support, but I couldn''t find any sort of > artificially imposed Hardware limitation. > At least, I find weird what happens on the AMD side: If the Memory > Controller was integrated to the Processor during the K8 era, I don''t get > why AMD-Vi is a Chipset feature. Yet there is no mention that you need a > specific Processor to use it (Or some are artificially limited like Intel > does), if all them includes support for it, or if support isn''t actually > related to the Processor itself and only Chipset and BIOS. > > > > > Nehalem and the > > equivalent AMD chips also support Second Level Address Translation > > (EPT on Intel, RVI on AMD) which offers a substantial speedup for > > various operations, and in particular virtualised graphics cards in > > some instances. The latter is supposedly the reason why Windows Phone > > 8 development (which needs Hyper-V) also requires a SLAT capable CPU. > > At least from AMD side, RVI seems to be supported from pretty much > everything after K10 Barcelona, so it doesn''t seem to be any type of show > stopper: > http://support.amd.com/us/kbarticles/Pages/GPU120AMDRVICPUsHyperVWin8.aspx > It is still good to know that. > > > > > > As above, you need a processor that > > supports VT-d, a chipset that supports it and a BIOS that is enabled > > for it and actually works. For instance, I have passthrough working on > > my pre nehalem system with a basic Matrox card, but not yet an AMD > > configuration. The BIOS did not support VT-d initially until an update > > was applied. > > One of the things that I mentioned in the Thread that I linked in my > original Post here, is that is pretty much unknow if you have proper, > working BIOS support until you actually give the Motherboard a try, and > nothing guarantees you that the Motherboard manufacturer will provide you > later with a BIOS upgrade. At least on AMD Motherboards, this seems to be a > huge issue, no idea on current Intel platforms. > > I was thinking about going to Coreboot (An open source BIOS replacement) > mailing list precisely to ask about this matter, so in case that an AMD > Motherboard got a supporting Chipset but the BIOS doesn''t works properly > (If I recall correctly, a common issue on AMD Motherboards was that the > ACPI IVRS tables were missing), I could at least have a backup alternative. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
I hope that this revives my old Thread instead of creating a new one. In case it doesn''t, the original is here: http://lists.xen.org/archives/html/xen-users/2013-04/msg00115.html After googling around some more, it seems that what you do with IOMMU virtualization is just mere passthrough, and not even the Hypervisor can use directly the Hardware that was passed onto a VM. If you pass, say, a Video Card to the VM, the Hypervisor can''t use it any longer. This means that if you only have a single Video Card, you potentially will lose video output for anything but that VM. Best setup seems to be an integrated Video Card like the one that most Processors currently got to handle the Hypervisor + other VMs with the standard emulated VGA Drivers plus a discrete Video Card for the gaming VM. Virtualization is a different beast that must be supported by that Hardware part itself. Currently, nVidia got some Video Cards named GRID that supposedly got some extra Software for the Hypervisor that allows a GPU to be virtualized and shared among many VMs. XenServer currently seems to support this: http://www.nvidia.com/object/enterprise-virtualization.html#source=pr AMD was going to follow suit with their Radeon Sky line. I don''t know if there are Software workarounds to do the same with standard consumer parts. I suppose that is one of the premium features that gets disabled for the Desktop lines. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
There was no way I was going to pass on this conversation, even with my busy schedule.>> This means that if you only have a single Video Card, you potentially will lose >> video output for anything but that VM. Best setup seems to be an integrated Video Card >> like the one that most Processors currently got to handle the Hypervisor + other VMs with >> the standard emulated VGA Drivers plus a discrete Video Card for the gaming VM.My understanding was that this relied on the number of Virtual Functions a PCI device was equipped with in the firmware. This at least is the case for network cards... Not to chase tails here however, can we step back and figure out which of the chipset manufactures (AMD vs. Intel) provides a stable platform that can be used in production. We are not necessarily interested in GPUs but we are interested in passing through network cards QLogic, Intel etc... I would imagine this would still be important to the gamers, and Justin.tv broadcasters as well.... Secondly, are any of the manufactures such as IBM and DELL openly pushing stable virtualized platforms (chipsets, pci bus, cpus, bios), that we can again run in a production. And of course, there should be XEN compatibility on top of all this... We can understand why the chipset, cpu, and even pci hardware manufactures would play this cat and mouse game with virtualiztion since to them it equates to less sales......>> Casey DeLorme >> From my experience if VT-d or IOMMU are not explicitly mentioned in the user manuals >> (available for download off the net before you spend a dime on the board) then it likely >> does not have support for it.Interesting... We do something similar when purchasing IBMs. We look to see if there are BIOS firmware updates that involve virtualization such as this: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086623 I came in a little late in the game for this conversation however, can we please iron out some issues here. At an abstract level (i.e., chipsets, cpus, gpu, network interfaces), without mentioning any motherboard manufactures such as ASRock, Asus, Saphire etc.. can we determine which combination will work. Both on the AMD and Intel platform. The reason for this is because not too many people deploy white boxes for production. it''s strictly SuperMicro, IBM, Dell etc... Once determining this, it would be nice to discuss which of the manufactures are able to support stable platforms without rendering 3 out of the 4 PCIe/x slots useless. This would hurt! We also need to keep it fairly modern. For example, PCIe slots would preferably be PCIe v2 x8/x16 PCIe v3 x8/x16. For example, I mentioned IBM enabled virtualization server however, what I am really looking at are the guts. For example: Intel Based (x3550 M4) CPU: E5-2600 Chipset: Intel C604 All PCIe slots are PCIe 3.0 Slot 1: PCIe x16; low profile, half-length Slot 2: PCIe x8, opt. PCI-X or PCIe x16; full-height/half-length (PCIe x16 req. 2nd CPU) I am actually not sure if this platform support virtulaization since the little bit of googling did not yield much. AMD Based: x3755 M3 CPU: AMD Opteron 6200 Chipset: SR5690 All PCIe slots are PCIe 2.0 Slot 1: PCIe x16, full height, full length Slot 2: PCIe x8, low profile, half length Slot 3: PCIe x8 (x4 wired), low profile, half length Slot 4: PCIe x8, low profile, half length (internal only, reserved for RAID cont This machine I think does support IOMMU passthrough at the bios level since there has been firmware updates for it: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086623 Buying hardware just got a lot more trickier folks, let''s hope this thread can shed some light on our lost souls, and make us complete again.... Kind Regards, Nick.
>> Besides BIOS options another huge problem I encountered was PCI Bridges on boards with >> lots of PCI Express slots, which are an extra layer and not always compatible with >> IOMMU. This caused me a lot of headaches when I bought an ASRock Z68 Extreme7 >> Gen3, as the NF200 PCI Bridge broke all but one PCI Express slot from being IOMMU >> compatible. Knowing which controllers on the board map to which devices on the >> otherhand is not mentioned in the board manual, so for example the 5 USB controllers on >> my current board I had to figure out their mappings manually, and one is shared with the >> BlueTooth, so that''s another hardware concern.This is very interesting. Could you please explain this a little more. From what I understand we need to map the PCI bridge back to possible peripherals (usb, ethernet etc..), that can be sharing resources with them? Is this also an issue with a board with say 2 PCIe slots vs. many? Kind Regards, Nick.
On Tue, 4 Jun 2013 08:24:53 -0400, Nick Khamis <symack@gmail.com> wrote:> There was no way I was going to pass on this conversation, even with > my busy schedule. > >>> This means that if you only have a single Video Card, you >>> potentially will lose >>> video output for anything but that VM. Best setup seems to be an >>> integrated Video Card >>> like the one that most Processors currently got to handle the >>> Hypervisor + other VMs with >>> the standard emulated VGA Drivers plus a discrete Video Card for >>> the gaming VM. > > My understanding was that this relied on the number of Virtual > Functions a PCI device was equipped with in the firmware. This at > least is the case for network cards...Is it? I cannot say I tested PCI passthrough with all of my network cards, but if special support for this is required from the NIC firmware, I''d have expected at least some to not work, especially the low end ones. Yet this doesn''t seem to have been the case. As far as I can tell, driver quality, especially for Windows domUs, seems to be a more significant issue.> Not to chase tails here however, can we step back and figure out > which > of the chipset manufactures (AMD vs. Intel) provides a stable > platform > that can be used in production. We are not necessarily interested in > GPUs but we are interested in passing through network cards QLogic, > Intel etc... I would imagine this would still be important to the > gamers, and Justin.tv broadcasters as well.... > > Secondly, are any of the manufactures such as IBM and DELL openly > pushing stable virtualized platforms (chipsets, pci bus, cpus, bios), > that we can again run in a production. And of course, there should be > XEN compatibility on top of all this...As far as I can tell, the issue is not all that clear cut. Hardware and firmware these days are almsot as buggy as software, and two motherboards with near identical spec from different manufacturers may well produce different success rates.> We can understand why the chipset, cpu, and even pci hardware > manufactures would play this cat and mouse game with virtualiztion > since to them it equates to less sales......I am not sure that that follows as clearly. Take Nvidia for example. For bare metal use you can use a "cheap" GeForce card. If you want to use it with VGA passthrough, you have to shell out 3-4x the amount for a Quadro card that is by and large identical but for a few resistors and half a byte of firmware. On the flipside, they do seem to do a pretty decent job of making sure that those that pay 4x the amount for their hardware do in fact end up with a decent, stable solution.>>> Casey DeLorme >>> From my experience if VT-d or IOMMU are not explicitly mentioned in >>> the user manuals >>> (available for download off the net before you spend a dime on the >>> board) then it likely >>> does not have support for it. > > Interesting... We do something similar when purchasing IBMs. We look > to see if there are BIOS firmware updates that involve virtualization > such as this: > > > http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086623 > > > I came in a little late in the game for this conversation however, > can > we please iron out some issues here. At an abstract level (i.e., > chipsets, cpus, gpu, network interfaces), without mentioning any > motherboard manufactures such as ASRock, Asus, Saphire etc.. can we > determine which combination will work. Both on the AMD and Intel > platform. The reason for this is because not too many people deploy > white boxes for production. it''s strictly SuperMicro, IBM, Dell > etc...And now you know why you pay a huge premium for "certified" or "approved" hardware - somebody has spent a lot of man-hours and money testing the hardware/software combinations to ensure that what they end up selling actually works. Unless your time is close to worthless or your expected deployments are at least in the hundreds, it is infeasible to invest the required amount in testing to come up with your own white-box solution. Don''t get me wrong - I spend such an infeasible amount of time and effort on various projects all the time, but I do this because the involved technologies interest me, or because I want a solution that nobody has come up with exactly in the way I want it to work, not because it would ever be viable in the context of any commercial systems integration. This is also why off the shelf solutions tend to only focus on a small subset of the problem space - the simple stuff is what most people want and any more than that starts to get prohibitively expensive, especially when you get to the point of effectively debugging your hardware manufacturers firmware without any access to it''s source code.> Once determining this, it would be nice to discuss which of the > manufactures are able to support stable platforms without rendering 3 > out of the 4 PCIe/x slots useless. This would hurt!The only suggestion I can come up with is a wiki where those interested in contributing can post success/failure stories surrounding specific motherboards (I will be happy to contribute any recipes I come up with if/when I manage to get my EVGA SR-2 to work with my hardware the way I want to use it).> We also need to keep it fairly modern. For example, PCIe slots would > preferably be PCIe v2 x8/x16 PCIe v3 x8/x16. For example, I mentioned > IBM enabled virtualization server however, what I am really looking > at > are the guts. For example: > > Intel Based (x3550 M4) > CPU: E5-2600 > Chipset: Intel C604 > > All PCIe slots are PCIe 3.0 > Slot 1: PCIe x16; low profile, half-length > Slot 2: PCIe x8, opt. PCI-X or PCIe x16; full-height/half-length > (PCIe > x16 req. 2nd CPU) > > I am actually not sure if this platform support virtulaization since > the little bit of googling did not yield much. > > AMD Based: x3755 M3 > CPU: AMD Opteron 6200 > Chipset: SR5690 > > All PCIe slots are PCIe 2.0 > > Slot 1: PCIe x16, full height, full length > Slot 2: PCIe x8, low profile, half length > Slot 3: PCIe x8 (x4 wired), low profile, half length > Slot 4: PCIe x8, low profile, half length (internal only, reserved > for RAID contOh - so you actually want manufacturers to come up with hardware that they pre-certify for virtualization use? Nice idea if it happens. Gordan
On Tue, 4 Jun 2013 08:32:42 -0400, Nick Khamis <symack@gmail.com> wrote:>>> Besides BIOS options another huge problem I encountered was PCI >>> Bridges on boards with >>> lots of PCI Express slots, which are an extra layer and not always >>> compatible with >>> IOMMU. This caused me a lot of headaches when I bought an ASRock >>> Z68 Extreme7 >>> Gen3, as the NF200 PCI Bridge broke all but one PCI Express slot >>> from being IOMMU >>> compatible.NF200 doesn''t support ACS, but xen doesn''t treat ACS as mandatory. It is disablable In xend-config.sxp, set: (pci-passthrough-strict-check no) (pci-dev-assign-strict-check no) I am not sure what the correct way to set this for xl is, but xl works fine for me on my EVGA SR-2 on which ALL 7 PCIe slots are behind NF200 bridges, so mandatory ACS requirement must be disabled by default or something.>>> Knowing which controllers on the board map to which devices on the >>> otherhand is not mentioned in the board manual, so for example the >>> 5 USB controllers on >>> my current board I had to figure out their mappings manually, and >>> one is shared with the >>> BlueTooth, so that''s another hardware concern.Nothing that cannot be worked out in 5 minutes, though. I did just this exact thing yesterday with the 12 USB ports on my SR2.> This is very interesting. Could you please explain this a little > more. > From what I understand we need to map the PCI bridge back to possible > peripherals (usb, ethernet etc..), that can be sharing resources with > them? Is this also an issue with a board with say 2 PCIe slots vs. > many?"lspci -t" is usually a good start. Gordan
>> My understanding was that this relied on the number of Virtual >> Functions a PCI device was equipped with in the firmware. This at >> least is the case for network cards...According to this: http://wiki.xen.org/wiki/Xen_VGA_Passthrough#Why_is_VGA_passthrough_different_from_normal_PCI_passthrough.3F Xen uses a different type of passthrough with Video Cards that it does with Network Cards. That should be why you can''t do that.>> Not to chase tails here however, can we step back and figure out which >> of the chipset manufactures (AMD vs. Intel) provides a stable platform >> that can be used in production. We are not necessarily interested in >> GPUs but we are interested in passing through network cards QLogic, >> Intel etc... I would imagine this would still be important to the >> gamers, and Justin.tv broadcasters as well....I have made another Thread with that intention, here: http://lists.xen.org/archives/html/xen-users/2013-06/msg00010.html>> We can understand why the chipset, cpu, and even pci hardware >> manufactures would play this cat and mouse game with virtualiztion >> since to them it equates to less sales......Indeed. Manufacturers doesn''t want to let consumers get all the useful Server features that they sell at a premium on your typical Desktop computer. Its not that they will lose sales, but their sales will have a much lower profit margin. There are many enthusiasts users that would happily purchase a cheap Processor and overclock it instead of paying the full price of what a Processor with that nominal Frequency got (Some Server guys may argue with the reliability issue of running out-of-spec, but we''re assuming that the guy knows what he is doing and can stress test it to guarantee rock solid stability), and on the professional lines like Intel Xeon, AMD Opteron, nVidia Quadro and AMD FirePro, for many parts they''re selling you the same silicon that for the consumer market but with a much higher price. Just check around for the price on nVidia GRID K2 that is supposed to be used for cloud gaming. And from a Hardware perspective, it just looks like a GeForce 690.>> >> Casey DeLorme >> >> From my experience if VT-d or IOMMU are not explicitly mentioned in the user >> >> manuals >> (available for download off the net before you spend a dime on >> >> the board) then it likely >> >> does not have support for it. >> >> Interesting... We do something similar when purchasing IBMs. We look >> to see if there are BIOS firmware updates that involve virtualization >> such as this: >> >> http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086623I don''t agree with just checking the manual for a single reason: That doesn''t guarantees that it will work. Some people says that they have an option on the BIOS to enable VT-d/AMD-Vi, but the support is buggy or badly implemented. A BIOS upgrade can break support or fix it, so sometimes you have to downgrade, or expect your Motherboard manufacturer to be interesed in fixing it. Not all of them do. This means that your safest bet it to get someone with the Motherboard you want and ask him if he got it working, and in what BIOS version.>> I came in a little late in the game for this conversation however, can >> we please iron out some issues here. At an abstract level (i.e., >> chipsets, cpus, gpu, network interfaces), without mentioning any >> motherboard manufactures such as ASRock, Asus, Saphire etc.. can we >> determine which combination will work. Both on the AMD and Intel >> platform. The reason for this is because not too many people deploy >> white boxes for production. it''s strictly SuperMicro, IBM, Dell etc...I already did a recollection of possible supporting Hardware, that needs to be confirmed or discarded: AMD Socket AM3: Chipsets AMD 890FX and 990FX has official IOMMU support build in in the Chipset itself. Of interesing note, is that albeit there seems to be other people that got the other 9xx series Chipsets working with AMD-Vi (On Xen wiki 970 and 990X are included, but not the 980G), AMD says on a Tech Doc that only the 990FX Chipset got support for it (Page 9, 1.1): http://support.amd.com/us/ChipsetMotherboard_TechDocs/48691.pdf Additionally, I heared that Bulldozer based Processors (Including AMD FX series Zambezi and Vishera, APUs Trinity and Richland) have another IOMMU built in. Considering this, you could potentially have two IOMMUs on Socket AM3+ if you have a Bulldozer based Processor with one of the previous two Chipsets. I have not confirmed this through. So the following combinations are possible: K10 based Processor on 890FX or 990FX Chipset *MUST WORK* K10 based Processor on 970, 980G or 990X *SHOULD NOT WORK* Bulldozer based Processor on ANY Chipset *SHOULD WORK* Bulldozer based Processor on 890FX or 990FX Chipset *MUST WORK*... just what IOMMU it uses? AMD Socket FM1: There should be NO support on this platform. Llano, being K10 based, doesn''t have a build in IOMMU, and the Chipsets doesn''t have it, either. AMD Socket FM2: As every Processor here is Bulldozer derived, you should have IOMMU support in all them. Besides the newer A85X, the other Chipsets are the same that on FM1 platform. ASRock released two beta BIOSes that claims to include IOMMU support on at least two Motherboard models that includes A55 (FM2A55 Pro) and A75 (FM2A75M-DGS) Chipsets: http://www.asrock.com/mb/overview.asp?cat=Download&os=Beta&Model=FM2A75M-DGS http://www.asrock.com/mb/AMD/FM2A55%20Pro/?cat=Beta So I should suppose that information was correct. Intel platforms are a bit more complicated. Intel usually likes to sell you features in a Processor/Chipset combo, so you usually need support from both things or get that feature artifficially crippled. If you have a K series Processor and want to overclock the CPU component, you need a P or Z series Chipset, you can''t do it on a B, H or Q. I don''t know if VT-d recibes similar treatment, but at least for vPro you DO need a Q series Chipset. With just one LGA 2011 exception, Intel disabled VT-d, TXT, vPro, and on Haswell, the newly introduced TSX on ALL K series Processors. Seems that they don''t want overclockers virtualizing. I didn''t hear anyone claiming that you need a specific Chipset for VT-d support, until I asked yesterday a question related to this to a guy that work on ASUS, that claims that most VT-d features are getting moved to Q series Chipsets only: http://www.xtremesystems.org/forums/showthread.php?286345-ASUS-Z87-Motherboards-Overview-Guides-and-Official-Support&p=5191428&viewfull=1#post5191428 I don''t know if that info is true or not, as there are at least some guys with Z series Chipsets that can use VT-d. However, ASUS Motherboards do have a overally bad reputation regarding VT-d/AMD-Vi support, so I can''t say if it just a mere excuse to justify that. I need to gather a few links and facts to reply to him about that. ASRock has been steadily adding IOMMU support to many Motherboards, and I heared that Gigabyte also sends custom BIOS to those people that requested support for it.>> Once determining this, it would be nice to discuss which of the >> manufactures are able to support stable platforms without rendering 3 >> out of the 4 PCIe/x slots useless. This would hurt!This is one of the main issues. On a low level, you usually don''t know the exact layout of a Motherboard and what is connected where. For what I readed, you can get all the stuff from the CPU and Chipset working, the usual things that are not compatible are bridges (The nVidia chip for adding PCIe lanes that was already mentioned), external controllers like the extra SATA Controllers that high end Motherboards have to provide you with more SATA connectors, and the like. Considering this, a good quality mainstream Motherboard should be a safer bet that a high end Motherboard full of additional components and chips that you may not have success trying to make them to work.>> Buying hardware just got a lot more trickier folks, let''s hope this >> thread can shed some light on our lost souls, and make us complete >> again....With both Intel Haswell and AMD Richland have just been released, it may be worth to do this research on the new platforms for those that want to be early adopters. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Andrew Bobulsky
2013-Jun-08 07:39 UTC
Re: AMD-Vi/Intel VT-d: Passthrough or virtualization?
On Tue, Jun 4, 2013 at 2:52 PM, Zir Blazer <zir_blazer@hotmail.com> wrote:>>> My understanding was that this relied on the number of Virtual >>> Functions a PCI device was equipped with in the firmware. This at >>> least is the case for network cards... > > According to this: > > http://wiki.xen.org/wiki/Xen_VGA_Passthrough#Why_is_VGA_passthrough_different_from_normal_PCI_passthrough.3F > > Xen uses a different type of passthrough with Video Cards that it does with > Network Cards. That should be why you can''t do that. > > > > >>> Not to chase tails here however, can we step back and figure out which >>> of the chipset manufactures (AMD vs. Intel) provides a stable platform >>> that can be used in production. We are not necessarily interested in >>> GPUs but we are interested in passing through network cards QLogic, >>> Intel etc... I would imagine this would still be important to the >>> gamers, and Justin.tv broadcasters as well.... > > I have made another Thread with that intention, here: > > http://lists.xen.org/archives/html/xen-users/2013-06/msg00010.html > > > > >>> We can understand why the chipset, cpu, and even pci hardware >>> manufactures would play this cat and mouse game with virtualiztion >>> since to them it equates to less sales...... > > Indeed. Manufacturers doesn''t want to let consumers get all the useful > Server features that they sell at a premium on your typical Desktop > computer. Its not that they will lose sales, but their sales will have a > much lower profit margin. There are many enthusiasts users that would > happily purchase a cheap Processor and overclock it instead of paying the > full price of what a Processor with that nominal Frequency got (Some Server > guys may argue with the reliability issue of running out-of-spec, but we''re > assuming that the guy knows what he is doing and can stress test it to > guarantee rock solid stability), and on the professional lines like Intel > Xeon, AMD Opteron, nVidia Quadro and AMD FirePro, for many parts they''re > selling you the same silicon that for the consumer market but with a much > higher price. > > Just check around for the price on nVidia GRID K2 that is supposed to be > used for cloud gaming. And from a Hardware perspective, it just looks like a > GeForce 690. > > > > >>> >> Casey DeLorme >>> >> From my experience if VT-d or IOMMU are not explicitly mentioned in >>> >> the user >>> >> manuals >> (available for download off the net before you spend a dime >>> >> on >>> >> the board) then it likely >>> >> does not have support for it. >>> >>> Interesting... We do something similar when purchasing IBMs. We look >>> to see if there are BIOS firmware updates that involve virtualization >>> such as this: >>> >>> >>> http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086623 > > I don''t agree with just checking the manual for a single reason: That > doesn''t guarantees that it will work. Some people says that they have an > option on the BIOS to enable VT-d/AMD-Vi, but the support is buggy or badly > implemented. A BIOS upgrade can break support or fix it, so sometimes you > have to downgrade, or expect your Motherboard manufacturer to be interesed > in fixing it. Not all of them do. This means that your safest bet it to get > someone with the Motherboard you want and ask him if he got it working, and > in what BIOS version. > > > > >>> I came in a little late in the game for this conversation however, can >>> we please iron out some issues here. At an abstract level (i.e., >>> chipsets, cpus, gpu, network interfaces), without mentioning any >>> motherboard manufactures such as ASRock, Asus, Saphire etc.. can we >>> determine which combination will work. Both on the AMD and Intel >>> platform. The reason for this is because not too many people deploy >>> white boxes for production. it''s strictly SuperMicro, IBM, Dell etc... > > I already did a recollection of possible supporting Hardware, that needs to > be confirmed or discarded: > > > AMD Socket AM3: Chipsets AMD 890FX and 990FX has official IOMMU support > build in in the Chipset itself. Of interesing note, is that albeit there > seems to be other people that got the other 9xx series Chipsets working with > AMD-Vi (On Xen wiki 970 and 990X are included, but not the 980G), AMD says > on a Tech Doc that only the 990FX Chipset got support for it (Page 9, 1.1): > > http://support.amd.com/us/ChipsetMotherboard_TechDocs/48691.pdf > > Additionally, I heared that Bulldozer based Processors (Including AMD FX > series Zambezi and Vishera, APUs Trinity and Richland) have another IOMMU > built in. Considering this, you could potentially have two IOMMUs on Socket > AM3+ if you have a Bulldozer based Processor with one of the previous two > Chipsets. I have not confirmed this through. So the following combinations > are possible: > > K10 based Processor on 890FX or 990FX Chipset *MUST WORK* > K10 based Processor on 970, 980G or 990X *SHOULD NOT WORK* > Bulldozer based Processor on ANY Chipset *SHOULD WORK* > Bulldozer based Processor on 890FX or 990FX Chipset *MUST WORK*... just what > IOMMU it uses? > > > AMD Socket FM1: There should be NO support on this platform. Llano, being > K10 based, doesn''t have a build in IOMMU, and the Chipsets doesn''t have it, > either. > > AMD Socket FM2: As every Processor here is Bulldozer derived, you should > have IOMMU support in all them. Besides the newer A85X, the other Chipsets > are the same that on FM1 platform. ASRock released two beta BIOSes that > claims to include IOMMU support on at least two Motherboard models that > includes A55 (FM2A55 Pro) and A75 (FM2A75M-DGS) Chipsets: > > http://www.asrock.com/mb/overview.asp?cat=Download&os=Beta&Model=FM2A75M-DGS > http://www.asrock.com/mb/AMD/FM2A55%20Pro/?cat=Beta > > So I should suppose that information was correct. > > > Intel platforms are a bit more complicated. Intel usually likes to sell you > features in a Processor/Chipset combo, so you usually need support from both > things or get that feature artifficially crippled. If you have a K series > Processor and want to overclock the CPU component, you need a P or Z series > Chipset, you can''t do it on a B, H or Q. I don''t know if VT-d recibes > similar treatment, but at least for vPro you DO need a Q series Chipset. > With just one LGA 2011 exception, Intel disabled VT-d, TXT, vPro, and on > Haswell, the newly introduced TSX on ALL K series Processors. Seems that > they don''t want overclockers virtualizing. > I didn''t hear anyone claiming that you need a specific Chipset for VT-d > support, until I asked yesterday a question related to this to a guy that > work on ASUS, that claims that most VT-d features are getting moved to Q > series Chipsets only: > > http://www.xtremesystems.org/forums/showthread.php?286345-ASUS-Z87-Motherboards-Overview-Guides-and-Official-Support&p=5191428&viewfull=1#post5191428 >(snippity-snip!)I apologize that I don''t have links for you, but VT-d and the K-series chips was brought up specifically in a Reddit AMA with an Intel architect a few months ago. He wrote something along the lines of:>VT-d was planned for support in the K-series chips, but late testing of the features on >the pre-production runs showed that they were failing Intel''s feature tests, and VT-d >support was disabled on them as an unfortunate last-minute move.I got a little less upset about not being able to buy a 2600K, and decided to skip the generation of chips altogether. But reading that VT-d support will disappear entirely is hopefully too disappointing to be true.... :( Regards, Andrew Bobulsky
>> I got a little less upset about not being able to buy a 2600K, and >> decided to skip the generation of chips altogether. But reading that >> VT-d support will disappear entirely is hopefully too disappointing to >> be true.... :(Wait, what?
Forgot to send this to the Xen Mailing List two days ago. Was wondering why it didn''t got posted...> I apologize that I don''t have links for you, but VT-d and the K-series > chips was brought up specifically in a Reddit AMA with an Intel > architect a few months ago. He wrote something along the lines of: > > >VT-d was planned for support in the K-series chips, but late testing of the features on >the pre-production runs showed that they were failing Intel''s feature tests, and VT-d >support was disabled on them as an unfortunate last-minute move. > > I got a little less upset about not being able to buy a 2600K, and > decided to skip the generation of chips altogether. But reading that > VT-d support will disappear entirely is hopefully too disappointing to > be true.... :( > > Regards, > Andrew BobulskyThis Thread? http://www.reddit.com/r/IAmA/comments/15iaet/iama_cpu_architect_and_designer_at_intel_ama/c7mqoh5 I had to click tons of times to load the 2xxx comments to be able to search the full Thread. The only time he replied to a VT-d comment was there. The question was: 2) why do the K edition chips not support VT-d, all off the non K chips support it, but the K do not? as some one who likes to overclock and test server OS builds this seams like it may be a problem eventually. The response was: 2) I had not noticed this. I don''t work on the SKUing, but I agree with your reasoning. I''ll make a case for it thanks for bringing it to my attention. So if it was THAT Thread, you understanded something totally different to what he said. Where Intel did disabled VT-d support due to a bug was LGA 2011 Sandy Bridge-E Processors. The C1 Stepping didn''t had VT-d, but it was enabled after it got fixed on the C2 Stepping. And it didn''t affected LGA 1155 Processors. http://www.techpowerup.com/152978/sandy-bridge-e-vt-d-broken-in-c1-stepping-fixed-in-c2-stepping-shortly-after-launch.html For as long as I can remember, Intel likes to play enabling/disabling features on certain models just to sell them for more profit on more expensive lines. Do you want ECC Memory support on a Uniprocessor computer? Then you need pretty much a Xeon E3, that while physically is the same die that some of the Desktop counterparts, they don''t disable on purpose that feature on them. With VT-d is the same. And most non-K Haswell models launched a week or so ago do claim VT-d support. So your info seems to be totally wrong. Also, I did a follow up question to the ASUS guy regarding VT-d support on their Motherboards, here: http://www.xtremesystems.org/forums/showthread.php?286345-ASUS-Z87-Motherboards-Overview-Guides-and-Official-Support&p=5191955&viewfull=1#post5191955 His response in the next Post was this: Contact your local support rep and see what they say. The info I gave you pertains to how it has been from HQ when I have asked for the past two gens. All 50 tests of vt-D would only work on Q series boards and not Z series. Basically, he says that ASUS throwed a 50 test battery for VT-d on non-Q series Chipsets for Sandy Bridge and Ivy Bridge, and they don''t pass some of these test. Without hard data of what test fails and what specific feature they use, I can''t come up with a conclusion, as on Motherboards from other manufacturers things seems to work fine. I suppose a Xen developer would want to have a word with ASUS to get hard data about this issue. However, I could believe that it can be an excuse just because the ASUS BIOS developers are extremely lazy to make things work as they should, as ASUS got an extremely bad reputation when it comes to comply with standards. An example here: http://www.phoronix.com/scan.php?page=news_item&px=OTk4NQ So not only their track record on VT-d support is lame compared to other manufacturers, on other features that the BIOS should announce, they suck too. I would want to lobby to get a few hands working on a tool like the one I said here: http://lists.xen.org/archives/html/xen-users/2013-06/msg00010.html Even with my null Linux and Xen experience, I would still say that I would value a tool that says you what work and what doesn''t, before you try it on the real thing. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users