I''m looking to build a new personal computer. I want it to function as a Linux desktop, provide network services for my home, and lastly, occasional Windows gaming. From what I''ve gathered, virtualization using a Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or Xen would be an attractive solution for my needs. For reference, reading these threads on Ars Technica may be helpful to understand where I''m coming from, http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But basically, I use Linux as my primary OS and would rather avoid dual booting or building two boxes just to play Windows games when I want to play Windows games. I''m also intrigued by the concept of virtualization and would like to experiment with it as a solution for my case. My problem is isolating which hardware to choose, specifically which combination of CPU, motherboard and video card. Previously I had been relying on web searches to glean information from gaming and enthusiast web sites and tech specs from motherboard manufacturers. After what I learned during my participation in the referenced threads at Ars Technica, I find myself back at square one. Instead of trying to guess what hardware support KVM & Xen, and vice versa. I''d like to know what hardware KVM & Xen users are actually using to run KVM & Xen? Particularly with consideration for 3D gaming and current generation hardware, BTW. If there is need for further clarification, I''ll answer any queries you might have.
Hi, Just a Xen newbie myself, but from what i've gathered and fiddled, Xen (P)VMs don't come with a graphics card. You'd have to remote to your Windows HVM to play games. You can fiddle with PCI pass through for some video cards and there's some VGA passtrough as well, but i don't think running a recent game on a VM would provide good results (i'm thinking FPSs here, not solitaire or simcity). As for hardware i considered an ATI/Asus board with a Phenom II X6 and 16GB or DDR3 a while ago, plus box, PSU, no disks, around 500€... YMMV and your needs as well. Just my ill-informed 2c. HTH Nuno _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Thu, 2012-09-20 at 17:12 -0400, ShadesOfGrey wrote:> I''m looking to build a new personal computer. I want it to function as > a Linux desktop, provide network services for my home, and lastly, > occasional Windows gaming. From what I''ve gathered, virtualization > using a Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or > Xen would be an attractive solution for my needs. For reference, > reading these threads on Ars Technica may be helpful to understand where > I''m coming from, > http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and > http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But > basically, I use Linux as my primary OS and would rather avoid dual > booting or building two boxes just to play Windows games when I want to > play Windows games. I''m also intrigued by the concept of virtualization > and would like to experiment with it as a solution for my case. > > My problem is isolating which hardware to choose, specifically which > combination of CPU, motherboard and video card. Previously I had been > relying on web searches to glean information from gaming and enthusiast > web sites and tech specs from motherboard manufacturers. After what I > learned during my participation in the referenced threads at Ars > Technica, I find myself back at square one. Instead of trying to guess > what hardware support KVM & Xen, and vice versa. I''d like to know what > hardware KVM & Xen users are actually using to run KVM & Xen? > Particularly with consideration for 3D gaming and current generation > hardware, BTW. > > If there is need for further clarification, I''ll answer any queries you > might have.There have been a couple success reports of using AMD/ATI graphics cards on Intel VT-d systems with KVM device assignment. Nvidia cards have not met with the same degree of success. In both our cases, the graphics device was assigned to a Windows guest as a secondary graphics head. For me, Windows took over the assigned device as the primary graphics, disabling the emulated graphics. For my slow HD 5450, 3dMark seems to get a similar score to what others see on bare metal. Graphics device assignment on KVM is getting better, but should be considered experimental at best. Thanks, Alex
If you want to be able to use PCI and VGA passthrough you basically need to make sure that your hardware supports either AMD-Vi (formerly known as AMD-IOMMU) or Intel VT-d extensions. In the Intel case it limits your choice of Motherboard (it must be supported in the BIOS) and CPU. In the AMD case it limits only your choice of motherboard. A good start is to check out one of these pages: http://wiki.xensource.com/xenwiki/VTdHowTo http://wiki.xen.org/wiki/VTd_HowTo A word of warning here is that parts of the documentation is somewhat dated. You can also communicate with e.g. Gigabyte, Asus or ASRock customer support and ask them if a particular motherboard supports these extensions. Most motherboards also have downloadable user manuals, if the BIOS settings in those pages shows options to enable/disable VT-d or AMD-Vi/IOMMU extensions then you will be ok with that motherboard. The other thing is choice of GPU for VGA passthrough and it is preferable that the GPU supports FLR or Function Level Reset as it is called. Thing is that the hardware needs to be reset somehow as it is passed through to the host. This is best done with FLR and nVidia is known to supply firmware patches for some of their Geforce cards with this support and it is said to be supported by default with their Quadro cards. FLR is not the only way to reset a PCI device, a reset could be trigged through the ACPI power management framework by temporarily cutting power to the affected PCI slot. These reset methods are called d3d0 and bus reset. The question however, is if this works on PCI cards that use auxiliary power directly from the PSU. There is a pdf document on the VMWare website (http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf) about this: ----------------------- Reset Method Possible values for the reset method include flr, d3d0, link, bridge, or default. The default setting is described as follows. If a device supports function level reset (FLR), ESX always uses FLR. If the device does not support FLR, ESX next defaults to link reset and bus reset in that order. Link reset and bus reset might prevent some devices from being assigned to different virtual machines, or from being assigned between the VMkernel and virtual machines. In the absence of FLR, it is possible to use PCI Power Management capability (D3 to D0 transitions) to trigger a reset. Most of the Intel NICs and various other HBAs support this mode. ----------------------- There are indications from people that d3d0 also work with PCI cards that take power from auxiliary inputs. I suggest that you take a look at the following youtube clip and read the comments there: http://www.youtube.com/watch?v=Gtmwnx-k2qg So it seems that it works although it may be a bit more quirky. It doesn''t hurt to take that discussion (particularly about FLR support) with nVidia and/or AMD. When it comes to virtualization, the technology has come very far, but it is still lacking considerably when it comes to sharing GPUs and also to some degree when it comes to sharing I/O devices (especially when you intend to run many virtual machines on a single system). The GPU today consists of three types of components; the processing unit, graphics memory and the video output unit/adapter and it is not clear as to how to share these components seamlessly between the host and virtual machines with minimal overhead. Whereas there are VT-x extensions that allows you to pretty seamlessly share CPU cores between VMs and the host there are currently none for the processing unit. It is also not clear how the hardware can assist with sharing TV/monitor screen estate between machines with all 3D effects such as Aqua for Win7 and the whatnot enabled for all machines. Especially when considering the dynamics of plugging and unplugging computer monitors to multiport/eyefinity graphics cards and the ability to change screen resolution. Things are improving for sure and a lot of research is likely going into this. I don''t know what''s happening in the GPU frontline but I know that the next thing with passthrough is the SR-IOV that allows PCI units to present several virtual instances of oneself to several virtual machines. It''s a cool thing, I recommend further reading about this here: http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/ It is likely to take a few years before something useful will come out of it. In the meanwhile, unless you want to use several GPUs which might not be a bad thing as a lot of monitors these days have several inputs, you can resort to using a remote desktop client to integrate one machine with another. Virtualbox for example use RDP through which you can interact with your virtual machine. In a similar manner you can set up a VNC server on your Linux host and establish a connection to it through your Windows VM. You will not get full 3D functionality (such as Aqua) through the client although there is a growing support for it through VirtualGL extensions that are coming to VNC and perhaps the Spice protocol. But some clients might even allow for seamless mode that lets you mix Linux and Windows windows on the same desktop like this for example: http://i.techrepublic.com.com/blogs/seamless.png http://www.youtube.com/watch?v=eQr8iI0yZH4 Just keep in mind that this is still a little bit of uncharted territory so there may be a few bumps on the way and it may not work as smooth as you would desire. I see that your demands are somewhat multifaceted. I believe that you also want to use diffent services such as using your machine as a file server with the possible intention of using filesystems such as ZFS. If you do, you should be careful with your selection of hardware for these particular purposes. If you want to get full protection against data corruption from ZFS, your choice of hardware gets rather limited when it comes to choice of hard drives, host bus adapter and network controller. The most stable implementation of ZFS is found with Illumos based operating systems (such as OpenIndiana, SmartOS, OmniOS, Belenix etc) or Solaris if you choose to download it from Oracle''s website. With these operating systems you are most likely to want to use hardware that has certified drivers for it. That way you are less likely to run into problems later on. That implies that you will be limited to choosing Intel based network adapters and LSI based SAS controllers. There should be _no_ hardware RAID functionality in the SAS controller that merely should be run in IT mode (or Initiator-Target mode). That requires the LSI controller to be flashed with IT firmware in most cases. The objective here is to make sure that _all_ errors that might occur with the hard drives are reported all the way to the software level and that nothing is concealed of obfuscated by internal error handling in the hardware. It is therefore recommended to use SAS hard drives instead of S-ATA (which also are fully compatible with SAS controllers). SAS hard drives are not much more expensive than similar SATA drives and you get a higher reliability out of them. It is also recommended to have at least two drive redundancy simply because if one drive is dead and you swap it, it is not uncommon that another drive dies in the rebuild process of the RAID cluster because of the added strain the rebuild process (or ''resilvering'' as it is called in Solaris terms) put on the drives. Of course, the system should communicate directly to the hard drive hardware and not be obfuscated by some virtual abstraction layer in between which means that you either run ZFS on the metal or through PCI passthrough of the SAS (and perhaps also network) adapters. Also, it is highly recommended that you use ECC RAM for such applications and it doesn''t hurt to dedicate a few gigs of it to the ZFS as RAM is used for cache. The good news is that most motherboards with good chipsets support ECC RAM even though you might not find anything about it in the user manuals. I admire your persistence with pursuing this undertaking and wish you the best of luck with it! Robin. On 2012-09-20 23:12, ShadesOfGrey wrote:> I''m looking to build a new personal computer. I want it to function > as a Linux desktop, provide network services for my home, and lastly, > occasional Windows gaming. From what I''ve gathered, virtualization > using a Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or > Xen would be an attractive solution for my needs. For reference, > reading these threads on Ars Technica may be helpful to understand > where I''m coming from, > http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and > http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But > basically, I use Linux as my primary OS and would rather avoid dual > booting or building two boxes just to play Windows games when I want > to play Windows games. I''m also intrigued by the concept of > virtualization and would like to experiment with it as a solution for > my case. > > My problem is isolating which hardware to choose, specifically which > combination of CPU, motherboard and video card. Previously I had been > relying on web searches to glean information from gaming and > enthusiast web sites and tech specs from motherboard manufacturers. > After what I learned during my participation in the referenced threads > at Ars Technica, I find myself back at square one. Instead of trying > to guess what hardware support KVM & Xen, and vice versa. I''d like to > know what hardware KVM & Xen users are actually using to run KVM & > Xen? Particularly with consideration for 3D gaming and current > generation hardware, BTW. > > If there is need for further clarification, I''ll answer any queries > you might have. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users > >
(CC''ing Casey on this, as I recommend his setup for an Intel-based solution) Hello ShadesOfGrey, Hehehe, talk about timing ;) On Sep 20, 2012, at 5:15 PM, ShadesOfGrey <shades_of_grey@earthlink.net> wrote:> I''m looking to build a new personal computer. I want it to function as a Linux desktop, provide network services for my home, and lastly, occasional Windows gaming. From what I''ve gathered, virtualization using a Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or Xen would be an attractive solution for my needs. For reference, reading these threads on Ars Technica may be helpful to understand where I''m coming from, http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But basically, I use Linux as my primary OS and would rather avoid dual booting or building two boxes just to play Windows games when I want to play Windows games. I''m also intrigued by the concept of virtualization and would like to experimentwith it as a solution for my case.> > My problem is isolating which hardware to choose, specifically which combination of CPU, motherboard and video card. Previously I had been relying on web searches to glean information from gaming and enthusiast web sites and tech specs from motherboard manufacturers. After what I learned during my participation in the referenced threads at Ars Technica, I find myself back at square one. Instead of trying to guess what hardware support KVM & Xen, and vice versa. I''d like to know what hardware KVM & Xen users are actually using to run KVM & Xen? Particularly with consideration for 3D gaming and current generation hardware, BTW. > > If there is need for further clarification, I''ll answer any queries you might have. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-usersAs one of the folks on the list who has done this---probably to the most extreme degree---I can tell you it''s good stuff. It brings the joys of datacenter consolidation to your gaming desktop, and also to your wallet ;) While my setup is now slightly dated, the 990FX chipset is still at the top of the AMD offering, so you can shop around on CPUs, and buy a cheaper "secondary" USB controller if you''re not looking to cram in a 4-to-1 ratio. I''ve never had much success passing through the onboard USB from an AMD system, so I highly recommend picking up a little PCIe x1 controller at the least. That said, I''m convinced that highpoint has one of the coincidentally-best products on the market for people looking to do this, but I digress! Take a look at, specifically, this post I made to the list some months back, and I''ll follow with some errata: http://lists.xen.org/archives/html/xen-users/2012-05/msg00328.html First, I''ve tested all of the hardware in the build that I recommended, and indeed ended up building a four-headed unit. It works like magic. Came in handy a few weeks ago when several of my friends and I piled into a couple cars for a vacation where we wanted to play games (yup, we''re total nerds), but we couldn''t fit four desktop cases in addition to our stuff in the cars. :) Second, by the time I got around to building it, the Antec One Hundred wasn''t available. Finding a case that supports 8 expansion slots is a tough thing, but I found another similarly priced one, and it was a dream to build. I recommend it highly if you think you may want to max out your slots and/or go deeper down the rabbit hole with consolidated desktops: http://www.newegg.com/Product/Product.aspx?Item=N82E16811112238 Aside from being a very solid case for the price point (good features for screwless installation as well), to give you an idea of the size, it is laid out in such a way that I could fit dual-GPU cards in it (Radeon 5970s). I ultimately had to remove the HDD mounts to pull it off, but you shouldn''t have that problem... Mostly because AMDs dual GPU cards won''t work for this, so don''t buy one for this build. It''s a problem with the PCIe switching hardware (well, the firmware thereof, probably) that they use. I''ll save you the rambling, but let''s just say that it should work, but doesn''t :( Also, the case does look good! ;) Finally, and this is unfortunate, for the AMD build, *I* recommend you use ESXi. While Xen _does_ work with the hardware that I''ve listed, I''ve never been able to get the VMs to work properly with the GPLPV drivers, and these are crucial to performance. I really, really want to bring this project back up on Xen though, and will try again now that 4.2 has gone RTM. If you aren''t buying anytime soon and would like to hit me up in a few weeks, by all means drop me a line, and I''ll let you know if I''ve gotten around to it. ---------------- So, for the Intel route! Casey DeLorme has, just this week, posted a fantastic set of detailed videos and documentation on his setup, where he basically does exactly what you''re trying to accomplish. You can find links to all of the documentation, which I''m pretty sure covers his exact hardware, along with videos of the installation process he used and a detailed, written guide. Fine work if you ask me ;) http://lists.xen.org/archives/html/xen-users/2012-09/msg00191.html As far as his hardware goes, I''m not sure if it''s the latest Intel chips or not. I''ve been eyeballing the i7-3770 myself (NOT the 3770K, that one will not work, as [in my opinion] Intel has a pension for artificially crippling their products for profit). Haven''t found a board yet, but then again I started eyeballing hardware a day or two ago. Cheers, Andrew Bobulsky
Thanks for the CC Andrew, I responded in private since I didn''t want to spam the list. Since we''re on the subject of the latest hardware, my system was using a Z68 Motherboard Chipset and Core i7 SandyBridge CPU (2600). I just sold it to a friend and upgraded to the latest. Now running an IvyBridge 3770, Z77 Motherboard Chipset (ASRock Z77 Extreme 9), and similar components for the rest. The Extreme 9 price is extremely high, I wouldn''t recommend it unless you have the money, their Z77 Extreme 6 is priced at the same as the Z68 Extreme4 Gen3, and has all the same features plus more USB 3.0. I would give that one a shake if you are looking for newest components. The hardest part of switching from the Extreme4 Gen3 was remapping the USB ports to USB controllers by PCI BDF in Linux, because I haven''t found an easy way to do that using the systems information yet. As soon as I''m done putting together the details, I''ll append the new hardware information to the wiki. As for changes, the new Z77 chipset has onboard USB 3.0 in exchange for some of the formerly USB 2.0 ports. In my performance demo video I showed that USB 2.0 speeds are sub-par, but even SUB 2.0 devices can hit higher rates when connected to a USB 3.0 port. As far as the IvyBridge CPU, onboard graphics are noticeably better in Dom0, and file decompression in my Windows HVM appears to be faster. I have not run any benchmark tests yet. I compiled the new Xen 4.2 stable, as tagged in the new 4.2 testing branch, but I may try the older revision of Xen in my guide. I am getting an error on the first-launch attempt of any virtual machine with passed devices. When I try again it works, but it''s 5 lines worth of errors which makes the following success seem less joyful. I tried kernel 3.5.4, and it appears to be suffering the same passthrough bugs as 3.5.2, so I would avoid 3.5+ still. I ended up with Kernel 3.4.11, which is working. One good bit of news was I kept my SSD, so I was able to just recreate the HVM configuration and launch Windows without problems. Only issues were new drivers, and Windows asking to reactivate due to HW changes. Hope this information helps, ~Casey On Fri, Sep 21, 2012 at 12:30 PM, Andrew Bobulsky <rulerof@gmail.com> wrote:> (CC''ing Casey on this, as I recommend his setup for an Intel-based > solution) > > Hello ShadesOfGrey, > > Hehehe, talk about timing ;) > > On Sep 20, 2012, at 5:15 PM, ShadesOfGrey <shades_of_grey@earthlink.net> > wrote: > > > I''m looking to build a new personal computer. I want it to function as > a Linux desktop, provide network services for my home, and lastly, > occasional Windows gaming. From what I''ve gathered, virtualization using a > Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or Xen would be > an attractive solution for my needs. For reference, reading these threads > on Ars Technica may be helpful to understand where I''m coming from, > http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and > http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But basically, > I use Linux as my primary OS and would rather avoid dual booting or > building two boxes just to play Windows games when I want to play Windows > games. I''m also intrigued by the concept of virtualization and would like > to experiment with it as a solution for my case. > > > > My problem is isolating which hardware to choose, specifically which > combination of CPU, motherboard and video card. Previously I had been > relying on web searches to glean information from gaming and enthusiast web > sites and tech specs from motherboard manufacturers. After what I learned > during my participation in the referenced threads at Ars Technica, I find > myself back at square one. Instead of trying to guess what hardware > support KVM & Xen, and vice versa. I''d like to know what hardware KVM & > Xen users are actually using to run KVM & Xen? Particularly with > consideration for 3D gaming and current generation hardware, BTW. > > > > If there is need for further clarification, I''ll answer any queries you > might have. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xen.org > > http://lists.xen.org/xen-users > > As one of the folks on the list who has done this---probably to the > most extreme degree---I can tell you it''s good stuff. It brings the > joys of datacenter consolidation to your gaming desktop, and also to > your wallet ;) > > While my setup is now slightly dated, the 990FX chipset is still at > the top of the AMD offering, so you can shop around on CPUs, and buy a > cheaper "secondary" USB controller if you''re not looking to cram in a > 4-to-1 ratio. I''ve never had much success passing through the onboard > USB from an AMD system, so I highly recommend picking up a little PCIe > x1 controller at the least. That said, I''m convinced that highpoint > has one of the coincidentally-best products on the market for people > looking to do this, but I digress! > > Take a look at, specifically, this post I made to the list some months > back, and I''ll follow with some errata: > http://lists.xen.org/archives/html/xen-users/2012-05/msg00328.html > > First, I''ve tested all of the hardware in the build that I > recommended, and indeed ended up building a four-headed unit. It works > like magic. Came in handy a few weeks ago when several of my friends > and I piled into a couple cars for a vacation where we wanted to play > games (yup, we''re total nerds), but we couldn''t fit four desktop cases > in addition to our stuff in the cars. :) > > Second, by the time I got around to building it, the Antec One Hundred > wasn''t available. Finding a case that supports 8 expansion slots is a > tough thing, but I found another similarly priced one, and it was a > dream to build. I recommend it highly if you think you may want to > max out your slots and/or go deeper down the rabbit hole with > consolidated desktops: > http://www.newegg.com/Product/Product.aspx?Item=N82E16811112238 > > Aside from being a very solid case for the price point (good features > for screwless installation as well), to give you an idea of the size, > it is laid out in such a way that I could fit dual-GPU cards in it > (Radeon 5970s). I ultimately had to remove the HDD mounts to pull it > off, but you shouldn''t have that problem... Mostly because AMDs dual > GPU cards won''t work for this, so don''t buy one for this build. It''s > a problem with the PCIe switching hardware (well, the firmware > thereof, probably) that they use. I''ll save you the rambling, but > let''s just say that it should work, but doesn''t :( > > Also, the case does look good! ;) > > Finally, and this is unfortunate, for the AMD build, *I* recommend you > use ESXi. While Xen _does_ work with the hardware that I''ve listed, > I''ve never been able to get the VMs to work properly with the GPLPV > drivers, and these are crucial to performance. I really, really want > to bring this project back up on Xen though, and will try again now > that 4.2 has gone RTM. If you aren''t buying anytime soon and would > like to hit me up in a few weeks, by all means drop me a line, and > I''ll let you know if I''ve gotten around to it. > > > ---------------- > > > So, for the Intel route! > > Casey DeLorme has, just this week, posted a fantastic set of detailed > videos and documentation on his setup, where he basically does exactly > what you''re trying to accomplish. You can find links to all of the > documentation, which I''m pretty sure covers his exact hardware, along > with videos of the installation process he used and a detailed, > written guide. Fine work if you ask me ;) > > http://lists.xen.org/archives/html/xen-users/2012-09/msg00191.html > > As far as his hardware goes, I''m not sure if it''s the latest Intel > chips or not. I''ve been eyeballing the i7-3770 myself (NOT the 3770K, > that one will not work, as [in my opinion] Intel has a pension for > artificially crippling their products for profit). Haven''t found a > board yet, but then again I started eyeballing hardware a day or two > ago. > > > > Cheers, > Andrew Bobulsky >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Sorry for the late response, I''ve had a lot to digest. On 09/20/2012 05:53 PM, Alex Williamson wrote:> > There have been a couple success reports of using AMD/ATI graphics cards > on Intel VT-d systems with KVM device assignment. Nvidia cards have not > met with the same degree of success. In both our cases, the graphics > device was assigned to a Windows guest as a secondary graphics head. > For me, Windows took over the assigned device as the primary graphics, > disabling the emulated graphics. For my slow HD 5450, 3dMark seems to > get a similar score to what others see on bare metal. > > Graphics device assignment on KVM is getting better, but should be > considered experimental at best. Thanks, > > Alex > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >I had noticed that KVM seems to be behind Xen with respect to PCI/VGA passthrough (among other things). Then again, with information generally being a bit outdated all around, I figured I should at least give KVM a shot. Worst case scenario, I''d try KVM and Xen (or some other Hypervisor) and both fail to offer what I want. My contingency plan is to build a modest second box. Best case, both work and it''s just a question of choosing which Hypervisor best fits. In any case, I plan on keeping an eye on both as they develop.
Sorry for the late response, I''ve had a lot to digest. On 09/21/2012 11:22 AM, Robin Axelsson wrote:> If you want to be able to use PCI and VGA passthrough you basically > need to make sure that your hardware supports either AMD-Vi (formerly > known as AMD-IOMMU) or Intel VT-d extensions. In the Intel case it > limits your choice of Motherboard (it must be supported in the BIOS) > and CPU. In the AMD case it limits only your choice of motherboard. A > good start is to check out one of these pages: > > http://wiki.xensource.com/xenwiki/VTdHowTo > http://wiki.xen.org/wiki/VTd_HowTo > > A word of warning here is that parts of the documentation is somewhat > dated. You can also communicate with e.g. Gigabyte, Asus or ASRock > customer support and ask them if a particular motherboard supports > these extensions. Most motherboards also have downloadable user > manuals, if the BIOS settings in those pages shows options to > enable/disable VT-d or AMD-Vi/IOMMU extensions then you will be ok > with that motherboard. >The lack of current information about Xen (and KVM) online has been frustrating — especially finding the many proof of concept videos that demonstrated possibilities but offered no real specifics. Looking for specifics, I sought information from gaming and enthusiast sites; I figured finding confirmation of VT-d and AMD-Vi support on such sites would be more likely. However, I found that wasn''t often the case. I did determine that ASRock motherboards seem to be the most likely to support VT-d, ASUS least likely (unless equipped with an Intel ''sanctioned'' VT-d chipset). I had narrowed my choices to two motherboards that appear to offer VT-d support and was intending to contact the manufacturer before purchase. Both choices are a bit pricey and I''ve been reconsidering whether I should look to other motherboards to reduce costs.> The other thing is choice of GPU for VGA passthrough and it is > preferable that the GPU supports FLR or Function Level Reset as it is > called. Thing is that the hardware needs to be reset somehow as it is > passed through to the host. This is best done with FLR and nVidia is > known to supply firmware patches for some of their Geforce cards with > this support and it is said to be supported by default with their > Quadro cards. FLR is not the only way to reset a PCI device, a reset > could be trigged through the ACPI power management framework by > temporarily cutting power to the affected PCI slot. These reset > methods are called d3d0 and bus reset. The question however, is if > this works on PCI cards that use auxiliary power directly from the > PSU. There is a pdf document on the VMWare website > (http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf) about > this: > > ----------------------- > Reset Method > > Possible values for the reset method include flr, d3d0, link, bridge, > or default. > > The default setting is described as follows. If a device supports > function level reset (FLR), ESX always uses FLR. If the device does > not support FLR, ESX next defaults to link reset and bus reset in that > order. Link reset and bus reset might prevent some devices from being > assigned to different virtual machines, or from being assigned between > the VMkernel and virtual machines. In the absence of FLR, it is > possible to use PCI Power Management capability (D3 to D0 transitions) > to trigger a reset. Most of the Intel NICs and various other HBAs > support this mode. > ----------------------- > > > There are indications from people that d3d0 also work with PCI cards > that take power from auxiliary inputs. I suggest that you take a look > at the following youtube clip and read the comments there: > > http://www.youtube.com/watch?v=Gtmwnx-k2qg > > So it seems that it works although it may be a bit more quirky. It > doesn''t hurt to take that discussion (particularly about FLR support) > with nVidia and/or AMD. >This is precisely the kind of information I was looking for from the threads I started on Ars Technica. It''s just unfortunate that FLR and D3 D0 support aren''t often found in the tech specs of must expansion hardware. However, now that I know what to ask, I''ll try contacting hardware manufacturers prior to purchasing any expansion hardware. Thank you!> When it comes to virtualization, the technology has come very far, but > it is still lacking considerably when it comes to sharing GPUs and > also to some degree when it comes to sharing I/O devices (especially > when you intend to run many virtual machines on a single system). The > GPU today consists of three types of components; the processing unit, > graphics memory and the video output unit/adapter and it is not clear > as to how to share these components seamlessly between the host and > virtual machines with minimal overhead. Whereas there are VT-x > extensions that allows you to pretty seamlessly share CPU cores > between VMs and the host there are currently none for the processing > unit. It is also not clear how the hardware can assist with sharing > TV/monitor screen estate between machines with all 3D effects such as > Aqua for Win7 and the whatnot enabled for all machines. Especially > when considering the dynamics of plugging and unplugging computer > monitors to multiport/eyefinity graphics cards and the ability to > change screen resolution. Things are improving for sure and a lot of > research is likely going into this. I don''t know what''s happening in > the GPU frontline but I know that the next thing with passthrough is > the SR-IOV that allows PCI units to present several virtual instances > of oneself to several virtual machines. It''s a cool thing, I recommend > further reading about this here: > > http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html > > http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/ >That is fascinating. Extending virtualization to expansion hardware via SR-IOV, sure would make the kind of setup I''m attempting a lot easier. However, if I can replicate what I''ve seen in proof of concept videos (namely Casey DeLorme''s), I think that will meet my needs for now. As it stands, I initially intend to reserve any discrete GPU(s) for Windows and rely on an integrated GPU for all other VMs using PV drivers (wherever possible). Afterward, I want to experiment with re-assigning the whatever discrete GPU(s) for GPGPU functions under a Linux VM whenever the GPU is not going to used for gaming (if at all possible).> It is likely to take a few years before something useful will come out > of it. In the meanwhile, unless you want to use several GPUs which > might not be a bad thing as a lot of monitors these days have several > inputs, you can resort to using a remote desktop client to integrate > one machine with another. Virtualbox for example use RDP through which > you can interact with your virtual machine. In a similar manner you > can set up a VNC server on your Linux host and establish a connection > to it through your Windows VM. You will not get full 3D functionality > (such as Aqua) through the client although there is a growing support > for it through VirtualGL extensions that are coming to VNC and perhaps > the Spice protocol. But some clients might even allow for seamless > mode that lets you mix Linux and Windows windows on the same desktop > like this for example: > > http://i.techrepublic.com.com/blogs/seamless.png > http://www.youtube.com/watch?v=eQr8iI0yZH4 > > > Just keep in mind that this is still a little bit of uncharted > territory so there may be a few bumps on the way and it may not work > as smooth as you would desire. > >From everything I''ve read, solutions that rely on any form of remote display protocols would be limited to a subset of Direct3D functions. Furthermore, these would vary from one implementation to another, thus making them far less attractive for gaming than VGA passthrough... Well, in my opinion anyway. VirtualBox''s seamless mode is pretty nifty. But it''s a Type 2 Hypervisor and relies on paravirtualized drivers that also suffer from the same limitations as remote display protocols. It''s great for most things, but gaming is not one of them. And I''m speaking from personal experience. Though I haven''t used them myself, the same would seem to hold true of Parallel''s and VMWare''s ''Workstation'' offerings. At least, as far as I''ve gathered. FYI, the Type 1 Hypervisors from Parallel''s and VMWare* are priced waaayyy outside my budget. *I only found out about VMWare''s ''free'' vSphere after I''d written this response.> > I see that your demands are somewhat multifaceted. I believe that you > also want to use diffent services such as using your machine as a file > server with the possible intention of using filesystems such as ZFS. > If you do, you should be careful with your selection of hardware for > these particular purposes. If you want to get full protection against > data corruption from ZFS, your choice of hardware gets rather limited > when it comes to choice of hard drives, host bus adapter and network > controller. The most stable implementation of ZFS is found with > Illumos based operating systems (such as OpenIndiana, SmartOS, OmniOS, > Belenix etc) or Solaris if you choose to download it from Oracle''s > website. With these operating systems you are most likely to want to > use hardware that has certified drivers for it. That way you are less > likely to run into problems later on. That implies that you will be > limited to choosing Intel based network adapters and LSI based SAS > controllers. There should be _no_ hardware RAID functionality in the > SAS controller that merely should be run in IT mode (or > Initiator-Target mode). That requires the LSI controller to be flashed > with IT firmware in most cases. The objective here is to make sure > that _all_ errors that might occur with the hard drives are reported > all the way to the software level and that nothing is concealed of > obfuscated by internal error handling in the hardware. It is therefore > recommended to use SAS hard drives instead of S-ATA (which also are > fully compatible with SAS controllers). SAS hard drives are not much > more expensive than similar SATA drives and you get a higher > reliability out of them. It is also recommended to have at least two > drive redundancy simply because if one drive is dead and you swap it, > it is not uncommon that another drive dies in the rebuild process of > the RAID cluster because of the added strain the rebuild process (or > ''resilvering'' as it is called in Solaris terms) put on the drives. Of > course, the system should communicate directly to the hard drive > hardware and not be obfuscated by some virtual abstraction layer in > between which means that you either run ZFS on the metal or through > PCI passthrough of the SAS (and perhaps also network) adapters. Also, > it is highly recommended that you use ECC RAM for such applications > and it doesn''t hurt to dedicate a few gigs of it to the ZFS as RAM is > used for cache. The good news is that most motherboards with good > chipsets support ECC RAM even though you might not find anything about > it in the user manuals.Again, thanks for the thorough explanation. This gives me a great deal to think about. The more I learn about ZFS, the less appealing it becomes. And by that I mean the confusion over which version of ZFS is in what OS? And just how well maintained the OSes supporting ZFS are? Now I have additional hardware considerations to keep in mind that may (or may not) make the cost of ZFS RAID-Z pool comparable to a hardware RAID5/6 solution anyway. Do you have any suggestions as to which of LSI HBAs I should be considering? I haven''t found an HCL for ZFS in my searches. Out of curiosity — and if you would happen to know — do you think what you suggest about the HBA and SAS drives for ZFS also applies to Btrfs? I''m assuming it would, but I''d appreciate some confirmation. It''s funny how the "I" in RAID never really seems to apply... Especially since it looks more and more like using ZFS or Btrfs will require I commit myself, from the start, to one or the other and a discrete HBA. Transitioning from an integrated SATA controller(s) and mdadm seems rather impractical. If I understand what''s involved in doing so correctly. It may turn out that anything other than mdadm is price prohibitive.> I admire your persistence with pursuing this undertaking and wish you > the best of luck with it! > Robin. > >Thanks. I''ve invested too much time in research to not at least make the attempt. Besides, if all else fails, I can fallback to a two box solution. That is, if I can get my hypothetical virtualization box to fit in my budget envelope...
Sorry for the late response, I''ve had a lot to digest. On 09/21/2012 12:30 PM, Andrew Bobulsky wrote:> (CC''ing Casey on this, as I recommend his setup for an Intel-based solution) > > Hello ShadesOfGrey, > > Hehehe, talk about timing ;) > > > As one of the folks on the list who has done this---probably to the > most extreme degree---I can tell you it''s good stuff. It brings the > joys of datacenter consolidation to your gaming desktop, and also to > your wallet ;) > > While my setup is now slightly dated, the 990FX chipset is still at > the top of the AMD offering, so you can shop around on CPUs, and buy a > cheaper "secondary" USB controller if you''re not looking to cram in a > 4-to-1 ratio. I''ve never had much success passing through the onboard > USB from an AMD system, so I highly recommend picking up a little PCIe > x1 controller at the least. That said, I''m convinced that highpoint > has one of the coincidentally-best products on the market for people > looking to do this, but I digress! > > Take a look at, specifically, this post I made to the list some months > back, and I''ll follow with some errata: > http://lists.xen.org/archives/html/xen-users/2012-05/msg00328.htmlI''d like to support AMD... I really would!... if for no other reason than I''d hate to see what would happen if Intel were the only choice for x86 CPUs. But I just can''t see myself justifying an AMD build. IIRC, the AMD FX-8150 was roughly on par with an Intel Core i5 2600. I don''t think I''ll be able to get the level of performance or longevity (five to ten years) I want from an AMD processor. From what Casey DeLorme said in his response, I don''t think I''ll have a problem with USB if I follow his advice on motherboards. But I''ll certainly keep in mind that I may need to pick up additional hardware like an USB HBA. But I''m more concerned about audio. You said you used a "Generic Piece of Crap" USB adapter and advised getting several in case of failure. I''d prefer to get something, shall we say, "more reliable." ;-) To that end, I was thinking of picking up a PCIe or PCI sound card from the Xonar series. But there''s a potential problem when passing the PCIe versions through: From what I understand, the PCIe Xonar cards use a PCI to PCIe bridge, which is not uncommon for a lot of hardware. The problem arises in the way in which ASUS implemented this bridge. IIRC, the sound device doesn''t appear to be recognized by Xen (or the VM, I forget which), even though it is present. Unfortunately, I can''t find the exact reference now. But this posting <http://markmail.org/message/uqhetcusgxfjyvpy> on the Xen Development list sounds like it resembles the issue I read about. Assuming that''s the issue, has Xen been patched to deal with this? If not, could someone recommend a decent PCIe sound card that either doesn''t rely on a PCI bridge or presently works with Xen (if not KVM, too)? Of course, this would be moot if I go with a motherboard that has legacy PCI slots. But one of the candidates I was considering is PCIe only.> First, I''ve tested all of the hardware in the build that I > recommended, and indeed ended up building a four-headed unit. It works > like magic. Came in handy a few weeks ago when several of my friends > and I piled into a couple cars for a vacation where we wanted to play > games (yup, we''re total nerds), but we couldn''t fit four desktop cases > in addition to our stuff in the cars. :)That sounds sweet. I might ask later how exactly you multiplexed things so you could get four users playing games, at the same time. Because, I have two cases where my original plan may not work as expected. First case, I was going to use Synergy <http://synergy-foss.org/> to handle sharing the keyboard and mouse. But that may not work for all potential VMs, as Synergy may not be available to the given OS of a specific VM. Secondly, I have an adolescent nephew who is always disappointed when he visits. The reason being that the only system I have ATM that has the hardware for gaming is a Linux box. Unfortunately for me, he doesn''t like any of the Linux games I can acquire (and that my sister would let him play anyway), save one.> > Second, by the time I got around to building it, the Antec One Hundred > wasn''t available. Finding a case that supports 8 expansion slots is a > tough thing, but I found another similarly priced one, and it was a > dream to build. I recommend it highly if you think you may want to > max out your slots and/or go deeper down the rabbit hole with > consolidated desktops: > http://www.newegg.com/Product/Product.aspx?Item=N82E16811112238 > > Aside from being a very solid case for the price point (good features > for screwless installation as well), to give you an idea of the size, > it is laid out in such a way that I could fit dual-GPU cards in it > (Radeon 5970s). I ultimately had to remove the HDD mounts to pull it > off, but you shouldn''t have that problem... Mostly because AMDs dual > GPU cards won''t work for this, so don''t buy one for this build. It''s > a problem with the PCIe switching hardware (well, the firmware > thereof, probably) that they use. I''ll save you the rambling, but > let''s just say that it should work, but doesn''t :( > > Also, the case does look good! ;)Personally, I''ve gravitated toward the Xigmatek Elysium. Yeah, it''s quite a bit more expensive than either the Antec or Lian Li you recommended. But, then again, it is a monster of a case. My only regret is that I couldn''t find the CCC-HSA0DS-U04 model anywhere (well, within the states anyway). Now it appears my second choice, the CCC-HSA0DS-U03 version has been discontinued (per Newegg) and is considerably more expensive than other the models among the retailers I found that carry it... Sorry for the mini-rant. I just never could comprehend the windowed side panel fad. And the two remaining available SKUs (U01 & U02) for the Elysium case have side panel windows.> Finally, and this is unfortunate, for the AMD build, *I* recommend you > use ESXi. While Xen _does_ work with the hardware that I''ve listed, > I''ve never been able to get the VMs to work properly with the GPLPV > drivers, and these are crucial to performance. I really, really want > to bring this project back up on Xen though, and will try again now > that 4.2 has gone RTM. If you aren''t buying anytime soon and would > like to hit me up in a few weeks, by all means drop me a line, and > I''ll let you know if I''ve gotten around to it. >Would you happen to know if the "free" version of VMware''s vSphere Hypervisor is a capable ''alternative'' to KVM or Xen for what I''m looking to do? VMWare''s FAQ <http://www.vmware.com/products/vsphere-hypervisor/faq.html> concerning vSphere states that it "was formerly known as VMware ESXi Single Server or free ESXi (often abbreviated to simply "VMware ESXi")." But the VMWare Compatibility Guide <http://partnerweb.vmware.com/comp_guide2/search.php?deviceCategory=server>, doesn''t have an entry for vSphere and the the FAQ doesn''t specify which version of ESXi vSphere it is based on. I''d like to at least test as many ''free'' solutions as possible. As long as they have the potential to meet my stated requirements, anyway.> > ---------------- > > > So, for the Intel route! > > Casey DeLorme has, just this week, posted a fantastic set of detailed > videos and documentation on his setup, where he basically does exactly > what you''re trying to accomplish. You can find links to all of the > documentation, which I''m pretty sure covers his exact hardware, along > with videos of the installation process he used and a detailed, > written guide. Fine work if you ask me ;) > > http://lists.xen.org/archives/html/xen-users/2012-09/msg00191.html > > As far as his hardware goes, I''m not sure if it''s the latest Intel > chips or not. I''ve been eyeballing the i7-3770 myself (NOT the 3770K, > that one will not work, as [in my opinion] Intel has a pension for > artificially crippling their products for profit). Haven''t found a > board yet, but then again I started eyeballing hardware a day or two > ago. >It. Is. An. AWESOME. Thing. If Casey hadn''t written such a guide, that''s what I had intended to do myself. I may yet do so if I use different hardware and/or software... Or perhaps he''d be so kind enough to allow me to piggyback off of his wiki entry if I (as I suspect) settle on Xen? Anyway, as he said in his response, he has upgraded his hardware since writing the guide. And the motherboard he is currently using was one I originally selected as a candidate for my build. Though, I''m now considering other motherboards. I''ve already exceeded the upper limits of my budget and I have yet to factor in the cost of a discrete GPU. I was looking for more information on how well any given GPU would behave under virtualization and meet certain use-case scenarios, Robin Axelsson really helped there. So, finding way to cut costs here and there, without sacrificing core functionality, really wouldn''t hurt. ;-) I do have to agree with you about Intel and their artificial profit centers. I was surprised they bothered with VT-d support in anything other than the Xeon line after all I had learned. Besides the inconsistent support for VT-d among K series Sandy Bridge Core i processors, there''s the issue of chipset support. The Zxx desktop chipset is intended for enthusiasts, yet VT-d support is not ''officially sanctioned'' by Intel on that chipset. Some nonsense about it not meeting their internal testing standards. No, instead you either use the Qxx or B75 desktop chipsets, switch to server chipsets (and the various tradeoffs that come with them), or hope that your chosen motherboard manufacturer ''unofficially'' officially support VT-d on motherboards with other chipsets. As ASRock does and Gigabyte may unofficially support through ''beta'' firmware versions (I still haven''t nailed down whether the G1.Sniper 3 has really and truly has VT-d support). MSI might, but so far their motherbaord offerings haven''t appealed to me in a more general sense. Anyway, the confusing nomenclature and vague technical specifications from Intel all remind me of the 386/486, SX/DX days. And this is why I really hope AMD can get their act together. Without competition, Intel will go back to really gouging its users!> > Cheers, > Andrew Bobulsky >Thanks a heap, Andrew. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Sorry for the late response, I''ve had a lot to digest. On 09/21/2012 01:02 PM, Casey DeLorme wrote:> Thanks for the CC Andrew, > > I responded in private since I didn''t want to spam the list. > > Since we''re on the subject of the latest hardware, my system was using > a Z68 Motherboard Chipset and Core i7 SandyBridge CPU (2600). I just > sold it to a friend and upgraded to the latest. > > Now running an IvyBridge 3770, Z77 Motherboard Chipset (ASRock Z77 > Extreme 9), and similar components for the rest. The Extreme 9 price > is extremely high, I wouldn''t recommend it unless you have the money, > their Z77 Extreme 6 is priced at the same as the Z68 Extreme4 Gen3, > and has all the same features plus more USB 3.0. I would give that > one a shake if you are looking for newest components. The hardest > part of switching from the Extreme4 Gen3 was remapping the USB ports > to USB controllers by PCI BDF in Linux, because I haven''t found an > easy way to do that using the systems information yet. >Funnily enough, the Extreme9 was one of my two original motherboard candidates, along with the Gigabyte G1.Sniper 3. Though I''ve expanded that list now to include all of ASRock''s Z77 line of motherboards. I might even consider Z77 motherboards from other manufacturers, if I can confirm they support VT-d. I''ve also been contemplating going with a Q77 motherboard, just to avoid the hassles of determining if a given Z77 motherboard actually has VT-d support. The downside to the Q77 motherboards I have looked at is that they typically lack the integrated extras (e.g. additional SATA/USB controllers) that you find on Z77 motherboards. This makes them less attractive for PCI passthrough. Furthermore, a respondent on the Ars Technica threads suggested that an LGA2011 could be worth a gander. Now, when I started investigating hardware, I had dismissed this option, mostly because they were often very expensive (incl. CPU). More aggravating, the X79 chipset used on desktop LGA2011 motherboards is not officially sanctioned for VT-d. Given it''s placement as a desktop workstation chipset, the omission of VT-d just pissed me off. I didn''t even bother investigating the possibility that X79 motherboard ''unofficially'' supported VT-d. So, in order to get VT-d on LGA2011, you had to level up to a server motherboard. Unfortunately, most of the options that had the mix of features I wanted, ended up being ridiculously expensive, particularly because those motherboards were designed with dual processors in mind. Fortunately, there have been some changes since then. For example, I found this Gigabyte motherboard <http://www.gigabyte.com/products/product-page.aspx?pid=4287#sp>. It''s based on the C606 server chipset but packaged as a desktop motherboard. It is sanctioned by Intel for VT-d and has more RAM capacity. As an added bonus, it includes an integrated SAS controller supplying eight SAS 3GB/s ports. I''m pretty sure it''s fake RAID though. No big deal, I''ll probably end up using Btrfs, mdadm, or ZFS on that controller anyway. As things stand now, this motherboard would pretty much cost the same as the Extreme9. Going the LGA 2011 route, however, would require I start with two discrete GPUs instead of one. Maybe something like this <http://www.newegg.com/Product/Product.aspx?Item=N82E16814131339>, combined with something like this <http://www.newegg.com/Product/Product.aspx?Item=N82E16814150549> or this <http://www.newegg.com/Product/Product.aspx?Item=N82E16814102989>. All of these considerations have seriously complicated my decision-making process. I still haven''t managed to pare down my hardware choices to fit the $2000 budget I have imposed on myself. And those calculations don''t even include one discrete GPU! So far, most (not all, e.g. Extreme6) of the suggestions I get haven''t helped to reduce costs. Based on your success, I''m sorely tempted to dip into my reserve fund for building a second, more modest box: My contingency plan if my experiments in virtualization failed to produce satisfactory results.> As soon as I''m done putting together the details, I''ll append the new > hardware information to the wiki. > > As for changes, the new Z77 chipset has onboard USB 3.0 in exchange > for some of the formerly USB 2.0 ports. In my performance demo video > I showed that USB 2.0 speeds are sub-par, but even SUB 2.0 devices can > hit higher rates when connected to a USB 3.0 port. As far as the > IvyBridge CPU, onboard graphics are noticeably better in Dom0, and > file decompression in my Windows HVM appears to be faster. I have not > run any benchmark tests yet. >I was kind of surprised by that. I can understand that there would be some overhead due to the virtualized environment, but I wondered what would cause such a degradation of performance? I mean I know USB is pretty CPU dependent. However, with a Core i7, I figure there would be enough CPU cycles to spare that only multiple asynchronous connections would cause that kind of performance hit.> > I compiled the new Xen 4.2 stable, as tagged in the new 4.2 testing > branch, but I may try the older revision of Xen in my guide. I am > getting an error on the first-launch attempt of any virtual machine > with passed devices. When I try again it works, but it''s 5 lines > worth of errors which makes the following success seem less joyful. > > I tried kernel 3.5.4, and it appears to be suffering the same > passthrough bugs as 3.5.2, so I would avoid 3.5+ still. I ended up > with Kernel 3.4.11, which is working. >I''ll keep all that in mind, though I don''t have a problem compiling sever kernel images to play with... BTW, would there be any significant problems, that you know of, to compiling and trying several combinations of Linux and Xen during my testing phase? If I went with significantly different hardware, I imagine I might need different combinations of both. But, since I haven''t actually gotten to the testing phase, I don''t know what easily avoidable pitfalls might be in my path.> One good bit of news was I kept my SSD, so I was able to just recreate > the HVM configuration and launch Windows without problems. Only > issues were new drivers, and Windows asking to reactivate due to HW > changes. > > > Hope this information helps, > > ~Casey >It''s all been very helpful. I am in your debt! I''m sure others will be too, once they make use of your guide. BTW, another thing that I''ve been wondering about. Should I treat Dom0 as an administrative domain and create a separate DomU for my Linux desktop? Or is it safe to reduce the overhead of having a Linux DomU and just use Dom0? I know this is probably more of a stylistic question; after all, you were using your Dom0 for development purposes... but that''s significantly different from a general purpose desktop environment. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 2012-09-24 05:45, ShadesOfGrey wrote:> Sorry for the late response, I''ve had a lot to digest. > > On 09/21/2012 11:22 AM, Robin Axelsson wrote: >> If you want to be able to use PCI and VGA passthrough you basically >> need to make sure that your hardware supports either AMD-Vi (formerly >> known as AMD-IOMMU) or Intel VT-d extensions. In the Intel case it >> limits your choice of Motherboard (it must be supported in the BIOS) >> and CPU. In the AMD case it limits only your choice of motherboard. A >> good start is to check out one of these pages: >> >> http://wiki.xensource.com/xenwiki/VTdHowTo >> http://wiki.xen.org/wiki/VTd_HowTo >> >> A word of warning here is that parts of the documentation is somewhat >> dated. You can also communicate with e.g. Gigabyte, Asus or ASRock >> customer support and ask them if a particular motherboard supports >> these extensions. Most motherboards also have downloadable user >> manuals, if the BIOS settings in those pages shows options to >> enable/disable VT-d or AMD-Vi/IOMMU extensions then you will be ok >> with that motherboard. >> > The lack of current information about Xen (and KVM) online has been > frustrating — especially finding the many proof of concept videos that > demonstrated possibilities but offered no real specifics. Looking for > specifics, I sought information from gaming and enthusiast sites; I > figured finding confirmation of VT-d and AMD-Vi support on such sites > would be more likely. However, I found that wasn''t often the case. I > did determine that ASRock motherboards seem to be the most likely to > support VT-d, ASUS least likely (unless equipped with an Intel > ''sanctioned'' VT-d chipset). I had narrowed my choices to two > motherboards that appear to offer VT-d support and was intending to > contact the manufacturer before purchase. Both choices are a bit > pricey and I''ve been reconsidering whether I should look to other > motherboards to reduce costs.Some motherboards support IOMMU even though it is not found in the user manual or specified on the website. Your best bet is to ask customer support. A guy posted here that he got it working on an Intel motherboard that doesn''t even have options for it in the BIOS, so it seems that in some cases it is only up to the CPU. This is not the case with AMD though as I stated before. I have bought a couple of Gigabyte GA990FX-UD7 myself, they are stable and have a good layout. They have support for IOMMU but I haven''t tested it thoroughly enough to fully confirm this although I don''t believe there would be any problem. It surprises me that ASRock and ASUS are so different. ASRock is, or at least used to be a subsidiary of ASUS so there shouldn''t be that much difference between them.>> The other thing is choice of GPU for VGA passthrough and it is >> preferable that the GPU supports FLR or Function Level Reset as it is >> called. Thing is that the hardware needs to be reset somehow as it is >> passed through to the host. This is best done with FLR and nVidia is >> known to supply firmware patches for some of their Geforce cards with >> this support and it is said to be supported by default with their >> Quadro cards. FLR is not the only way to reset a PCI device, a reset >> could be trigged through the ACPI power management framework by >> temporarily cutting power to the affected PCI slot. These reset >> methods are called d3d0 and bus reset. The question however, is if >> this works on PCI cards that use auxiliary power directly from the >> PSU. There is a pdf document on the VMWare website >> (http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf) >> about this: >> >> ----------------------- >> Reset Method >> >> Possible values for the reset method include flr, d3d0, link, bridge, >> or default. >> >> The default setting is described as follows. If a device supports >> function level reset (FLR), ESX always uses FLR. If the device does >> not support FLR, ESX next defaults to link reset and bus reset in >> that order. Link reset and bus reset might prevent some devices from >> being assigned to different virtual machines, or from being assigned >> between the VMkernel and virtual machines. In the absence of FLR, it >> is possible to use PCI Power Management capability (D3 to D0 >> transitions) to trigger a reset. Most of the Intel NICs and various >> other HBAs support this mode. >> ----------------------- >> >> >> There are indications from people that d3d0 also work with PCI cards >> that take power from auxiliary inputs. I suggest that you take a look >> at the following youtube clip and read the comments there: >> >> http://www.youtube.com/watch?v=Gtmwnx-k2qg >> >> So it seems that it works although it may be a bit more quirky. It >> doesn''t hurt to take that discussion (particularly about FLR support) >> with nVidia and/or AMD. >> > This is precisely the kind of information I was looking for from the > threads I started on Ars Technica. It''s just unfortunate that FLR and > D3 D0 support aren''t often found in the tech specs of must expansion > hardware. However, now that I know what to ask, I''ll try contacting > hardware manufacturers prior to purchasing any expansion hardware. > Thank you!D3 and D0 are power states defined for devices in the ACPI specification and can be used to control the supply voltage (Vcc) to PCI and PCIe devices. You can find more information about it here for example: http://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface --------------- Device states The device states /D0/-/D3/ are device-dependent: * *D0* /Fully On/ is the operating state. * *D1* and *D2* are intermediate power-states whose definition varies by device. * *D3* /Off/ has the device powered off and unresponsive to its bus. --------------- So, either it works for a certain type of hardware or it doesn''t and I wouldn''t expect a vendor to state this "support" in the specifications since it isn''t a "feature" in and of itself if you get me. But maybe this will change and maybe FLR support will become more widespread.>> When it comes to virtualization, the technology has come very far, >> but it is still lacking considerably when it comes to sharing GPUs >> and also to some degree when it comes to sharing I/O devices >> (especially when you intend to run many virtual machines on a single >> system). The GPU today consists of three types of components; the >> processing unit, graphics memory and the video output unit/adapter >> and it is not clear as to how to share these components seamlessly >> between the host and virtual machines with minimal overhead. Whereas >> there are VT-x extensions that allows you to pretty seamlessly share >> CPU cores between VMs and the host there are currently none for the >> processing unit. It is also not clear how the hardware can assist >> with sharing TV/monitor screen estate between machines with all 3D >> effects such as Aqua for Win7 and the whatnot enabled for all >> machines. Especially when considering the dynamics of plugging and >> unplugging computer monitors to multiport/eyefinity graphics cards >> and the ability to change screen resolution. Things are improving for >> sure and a lot of research is likely going into this. I don''t know >> what''s happening in the GPU frontline but I know that the next thing >> with passthrough is the SR-IOV that allows PCI units to present >> several virtual instances of oneself to several virtual machines. >> It''s a cool thing, I recommend further reading about this here: >> >> http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html >> >> http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/ >> > That is fascinating. Extending virtualization to expansion hardware > via SR-IOV, sure would make the kind of setup I''m attempting a lot > easier. However, if I can replicate what I''ve seen in proof of concept > videos (namely Casey DeLorme''s), I think that will meet my needs for > now. As it stands, I initially intend to reserve any discrete GPU(s) > for Windows and rely on an integrated GPU for all other VMs using PV > drivers (wherever possible). Afterward, I want to experiment with > re-assigning the whatever discrete GPU(s) for GPGPU functions under a > Linux VM whenever the GPU is not going to used for gaming (if at all > possible). >> It is likely to take a few years before something useful will come >> out of it. In the meanwhile, unless you want to use several GPUs >> which might not be a bad thing as a lot of monitors these days have >> several inputs, you can resort to using a remote desktop client to >> integrate one machine with another. Virtualbox for example use RDP >> through which you can interact with your virtual machine. In a >> similar manner you can set up a VNC server on your Linux host and >> establish a connection to it through your Windows VM. You will not >> get full 3D functionality (such as Aqua) through the client although >> there is a growing support for it through VirtualGL extensions that >> are coming to VNC and perhaps the Spice protocol. But some clients >> might even allow for seamless mode that lets you mix Linux and >> Windows windows on the same desktop like this for example: >> >> http://i.techrepublic.com.com/blogs/seamless.png >> http://www.youtube.com/watch?v=eQr8iI0yZH4 >> >> >> Just keep in mind that this is still a little bit of uncharted >> territory so there may be a few bumps on the way and it may not work >> as smooth as you would desire. >> >> > From everything I''ve read, solutions that rely on any form of remote > display protocols would be limited to a subset of Direct3D functions. > Furthermore, these would vary from one implementation to another, thus > making them far less attractive for gaming than VGA passthrough... > Well, in my opinion anyway. > > VirtualBox''s seamless mode is pretty nifty. But it''s a Type 2 > Hypervisor and relies on paravirtualized drivers that also suffer from > the same limitations as remote display protocols. It''s great for most > things, but gaming is not one of them. And I''m speaking from personal > experience. Though I haven''t used them myself, the same would seem to > hold true of Parallel''s and VMWare''s ''Workstation'' offerings. At > least, as far as I''ve gathered. > > FYI, the Type 1 Hypervisors from Parallel''s and VMWare* are priced > waaayyy outside my budget. >I understand that you want full 3D functionality for Windows gaming but maybe you''ll find the subset of 3D functionality for the Linux machine acceptable. I have looked into VirtualGL and with TurboVNC, you might get a pretty decent desktop environment and it seems like most of the features are there already. It appears that the 3D is rendered by hardware/GPU before it is streamed through VNC or Spice. So it seems that you would need another GPU for that. You can find more info on VirtualGL here: http://www.virtualgl.org/ Also the line between a type 1 and type 2 hypervisor tend to get a bit blurry. The point with type 1 is that it has access to ring-0 so that it can get access directly to the hardware to be passed through to the guests (I did confuse ''host'' and ''guest'' in my prior post). It also doesn''t need to ask the host OS for permission in the same way as a type 2 hypervisor which is likely to give performance advantages in some cases. However, even a type 2 hypervisor, although it is run as an application inside the OS can get "type 1" like privileges. By patching into the kernel and/or using special "dummy drivers" for hardware to be shared with VMs you can achieve pretty much the same thing, ergo it is no longer clear whether the hypervisor is a type 1 or type 2. There is an article about it from the old IBM Mainframe days but I can''t seem to find it.> > *I only found out about VMWare''s ''free'' vSphere after I''d written this > response. >> >> I see that your demands are somewhat multifaceted. I believe that you >> also want to use diffent services such as using your machine as a >> file server with the possible intention of using filesystems such as >> ZFS. If you do, you should be careful with your selection of hardware >> for these particular purposes. If you want to get full protection >> against data corruption from ZFS, your choice of hardware gets rather >> limited when it comes to choice of hard drives, host bus adapter and >> network controller. The most stable implementation of ZFS is found >> with Illumos based operating systems (such as OpenIndiana, SmartOS, >> OmniOS, Belenix etc) or Solaris if you choose to download it from >> Oracle''s website. With these operating systems you are most likely to >> want to use hardware that has certified drivers for it. That way you >> are less likely to run into problems later on. That implies that you >> will be limited to choosing Intel based network adapters and LSI >> based SAS controllers. There should be _no_ hardware RAID >> functionality in the SAS controller that merely should be run in IT >> mode (or Initiator-Target mode). That requires the LSI controller to >> be flashed with IT firmware in most cases. The objective here is to >> make sure that _all_ errors that might occur with the hard drives are >> reported all the way to the software level and that nothing is >> concealed of obfuscated by internal error handling in the hardware. >> It is therefore recommended to use SAS hard drives instead of S-ATA >> (which also are fully compatible with SAS controllers). SAS hard >> drives are not much more expensive than similar SATA drives and you >> get a higher reliability out of them. It is also recommended to have >> at least two drive redundancy simply because if one drive is dead and >> you swap it, it is not uncommon that another drive dies in the >> rebuild process of the RAID cluster because of the added strain the >> rebuild process (or ''resilvering'' as it is called in Solaris terms) >> put on the drives. Of course, the system should communicate directly >> to the hard drive hardware and not be obfuscated by some virtual >> abstraction layer in between which means that you either run ZFS on >> the metal or through PCI passthrough of the SAS (and perhaps also >> network) adapters. Also, it is highly recommended that you use ECC >> RAM for such applications and it doesn''t hurt to dedicate a few gigs >> of it to the ZFS as RAM is used for cache. The good news is that most >> motherboards with good chipsets support ECC RAM even though you might >> not find anything about it in the user manuals. > Again, thanks for the thorough explanation. This gives me a great deal > to think about. The more I learn about ZFS, the less appealing it > becomes. And by that I mean the confusion over which version of ZFS is > in what OS? And just how well maintained the OSes supporting ZFS are? > Now I have additional hardware considerations to keep in mind that may > (or may not) make the cost of ZFS RAID-Z pool comparable to a hardware > RAID5/6 solution anyway. Do you have any suggestions as to which of > LSI HBAs I should be considering? I haven''t found an HCL for ZFS in my > searches. > > Out of curiosity — and if you would happen to know — do you think what > you suggest about the HBA and SAS drives for ZFS also applies to > Btrfs? I''m assuming it would, but I''d appreciate some confirmation. > > It''s funny how the "I" in RAID never really seems to apply... > Especially since it looks more and more like using ZFS or Btrfs will > require I commit myself, from the start, to one or the other and a > discrete HBA. Transitioning from an integrated SATA controller(s) and > mdadm seems rather impractical. If I understand what''s involved in > doing so correctly. It may turn out that anything other than mdadm is > price prohibitive. >I don''t think you will have a problem with getting ZFS to run and if that''s your only goal then you don''t need to be very picky with your choice of hardware. I find ZFS pretty easy and handy to use. I has really great functionality and I don''t have many bad things to say about it so far. ZFS is a filesystem (along with a couple of software tools to administrate it) just like EXT4 or NTFS so hardware support depends on the platform it runs on. But the point with using ZFS is to get maximum protection against data corruption and that''s where the selection of hardware gets limited and there are "best practices" set up to achieve that. I have not tested ZFS on any other platform than on OpenSolaris and OpenIndiana but I do know that it is well implemented on that platform and more mature there than on any other (non-solaris) platform. Another advantage with the OSOL/OI platform is that the CIFS functionality is implemented in the kernel space and not in the userland which will give advantages performance wise if you intend to share files with other windows computers. (I don''t deny that Samba is pretty good on Linux too. There are some benchmarks on the phoronix website comparing samba with NFS and they are in favor of Samba on those benchmarks...) The second best implementation is found with FreeBSD and it is probably fairly mature but I haven''t tested it myself and some people have run into problems with it in the past. The Linux version is probably merely at infancy stage and likely not yet mature enough for regular use. It is probably not as "bad" as btrfs though. There is quite a bit of information about it on the phoronix.com website (and probably also at lwn.net): http://www.phoronix.com/scan.php?page=news_item&px=MTE4Nzc A search there on ZFS will give more articles. The latest official version of ZFS is 28 and is probably implemented in both Linux, and FreeBSD by now. Later versions have been released since Oracle killed the OpenSolaris project and can be found with the commercial closed-source Solaris platform that is supplied by Oracle. Things have happened since Oracle pulled the plug on OSOL project and leading developers behind the ZFS project such as Jeff Bonwick left Sun (after the acquisition by Oracle) and joined up with the Illumos team instead. So you cannot determine the stability of ZFS and ZPOOL merely by looking into the version number unfortunately and I wouldn''t expect the FreeBSD implementation to be as stable as the Solaris implementation. It just takes time for the implementation to mature and the bugs to be weeded out and it just happens to have been around for Solaris/OpenSolaris/Illumos for much longer than the other platforms and the Solaris/Illumos version also happens to get first dibs on the features. Among the Illumos people there is an ambition to drop the version numbering altogether and instead talk about available features. The recommendation to use SAS hard drives is not so much about the quality of the hard drives themselves as it is about the SAS protocol. The SAS protocol simply handles SCSI transport commands in a better and more reliable manner than do SATA. I believe any decent SAS drive would do. As for HBAs I wrote a list with LSI based hardware a while ago here: https://www.illumos.org/boards/1/topics/572 the thing is that a lot of OEMs such as IBM, HP, Cisco, Fujitsu-Siemens, Dell, ... supply their branded HBAs with LSI circuitry on them. What hardware to choose depends on what you''re looking for. If you want an 8-port controller I would go for Intel SASUC8I or LSI SAS3801E-R. If you want SAS/SATA3 with 6.0 Gb/s then LSI''s SAS 9200 series cards would be a better choice. I don''t know what OEMs have come up with in the SATA3 department since I wrote that list but the chips to look for in that case are the LSI MegaRAID 2004/2008/2016e depending on how many ports you want. If you want to read a further discussion about reliability of different RAID setups I made a post about this in the following thread (last post): http://communities.intel.com/thread/25945>> I admire your persistence with pursuing this undertaking and wish you >> the best of luck with it! >> Robin. >> >> > Thanks. I''ve invested too much time in research to not at least make > the attempt. Besides, if all else fails, I can fallback to a two box > solution. That is, if I can get my hypothetical virtualization box to > fit in my budget envelope... > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users > > . >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 09/24/2012 08:49 AM, Robin Axelsson wrote:> On 2012-09-24 05:45, ShadesOfGrey wrote: >> Sorry for the late response, I''ve had a lot to digest.<snip>>> The lack of current information about Xen (and KVM) online has been >> frustrating --- especially finding the many proof of concept videos >> that demonstrated possibilities but offered no real specifics. >> Looking for specifics, I sought information from gaming and >> enthusiast sites; I figured finding confirmation of VT-d and AMD-Vi >> support on such sites would be more likely. However, I found that >> wasn''t often the case. I did determine that ASRock motherboards seem >> to be the most likely to support VT-d, ASUS least likely (unless >> equipped with an Intel ''sanctioned'' VT-d chipset). I had narrowed my >> choices to two motherboards that appear to offer VT-d support and was >> intending to contact the manufacturer before purchase. Both choices >> are a bit pricey and I''ve been reconsidering whether I should look to >> other motherboards to reduce costs. > > Some motherboards support IOMMU even though it is not found in the > user manual or specified on the website. Your best bet is to ask > customer support. A guy posted here that he got it working on an Intel > motherboard that doesn''t even have options for it in the BIOS, so it > seems that in some cases it is only up to the CPU. This is not the > case with AMD though as I stated before. I have bought a couple of > Gigabyte GA990FX-UD7 myself, they are stable and have a good layout. > They have support for IOMMU but I haven''t tested it thoroughly enough > to fully confirm this although I don''t believe there would be any > problem. >I''m aware of this. In fact, I only have anecdotal evidence that the Gigabyte G1.Sniper 3 has IOMMU (VT-d) support. I intend to query the manufacturers, seeking confirmation of IOMMU support, of every motherboard that ends up on my short list. I may include other Z77 motherboards from MSI and Gigabyte, since there is concrete evidence that the Z77 chipset does support IOMMU. I did focus, however, on those motherboards I had some inkling could support IOMMU. Now I''m trying to expand my selection process to include less expensive options.> It surprises me that ASRock and ASUS are so different. ASRock is, or > at least used to be a subsidiary of ASUS so there shouldn''t be that > much difference between them.Same here. But I guess things changed after ASRock was spun-off. <snip>>> This is precisely the kind of information I was looking for from the >> threads I started on Ars Technica. It''s just unfortunate that FLR and >> D3 D0 support aren''t often found in the tech specs of must expansion >> hardware. However, now that I know what to ask, I''ll try contacting >> hardware manufacturers prior to purchasing any expansion hardware. >> Thank you! > > D3 and D0 are power states defined for devices in the ACPI > specification and can be used to control the supply voltage (Vcc) to > PCI and PCIe devices. You can find more information about it here for > example: > > http://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface > > --------------- > > > Device states > > The device states /D0/-/D3/ are device-dependent: > > * *D0* /Fully On/ is the operating state. > * *D1* and *D2* are intermediate power-states whose definition > varies by device. > * *D3* /Off/ has the device powered off and unresponsive to its bus. > > --------------- > > So, either it works for a certain type of hardware or it doesn''t and I > wouldn''t expect a vendor to state this "support" in the specifications > since it isn''t a "feature" in and of itself if you get me. But maybe > this will change and maybe FLR support will become more widespread.I did read that Wikipedia entry after your referenced excerpt. To my mind, if a manufacturer claims support for the ACPI or PCIe spec and doesn''t implement certain portions of those specs(some are optional after all), said manufacturer has an obligation to make that clear to your customers and users. But then, that''s just me. Anyway, this information gives me what I need to ask the right questions of tech support before purchasing. <snip>>> From everything I''ve read, solutions that rely on any form of remote >> display protocols would be limited to a subset of Direct3D functions. >> Furthermore, these would vary from one implementation to another, >> thus making them far less attractive for gaming than VGA >> passthrough... Well, in my opinion anyway. >> >> VirtualBox''s seamless mode is pretty nifty. But it''s a Type 2 >> Hypervisor and relies on paravirtualized drivers that also suffer >> from the same limitations as remote display protocols. It''s great for >> most things, but gaming is not one of them. And I''m speaking from >> personal experience. Though I haven''t used them myself, the same >> would seem to hold true of Parallel''s and VMWare''s ''Workstation'' >> offerings. At least, as far as I''ve gathered. >> >> FYI, the Type 1 Hypervisors from Parallel''s and VMWare* are priced >> waaayyy outside my budget. >> > I understand that you want full 3D functionality for Windows gaming > but maybe you''ll find the subset of 3D functionality for the Linux > machine acceptable. I have looked into VirtualGL and with TurboVNC, > you might get a pretty decent desktop environment and it seems like > most of the features are there already. It appears that the 3D is > rendered by hardware/GPU before it is streamed through VNC or Spice. > So it seems that you would need another GPU for that. You can find > more info on VirtualGL here: > > http://www.virtualgl.org/Upon further investigation, the only option that is available to remotely translate Windows 3D apps is Microsoft''s own RemoteFX. In which case, Microsoft products would be the foundation of my software stack... Something I''m trying to avoid.> > Also the line between a type 1 and type 2 hypervisor tend to get a bit > blurry. The point with type 1 is that it has access to ring-0 so that > it can get access directly to the hardware to be passed through to the > guests (I did confuse ''host'' and ''guest'' in my prior post). It also > doesn''t need to ask the host OS for permission in the same way as a > type 2 hypervisor which is likely to give performance advantages in > some cases. > > However, even a type 2 hypervisor, although it is run as an > application inside the OS can get "type 1" like privileges. By > patching into the kernel and/or using special "dummy drivers" for > hardware to be shared with VMs you can achieve pretty much the same > thing, ergo it is no longer clear whether the hypervisor is a type 1 > or type 2. > > There is an article about it from the old IBM Mainframe days but I > can''t seem to find it. >You might be right, the performance of a Type 2 Hypervisor may be sufficient. Regardless, I''d still have roughly the same hardware requirements as if I were going to use a Type 1 Hypervisor. I''d still need the same CPU, RAM, GPU, and storage requirements as I''ve already put forth. I could omit the potentially necessary additional components for use with a Type 1 Hypervisor. Things like USB controller, NIC, or sound card. But those costs would be replaced with the cost of licensing the Type2 Hypervisor. In which case, I don''t really gain or lose anything by experimenting with Type 1 Hypervisors... At any rate, I could compare and contrast the performance of Type 1 and Type 2 Hypervisors using Paralles'' and VMWare''s trialware. It might take a good long while, but it would be an adventure.>> >> *I only found out about VMWare''s ''free'' vSphere after I''d written >> this response. >><snip>>>> Also, it is highly recommended that you use ECC RAM for such >>> applications and it doesn''t hurt to dedicate a few gigs of it to the >>> ZFS as RAM is used for cache. The good news is that most >>> motherboards with good chipsets support ECC RAM even though you >>> might not find anything about it in the user manuals. >> Again, thanks for the thorough explanation. This gives me a great >> deal to think about. The more I learn about ZFS, the less appealing >> it becomes. And by that I mean the confusion over which version of >> ZFS is in what OS? And just how well maintained the OSes supporting >> ZFS are? Now I have additional hardware considerations to keep in >> mind that may (or may not) make the cost of ZFS RAID-Z pool >> comparable to a hardware RAID5/6 solution anyway. Do you have any >> suggestions as to which of LSI HBAs I should be considering? I >> haven''t found an HCL for ZFS in my searches. >> >> Out of curiosity --- and if you would happen to know --- do you think >> what you suggest about the HBA and SAS drives for ZFS also applies to >> Btrfs? I''m assuming it would, but I''d appreciate some confirmation. >> >> It''s funny how the "I" in RAID never really seems to apply... >> Especially since it looks more and more like using ZFS or Btrfs will >> require I commit myself, from the start, to one or the other and a >> discrete HBA. Transitioning from an integrated SATA controller(s) and >> mdadm seems rather impractical. If I understand what''s involved in >> doing so correctly. It may turn out that anything other than mdadm is >> price prohibitive. >> > > I don''t think you will have a problem with getting ZFS to run and if > that''s your only goal then you don''t need to be very picky with your > choice of hardware. I find ZFS pretty easy and handy to use. I has > really great functionality and I don''t have many bad things to say > about it so far. ZFS is a filesystem (along with a couple of software > tools to administrate it) just like EXT4 or NTFS so hardware support > depends on the platform it runs on. > > But the point with using ZFS is to get maximum protection against data > corruption and that''s where the selection of hardware gets limited and > there are "best practices" set up to achieve that. I have not tested > ZFS on any other platform than on OpenSolaris and OpenIndiana but I do > know that it is well implemented on that platform and more mature > there than on any other (non-solaris) platform. Another advantage with > the OSOL/OI platform is that the CIFS functionality is implemented in > the kernel space and not in the userland which will give advantages > performance wise if you intend to share files with other windows > computers. (I don''t deny that Samba is pretty good on Linux too. There > are some benchmarks on the phoronix website comparing samba with NFS > and they are in favor of Samba on those benchmarks...) The second best > implementation is found with FreeBSD and it is probably fairly mature > but I haven''t tested it myself and some people have run into problems > with it in the past. The Linux version is probably merely at infancy > stage and likely not yet mature enough for regular use. It is probably > not as "bad" as btrfs though. There is quite a bit of information > about it on the phoronix.com website (and probably also at lwn.net): > > http://www.phoronix.com/scan.php?page=news_item&px=MTE4Nzc >My goal wasn''t just to experiment with Btrfs, mdamd, or ZFS, if that''s what you meant by, "with getting ZFS to run and if that''s your only goal". I actually intend to use it ''in production''. In fact, the secondary role (Linux desktop being primary) for my proposed Virtualization rig is as file server (incl. httpd). Windows gaming is a tertiary or quaternary concern for me. So, finding out that the "best practices" for implementing ZFS include hardware I hadn''t anticipated, is a bit off-putting. The diversity in ZFS implementations is what I meant about being confused. I''m not as familiar with the underlying platforms that utilize ZFS (other than Linux, and not with ZFS in use). Without that familiarity, it makes it more difficult to gauge which of those platforms would suit my purposes. For example, I had read a little bit about the degraded CIFS performance on FreeBSD and Linux due to their reliance on Samba (it residing in user-space being the issue). BTW, I meant to ask how you came to the conclusion that ECC RAM is supported on desktop motherboards? It''s always been my understanding that you could use ECC RAM on such hardware, but there was no added benefit.> A search there on ZFS will give more articles. The latest official > version of ZFS is 28 and is probably implemented in both Linux, and > FreeBSD by now. Later versions have been released since Oracle killed > the OpenSolaris project and can be found with the commercial > closed-source Solaris platform that is supplied by Oracle. Things have > happened since Oracle pulled the plug on OSOL project and leading > developers behind the ZFS project such as Jeff Bonwick left Sun (after > the acquisition by Oracle) and joined up with the Illumos team > instead. So you cannot determine the stability of ZFS and ZPOOL merely > by looking into the version number unfortunately and I wouldn''t expect > the FreeBSD implementation to be as stable as the Solaris > implementation. It just takes time for the implementation to mature > and the bugs to be weeded out and it just happens to have been around > for Solaris/OpenSolaris/Illumos for much longer than the other > platforms and the Solaris/Illumos version also happens to get first > dibs on the features. Among the Illumos people there is an ambition to > drop the version numbering altogether and instead talk about available > features. >One of the two ZFS implementations on Linux has reached version 28, the other is at version 23 and seems to be abandoned or stagnant (last release was May 2011). I''m not quite sure what the status of ZFS on FreeBSD is. From this table <https://en.wikipedia.org/wiki/ZFS#Comparisons> on Wikipedia, ZFS is at version 28. However the table notes that there is no CIFS or iSCSI support, which I''ll try to independently confirm. And, as you say, the fact that ZFS (and Solaris as a whole) has essentially been forked, with the re-consolidation of Solaris as closed-source, just adds to the confusion. It''s possible any given build of Illumos'' version 28 of ZFS could have features or bug fixes no present in Oracle''s version 33.> The recommendation to use SAS hard drives is not so much about the > quality of the hard drives themselves as it is about the SAS protocol. > The SAS protocol simply handles SCSI transport commands in a better > and more reliable manner than do SATA. I believe any decent SAS drive > would do. As for HBAs I wrote a list with LSI based hardware a while > ago here: > > https://www.illumos.org/boards/1/topics/572 > > the thing is that a lot of OEMs such as IBM, HP, Cisco, > Fujitsu-Siemens, Dell, ... supply their branded HBAs with LSI > circuitry on them. What hardware to choose depends on what you''re > looking for. If you want an 8-port controller I would go for Intel > SASUC8I or LSI SAS3801E-R. If you want SAS/SATA3 with 6.0 Gb/s then > LSI''s LSI SAS3801E-R series cards would be a better choice. I don''t > know what OEMs have come up with in the SATA3 department since I wrote > that list but the chips to look for in that case are the LSI MegaRAID > 2004/2008/2016e depending on how many ports you want. > > If you want to read a further discussion about reliability of > different RAID setups I made a post about this in the following thread > (last post): > > http://communities.intel.com/thread/25945 >The cost of the drives is marginally higher than the drives I had budgeted, but including the cost of an SAS HBA is problematic. I hadn''t expected purchasing an HBA from the very start. I''d hoped to deffer such a purchase for a bit, but you are saying that would be ill-advised. How likely is it I would encounter problems using SATA drives in the interim? Have you seen any price/feature advantage to reseller versions of LSI OEM products? I would think the offerings from Cisco, Dell, HP, and IBM would come at a premium as opposed to purchasing LSI branded hardware. Anyway, I thought there might be specific models from particular manufacturers to meet the "best practices" for ZFS. However, from your suggestions, I gather it really doesn''t matter as long as the HBA is Intel or LSI? I have seen discussions in support of file system level redundancy like that of ZFS, precisely for the reason you cited. I guess I can add your link to the pro-FS pile of reference material. I wonder, should we perhaps move the discussion of ZFS to private email? It is a bit off-topic. <http://www.gigabyte.com/products/product-page.aspx?pid=4287#sp> _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 9/26/12 8:34 PM, ShadesOfGrey wrote:> On 09/24/2012 08:49 AM, Robin Axelsson wrote: >> On 2012-09-24 05:45, ShadesOfGrey wrote: >>> Sorry for the late response, I''ve had a lot to digest. > <snip> >>> The lack of current information about Xen (and KVM) online has been >>> frustrating — especially finding the many proof of concept videos >>> that demonstrated possibilities but offered no real specifics. >>> Looking for specifics, I sought information from gaming and >>> enthusiast sites; I figured finding confirmation of VT-d and AMD-Vi >>> support on such sites would be more likely. However, I found that >>> wasn''t often the case. I did determine that ASRock motherboards seem >>> to be the most likely to support VT-d, ASUS least likely (unless >>> equipped with an Intel ''sanctioned'' VT-d chipset). I had narrowed my >>> choices to two motherboards that appear to offer VT-d support and was >>> intending to contact the manufacturer before purchase. Both choices >>> are a bit pricey and I''ve been reconsidering whether I should look to >>> other motherboards to reduce costs. >> >> Some motherboards support IOMMU even though it is not found in the >> user manual or specified on the website. Your best bet is to ask >> customer support. A guy posted here that he got it working on an Intel >> motherboard that doesn''t even have options for it in the BIOS, so it >> seems that in some cases it is only up to the CPU. This is not the >> case with AMD though as I stated before. I have bought a couple of >> Gigabyte GA990FX-UD7 myself, they are stable and have a good layout. >> They have support for IOMMU but I haven''t tested it thoroughly enough >> to fully confirm this although I don''t believe there would be any >> problem. >> > I''m aware of this. In fact, I only have anecdotal evidence that the > Gigabyte G1.Sniper 3 has IOMMU (VT-d) support. I intend to query the > manufacturers, seeking confirmation of IOMMU support, of every > motherboard that ends up on my short list. I may include other Z77 > motherboards from MSI and Gigabyte, since there is concrete evidence > that the Z77 chipset does support IOMMU. I did focus, however, on those > motherboards I had some inkling could support IOMMU. Now I''m trying to > expand my selection process to include less expensive options.I have a Gigabyte GA-Z77-DS3H, which I can confirm does support IOMMU ... I''m passing PCI video tuner cards to two different DomUs and a PCI-Express 1x USB card to another DomU. I also have an Intel DH77KC that does IOMMU but the H77 chipset support was (is?) buggy in the current mainstream kernel releases so it''s still in the box on the shelf. -Robert
On 09/26/2012 09:57 PM, Robert Rust wrote:> On 9/26/12 8:34 PM, ShadesOfGrey wrote: >> On 09/24/2012 08:49 AM, Robin Axelsson wrote: >>> On 2012-09-24 05:45, ShadesOfGrey wrote: >>>> Sorry for the late response, I''ve had a lot to digest. >> <snip> >>>> The lack of current information about Xen (and KVM) online has been >>>> frustrating — especially finding the many proof of concept videos >>>> that demonstrated possibilities but offered no real specifics. >>>> Looking for specifics, I sought information from gaming and >>>> enthusiast sites; I figured finding confirmation of VT-d and AMD-Vi >>>> support on such sites would be more likely. However, I found that >>>> wasn''t often the case. I did determine that ASRock motherboards seem >>>> to be the most likely to support VT-d, ASUS least likely (unless >>>> equipped with an Intel ''sanctioned'' VT-d chipset). I had narrowed my >>>> choices to two motherboards that appear to offer VT-d support and was >>>> intending to contact the manufacturer before purchase. Both choices >>>> are a bit pricey and I''ve been reconsidering whether I should look to >>>> other motherboards to reduce costs. >>> >>> Some motherboards support IOMMU even though it is not found in the >>> user manual or specified on the website. Your best bet is to ask >>> customer support. A guy posted here that he got it working on an Intel >>> motherboard that doesn''t even have options for it in the BIOS, so it >>> seems that in some cases it is only up to the CPU. This is not the >>> case with AMD though as I stated before. I have bought a couple of >>> Gigabyte GA990FX-UD7 myself, they are stable and have a good layout. >>> They have support for IOMMU but I haven''t tested it thoroughly enough >>> to fully confirm this although I don''t believe there would be any >>> problem. >>> >> I''m aware of this. In fact, I only have anecdotal evidence that the >> Gigabyte G1.Sniper 3 has IOMMU (VT-d) support. I intend to query the >> manufacturers, seeking confirmation of IOMMU support, of every >> motherboard that ends up on my short list. I may include other Z77 >> motherboards from MSI and Gigabyte, since there is concrete evidence >> that the Z77 chipset does support IOMMU. I did focus, however, on those >> motherboards I had some inkling could support IOMMU. Now I''m trying to >> expand my selection process to include less expensive options. > > I have a Gigabyte GA-Z77-DS3H, which I can confirm does support IOMMU > ... I''m passing PCI video tuner cards to two different DomUs and a > PCI-Express 1x USB card to another DomU. > I also have an Intel DH77KC that does IOMMU but the H77 chipset > support was (is?) buggy in the current mainstream kernel releases so > it''s still in the box on the shelf. > > -Robert > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >The GA-Z77-DS3H doesn''t fit my particular needs, but I appreciate your confirming IOMMU support on that motherboard. If you ever get IOMMU working on the DH77KC, please let us know.
On 2012-09-27 03:34, ShadesOfGrey wrote:> On 09/24/2012 08:49 AM, Robin Axelsson wrote: >> On 2012-09-24 05:45, ShadesOfGrey wrote: >>> Sorry for the late response, I''ve had a lot to digest. > <snip> >>> The lack of current information about Xen (and KVM) online has been >>> frustrating --- especially finding the many proof of concept videos >>> that demonstrated possibilities but offered no real specifics. >>> Looking for specifics, I sought information from gaming and >>> enthusiast sites; I figured finding confirmation of VT-d and AMD-Vi >>> support on such sites would be more likely. However, I found that >>> wasn''t often the case. I did determine that ASRock motherboards seem >>> to be the most likely to support VT-d, ASUS least likely (unless >>> equipped with an Intel ''sanctioned'' VT-d chipset). I had narrowed my >>> choices to two motherboards that appear to offer VT-d support and >>> was intending to contact the manufacturer before purchase. Both >>> choices are a bit pricey and I''ve been reconsidering whether I >>> should look to other motherboards to reduce costs. >> >> Some motherboards support IOMMU even though it is not found in the >> user manual or specified on the website. Your best bet is to ask >> customer support. A guy posted here that he got it working on an >> Intel motherboard that doesn''t even have options for it in the BIOS, >> so it seems that in some cases it is only up to the CPU. This is not >> the case with AMD though as I stated before. I have bought a couple >> of Gigabyte GA990FX-UD7 myself, they are stable and have a good >> layout. They have support for IOMMU but I haven''t tested it >> thoroughly enough to fully confirm this although I don''t believe >> there would be any problem. >> > I''m aware of this. In fact, I only have anecdotal evidence that the > Gigabyte G1.Sniper 3 has IOMMU (VT-d) support. I intend to query the > manufacturers, seeking confirmation of IOMMU support, of every > motherboard that ends up on my short list. I may include other Z77 > motherboards from MSI and Gigabyte, since there is concrete evidence > that the Z77 chipset does support IOMMU. I did focus, however, on > those motherboards I had some inkling could support IOMMU. Now I''m > trying to expand my selection process to include less expensive options. >> It surprises me that ASRock and ASUS are so different. ASRock is, or >> at least used to be a subsidiary of ASUS so there shouldn''t be that >> much difference between them. > Same here. But I guess things changed after ASRock was spun-off. > > <snip> >>> This is precisely the kind of information I was looking for from the >>> threads I started on Ars Technica. It''s just unfortunate that FLR >>> and D3 D0 support aren''t often found in the tech specs of must >>> expansion hardware. However, now that I know what to ask, I''ll try >>> contacting hardware manufacturers prior to purchasing any expansion >>> hardware. Thank you! >> >> D3 and D0 are power states defined for devices in the ACPI >> specification and can be used to control the supply voltage (Vcc) to >> PCI and PCIe devices. You can find more information about it here for >> example: >> >> http://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface >> >> --------------- >> >> >> Device states >> >> The device states /D0/-/D3/ are device-dependent: >> >> * *D0* /Fully On/ is the operating state. >> * *D1* and *D2* are intermediate power-states whose definition >> varies by device. >> * *D3* /Off/ has the device powered off and unresponsive to its bus. >> >> --------------- >> >> So, either it works for a certain type of hardware or it doesn''t and >> I wouldn''t expect a vendor to state this "support" in the >> specifications since it isn''t a "feature" in and of itself if you get >> me. But maybe this will change and maybe FLR support will become more >> widespread. > I did read that Wikipedia entry after your referenced excerpt. To my > mind, if a manufacturer claims support for the ACPI or PCIe spec and > doesn''t implement certain portions of those specs(some are optional > after all), said manufacturer has an obligation to make that clear to > your customers and users. But then, that''s just me. Anyway, this > information gives me what I need to ask the right questions of tech > support before purchasing. >I think most motherboards should support this. I''ve also had it confirmed by nVidia that they support FLR on all current Quadro cards greater than or equal to Quadro 2000, on Tesla C2050 and higher and M2050 and higher, and on new VGX and GRID cards.> <snip> >>> From everything I''ve read, solutions that rely on any form of remote >>> display protocols would be limited to a subset of Direct3D >>> functions. Furthermore, these would vary from one implementation to >>> another, thus making them far less attractive for gaming than VGA >>> passthrough... Well, in my opinion anyway. >>> >>> VirtualBox''s seamless mode is pretty nifty. But it''s a Type 2 >>> Hypervisor and relies on paravirtualized drivers that also suffer >>> from the same limitations as remote display protocols. It''s great >>> for most things, but gaming is not one of them. And I''m speaking >>> from personal experience. Though I haven''t used them myself, the >>> same would seem to hold true of Parallel''s and VMWare''s >>> ''Workstation'' offerings. At least, as far as I''ve gathered. >>> >>> FYI, the Type 1 Hypervisors from Parallel''s and VMWare* are priced >>> waaayyy outside my budget. >>> >> I understand that you want full 3D functionality for Windows gaming >> but maybe you''ll find the subset of 3D functionality for the Linux >> machine acceptable. I have looked into VirtualGL and with TurboVNC, >> you might get a pretty decent desktop environment and it seems like >> most of the features are there already. It appears that the 3D is >> rendered by hardware/GPU before it is streamed through VNC or Spice. >> So it seems that you would need another GPU for that. You can find >> more info on VirtualGL here: >> >> http://www.virtualgl.org/ > Upon further investigation, the only option that is available to > remotely translate Windows 3D apps is Microsoft''s own RemoteFX. In > which case, Microsoft products would be the foundation of my software > stack... Something I''m trying to avoid.I was just giving you the options I could think of OTOH that would let you share a Windows desktop with a Linux desktop, or rather share the desktop of one VM with the host machine.>> >> Also the line between a type 1 and type 2 hypervisor tend to get a >> bit blurry. The point with type 1 is that it has access to ring-0 so >> that it can get access directly to the hardware to be passed through >> to the guests (I did confuse ''host'' and ''guest'' in my prior post). It >> also doesn''t need to ask the host OS for permission in the same way >> as a type 2 hypervisor which is likely to give performance advantages >> in some cases. >> >> However, even a type 2 hypervisor, although it is run as an >> application inside the OS can get "type 1" like privileges. By >> patching into the kernel and/or using special "dummy drivers" for >> hardware to be shared with VMs you can achieve pretty much the same >> thing, ergo it is no longer clear whether the hypervisor is a type 1 >> or type 2. >> >> There is an article about it from the old IBM Mainframe days but I >> can''t seem to find it. >> > You might be right, the performance of a Type 2 Hypervisor may be > sufficient. Regardless, I''d still have roughly the same hardware > requirements as if I were going to use a Type 1 Hypervisor. I''d still > need the same CPU, RAM, GPU, and storage requirements as I''ve already > put forth. I could omit the potentially necessary additional > components for use with a Type 1 Hypervisor. Things like USB > controller, NIC, or sound card. But those costs would be replaced > with the cost of licensing the Type2 Hypervisor. > > In which case, I don''t really gain or lose anything by experimenting > with Type 1 Hypervisors... At any rate, I could compare and contrast > the performance of Type 1 and Type 2 Hypervisors using Paralles'' and > VMWare''s trialware. It might take a good long while, but it would be > an adventure. >>> >>> *I only found out about VMWare''s ''free'' vSphere after I''d written >>> this response. >>> > <snip> >>>> Also, it is highly recommended that you use ECC RAM for such >>>> applications and it doesn''t hurt to dedicate a few gigs of it to >>>> the ZFS as RAM is used for cache. The good news is that most >>>> motherboards with good chipsets support ECC RAM even though you >>>> might not find anything about it in the user manuals. >>> Again, thanks for the thorough explanation. This gives me a great >>> deal to think about. The more I learn about ZFS, the less appealing >>> it becomes. And by that I mean the confusion over which version of >>> ZFS is in what OS? And just how well maintained the OSes supporting >>> ZFS are? Now I have additional hardware considerations to keep in >>> mind that may (or may not) make the cost of ZFS RAID-Z pool >>> comparable to a hardware RAID5/6 solution anyway. Do you have any >>> suggestions as to which of LSI HBAs I should be considering? I >>> haven''t found an HCL for ZFS in my searches. >>> >>> Out of curiosity --- and if you would happen to know --- do you >>> think what you suggest about the HBA and SAS drives for ZFS also >>> applies to Btrfs? I''m assuming it would, but I''d appreciate some >>> confirmation. >>> >>> It''s funny how the "I" in RAID never really seems to apply... >>> Especially since it looks more and more like using ZFS or Btrfs will >>> require I commit myself, from the start, to one or the other and a >>> discrete HBA. Transitioning from an integrated SATA controller(s) >>> and mdadm seems rather impractical. If I understand what''s involved >>> in doing so correctly. It may turn out that anything other than >>> mdadm is price prohibitive. >>> >> >> I don''t think you will have a problem with getting ZFS to run and if >> that''s your only goal then you don''t need to be very picky with your >> choice of hardware. I find ZFS pretty easy and handy to use. I has >> really great functionality and I don''t have many bad things to say >> about it so far. ZFS is a filesystem (along with a couple of software >> tools to administrate it) just like EXT4 or NTFS so hardware support >> depends on the platform it runs on. >> >> But the point with using ZFS is to get maximum protection against >> data corruption and that''s where the selection of hardware gets >> limited and there are "best practices" set up to achieve that. I have >> not tested ZFS on any other platform than on OpenSolaris and >> OpenIndiana but I do know that it is well implemented on that >> platform and more mature there than on any other (non-solaris) >> platform. Another advantage with the OSOL/OI platform is that the >> CIFS functionality is implemented in the kernel space and not in the >> userland which will give advantages performance wise if you intend to >> share files with other windows computers. (I don''t deny that Samba is >> pretty good on Linux too. There are some benchmarks on the phoronix >> website comparing samba with NFS and they are in favor of Samba on >> those benchmarks...) The second best implementation is found with >> FreeBSD and it is probably fairly mature but I haven''t tested it >> myself and some people have run into problems with it in the past. >> The Linux version is probably merely at infancy stage and likely not >> yet mature enough for regular use. It is probably not as "bad" as >> btrfs though. There is quite a bit of information about it on the >> phoronix.com website (and probably also at lwn.net): >> >> http://www.phoronix.com/scan.php?page=news_item&px=MTE4Nzc >> > My goal wasn''t just to experiment with Btrfs, mdamd, or ZFS, if that''s > what you meant by, "with getting ZFS to run and if that''s your only > goal". I actually intend to use it ''in production''. In fact, the > secondary role (Linux desktop being primary) for my proposed > Virtualization rig is as file server (incl. httpd). Windows gaming is > a tertiary or quaternary concern for me. So, finding out that the > "best practices" for implementing ZFS include hardware I hadn''t > anticipated, is a bit off-putting. > > The diversity in ZFS implementations is what I meant about being > confused. I''m not as familiar with the underlying platforms that > utilize ZFS (other than Linux, and not with ZFS in use). Without that > familiarity, it makes it more difficult to gauge which of those > platforms would suit my purposes. For example, I had read a little > bit about the degraded CIFS performance on FreeBSD and Linux due to > their reliance on Samba (it residing in user-space being the issue). > > BTW, I meant to ask how you came to the conclusion that ECC RAM is > supported on desktop motherboards? It''s always been my understanding > that you could use ECC RAM on such hardware, but there was no added > benefit.When I looked at the Gigabyte GA990FX motherboard there was no documentation about it anywhere, nor was it stated in the specifications. When I contacted customer support and asked about it, they sent me screen dumps of BIOS showing that it in fact does support ECC providing different ECC scrubbing options. So you can ask support if you are unsure. The benefit of ECC is that there is a parity bit that checks the integrity of the data that is present in the RAM. We have cosmic radiation and bit flips do occur, also the memory can turn out to be faulty. It has been debated whether we really "need" this on a desktop. The finer litography of the hardware is likely to make it more sensitive to such failures and bit flips than before so it makes more sense to use ECC today than in the past. If you are unlucky, you computer will crash because of that. Some files may get corrupted in the process. It may not happen very often but it does happen occasionally, hopefully, the bit flips happen in an address space that currently is not in use. If you run a server on the other hand then you are likely to run into freezes and crashes eventually (it may depend on how long uptimes were talking about). Using ECC RAM will prevent those crashes and add extra protection. In fact, Microsoft recommends ECC RAM even on desktop computers. ZFS does offer protection against data corruption but if you are unlucky, some data corruption may go by undetected when written from RAM. ZFS (or any other filesystem) will write the damaged data to disk and be unable to automatically detect the corruption. This is why using ECC RAM with ZFS is "strongly recommended" in the best practices.>> A search there on ZFS will give more articles. The latest official >> version of ZFS is 28 and is probably implemented in both Linux, and >> FreeBSD by now. Later versions have been released since Oracle killed >> the OpenSolaris project and can be found with the commercial >> closed-source Solaris platform that is supplied by Oracle. Things >> have happened since Oracle pulled the plug on OSOL project and >> leading developers behind the ZFS project such as Jeff Bonwick left >> Sun (after the acquisition by Oracle) and joined up with the Illumos >> team instead. So you cannot determine the stability of ZFS and ZPOOL >> merely by looking into the version number unfortunately and I >> wouldn''t expect the FreeBSD implementation to be as stable as the >> Solaris implementation. It just takes time for the implementation to >> mature and the bugs to be weeded out and it just happens to have been >> around for Solaris/OpenSolaris/Illumos for much longer than the other >> platforms and the Solaris/Illumos version also happens to get first >> dibs on the features. Among the Illumos people there is an ambition >> to drop the version numbering altogether and instead talk about >> available features. >> > One of the two ZFS implementations on Linux has reached version 28, > the other is at version 23 and seems to be abandoned or stagnant (last > release was May 2011). I''m not quite sure what the status of ZFS on > FreeBSD is. From this table > <https://en.wikipedia.org/wiki/ZFS#Comparisons> on Wikipedia, ZFS is > at version 28. However the table notes that there is no CIFS or iSCSI > support, which I''ll try to independently confirm. And, as you say, > the fact that ZFS (and Solaris as a whole) has essentially been > forked, with the re-consolidation of Solaris as closed-source, just > adds to the confusion. It''s possible any given build of Illumos'' > version 28 of ZFS could have features or bug fixes no present in > Oracle''s version 33.I don''t know much about FreeBSD but in Solaris/Illumos the CIFS is implemented in the kernel space, in platforms such as Linux and BSD it is provided by the Samba framework in the userspace. If you want iSCSI then you have full support for it in Illumos based operating systems. Also keep in mind that later version storage pools are not readable on systems with lower versions of zfs/zpool. zpool 28 is implemented in FreeBSD 9.0 and onwards. To my knowledge the ZFSonLinux project also has implemented zpool version 28. It should be noted that people also have had problems with migrating a ZFS pool from one platform to another even though the version numbers matched.>> The recommendation to use SAS hard drives is not so much about the >> quality of the hard drives themselves as it is about the SAS >> protocol. The SAS protocol simply handles SCSI transport commands in >> a better and more reliable manner than do SATA. I believe any decent >> SAS drive would do. As for HBAs I wrote a list with LSI based >> hardware a while ago here: >> >> https://www.illumos.org/boards/1/topics/572 >> >> the thing is that a lot of OEMs such as IBM, HP, Cisco, >> Fujitsu-Siemens, Dell, ... supply their branded HBAs with LSI >> circuitry on them. What hardware to choose depends on what you''re >> looking for. If you want an 8-port controller I would go for Intel >> SASUC8I or LSI SAS3801E-R. If you want SAS/SATA3 with 6.0 Gb/s then >> LSI''s LSI SAS3801E-R series cards would be a better choice. I don''t >> know what OEMs have come up with in the SATA3 department since I >> wrote that list but the chips to look for in that case are the LSI >> MegaRAID 2004/2008/2016e depending on how many ports you want. >> >> If you want to read a further discussion about reliability of >> different RAID setups I made a post about this in the following >> thread (last post): >> >> http://communities.intel.com/thread/25945 >> > The cost of the drives is marginally higher than the drives I had > budgeted, but including the cost of an SAS HBA is problematic. I > hadn''t expected purchasing an HBA from the very start. I''d hoped to > deffer such a purchase for a bit, but you are saying that would be > ill-advised. How likely is it I would encounter problems using SATA > drives in the interim? >The problem with "normal" hardware is that the error handling is internal. If a hard drive is starting to have bad sectors, the drive will handle them internally and reallocate the bad sectors. You will not notice anything and the errors will not be reported to the system. At most you will experience sluggish performance, perhaps a freeze. When things go so bad that errors are starting to become visible, then it is too late to do anything about it, the drive is already dead. In the past the drives would return "garbage", in the modern days the drive will reread the sector over and over again while trying to recalibrate the head which gives rise tio the characteristic click-of-death noise. If you use a "cheap" HBA with say ZFS, the tools for monitoring the hardware such as iostat will be pretty much useless. Things will ''look'' just fine with no errors and when errors start to show, they will happen out of the blue with no prior warning. The system drive of a ZFS file server that I run just crashed like that without warning. Prior to that, all I had on the drive were some corrupt blocks of the hard drive image of a virtual machine. It was connected to the Southbridge OnChip SATA controller of the motherboard. I didn''t cry about it although there were some system files that I would have liked to keep though. I replaced it with a server grade SATA drive (WD RE4). I have suffered corruption on that drive on one of the virtual hard drive images. Since I don''t see much sense in using RAID on the system drive (other people may beg to differ) I enabled the "ditto blocks" feature on that drive and recovered the image file using ''ddrescue''. The drive is doing fine ever since. On the storage pool of that system I have already replaced two SATA drives on my SAS controller bacause they were going to die. What happened was that the server started to freeze but I couldn''t tell what was the problem. Everything looked ok with iostat, was it a driver issue? I couldn''t tell. Then eventually I started to see ''media errors'' with iostat and when I replaced the drive the pool didn''t freeze anymore. Then the same thing happened again and I replaced another hard drive like that. That''s the reality of that, if the drives were SAS I would have been able to tell what''s wrong at an earlier stage. There was a discussion about this about 6 months ago on the openindiana-discuss mailing list.> Have you seen any price/feature advantage to reseller versions of LSI > OEM products? I would think the offerings from Cisco, Dell, HP, and > IBM would come at a premium as opposed to purchasing LSI branded > hardware. Anyway, I thought there might be specific models from > particular manufacturers to meet the "best practices" for ZFS. > However, from your suggestions, I gather it really doesn''t matter as > long as the HBA is Intel or LSI? >That''s something you have to look for at your dealers. My experience is that the OEM versions generally are cheaper than LSI branded products. You can compare yourself here: Intel SASUC8I http://amzn.to/QjtOR9 LSI 3081E-R http://amzn.to/USps9x Both are essentially the same card but the original LSI one is more expensive than the one from Intel.> I wonder, should we perhaps move the discussion of ZFS to private > email? It is a bit off-topic.I think a better idea is to take this to the openindiana-discuss mailing list and/or the FreeNAS mailing lists/forums as other people with ZFS experience will read your posts. If you post on the openindiana-discuss I will read your post there as well.> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users