libvirt users - Oct 2014 - Re: Virtual gaming/workstation desktops using KVM VGA passthrough?

GizmoChicken,

Read that there's a problem with Nvidia GeForce GTX cards and both Xen and
KVM, like it's not on a Nvidia whitelist, I'm a bit concerned.  From the
ArchLinux thread I've seen some success with Nvidia GeForce GTX cards, this
is confusing.  The bit I read here was:

" That would mean you are stuck with the GTX for that, but Nvidia nerfs
their cards to stop passthrough, attempting to force you to pay for a
special card that allows it, costing in the neighborhood of like 2000$ for
a cheap one. "
http://www.reddit.com/r/linux/comments/23lpa6/question_where_are_we_with_video_card_passthrough/

Do I need a Quadro for this?  In that case it would be impossible to
justify.  Originally the GeForce GTX 750 Ti seemed like the ideal card for
this type of deployment.  Now I'm wondering if AMD is safer for the VDI
guest.  What should I do?

CC'd Alex Williamson into the thread so we can get some verification here
that Nvidia GeForce GTX are in fact able to do VGA passthrough with a stock
Fedora 20 virtualization host using VFIO with a KVM hypervisor.

Thanks.

On Sat, Oct 4, 2014 at 5:13 PM, Gizmo Chicken <gizmochicken@gmail.com>
wrote:
> Hi Alex,
>
> The first bit of advice that I was about to give was that you should
> have a look at a particular thread on the Arch Linux user forum.  But
> as it turns out, the article that you found (which seems really
> well-written) already mentions that thread, and in fact even links to
> one of my comments on that thread, namely this comment:
> https://bbs.archlinux.org/viewtopic.php?pid=1313007#p1313007
>
> With regard to CentOS 7 vs Ubuntu 14.04, keep in mind that VFIO (which
> is part of the Linux kernel) is relatively new and is still rapidly
> evolving.
>
> I'm not very familiar with CentOS, but if I'm not mistaken, CentOS
7
> uses kernel 3.10 by default.  You'll probably want to use a newer
> kernel version, which would have better VFIO support.  Again, I'm not
> really familiar with CentOS, but my understanding is that many feel
> that updating the kernel in CentOS isn't a good idea. So with this in
> mind, you might be better off using a distro that ships with a newer
> kernel version, such as Fedora or Ubuntu, etc.
>
> In my experience, Ubuntu 14.04, which ships with kernel 3.13 and QEMU
> 2.0, works pretty well with VFIO.  However, Ubuntu 14.10, which will
> ship with kernel 3.16 and QEMU 2.1, seems to work a little better, but
> is still pre-release and so has a few other bugs.  If you go with
> Ubuntu 14.04, you might want to install a newer "mainline"
kernel.
> But in any case, please do keep in mind that VFIO is still new and so
> you will almost assuredly encounter some bugs.  So don't expect 100%
> stability.
>
> Hope the above is of some help.
>
> Regards,
> GizmoChicken
>
> On Fri, Oct 3, 2014 at 5:19 PM, Alex G.S. <alxgrtnstrngl@gmail.com>
wrote:
> > Dear List,
> >
> > Recently I read an article [1] about how to use KVM and VGA
passthrough
> to
> > create multiple GPU accelerated VM's for use in gaming or as
virtual
> > workstations:
> >
> > [1]
> >
>
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
> >
> > I'm planning on building a KVM workstation to do this using CentOS
7 or
> > Ubuntu 14.04 LTS as the base virtualization host and then an Intel
Xeon
> CPU
> > and two Nvidia GeForce GTX 750's as GPU's for the virtual
machines.
> >
> > I'm really new to this sort of GPU passthrough on KVM and am
wondering if
> > anyone has any advice or has done this before?
> >
> > Thank you!
> >
> >
> >
> >
> > _______________________________________________
> > libvirt-users mailing list
> > libvirt-users@redhat.com
> > https://www.redhat.com/mailman/listinfo/libvirt-users
>

On Mon, 2014-10-06 at 22:59 -0400, Alex G.S. wrote:
> GizmoChicken,
> 
> Read that there's a problem with Nvidia GeForce GTX cards and both Xen
and
> KVM, like it's not on a Nvidia whitelist, I'm a bit concerned. 
From the
> ArchLinux thread I've seen some success with Nvidia GeForce GTX cards,
this
> is confusing.  The bit I read here was:
> 
> " That would mean you are stuck with the GTX for that, but Nvidia
nerfs
> their cards to stop passthrough, attempting to force you to pay for a
> special card that allows it, costing in the neighborhood of like 2000$ for
> a cheap one. "
>
http://www.reddit.com/r/linux/comments/23lpa6/question_where_are_we_with_video_card_passthrough/
> 
> Do I need a Quadro for this?  In that case it would be impossible to
> justify.  Originally the GeForce GTX 750 Ti seemed like the ideal card for
> this type of deployment.  Now I'm wondering if AMD is safer for the VDI
> guest.  What should I do?
> 
> CC'd Alex Williamson into the thread so we can get some verification
here
> that Nvidia GeForce GTX are in fact able to do VGA passthrough with a stock
> Fedora 20 virtualization host using VFIO with a KVM hypervisor.
If you want to use a stock distro kernel then you need to use OVMF
(UEFI) for the guest or get lucky and not use IGD on the host (and use a
very new kernel and use a wrapper around qemu for the x-vga option).
See:

http://vfio.blogspot.com/2014/08/primary-graphics-assignment-without-vga.html
http://vfio.blogspot.com/2014/09/libvirt-now-supports-ovmf.html
http://vfio.blogspot.com/2014/09/ovmf-split-image-support.html

Nvidia has been throwing us some curve balls with their driver and
hypervisor detection, but there are still combinations that work.  For
the latest driver it's required to hide the KVM hypervisor signature and
disable Hyper-V enlightenment.  If you're looking for something
supported in a VM, Quadro is the only game in town afaik.  I have a
GTX750 (not Ti) that works great on stock Fedora 20 kernel, upstream
qemu + ovmf + libvirt, win8.1 guest (obligatory ymmv).  Thanks,

Alex
> On Sat, Oct 4, 2014 at 5:13 PM, Gizmo Chicken
<gizmochicken@gmail.com>
> wrote:
> 
> > Hi Alex,
> >
> > The first bit of advice that I was about to give was that you should
> > have a look at a particular thread on the Arch Linux user forum.  But
> > as it turns out, the article that you found (which seems really
> > well-written) already mentions that thread, and in fact even links to
> > one of my comments on that thread, namely this comment:
> > https://bbs.archlinux.org/viewtopic.php?pid=1313007#p1313007
> >
> > With regard to CentOS 7 vs Ubuntu 14.04, keep in mind that VFIO (which
> > is part of the Linux kernel) is relatively new and is still rapidly
> > evolving.
> >
> > I'm not very familiar with CentOS, but if I'm not mistaken,
CentOS 7
> > uses kernel 3.10 by default.  You'll probably want to use a newer
> > kernel version, which would have better VFIO support.  Again, I'm
not
> > really familiar with CentOS, but my understanding is that many feel
> > that updating the kernel in CentOS isn't a good idea. So with this
in
> > mind, you might be better off using a distro that ships with a newer
> > kernel version, such as Fedora or Ubuntu, etc.
> >
> > In my experience, Ubuntu 14.04, which ships with kernel 3.13 and QEMU
> > 2.0, works pretty well with VFIO.  However, Ubuntu 14.10, which will
> > ship with kernel 3.16 and QEMU 2.1, seems to work a little better, but
> > is still pre-release and so has a few other bugs.  If you go with
> > Ubuntu 14.04, you might want to install a newer "mainline"
kernel.
> > But in any case, please do keep in mind that VFIO is still new and so
> > you will almost assuredly encounter some bugs.  So don't expect
100%
> > stability.
> >
> > Hope the above is of some help.
> >
> > Regards,
> > GizmoChicken
> >
> > On Fri, Oct 3, 2014 at 5:19 PM, Alex G.S.
<alxgrtnstrngl@gmail.com> wrote:
> > > Dear List,
> > >
> > > Recently I read an article [1] about how to use KVM and VGA
passthrough
> > to
> > > create multiple GPU accelerated VM's for use in gaming or as
virtual
> > > workstations:
> > >
> > > [1]
> > >
> >
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
> > >
> > > I'm planning on building a KVM workstation to do this using
CentOS 7 or
> > > Ubuntu 14.04 LTS as the base virtualization host and then an
Intel Xeon
> > CPU
> > > and two Nvidia GeForce GTX 750's as GPU's for the virtual
machines.
> > >
> > > I'm really new to this sort of GPU passthrough on KVM and am
wondering if
> > > anyone has any advice or has done this before?
> > >
> > > Thank you!
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > libvirt-users mailing list
> > > libvirt-users@redhat.com
> > > https://www.redhat.com/mailman/listinfo/libvirt-users
> >

>
> Nvidia has been throwing us some curve balls with their driver and
> hypervisor detection, but there are still combinations that work.

Okay, thanks for the information.  Would it better then to go with an AMD
Radeon R7 or R9 for VGA passthrough to the Linux or Windows VM?  Are they
problem free and more predictable when it comes to passthrough?

Best,
AlexGS

On Mon, Oct 6, 2014 at 11:46 PM, Alex Williamson
<alex.williamson@redhat.com
> wrote:
> On Mon, 2014-10-06 at 22:59 -0400, Alex G.S. wrote:
> > GizmoChicken,
> >
> > Read that there's a problem with Nvidia GeForce GTX cards and both
Xen
> and
> > KVM, like it's not on a Nvidia whitelist, I'm a bit concerned.
From the
> > ArchLinux thread I've seen some success with Nvidia GeForce GTX
cards,
> this
> > is confusing.  The bit I read here was:
> >
> > " That would mean you are stuck with the GTX for that, but Nvidia
nerfs
> > their cards to stop passthrough, attempting to force you to pay for a
> > special card that allows it, costing in the neighborhood of like 2000$
> for
> > a cheap one. "
> >
>
http://www.reddit.com/r/linux/comments/23lpa6/question_where_are_we_with_video_card_passthrough/
> >
> > Do I need a Quadro for this?  In that case it would be impossible to
> > justify.  Originally the GeForce GTX 750 Ti seemed like the ideal card
> for
> > this type of deployment.  Now I'm wondering if AMD is safer for
the VDI
> > guest.  What should I do?
> >
> > CC'd Alex Williamson into the thread so we can get some
verification here
> > that Nvidia GeForce GTX are in fact able to do VGA passthrough with a
> stock
> > Fedora 20 virtualization host using VFIO with a KVM hypervisor.
>
> If you want to use a stock distro kernel then you need to use OVMF
> (UEFI) for the guest or get lucky and not use IGD on the host (and use a
> very new kernel and use a wrapper around qemu for the x-vga option).
> See:
>
>
>
http://vfio.blogspot.com/2014/08/primary-graphics-assignment-without-vga.html
> http://vfio.blogspot.com/2014/09/libvirt-now-supports-ovmf.html
> http://vfio.blogspot.com/2014/09/ovmf-split-image-support.html
>
> Nvidia has been throwing us some curve balls with their driver and
> hypervisor detection, but there are still combinations that work.  For
> the latest driver it's required to hide the KVM hypervisor signature
and
> disable Hyper-V enlightenment.  If you're looking for something
> supported in a VM, Quadro is the only game in town afaik.  I have a
> GTX750 (not Ti) that works great on stock Fedora 20 kernel, upstream
> qemu + ovmf + libvirt, win8.1 guest (obligatory ymmv).  Thanks,
>
> Alex
>
> > On Sat, Oct 4, 2014 at 5:13 PM, Gizmo Chicken
<gizmochicken@gmail.com>
> > wrote:
> >
> > > Hi Alex,
> > >
> > > The first bit of advice that I was about to give was that you
should
> > > have a look at a particular thread on the Arch Linux user forum. 
But
> > > as it turns out, the article that you found (which seems really
> > > well-written) already mentions that thread, and in fact even
links to
> > > one of my comments on that thread, namely this comment:
> > > https://bbs.archlinux.org/viewtopic.php?pid=1313007#p1313007
> > >
> > > With regard to CentOS 7 vs Ubuntu 14.04, keep in mind that VFIO
(which
> > > is part of the Linux kernel) is relatively new and is still
rapidly
> > > evolving.
> > >
> > > I'm not very familiar with CentOS, but if I'm not
mistaken, CentOS 7
> > > uses kernel 3.10 by default.  You'll probably want to use a
newer
> > > kernel version, which would have better VFIO support.  Again,
I'm not
> > > really familiar with CentOS, but my understanding is that many
feel
> > > that updating the kernel in CentOS isn't a good idea. So with
this in
> > > mind, you might be better off using a distro that ships with a
newer
> > > kernel version, such as Fedora or Ubuntu, etc.
> > >
> > > In my experience, Ubuntu 14.04, which ships with kernel 3.13 and
QEMU
> > > 2.0, works pretty well with VFIO.  However, Ubuntu 14.10, which
will
> > > ship with kernel 3.16 and QEMU 2.1, seems to work a little
better, but
> > > is still pre-release and so has a few other bugs.  If you go with
> > > Ubuntu 14.04, you might want to install a newer
"mainline" kernel.
> > > But in any case, please do keep in mind that VFIO is still new
and so
> > > you will almost assuredly encounter some bugs.  So don't
expect 100%
> > > stability.
> > >
> > > Hope the above is of some help.
> > >
> > > Regards,
> > > GizmoChicken
> > >
> > > On Fri, Oct 3, 2014 at 5:19 PM, Alex G.S.
<alxgrtnstrngl@gmail.com>
> wrote:
> > > > Dear List,
> > > >
> > > > Recently I read an article [1] about how to use KVM and VGA
> passthrough
> > > to
> > > > create multiple GPU accelerated VM's for use in gaming
or as virtual
> > > > workstations:
> > > >
> > > > [1]
> > > >
> > >
>
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
> > > >
> > > > I'm planning on building a KVM workstation to do this
using CentOS 7
> or
> > > > Ubuntu 14.04 LTS as the base virtualization host and then an
Intel
> Xeon
> > > CPU
> > > > and two Nvidia GeForce GTX 750's as GPU's for the
virtual machines.
> > > >
> > > > I'm really new to this sort of GPU passthrough on KVM
and am
> wondering if
> > > > anyone has any advice or has done this before?
> > > >
> > > > Thank you!
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > libvirt-users mailing list
> > > > libvirt-users@redhat.com
> > > > https://www.redhat.com/mailman/listinfo/libvirt-users
> > >
>
>
>
>