I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. The hardware does work - I''ve passed the embedded G200e through using ESXi 5.1 although other devices (a Radeon 6950) don''t work. Issues are - both with XCP1.6 and xen 4.2 (with some small AMD VGA passthrough patches)/xen 4.3 unstable on Debian Wheezy : Pciback is a module. It recognises the passthrough keyword but PCI IDs are still modified in VMs. Pt_iomul errors in qemu log even though the status is ''successful'' Xl create errors out with a gnttab error both in 4.2/4.3. Xm works. When passing through an old 3C900 PCI NIC as a test, this causes noise in the audigy 4 PCI soundcard in the next slot unless that is also passed through. I suspect this is due to the bus layout of the S3210SHLC. Is it always necessary to pass all PCI devices on the same root through? Also should it be necessary/possible to pass a PCI root? The VMs think the device was successfully passed through (a wheezy hvm binds the appropriate modules and a Windows 7 hvm has seen the radeon) but they don''t actually work. There are so many combinations of hardware and kernel finding out what is wrong is tricky. I''d appreciate some advice diagnosing it. Hardware is S3210SHLC with latest (52) BIOS, embedded Matrox G200e, two E1000s of some type, a GeForce G210 (for console), Reference AMD 6950 (on the PCH 4x slot as MCH slot bandwidths are restricted for graphics cards), audigy 4 PCI and 3C900. Cheers! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote:> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. >This hardware has actual IOMMU?> The hardware does work - I''ve passed the embedded G200e through using ESXi 5.1 although other devices (a Radeon 6950) don''t work. > > Issues are - both with XCP1.6 and xen 4.2 (with some small AMD VGA passthrough patches)/xen 4.3 unstable on Debian Wheezy : > > Pciback is a module. It recognises the passthrough keyword but PCI IDs are still modified in VMs.Not sure what you mean by that. Modified in VMs? You mean that the BDF value is different?> > Pt_iomul errors in qemu log even though the status is ''successful'' > > Xl create errors out with a gnttab error both in 4.2/4.3. Xm works.Uh, that is not good. Why would ''xl'' die on that? Is that fixed with a different version of the Linux kernel?> > When passing through an old 3C900 PCI NIC as a test, this causes noise in the audigy 4 PCI soundcard in the next slot unless that is also passed through. I suspect this is due to the bus layout of the S3210SHLC. Is it always necessary to pass all PCI devices on the same root through? Also should it be necessary/possible to pass a PCI root? >Isn''t that an ISA card? Do you mean 3c590 perhaps?> The VMs think the device was successfully passed through (a wheezy hvm binds the appropriate modules and a Windows 7 hvm has seen the radeon) but they don''t actually work. > > There are so many combinations of hardware and kernel finding out what is wrong is tricky. I''d appreciate some advice diagnosing it.So I''ve been quite successfull with both AMD and Intel to pass in a PCIe type NIC. And also an Radeon card in Windows 7: 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 06:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV770 [Radeon HD 4870] 06:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV770 HDMI Audio [Radeon HD 4850/4870] Without any modifications to Xen. Just used Fedora 18 and use xen-pciback.hide to hide the 04:00.0 and 06:00.* and passed them in to the guest: builder=''hvm'' memory = 2048 name = "Windows7" vcpus=3 disk = [ ''phy:/dev/vg_guest/Win7_Home,hda,w''] vnc=1 videoram=8 vnclisten="0.0.0.0" vncpasswd='''' stdvga=0 serial=''pty'' tsc_mode=0 usb=1 usb_add="host:045e:0040" usbdevice=''tablet'' xen_platform_pci=1 # Remember QEMU_AUDIO_DRV=pa soundhw=''es1370'' pci=[''04:00.0'',''06:00.1'',''06:00.0'']> > Hardware is S3210SHLC with latest (52) BIOS, embedded Matrox G200e, two E1000s of some type, a GeForce G210 (for console), Reference AMD 6950 (on the PCH 4x slot as MCH slot bandwidths are restricted for graphics cards), audigy 4 PCI and 3C900. > > Cheers!> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On 15 March 2013 21:02, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote: >> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. >> > > This hardware has actual IOMMU?Yes, and I''ve passed hardware through in Esxi 5.1. It''s a server chipset with IOMMU, but pre Nehalem so the features are less complete.> >> The hardware does work - I''ve passed the embedded G200e through using ESXi 5.1 although other devices (a Radeon 6950) don''t work. >> >> Issues are - both with XCP1.6 and xen 4.2 (with some small AMD VGA passthrough patches)/xen 4.3 unstable on Debian Wheezy : >> >> Pciback is a module. It recognises the passthrough keyword but PCI IDs are still modified in VMs. > > > Not sure what you mean by that. Modified in VMs? You mean that the BDF value is different? >Yes. I''m given to understand the passthrough parameter should maintain the BDF - but it doesn''t, and there appears to be no advice on this on the Internet other than ''it should work'' (but doesn''t for some people).>> Pt_iomul errors in qemu log even though the status is ''successful'' >> >> Xl create errors out with a gnttab error both in 4.2/4.3. Xm works. > > Uh, that is not good. Why would ''xl'' die on that? Is that fixed with a different > version of the Linux kernel?I don''t know - that''s why I''m asking :). I don''t mind doing a bit of debug work but I''m getting a little tired of trying kernels just in case it might work.>> >> When passing through an old 3C900 PCI NIC as a test, this causes noise in the audigy 4 PCI soundcard in the next slot unless that is also passed through. I suspect this is due to the bus layout of the S3210SHLC. Is it always necessary to pass all PCI devices on the same root through? Also should it be necessary/possible to pass a PCI root? >> > > Isn''t that an ISA card? Do you mean 3c590 perhaps?No, you''re thinking of the 3C509. The 3C900 is an old PCI 10Mb NIC. It may be this is far too old and I need to try something newer (it was only as a test and was to hand).>> The VMs think the device was successfully passed through (a wheezy hvm binds the appropriate modules and a Windows 7 hvm has seen the radeon) but they don''t actually work. >> >> There are so many combinations of hardware and kernel finding out what is wrong is tricky. I''d appreciate some advice diagnosing it. > > So I''ve been quite successfull with both AMD and Intel to pass in a PCIe type NIC. And also > an Radeon card in Windows 7: > > 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection > 06:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV770 [Radeon HD 4870] > 06:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV770 HDMI Audio [Radeon HD 4850/4870] > > Without any modifications to Xen. Just used Fedora 18 and use xen-pciback.hide to hide > the 04:00.0 and 06:00.* and passed them in to the guest:Can I ask which kernel you are using, please?> builder=''hvm'' > memory = 2048 > name = "Windows7" > vcpus=3 > disk = [ ''phy:/dev/vg_guest/Win7_Home,hda,w''] > vnc=1 > videoram=8 > vnclisten="0.0.0.0" > vncpasswd='''' > stdvga=0 > serial=''pty'' > tsc_mode=0 > usb=1 > usb_add="host:045e:0040" > usbdevice=''tablet'' > xen_platform_pci=1 > # Remember QEMU_AUDIO_DRV=pa > soundhw=''es1370'' > pci=[''04:00.0'',''06:00.1'',''06:00.0''] > >> >> Hardware is S3210SHLC with latest (52) BIOS, embedded Matrox G200e, two E1000s of some type, a GeForce G210 (for console), Reference AMD 6950 (on the PCH 4x slot as MCH slot bandwidths are restricted for graphics cards), audigy 4 PCI and 3C900.Thanks! PK
On Sat, Mar 16, 2013 at 04:05:19PM +0000, Peter Kay wrote:> On 15 March 2013 21:02, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote: > >> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. > >> > > > > This hardware has actual IOMMU? > Yes, and I''ve passed hardware through in Esxi 5.1. It''s a server > chipset with IOMMU, but pre Nehalem so the features are less complete.Oh. I didn''t know that such mutants existed :-)> > > > >> The hardware does work - I''ve passed the embedded G200e through using ESXi 5.1 although other devices (a Radeon 6950) don''t work. > >> > >> Issues are - both with XCP1.6 and xen 4.2 (with some small AMD VGA passthrough patches)/xen 4.3 unstable on Debian Wheezy : > >> > >> Pciback is a module. It recognises the passthrough keyword but PCI IDs are still modified in VMs. > > > > > > Not sure what you mean by that. Modified in VMs? You mean that the BDF value is different? > > > Yes. I''m given to understand the passthrough parameter should maintain > the BDF - but it doesn''t, and there appears to be no advice on this on > the Internet other than ''it should work'' (but doesn''t for some > people).That depends. The xen-pciback.vpci argument that define whether you want them to be virtual (so they start at 00) or match your host. But that option is only usefull for PV guests. I think you are using an HVM one. In which that does not matter that much I would think.> > >> Pt_iomul errors in qemu log even though the status is ''successful'' > >> > >> Xl create errors out with a gnttab error both in 4.2/4.3. Xm works. > > > > Uh, that is not good. Why would ''xl'' die on that? Is that fixed with a different > > version of the Linux kernel? > > I don''t know - that''s why I''m asking :). I don''t mind doing a bit of > debug work but I''m getting a little tired of trying kernels just in > case it might work.I just used the distro''s default and also the distro''s default Xen. Fedora 17 or Fedora 18.> > >> > >> When passing through an old 3C900 PCI NIC as a test, this causes noise in the audigy 4 PCI soundcard in the next slot unless that is also passed through. I suspect this is due to the bus layout of the S3210SHLC. Is it always necessary to pass all PCI devices on the same root through? Also should it be necessary/possible to pass a PCI root? > >> > > > > Isn''t that an ISA card? Do you mean 3c590 perhaps? > > No, you''re thinking of the 3C509. The 3C900 is an old PCI 10Mb NIC. It > may be this is far too old and I need to try something newer (it was > only as a test and was to hand). > > >> The VMs think the device was successfully passed through (a wheezy hvm binds the appropriate modules and a Windows 7 hvm has seen the radeon) but they don''t actually work. > >> > >> There are so many combinations of hardware and kernel finding out what is wrong is tricky. I''d appreciate some advice diagnosing it. > > > > So I''ve been quite successfull with both AMD and Intel to pass in a PCIe type NIC. And also > > an Radeon card in Windows 7: > > > > 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection > > 06:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV770 [Radeon HD 4870] > > 06:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV770 HDMI Audio [Radeon HD 4850/4870] > > > > Without any modifications to Xen. Just used Fedora 18 and use xen-pciback.hide to hide > > the 04:00.0 and 06:00.* and passed them in to the guest: > > Can I ask which kernel you are using, please?Stock Fedora. Thought right now I rebuilt it with v3.9-rc2 with an Fedora config.> > > builder=''hvm'' > > memory = 2048 > > name = "Windows7" > > vcpus=3 > > disk = [ ''phy:/dev/vg_guest/Win7_Home,hda,w''] > > vnc=1 > > videoram=8 > > vnclisten="0.0.0.0" > > vncpasswd='''' > > stdvga=0 > > serial=''pty'' > > tsc_mode=0 > > usb=1 > > usb_add="host:045e:0040" > > usbdevice=''tablet'' > > xen_platform_pci=1 > > # Remember QEMU_AUDIO_DRV=pa > > soundhw=''es1370'' > > pci=[''04:00.0'',''06:00.1'',''06:00.0''] > > > >> > >> Hardware is S3210SHLC with latest (52) BIOS, embedded Matrox G200e, two E1000s of some type, a GeForce G210 (for console), Reference AMD 6950 (on the PCH 4x slot as MCH slot bandwidths are restricted for graphics cards), audigy 4 PCI and 3C900. > > Thanks!> > PK
On 19 March 2013 14:54, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Sat, Mar 16, 2013 at 04:05:19PM +0000, Peter Kay wrote: >> On 15 March 2013 21:02, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: >> > On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote: >> >> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. >> >> >> > >> > This hardware has actual IOMMU? >> Yes, and I''ve passed hardware through in Esxi 5.1. It''s a server >> chipset with IOMMU, but pre Nehalem so the features are less complete. > > Oh. I didn''t know that such mutants existed :-)Yes - it''s the original VT-d, as supported by the Q35, Q45, X38, X48 and S3200/S3210 chipsets (not on all motherboards). The Nehalem and later supporting chipsets is termed by Intel as VT-d2 as it features interrupt remapping, etc (except where it doesn''t due to errata, as per recent messages to the devel list).>> Yes. I''m given to understand the passthrough parameter should maintain >> the BDF - but it doesn''t, and there appears to be no advice on this on >> the Internet other than ''it should work'' (but doesn''t for some >> people). > > That depends. The xen-pciback.vpci argument that define whether you > want them to be virtual (so they start at 00) or match your host. > > But that option is only usefull for PV guests. I think you are using > an HVM one. In which that does not matter that much I would think.Yes, I''m using HVM. I''ll have a look at vpci though, thanks! Anyway, thank for the other recommendations - I''ve actually made a fair bit of progress. I swapped out the 3C900 for an 82557 (Pro 100) and everything started working. Once I turned off video output completely and passed through the embedded G200e (primary video on) I was also successfully able to use X with it on a Wheezy HVM. I''ve yet to get the 6950 working - it''s seen, identified and bound to, but then I get drm errors - there''s no /dev/dri/card* devices. I''ll try swapping the fglrx driver for radeon and fiddling. Failing all that I may try with the Fedora config that works for you. The one time I got the 6950 to do something the fan ran at full pelt and hung the whole machine, which really was a bit suboptimal.. Then again, I do also have the relaxed option set for xen. Thanks! PK
On Fri, Mar 22, 2013 at 07:24:43PM +0000, Peter Kay wrote:> On 19 March 2013 14:54, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > On Sat, Mar 16, 2013 at 04:05:19PM +0000, Peter Kay wrote: > >> On 15 March 2013 21:02, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > >> > On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote: > >> >> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. > >> >> > >> > > >> > This hardware has actual IOMMU? > >> Yes, and I''ve passed hardware through in Esxi 5.1. It''s a server > >> chipset with IOMMU, but pre Nehalem so the features are less complete. > > > > Oh. I didn''t know that such mutants existed :-) > Yes - it''s the original VT-d, as supported by the Q35, Q45, X38, X48 > and S3200/S3210 chipsets (not on all motherboards). The Nehalem and > later supporting chipsets is termed by Intel as VT-d2 as it features > interrupt remapping, etc (except where it doesn''t due to errata, as > per recent messages to the devel list). > > >> Yes. I''m given to understand the passthrough parameter should maintain > >> the BDF - but it doesn''t, and there appears to be no advice on this on > >> the Internet other than ''it should work'' (but doesn''t for some > >> people). > > > > That depends. The xen-pciback.vpci argument that define whether you > > want them to be virtual (so they start at 00) or match your host. > > > > But that option is only usefull for PV guests. I think you are using > > an HVM one. In which that does not matter that much I would think. > Yes, I''m using HVM. I''ll have a look at vpci though, thanks! > > Anyway, thank for the other recommendations - I''ve actually made a > fair bit of progress. > > I swapped out the 3C900 for an 82557 (Pro 100) and everything started > working. Once I turned off video output completely and passed through > the embedded G200e (primary video on) I was also successfully able to > use X with it on a Wheezy HVM.Nice!> > I''ve yet to get the 6950 working - it''s seen, identified and bound to, > but then I get drm errors - there''s no /dev/dri/card* devices. I''ll > try swapping the fglrx driver for radeon and fiddling. Failing all > that I may try with the Fedora config that works for you. The one time > I got the 6950 to do something the fan ran at full pelt and hung the > whole machine, which really was a bit suboptimal.. Then again, I do > also have the relaxed option set for xen.I only had tried with Windows 7 guest doing the passthrough. The one time I did try with Linux the issues I saw were that the BIOS was not passed in - but I cannot remember if this was just as a PV (so was using xen-pcifront, which does not expose the ROM BAR), or HVM. For HVM, I would think I need to stuff the BIOS of the radeon card somewhere in QEMU similary to how some people need to do it with Nvidia. If you look around on xen-devel there are some patches floating around from David TECHER that explain what he had to do. Note, for debugging the Linux, I would suggest drm.debug=255 on the Linux command line. There is probably some radeon.XXX option as well but I can''t remember what it is called.> > Thanks! > > PK > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Monday, March 25, 2013, 2:15:17 PM, you wrote:> On Fri, Mar 22, 2013 at 07:24:43PM +0000, Peter Kay wrote: >> On 19 March 2013 14:54, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: >> > On Sat, Mar 16, 2013 at 04:05:19PM +0000, Peter Kay wrote: >> >> On 15 March 2013 21:02, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: >> >> > On Fri, Mar 01, 2013 at 02:00:08PM +0000, Peter Kay wrote: >> >> >> I''m having difficulty getting any passthrough working on an S3210SHLC (X38 derived S3210 socket 775 server chipset, thus pre SLAT) - this is on the list of supported hardware. Is there any guide as to what can be done - I''ve tried iommu=verbose, so is the next step sending debug printfs to the xen console? I realise this is a development list so I''d like advice on which bits of source to poke so I can find the root issue. >> >> >> >> >> > >> >> > This hardware has actual IOMMU? >> >> Yes, and I''ve passed hardware through in Esxi 5.1. It''s a server >> >> chipset with IOMMU, but pre Nehalem so the features are less complete. >> > >> > Oh. I didn''t know that such mutants existed :-) >> Yes - it''s the original VT-d, as supported by the Q35, Q45, X38, X48 >> and S3200/S3210 chipsets (not on all motherboards). The Nehalem and >> later supporting chipsets is termed by Intel as VT-d2 as it features >> interrupt remapping, etc (except where it doesn''t due to errata, as >> per recent messages to the devel list). >> >> >> Yes. I''m given to understand the passthrough parameter should maintain >> >> the BDF - but it doesn''t, and there appears to be no advice on this on >> >> the Internet other than ''it should work'' (but doesn''t for some >> >> people). >> > >> > That depends. The xen-pciback.vpci argument that define whether you >> > want them to be virtual (so they start at 00) or match your host. >> > >> > But that option is only usefull for PV guests. I think you are using >> > an HVM one. In which that does not matter that much I would think. >> Yes, I''m using HVM. I''ll have a look at vpci though, thanks! >> >> Anyway, thank for the other recommendations - I''ve actually made a >> fair bit of progress. >> >> I swapped out the 3C900 for an 82557 (Pro 100) and everything started >> working. Once I turned off video output completely and passed through >> the embedded G200e (primary video on) I was also successfully able to >> use X with it on a Wheezy HVM.> Nice! >> >> I''ve yet to get the 6950 working - it''s seen, identified and bound to, >> but then I get drm errors - there''s no /dev/dri/card* devices. I''ll >> try swapping the fglrx driver for radeon and fiddling. Failing all >> that I may try with the Fedora config that works for you. The one time >> I got the 6950 to do something the fan ran at full pelt and hung the >> whole machine, which really was a bit suboptimal.. Then again, I do >> also have the relaxed option set for xen.> I only had tried with Windows 7 guest doing the passthrough. The one time > I did try with Linux the issues I saw were that the BIOS was not > passed in - but I cannot remember if this was just as a PV (so was > using xen-pcifront, which does not expose the ROM BAR), or HVM.> For HVM, I would think I need to stuff the BIOS of the radeon card > somewhere in QEMU similary to how some people need to do it with Nvidia. > If you look around on xen-devel there are some patches floating around > from David TECHER that explain what he had to do.When using secondary passthrough to a hvm (on xen-unstable and qemu upstream and recent kernel) it can work out out of the box (on a HD-65xx) (tested with debian wheezy and ubuntu). Don''t know when you try to use it as primary though. Suing the fglrx is quite a pain in the ass on kernels > 3.5 due to the UAPI reshuffling. -- Sander> Note, for debugging the Linux, I would suggest drm.debug=255 > on the Linux command line. There is probably some radeon.XXX option as well > but I can''t remember what it is called.>> >> Thanks! >> >> PK >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>