Konrad Rzeszutek Wilk
2011-Oct-17 16:40 UTC
Re: [Xen-devel] Issue with PCI-passthrough and pvops
On Mon, Oct 17, 2011 at 05:36:27PM +0200, Dario Faggioli wrote:> Hi everyone, > > I''m trying to setup PCI-passthrough for a network card on a testbox. > With HVM, everything seems to work, while if I try with a pv-guest the > domain crashes! > > Here''s the thing: > -- > # xl pci-list-assignable-devices > 0000:07:00.0 > 0000:07:00.1 > > # cat xen/VMs/Debian-squeeze.pv | grep pci> # pci=[ ''[SSSS:]BB:DD.F[,option1[,option2[...]]]'', ... ] > pci=[ ''07:00.0'' ] > > # xl create xen/VMs/Debian-squeeze.pv > Parsing config file xen/VMs/Debian-squeeze.pv > Daemon running with PID 5588 > > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 750 16 r----- 2205.3 > Debian-squeeze_pv 3 128 2 ---sc- 19.8 > -- > > Again, the very same config and PCI device for an HVM domain starts, and > I can see and bring up the NIC. > > What am I doing wrong?Do you have ''iommu=soft'' in your guest config? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-19 01:10 UTC
Re: [Xen-devel] Issue with PCI-passthrough and pvops
On Wed, Oct 19, 2011 at 01:54:07AM +0200, Dario Faggioli wrote:> On Mon, 2011-10-17 at 12:40 -0400, Konrad Rzeszutek Wilk wrote: > > > Here''s the thing: > > > -- > > > # xl pci-list-assignable-devices > > > 0000:07:00.0 > > > 0000:07:00.1 > > > > > > # cat xen/VMs/Debian-squeeze.pv | grep pci> > > # pci=[ ''[SSSS:]BB:DD.F[,option1[,option2[...]]]'', ... ] > > > pci=[ ''07:00.0'' ] > > > > > > # xl list > > > Name ID Mem VCPUs State Time(s) > > > Domain-0 0 750 16 r----- 2205.3 > > > Debian-squeeze_pv 3 128 2 ---sc- 19.8 > > > -- > > > > > > > Do you have ''iommu=soft'' in your guest config? > > > I do... BTW, it turned out that was an out-of-memory issue, which went > away after increasing VM''s memory (although the old amount of RAM was > enough without PCI-passthrough and still is for HVM, but anyway...) > > Now I have a pv-guest that boots but here''s what the host and the guest > are saying. > > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 750 16 r----- 45807.9 > Debian-squeeze_pv 1 512 2 r----- 2116.6 > > Host: > [42515.533157] pciback 0000:07:00.0: restoring config space at offset 0xf (was 0x100, writing 0x10f) > [42515.533194] pciback 0000:07:00.0: restoring config space at offset 0x8 (was 0xc, writing 0xd58f800c) > [42515.533209] pciback 0000:07:00.0: restoring config space at offset 0x6 (was 0x1, writing 0xecc1) > [42515.533224] pciback 0000:07:00.0: restoring config space at offset 0x4 (was 0xc, writing 0xd590000c) > [42515.533290] pciback 0000:07:00.0: BAR 7: set to [mem 0xdf200000-0xdf2fffff 64bit] (PCI address [0xdf200000-0xdf2fffff]) > [42515.533302] pciback 0000:07:00.0: BAR 10: set to [mem 0xdf300000-0xdf3fffff 64bit] (PCI address [0xdf300000-0xdf3fffff]) > (XEN) [VT-D]iommu.c:1543: d0:PCIe: unmap 0000:07:00.0 > (XEN) [VT-D]iommu.c:1412: d1:PCIe: map 0000:07:00.0 > [42515.556555] xen-pciback: vpci: 0000:07:00.0: assign to virtual slot 0 > mapping kernel into physical memory > about to get started... > [42526.391448] pciback 0000:07:00.0: Driver tried to write to a read-only configuration space field at offset 0x168, size 2. This may be harmless, but if you have problems with your device: > [42526.391450] 1) see permissive attribute in sysfs > [42526.391451] 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci. > > Guest: > [ 19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k > [ 19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation. > [ 19.609465] ixgbe 0000:00:00.0: device not available (can''t reserve [mem 0xdf300000-0xe32fffff 64bit]) > [ 19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22Well, that is the problem.> [ 19.611764] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k > [ 19.612656] Copyright (c) 2009 - 2010 Intel Corporation. > [ 19.614144] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI > [ 19.614865] ixgb: Copyright (c) 1999-2008 Intel Corporation. > > While in the guest, I can see the NIC with `lspci'' but I can''t bring it > up. Also, trying to check in /sys/bus/pci/..., the device does not seem > to be claimed by anyone (no driver file present). > > Moreover, when trying to kill the domain, the following happens: > # xl destroy 1 > libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40 > AbortedUgh, that looks like a bug. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-10-19 at 00:54 +0100, Dario Faggioli wrote:> when trying to kill the domain, the following happens: > # xl destroy 1 > libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40 > AbortedYou most probably need ''libxl: add missing "break;" to do_pci_remove'' which I posted v3 of yesterday. If not then "gdb xl destroy" and "bt" would be useful. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-10-19 at 02:10 +0100, Konrad Rzeszutek Wilk wrote:> On Wed, Oct 19, 2011 at 01:54:07AM +0200, Dario Faggioli wrote: > > Guest: > > [ 19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k > > [ 19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation. > > [ 19.609465] ixgbe 0000:00:00.0: device not available (can''t reserve [mem 0xdf300000-0xe32fffff 64bit]) > > [ 19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22 > > Well, that is the problem.Does he need "e820_host=1" in his cfg? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 19, 2011 at 9:40 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:>> > Guest: >> > [ 19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k >> > [ 19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation. >> > [ 19.609465] ixgbe 0000:00:00.0: device not available (can''t reserve [mem 0xdf300000-0xe32fffff 64bit]) >> > [ 19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22 >> >> Well, that is the problem. > > Does he need "e820_host=1" in his cfg? >Mmm... If you mean putting that line in my DomU config file (I checked 23428:131f19c67d85, and it seems so), that is not helping. Thanks and regards, Dario PS. sorry for the webmail... Installation of new laptop going on here :-P -- <<This happens because I choose it to happen!>> (Raistlin Majere) ---------------------------------------------------------------------- Dario Faggioli, <http://retis.sssup.it/people/faggioli> PhD Candidate, ReTiS Lab, Scuola Superiore Sant''Anna, Pisa (Italy) Senior Software Engineer, Citrix Systems R&D, Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-19 14:10 UTC
Re: [Xen-devel] Issue with PCI-passthrough and pvops
On Wed, Oct 19, 2011 at 03:46:26PM +0200, Dario Faggioli wrote:> On Wed, Oct 19, 2011 at 9:40 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >> > Guest: > >> > [ 19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k > >> > [ 19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation. > >> > [ 19.609465] ixgbe 0000:00:00.0: device not available (can''t reserve [mem 0xdf300000-0xe32fffff 64bit]) > >> > [ 19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22 > >> > >> Well, that is the problem. > > > > Does he need "e820_host=1" in his cfg? > > > Mmm... If you mean putting that line in my DomU config file > (I checked 23428:131f19c67d85, and it seems so), that is not > helping.That option is usually required if the guest has more than 2GB. As we would end up setting an E820 that would trample over the PCI hole. (In this case it looks like part of the PCI hole is DF300000). Dario, I''ve seen this bug before with .. Hm, some similar adapter. I know that if pass in ''igb.max_vfs=2'' and passed in the igbvf PCI cards the guest worked just fine. It just did not like being passed in as a real device. Otherwise, older MSI/MSI-X (non SR-IOV) cards worked fine so I never go further in debugging this. I would recommend you take a look at the probe function and figure out why it can''t reserve that region. And easy way to figure that out is to boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff region. Perhaps something else is overlapping it? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 19, 2011 at 9:39 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:>> # xl destroy 1 >> libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40 >> Aborted > > You most probably need ''libxl: add missing "break;" to do_pci_remove'' > which I posted v3 of yesterday. >Yep, error is gone, thanks! Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ---------------------------------------------------------------------- Dario Faggioli, <http://retis.sssup.it/people/faggioli> PhD Candidate, ReTiS Lab, Scuola Superiore Sant''Anna, Pisa (Italy) Senior Software Engineer, Citrix Systems R&D, Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 19, 2011 at 4:10 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> >> > Guest: >> >> > [ 19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k >> >> > [ 19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation. >> >> > [ 19.609465] ixgbe 0000:00:00.0: device not available (can''t reserve [mem 0xdf300000-0xe32fffff 64bit]) >> >> > [ 19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22 > > Dario, > > I''ve seen this bug before with .. Hm, some similar adapter. I know > that if pass in ''igb.max_vfs=2'' and passed in the igbvf PCI cards the > guest worked just fine. It just did not like being passed in as a real device. > > Otherwise, older MSI/MSI-X (non SR-IOV) cards worked fine so I never go > further in debugging this. >Yeah, in fact passing other cards the box have to the very same domain seems to just work.> I would recommend you take a look at the probe function and figure out why > it can''t reserve that region. And easy way to figure that out is to > boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff > region. Perhaps something else is overlapping it? >I might be wrong, but it doesn''t seem quite so to me: # cat /proc/iomem 00000000-0000ffff : reserved 00010000-0009ffff : System RAM 000a0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-1fffffff : System RAM 01000000-01b88b4b : Kernel code 01b88b4c-02352a7f : Kernel data 02652000-02ee6fff : Kernel bss 20000000-bf698fff : Unusable memory bf6af000-bf6cdfff : ACPI Tables d58f8000-d58fbfff : 0000:00:00.0 d5900000-d597ffff : 0000:00:00.0 df200000-e71fffff : 0000:00:00.0 100000000-1000fffff : System RAM The probe function is huge... But I''ll se if I can find the time to take a look at it in the next days... Thanks a lot for now. :-) Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ---------------------------------------------------------------------- Dario Faggioli, <http://retis.sssup.it/people/faggioli> Senior Software Engineer, Citrix Systems R&D, Cambridge (UK) PhD Candidate, ReTiS Lab, Scuola Superiore Sant''Anna, Pisa (Italy) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-20 01:42 UTC
Re: [Xen-devel] Issue with PCI-passthrough and pvops
> > I would recommend you take a look at the probe function and figure out why > > it can''t reserve that region. And easy way to figure that out is to > > boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff > > region. Perhaps something else is overlapping it? > > > I might be wrong, but it doesn''t seem quite so to me: > > # cat /proc/iomem > 00000000-0000ffff : reserved > 00010000-0009ffff : System RAM > 000a0000-000fffff : reserved > 000f0000-000fffff : System ROM > 00100000-1fffffff : System RAM > 01000000-01b88b4b : Kernel code > 01b88b4c-02352a7f : Kernel data > 02652000-02ee6fff : Kernel bss > 20000000-bf698fff : Unusable memory > bf6af000-bf6cdfff : ACPI Tables > d58f8000-d58fbfff : 0000:00:00.0 > d5900000-d597ffff : 0000:00:00.0 > df200000-e71fffff : 0000:00:00.0 > 100000000-1000fffff : System RAM > > The probe function is huge... But I''ll se if I can find the time to take > a look at it in the next days...Hmmm. Instrumenting resources.c (__request_resource) might be the way to figure out where it chokes on. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Maybe Matching Threads
- [PATCHv2 0 of 2] Deal with IOMMU faults in softirq context.
- [PATCH 0 of 3] xen: sched_credit: fix tickling and add some tracing
- Re: [RFC/RFT][PATCH 0 of 3] rework locking in sched_adjust
- [PATCH] xl: fix sedf parameters checking
- [PATCH 00 of 10 v3] Automatic NUMA placement for xl