Sander Eikelenboom
2013-Feb-25 22:18 UTC
linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Hi Konrad, I can''t get linux-3.9 rc0 to boot under xen-unstable. It doesn''t detect the s-ata controller, so it ends op with udev timing and bailing out to busybox. I don''t see a obvious error in the logs. The same kernel boots fine on baremetal. With linux 3.8 it boots fine under this version of xen-unstable. Can you spot something from the logs or you have a hunch from where to start a bisect ? (bisecting could be a problem due to multiple boot issues and entangled patchsets i presume ..) 3.9-rc0 last commit is: ab7826595e9ec51a51f622c5fc91e2f59440481a Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6 -- Sander Attached: - Serial log with 3.9-rc0 kernel (missing sata) - Serial log with 3.8 kernel (booting fine - .config _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Jan Beulich
2013-Feb-26 08:41 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
>>> On 25.02.13 at 23:18, Sander Eikelenboom <linux@eikelenboom.it> wrote: > I can''t get linux-3.9 rc0 to boot under xen-unstable. > It doesn''t detect the s-ata controller, so it ends op with udev timing and > bailing out to busybox. > > I don''t see a obvious error in the logs.Perhaps because the log is far from being complete? There''s a huge gap right before the first pciback message, yet that''s quite likely the relevant part. Jan
Konrad Rzeszutek Wilk
2013-Feb-26 15:20 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
On Mon, Feb 25, 2013 at 11:18:30PM +0100, Sander Eikelenboom wrote:> Hi Konrad, > > I can''t get linux-3.9 rc0 to boot under xen-unstable. > It doesn''t detect the s-ata controller, so it ends op with udev timing and bailing out to busybox. > > I don''t see a obvious error in the logs. > > The same kernel boots fine on baremetal. > With linux 3.8 it boots fine under this version of xen-unstable. > > Can you spot something from the logs or you have a hunch from where to start a bisect ? > (bisecting could be a problem due to multiple boot issues and entangled patchsets i presume ..)There are at least two pitfalls - the 14e568e78f6f80ca1e27256641ddf524c7dbdc51 (which you can revert) and you need to make sure you have 0cc9129d75ef8993702d97ab0e49542c15ac6ab9 if you end up in the merge 2ef14f465b9e096531343f5b734cffc5f759f4a6 ("Merge branch ''x86-mm-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")> > 3.9-rc0 last commit is: ab7826595e9ec51a51f622c5fc91e2f59440481a Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6Interestingly I can reproduce it on my F1A75-M box as well. With baremetal I get: [ 12.701221] ahci 0000:00:11.0: version 3.0 [ 12.701424] initcall radeon_init+0x0/0x1000 [radeon] returned 0 after 528175 usecs [ 12.712999] ahci 0000:00:11.0: irq 46 for MSI/MSI-X [ 12.717967] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 12.726132] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 12.735405] scsi0 : ahci [ 12.738087] scsi1 : ahci [ 12.740734] scsi2 : ahci [ 12.743511] scsi3 : ahci [ 12.746278] scsi4 : ahci [ 12.748930] scsi5 : ahci And when booting under Xen 4.1 (note, no IOMMU): [ 19.178696] ahci 0000:00:11.0: version 3.0 [ 19.178721] xen: registering gsi 19 triggering 0 polarity 1 [ 19.178743] xen: --> pirq=19 -> irq=19 (gsi=19) [ 19.179040] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 19.179043] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 19.241748] ahci: probe of 0000:00:11.0 failed with error -22 [ 19.249124] initcall ahci_pci_driver_init+0x0/0x1000 [ahci] returned 0 after 460418 usecs Had you tried to checkout right before this mega update from Jeff Garzik: ab78265 Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6 21fbd58 Merge branch ''v4l_for_linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media d9978ec Merge tag ''upstream-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev ==> 77be36d Merge tag ''stable/for-linus-3.9-rc0-tag'' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen And seeing if that works?> > -- > Sander > > Attached: > - Serial log with 3.9-rc0 kernel (missing sata) > - Serial log with 3.8 kernel (booting fine > - .config
Sander Eikelenboom
2013-Feb-26 15:55 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Tuesday, February 26, 2013, 4:20:38 PM, you wrote:> On Mon, Feb 25, 2013 at 11:18:30PM +0100, Sander Eikelenboom wrote: >> Hi Konrad, >> >> I can''t get linux-3.9 rc0 to boot under xen-unstable. >> It doesn''t detect the s-ata controller, so it ends op with udev timing and bailing out to busybox. >> >> I don''t see a obvious error in the logs. >> >> The same kernel boots fine on baremetal. >> With linux 3.8 it boots fine under this version of xen-unstable. >> >> Can you spot something from the logs or you have a hunch from where to start a bisect ? >> (bisecting could be a problem due to multiple boot issues and entangled patchsets i presume ..)> There are at least two pitfalls - the > 14e568e78f6f80ca1e27256641ddf524c7dbdc51 (which you can revert) and you need > to make sure you have 0cc9129d75ef8993702d97ab0e49542c15ac6ab9 > if you end up in the merge 2ef14f465b9e096531343f5b734cffc5f759f4a6 > ("Merge branch ''x86-mm-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")>> >> 3.9-rc0 last commit is: ab7826595e9ec51a51f622c5fc91e2f59440481a Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6> Interestingly I can reproduce it on my F1A75-M box as well.> With baremetal I get: > [ 12.701221] ahci 0000:00:11.0: version 3.0 > [ 12.701424] initcall radeon_init+0x0/0x1000 [radeon] returned 0 after 528175 usecs > [ 12.712999] ahci 0000:00:11.0: irq 46 for MSI/MSI-X > [ 12.717967] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode > [ 12.726132] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 12.735405] scsi0 : ahci > [ 12.738087] scsi1 : ahci > [ 12.740734] scsi2 : ahci > [ 12.743511] scsi3 : ahci > [ 12.746278] scsi4 : ahci > [ 12.748930] scsi5 : ahci> And when booting under Xen 4.1 (note, no IOMMU):> [ 19.178696] ahci 0000:00:11.0: version 3.0 > [ 19.178721] xen: registering gsi 19 triggering 0 polarity 1 > [ 19.178743] xen: --> pirq=19 -> irq=19 (gsi=19) > [ 19.179040] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode > [ 19.179043] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 19.241748] ahci: probe of 0000:00:11.0 failed with error -22 > [ 19.249124] initcall ahci_pci_driver_init+0x0/0x1000 [ahci] returned 0 after 460418 usecs> Had you tried to checkout right before this mega update from Jeff Garzik:> ab78265 Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6 > 21fbd58 Merge branch ''v4l_for_linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media > d9978ec Merge tag ''upstream-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev==>> 77be36d Merge tag ''stable/for-linus-3.9-rc0-tag'' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen> And seeing if that works?Hmm that''s worth the try, one of the changes seems to be to try to enable multiple msi per device, somehow that sounds like something that could blow up under xen. Compiling now ..>> >> -- >> Sander >> >> Attached: >> - Serial log with 3.9-rc0 kernel (missing sata) >> - Serial log with 3.8 kernel (booting fine >> - .config
Sander Eikelenboom
2013-Feb-26 20:56 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Tuesday, February 26, 2013, 4:20:38 PM, you wrote:> On Mon, Feb 25, 2013 at 11:18:30PM +0100, Sander Eikelenboom wrote: >> Hi Konrad, >> >> I can''t get linux-3.9 rc0 to boot under xen-unstable. >> It doesn''t detect the s-ata controller, so it ends op with udev timing and bailing out to busybox. >> >> I don''t see a obvious error in the logs. >> >> The same kernel boots fine on baremetal. >> With linux 3.8 it boots fine under this version of xen-unstable. >> >> Can you spot something from the logs or you have a hunch from where to start a bisect ? >> (bisecting could be a problem due to multiple boot issues and entangled patchsets i presume ..)> There are at least two pitfalls - the > 14e568e78f6f80ca1e27256641ddf524c7dbdc51 (which you can revert) and you need > to make sure you have 0cc9129d75ef8993702d97ab0e49542c15ac6ab9 > if you end up in the merge 2ef14f465b9e096531343f5b734cffc5f759f4a6 > ("Merge branch ''x86-mm-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")>> >> 3.9-rc0 last commit is: ab7826595e9ec51a51f622c5fc91e2f59440481a Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6> Interestingly I can reproduce it on my F1A75-M box as well.> With baremetal I get: > [ 12.701221] ahci 0000:00:11.0: version 3.0 > [ 12.701424] initcall radeon_init+0x0/0x1000 [radeon] returned 0 after 528175 usecs > [ 12.712999] ahci 0000:00:11.0: irq 46 for MSI/MSI-X > [ 12.717967] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode > [ 12.726132] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 12.735405] scsi0 : ahci > [ 12.738087] scsi1 : ahci > [ 12.740734] scsi2 : ahci > [ 12.743511] scsi3 : ahci > [ 12.746278] scsi4 : ahci > [ 12.748930] scsi5 : ahci> And when booting under Xen 4.1 (note, no IOMMU):> [ 19.178696] ahci 0000:00:11.0: version 3.0 > [ 19.178721] xen: registering gsi 19 triggering 0 polarity 1 > [ 19.178743] xen: --> pirq=19 -> irq=19 (gsi=19) > [ 19.179040] ahci 0000:00:11.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode > [ 19.179043] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 19.241748] ahci: probe of 0000:00:11.0 failed with error -22 > [ 19.249124] initcall ahci_pci_driver_init+0x0/0x1000 [ahci] returned 0 after 460418 usecs> Had you tried to checkout right before this mega update from Jeff Garzik:> ab78265 Merge tag ''mfd-3.9-1'' of git://git.kernel.org/pub/scm/linux/kernel/git/sameo/mfd-2.6 > 21fbd58 Merge branch ''v4l_for_linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media > d9978ec Merge tag ''upstream-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev==>> 77be36d Merge tag ''stable/for-linus-3.9-rc0-tag'' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen> And seeing if that works?Ok reverting to just before the libata merge doesn''t help. Time to start digging further back ...>> >> -- >> Sander >> >> Attached: >> - Serial log with 3.9-rc0 kernel (missing sata) >> - Serial log with 3.8 kernel (booting fine >> - .config
Sander Eikelenboom
2013-Feb-27 10:57 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Tuesday, February 26, 2013, 9:41:53 AM, you wrote:>>>> On 25.02.13 at 23:18, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> I can''t get linux-3.9 rc0 to boot under xen-unstable. >> It doesn''t detect the s-ata controller, so it ends op with udev timing and >> bailing out to busybox. >> >> I don''t see a obvious error in the logs.> Perhaps because the log is far from being complete? There''s a huge > gap right before the first pciback message, yet that''s quite likely > the relevant part.Hi Jan / Konrad, Tried bisecting, but that ended up no where, so back to the logs... With v3.9-rc0 + xen, it''s indeed missing a part of the log: [ 4.141328] Brought up 6 CPUs [ 4.142654] Grant tables using version 2 layout. [ 4.142676] Grant table initialized [ 4.142813] NET: Registered protocol family 16 [ 4.145343] node 0 link 0: io port [1000, ffffff] [ 4.145354] TOM: 00000000b0000000 aka 2816M [ 4.145360] Fam 10h mmconf [mem 0xe0000000-0xefffffff] [ 4.145373] node 0 link 0: mmio [e0000000, efffffff] ==> none [ 4.145381] node 0 link 0: mmio [f0000000, ffffffff] [ 4.145388] node 0 link 0: mmio [a0000, bffff] [ 4.145395] node 0 link 0: mmio [b0000000, dfffffff] [ 4.145401] TOM2: 0000000250000000 aka 9472M [ 4.145406] bus: [bus 00-07] on node 0 link 0 [ 4.145411] bus: 00 [io 0x0000-0xffff] [ 4.145415] bus: 00 [mem 0xf0000000-0xffffffff] [ 4.145420] bus: 00 [mem 0x000a0000-0x000bffff] [ 4.145424] bus: 00 [mem 0xb0000000-0xdfffffff] [ 4.145429] bus: 00 [mem 0x250000000-0xfcffffffff] [ 4.145702] ACPI: bus type pci registered [ 4.146801] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 4.146812] PCI: not using MMCONFIG [ 4.146817] PCI: Using configuration type 1 for base access [ 4.146822] PCI: Using configuration type 1 for extended access [ 4.191935] bio: create slab <bio-0> at 0 [ 4.192623] ACPI: Added _OSI(Module Device) [ 4.192630] ACPI: Added _OSI(Processor Device) [ 4.192636] ACPI: Added _OSI(3.0 _SCP Extensions) [ 4.192642] ACPI: Added _OSI(Processor Aggregator Device) [ 4.195958] ACPI: EC: Look up EC in DSDT [ 4.199659] ACPI: Executed 3 blocks of module-level executable AML code [ 4.204162] ACPI: Interpreter enabled [ 4.204170] ACPI: (supports S0 S5) [ 4.204181] ACPI: Using IOAPIC for interrupt routing [ 4.204219] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 4.205800] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboa[ 7.107382] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001 The line in the serial-log also seems to be truncated somehow and the rest of the info missing .. When booting v3.9-rc0 Baremetal the complete pci enumeration seems to be in between: [ 0.288861] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.288867] PCI: not using MMCONFIG [ 0.288872] PCI: Using configuration type 1 for base access [ 0.288875] PCI: Using configuration type 1 for extended access [ 0.360826] bio: create slab <bio-0> at 0 [ 0.361873] ACPI: Added _OSI(Module Device) [ 0.361883] ACPI: Added _OSI(Processor Device) [ 0.361891] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.361899] ACPI: Added _OSI(Processor Aggregator Device) [ 0.365516] ACPI: EC: Look up EC in DSDT [ 0.373572] ACPI: Executed 3 blocks of module-level executable AML code [ 0.452047] ACPI: Interpreter enabled [ 0.452052] ACPI: (supports S0 S5) [ 0.452077] ACPI: Using IOAPIC for interrupt routing [ 0.452249] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.465175] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.479870] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.605451] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.609907] acpi PNP0A03:00: Requesting ACPI _OSC control (0x1d) [ 0.613973] acpi PNP0A03:00: ACPI _OSC control (0x1d) granted [ 0.617005] PCI host bridge to bus 0000:00 [ 0.617027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.617037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.617046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.617055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.617064] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000dffff] [ 0.617073] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xdfffffff] [ 0.617081] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfebfffff] [ 0.617094] pci_bus 0000:00: scanning bus Tried booting with pci=nommconf,nocrs but to no avail. Are there any other boot options i could try to narrow it down ? -- Sander> Jan
Jan Beulich
2013-Feb-27 11:06 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
>>> On 27.02.13 at 11:57, Sander Eikelenboom <linux@eikelenboom.it> wrote:> Tuesday, February 26, 2013, 9:41:53 AM, you wrote: > >>>>> On 25.02.13 at 23:18, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>> I can''t get linux-3.9 rc0 to boot under xen-unstable. >>> It doesn''t detect the s-ata controller, so it ends op with udev timing and >>> bailing out to busybox. >>> >>> I don''t see a obvious error in the logs. > >> Perhaps because the log is far from being complete? There''s a huge >> gap right before the first pciback message, yet that''s quite likely >> the relevant part. > > Hi Jan / Konrad, > > Tried bisecting, but that ended up no where, so back to the logs... > > With v3.9-rc0 + xen, it''s indeed missing a part of the log: > [ 4.141328] Brought up 6 CPUs > [ 4.142654] Grant tables using version 2 layout. > [ 4.142676] Grant table initialized > [ 4.142813] NET: Registered protocol family 16 > [ 4.145343] node 0 link 0: io port [1000, ffffff] > [ 4.145354] TOM: 00000000b0000000 aka 2816M > [ 4.145360] Fam 10h mmconf [mem 0xe0000000-0xefffffff] > [ 4.145373] node 0 link 0: mmio [e0000000, efffffff] ==> none > [ 4.145381] node 0 link 0: mmio [f0000000, ffffffff] > [ 4.145388] node 0 link 0: mmio [a0000, bffff] > [ 4.145395] node 0 link 0: mmio [b0000000, dfffffff] > [ 4.145401] TOM2: 0000000250000000 aka 9472M > [ 4.145406] bus: [bus 00-07] on node 0 link 0 > [ 4.145411] bus: 00 [io 0x0000-0xffff] > [ 4.145415] bus: 00 [mem 0xf0000000-0xffffffff] > [ 4.145420] bus: 00 [mem 0x000a0000-0x000bffff] > [ 4.145424] bus: 00 [mem 0xb0000000-0xdfffffff] > [ 4.145429] bus: 00 [mem 0x250000000-0xfcffffffff] > [ 4.145702] ACPI: bus type pci registered > [ 4.146801] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem > 0xe0000000-0xefffffff] (base 0xe0000000) > [ 4.146812] PCI: not using MMCONFIG > [ 4.146817] PCI: Using configuration type 1 for base access > [ 4.146822] PCI: Using configuration type 1 for extended access > [ 4.191935] bio: create slab <bio-0> at 0 > [ 4.192623] ACPI: Added _OSI(Module Device) > [ 4.192630] ACPI: Added _OSI(Processor Device) > [ 4.192636] ACPI: Added _OSI(3.0 _SCP Extensions) > [ 4.192642] ACPI: Added _OSI(Processor Aggregator Device) > [ 4.195958] ACPI: EC: Look up EC in DSDT > [ 4.199659] ACPI: Executed 3 blocks of module-level executable AML code > [ 4.204162] ACPI: Interpreter enabled > [ 4.204170] ACPI: (supports S0 S5) > [ 4.204181] ACPI: Using IOAPIC for interrupt routing > [ 4.204219] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem > 0xe0000000-0xefffffff] (base 0xe0000000) > [ 4.205800] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in > ACPI motherboa[ 7.107382] usb usb5: New USB device found, idVendor=1d6b, > idProduct=0001 > > > The line in the serial-log also seems to be truncated somehow and the rest of > the info missing .. > > When booting v3.9-rc0 Baremetal the complete pci enumeration seems to be in > between: > ... > Tried booting with pci=nommconf,nocrs but to no avail. > Are there any other boot options i could try to narrow it down ?You first of all want to make sure you get a complete log. "sync_console" or "serial_tx_buffer=<size>" are your friends... But then again I would guess it''s not bus enumeration related. Did the multiple-MSI-vectors suspicion lead nowhere? Jan
Sander Eikelenboom
2013-Feb-27 11:46 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Wednesday, February 27, 2013, 12:06:31 PM, you wrote:>>>> On 27.02.13 at 11:57, Sander Eikelenboom <linux@eikelenboom.it> wrote:>> Tuesday, February 26, 2013, 9:41:53 AM, you wrote: >> >>>>>> On 25.02.13 at 23:18, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>>> I can''t get linux-3.9 rc0 to boot under xen-unstable. >>>> It doesn''t detect the s-ata controller, so it ends op with udev timing and >>>> bailing out to busybox. >>>> >>>> I don''t see a obvious error in the logs. >> >>> Perhaps because the log is far from being complete? There''s a huge >>> gap right before the first pciback message, yet that''s quite likely >>> the relevant part. >> >> Hi Jan / Konrad, >> >> Tried bisecting, but that ended up no where, so back to the logs... >> >> With v3.9-rc0 + xen, it''s indeed missing a part of the log: >> [ 4.141328] Brought up 6 CPUs >> [ 4.142654] Grant tables using version 2 layout. >> [ 4.142676] Grant table initialized >> [ 4.142813] NET: Registered protocol family 16 >> [ 4.145343] node 0 link 0: io port [1000, ffffff] >> [ 4.145354] TOM: 00000000b0000000 aka 2816M >> [ 4.145360] Fam 10h mmconf [mem 0xe0000000-0xefffffff] >> [ 4.145373] node 0 link 0: mmio [e0000000, efffffff] ==> none >> [ 4.145381] node 0 link 0: mmio [f0000000, ffffffff] >> [ 4.145388] node 0 link 0: mmio [a0000, bffff] >> [ 4.145395] node 0 link 0: mmio [b0000000, dfffffff] >> [ 4.145401] TOM2: 0000000250000000 aka 9472M >> [ 4.145406] bus: [bus 00-07] on node 0 link 0 >> [ 4.145411] bus: 00 [io 0x0000-0xffff] >> [ 4.145415] bus: 00 [mem 0xf0000000-0xffffffff] >> [ 4.145420] bus: 00 [mem 0x000a0000-0x000bffff] >> [ 4.145424] bus: 00 [mem 0xb0000000-0xdfffffff] >> [ 4.145429] bus: 00 [mem 0x250000000-0xfcffffffff] >> [ 4.145702] ACPI: bus type pci registered >> [ 4.146801] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem >> 0xe0000000-0xefffffff] (base 0xe0000000) >> [ 4.146812] PCI: not using MMCONFIG >> [ 4.146817] PCI: Using configuration type 1 for base access >> [ 4.146822] PCI: Using configuration type 1 for extended access >> [ 4.191935] bio: create slab <bio-0> at 0 >> [ 4.192623] ACPI: Added _OSI(Module Device) >> [ 4.192630] ACPI: Added _OSI(Processor Device) >> [ 4.192636] ACPI: Added _OSI(3.0 _SCP Extensions) >> [ 4.192642] ACPI: Added _OSI(Processor Aggregator Device) >> [ 4.195958] ACPI: EC: Look up EC in DSDT >> [ 4.199659] ACPI: Executed 3 blocks of module-level executable AML code >> [ 4.204162] ACPI: Interpreter enabled >> [ 4.204170] ACPI: (supports S0 S5) >> [ 4.204181] ACPI: Using IOAPIC for interrupt routing >> [ 4.204219] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem >> 0xe0000000-0xefffffff] (base 0xe0000000) >> [ 4.205800] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in >> ACPI motherboa[ 7.107382] usb usb5: New USB device found, idVendor=1d6b, >> idProduct=0001 >> >> >> The line in the serial-log also seems to be truncated somehow and the rest of >> the info missing .. >> >> When booting v3.9-rc0 Baremetal the complete pci enumeration seems to be in >> between: >> ... >> Tried booting with pci=nommconf,nocrs but to no avail. >> Are there any other boot options i could try to narrow it down ?> You first of all want to make sure you get a complete log. > "sync_console" or "serial_tx_buffer=<size>" are your friends...Ah that worked :-) Complete serial log and lspci-v38 attached. Sata device is: 00:11.0 From the log i see: [ 16.132653] pci 0000:00:11.0: [1002:4390] type 00 class 0x01018f [ 16.150679] pci 0000:00:11.0: calling quirk_no_ata_d3+0x0/0x10 [ 16.168365] pci 0000:00:11.0: reg 10: [io 0x7000-0x7007] [ 16.184733] pci 0000:00:11.0: reg 14: [io 0x6000-0x6003] [ 16.201116] pci 0000:00:11.0: reg 18: [io 0x5000-0x5007] [ 16.217491] pci 0000:00:11.0: reg 1c: [io 0x3000-0x3003] [ 16.233868] pci 0000:00:11.0: reg 20: [io 0x2000-0x200f] [ 16.250249] pci 0000:00:11.0: reg 24: [mem 0xf96ff000-0xf96ff3ff] [ 16.268714] pci 0000:00:11.0: calling quirk_amd_ide_mode+0x0/0xe0 [ 16.287163] pci 0000:00:11.0: set SATA to AHCI mode <snip> [ 21.524945] pci 0000:00:11.0: BAR 0: reserving [io 0x7000-0x7007 flags 0x40101] (d=0, p=0) [ 21.550163] pci 0000:00:11.0: BAR 1: reserving [io 0x6000-0x6003 flags 0x40101] (d=0, p=0) [ 21.575411] pci 0000:00:11.0: BAR 2: reserving [io 0x5000-0x5007 flags 0x40101] (d=0, p=0) [ 21.600648] pci 0000:00:11.0: BAR 3: reserving [io 0x3000-0x3003 flags 0x40101] (d=0, p=0) [ 21.625849] pci 0000:00:11.0: BAR 4: reserving [io 0x2000-0x200f flags 0x40101] (d=0, p=0) [ 21.651069] pci 0000:00:11.0: BAR 5: reserving [mem 0xf96ff000-0xf96ff3ff flags 0x40200] (d=0, p=0) <snip> [ 24.957481] pci 0000:00:11.0: calling quirk_msi_intx_disable_ati_bug+0x0/0x50 <snip> [ 89.189428] ahci 0000:00:11.0: version 3.0 [ 89.207872] xen: registering gsi 19 triggering 0 polarity 1 [ 89.230656] xen: --> pirq=19 -> irq=19 (gsi=19) (XEN) [2013-02-27 11:21:21] IOAPIC[0]: Set PCI routing entry (6-19 -> 0xa9 -> IRQ 19 Mode:1 Active:1) [ 89.277231] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 89.307529] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22> But then again I would guess it''s not bus enumeration related. > Did the multiple-MSI-vectors suspicion lead nowhere?> Jan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Jan Beulich
2013-Feb-27 12:54 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: > [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22Which is -EINVAL. With nothing else printed, I''m afraid you need to find the origin of this return value by instrumenting the involved call tree. Jan
Sander Eikelenboom
2013-Feb-27 17:50 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Wednesday, February 27, 2013, 1:54:31 PM, you wrote:>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22> Which is -EINVAL. With nothing else printed, I''m afraid you need to > find the origin of this return value by instrumenting the involved > call tree.Just wondering, is multiple msi''s per device actually supported by xen ? -- Sander> Jan
Konrad Rzeszutek Wilk
2013-Feb-27 19:28 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote:> > Wednesday, February 27, 2013, 1:54:31 PM, you wrote: > > >>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: > >> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 > > > Which is -EINVAL. With nothing else printed, I''m afraid you need to > > find the origin of this return value by instrumenting the involved > > call tree. > > Just wondering, is multiple msi''s per device actually supported by xen ?That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs use them and they work great with Xen. BTW, this is merge: ommit 5800700f66678ea5c85e7d62b138416070bf7f60 Merge: 266d7ad af8d102 Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Tue Feb 19 19:07:27 2013 -0800 Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull x86/apic changes from Ingo Molnar: "Main changes: - Multiple MSI support added to the APIC, PCI and AHCI code - acked by all relevant maintainers, by Alexander Gordeev. The advantage is that multiple AHCI ports can have multiple MSI irqs assigned, and can thus spread to multiple CPUs. [ Drivers can make use of this new facility via the pci_enable_msi_block_auto() method ] With MSI per device, the hypercall that ends up happening is: PHYSDEVOP_map_pirq with: map_irq.domid = domid; map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; map_irq.index = -1; map_irq.pirq = -1; map_irq.bus = dev->bus->number | (pci_domain_nr(dev->bus) << 16); map_irq.devfn = dev->devfn; Which would imply that we are doing this call multiple times? (This is xen_initdom_setup_msi_irqs). It looks like pci_enable_msi_block_auto is the multiple MSI one and it should perculate down to xen_initdom_setup_msi_irqs. Granted the xen_init.. does not do anything with the ''nvec'' call. So could I ask you try out your hunch by doing three things: 1). Instrument xen_initdom_setup_msi_irqs to see if the nvec has anything but 1 and in its loop instrument to see if it has more than on MSI attribute? 2). The ahci driver has ahci_init_interrupts which only does the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) to have AHCI_HFLAG_NO_MSI flag (you probably want to do this seperatly from 1). 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab?> > -- > Sander > > > Jan > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Sander Eikelenboom
2013-Feb-27 19:56 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Wednesday, February 27, 2013, 8:28:10 PM, you wrote:> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: >> >> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >> >> >>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> >> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >> >> > Which is -EINVAL. With nothing else printed, I''m afraid you need to >> > find the origin of this return value by instrumenting the involved >> > call tree. >> >> Just wondering, is multiple msi''s per device actually supported by xen ?> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs > use them and they work great with Xen.> BTW, this is merge: > ommit 5800700f66678ea5c85e7d62b138416070bf7f60 > Merge: 266d7ad af8d102 > Author: Linus Torvalds <torvalds@linux-foundation.org> > Date: Tue Feb 19 19:07:27 2013 -0800> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > > Pull x86/apic changes from Ingo Molnar: > "Main changes: > > - Multiple MSI support added to the APIC, PCI and AHCI code - acked > by all relevant maintainers, by Alexander Gordeev. > > The advantage is that multiple AHCI ports can have multiple MSI > irqs assigned, and can thus spread to multiple CPUs. > > [ Drivers can make use of this new facility via the > pci_enable_msi_block_auto() method ]Ahh yes, i have added some debug info to ahci.c: [ 36.778395] SE | bus: ''pci'': really_probe: probing driver ahci with device 0000:00:11.0 [ 36.809777] really_probe: pinctrl_bind_pins(0000:00:11.0) ret: 0 [ 36.835136] ahci 0000:00:11.0: SE | ahci_init_one start [ 36.858284] ahci 0000:00:11.0: version 3.0 [ 36.877840] xen: registering gsi 19 triggering 0 polarity 1 [ 36.901791] xen: --> pirq=19 -> irq=19 (gsi=19) (XEN) [2013-02-27 19:43:07] IOAPIC[0]: Set PCI routing entry (6-19 -> 0x99 -> IRQ 19 Mode:1 Active:1) [ 36.949293] ahci 0000:00:11.0: SE | pcim_enable_device(pdev) rc:0 [ 36.974714] ahci 0000:00:11.0: SE pcim_iomap_regions_request_all(pdev, 1 << ahci_pci_bar, DRV_NAME) rc:0 [ 37.010706] ahci 0000:00:11.0: SE pci_enable_msi_block_auto(pdev, &maxvec) rc:4 [ 37.039878] ahci 0000:00:11.0: SE | n_msis: 4 [ 37.060115] ahci 0000:00:11.0: SE | ahci_configure_dma_masks(pdev, hpriv->cap & HOST_CAP_64) rc:0 [ 37.094135] ahci 0000:00:11.0: SE | ahci_pci_reset_controller(host) rc:0 [ 37.121658] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 37.153118] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 37.184265] ahci 0000:00:11.0: SE | me here 1 [ 37.204568] ahci 0000:00:11.0: SE | n_msis(4) host->n_ports(4) irq:121 [ 37.231748] ahci 0000:00:11.0: SE | ata_host_start(host) rc:0 [ 37.256222] ahci 0000:00:11.0: SE | devm_request_threaded_irq i:0 rc:0 [ 37.283023] ahci 0000:00:11.0: SE | devm_request_threaded_irq i:1 rc:-22 [ 37.310344] really_probe: dev->bus->probe(0000:00:11.0) ret: -22 [ 37.335467] ahci: probe of 0000:00:11.0 failed with error -22 [ 37.359552] really_probe: 0000:00:11.0 done ret: 0 So it bails out at the second devm_request_threaded_irq in: int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis) { int i, rc; dev_err(host->dev, "SE | n_msis(%d) host->n_ports(%d) irq:%d\n",n_msis , host->n_ports,irq); /* Sharing Last Message among several ports is not supported */ if (n_msis < host->n_ports){ dev_err(host->dev, "SE | uhoh n_msis(%d) < host->n_ports(%d) irq:%d\n",n_msis , host->n_ports,irq); return -EINVAL; } rc = ata_host_start(host); dev_err(host->dev, "SE | ata_host_start(host) rc:%d\n",rc); if (rc) return rc; for (i = 0; i < host->n_ports; i++) { rc = devm_request_threaded_irq(host->dev, irq + i, ahci_hw_interrupt, ahci_thread_fn, IRQF_SHARED, dev_driver_string(host->dev), host->ports[i]); dev_err(host->dev, "SE | devm_request_threaded_irq i:%d rc:%d\n",i,rc); if (rc) goto out_free_irqs; }> With MSI per device, the hypercall that ends up happening is: > PHYSDEVOP_map_pirq with:> map_irq.domid = domid; > map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; > map_irq.index = -1; > map_irq.pirq = -1; > map_irq.bus = dev->bus->number | > (pci_domain_nr(dev->bus) << 16); > map_irq.devfn = dev->devfn;> Which would imply that we are doing this call multiple times? > (This is xen_initdom_setup_msi_irqs).> It looks like pci_enable_msi_block_auto is the multiple MSI one > and it should perculate down to xen_initdom_setup_msi_irqs. > Granted the xen_init.. does not do anything with the ''nvec'' call.> So could I ask you try out your hunch by doing three things: > 1). Instrument xen_initdom_setup_msi_irqs to see if the > nvec has anything but 1 and in its loop instrument to > see if it has more than on MSI attribute?> 2). The ahci driver has ahci_init_interrupts which only does > the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. > If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) > to have AHCI_HFLAG_NO_MSI flag (you probably want to do this > seperatly from 1). > 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 > and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab? >> >> -- >> Sander >> >> > Jan >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>
Sander Eikelenboom
2013-Feb-27 20:41 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Wednesday, February 27, 2013, 8:28:10 PM, you wrote:> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: >> >> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >> >> >>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> >> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >> >> > Which is -EINVAL. With nothing else printed, I''m afraid you need to >> > find the origin of this return value by instrumenting the involved >> > call tree. >> >> Just wondering, is multiple msi''s per device actually supported by xen ?> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs > use them and they work great with Xen.> BTW, this is merge: > ommit 5800700f66678ea5c85e7d62b138416070bf7f60 > Merge: 266d7ad af8d102 > Author: Linus Torvalds <torvalds@linux-foundation.org> > Date: Tue Feb 19 19:07:27 2013 -0800> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > > Pull x86/apic changes from Ingo Molnar: > "Main changes: > > - Multiple MSI support added to the APIC, PCI and AHCI code - acked > by all relevant maintainers, by Alexander Gordeev. > > The advantage is that multiple AHCI ports can have multiple MSI > irqs assigned, and can thus spread to multiple CPUs. > > [ Drivers can make use of this new facility via the > pci_enable_msi_block_auto() method ]> With MSI per device, the hypercall that ends up happening is: > PHYSDEVOP_map_pirq with:> map_irq.domid = domid; > map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; > map_irq.index = -1; > map_irq.pirq = -1; > map_irq.bus = dev->bus->number | > (pci_domain_nr(dev->bus) << 16); > map_irq.devfn = dev->devfn;> Which would imply that we are doing this call multiple times? > (This is xen_initdom_setup_msi_irqs).> It looks like pci_enable_msi_block_auto is the multiple MSI one > and it should perculate down to xen_initdom_setup_msi_irqs. > Granted the xen_init.. does not do anything with the ''nvec'' call.> So could I ask you try out your hunch by doing three things: > 1). Instrument xen_initdom_setup_msi_irqs to see if the > nvec has anything but 1 and in its loop instrument to > see if it has more than on MSI attribute?> 2). The ahci driver has ahci_init_interrupts which only does > the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. > If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) > to have AHCI_HFLAG_NO_MSI flag (you probably want to do this > seperatly from 1). > 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 > and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab?So of interest are commits: - 5ca72c4f7c412c2002363218901eba5516c476b1 - 08261d87f7d1b6253ab3223756625a5c74532293 - 51906e779f2b13b38f8153774c4c7163d412ffd9 Hmmm reading the commit message of 51906e779f2b13b38f8153774c4c7163d412ffd9: x86/MSI: Support multiple MSIs in presense of IRQ remapping The MSI specification has several constraints in comparison with MSI-X, most notable of them is the inability to configure MSIs independently. As a result, it is impossible to dispatch interrupts from different queues to different CPUs. This is largely devalues the support of multiple MSIs in SMP systems. Also, a necessity to allocate a contiguous block of vector numbers for devices capable of multiple MSIs might cause a considerable pressure on x86 interrupt vector allocator and could lead to fragmentation of the interrupt vectors space. This patch overcomes both drawbacks in presense of IRQ remapping and lets devices take advantage of multiple queues and per-IRQ affinity assignments. At least makes clear why baremetal does boot and xen doesn''t: Baremetal behaves differently and thus boots because interrupt remapping gets disabled on boot by the kernel iommu code due to the buggy bios iommu errata, so according to the commit message above it doesn''t even try the multiple MSI per device scenario. So the question is if it can be enabled in Xen (and if it actually could be beneficial because the commit messages seems to indicate that could be questionable). If not, the check in arch/x86/kernel/apic/io_apic.c:setup_msi_irqs should fail -- Sander>> >> -- >> Sander >> >> > Jan >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>
konrad wilk
2013-Feb-27 22:22 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
On 2/27/2013 3:41 PM, Sander Eikelenboom wrote:> Wednesday, February 27, 2013, 8:28:10 PM, you wrote: > >> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: >>> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >>> >>>>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>>>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >>>> Which is -EINVAL. With nothing else printed, I''m afraid you need to >>>> find the origin of this return value by instrumenting the involved >>>> call tree. >>> Just wondering, is multiple msi''s per device actually supported by xen ? >> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs >> use them and they work great with Xen. >> BTW, this is merge: >> ommit 5800700f66678ea5c85e7d62b138416070bf7f60 >> Merge: 266d7ad af8d102 >> Author: Linus Torvalds <torvalds@linux-foundation.org> >> Date: Tue Feb 19 19:07:27 2013 -0800 >> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip >> >> Pull x86/apic changes from Ingo Molnar: >> "Main changes: >> >> - Multiple MSI support added to the APIC, PCI and AHCI code - acked >> by all relevant maintainers, by Alexander Gordeev. >> >> The advantage is that multiple AHCI ports can have multiple MSI >> irqs assigned, and can thus spread to multiple CPUs. >> >> [ Drivers can make use of this new facility via the >> pci_enable_msi_block_auto() method ] > > >> With MSI per device, the hypercall that ends up happening is: >> PHYSDEVOP_map_pirq with: >> map_irq.domid = domid; >> map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; >> map_irq.index = -1; >> map_irq.pirq = -1; >> map_irq.bus = dev->bus->number | >> (pci_domain_nr(dev->bus) << 16); >> map_irq.devfn = dev->devfn; >> Which would imply that we are doing this call multiple times? >> (This is xen_initdom_setup_msi_irqs). >> It looks like pci_enable_msi_block_auto is the multiple MSI one >> and it should perculate down to xen_initdom_setup_msi_irqs. >> Granted the xen_init.. does not do anything with the ''nvec'' call. >> So could I ask you try out your hunch by doing three things: >> 1). Instrument xen_initdom_setup_msi_irqs to see if the >> nvec has anything but 1 and in its loop instrument to >> see if it has more than on MSI attribute? >> 2). The ahci driver has ahci_init_interrupts which only does >> the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. >> If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) >> to have AHCI_HFLAG_NO_MSI flag (you probably want to do this >> seperatly from 1). >> 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 >> and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab? > > So of interest are commits: > - 5ca72c4f7c412c2002363218901eba5516c476b1 > - 08261d87f7d1b6253ab3223756625a5c74532293 > - 51906e779f2b13b38f8153774c4c7163d412ffd9 > > Hmmm reading the commit message of 51906e779f2b13b38f8153774c4c7163d412ffd9: > > x86/MSI: Support multiple MSIs in presense of IRQ remapping > > The MSI specification has several constraints in comparison with > MSI-X, most notable of them is the inability to configure MSIs > independently. As a result, it is impossible to dispatch > interrupts from different queues to different CPUs. This is > largely devalues the support of multiple MSIs in SMP systems. > > Also, a necessity to allocate a contiguous block of vector > numbers for devices capable of multiple MSIs might cause a > considerable pressure on x86 interrupt vector allocator and > could lead to fragmentation of the interrupt vectors space. > > This patch overcomes both drawbacks in presense of IRQ remapping > and lets devices take advantage of multiple queues and per-IRQ > affinity assignments. > > At least makes clear why baremetal does boot and xen doesn''t: > > Baremetal behaves differently and thus boots because interrupt remapping gets disabled on boot by the kernel iommu code due to the buggy bios iommu errata, so according to the commit message above it doesn''t even try the multiple MSI per device scenario. > > So the question is if it can be enabled in Xen (and if it actually could be beneficial because the commit messages seems to indicate that could be questionable). > If not, the check in arch/x86/kernel/apic/io_apic.c:setup_msi_irqs should failExcept that function in Xen is not run. that is b/c x86_msi_ops.setup_msi_irqs end up pointing to xen_initdom_setup_irqs. While if IOMMU is enabled it gets set to irq_remapping_setup_msi_irqs. So a fix like this: diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index 56ab749..47f8cca 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -263,6 +263,9 @@ static int xen_initdom_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) int ret = 0; struct msi_desc *msidesc; + if (type == PCI_CAP_ID_MSI && nvec > 1) + return 1; + list_for_each_entry(msidesc, &dev->msi_list, list) { struct physdev_map_pirq map_irq; domid_t domid; (sorry about the paste getting messed up here) - ought to do it? As for example on one of my AMD machines there is no IOMMU, and this is where AHCI does work under baremetal but not under Xen. We can future wise implement a better version of this to deal with multiple MSIs, but lets make sure to first get it booting.> -- > Sander > > > > >>> -- >>> Sander >>> >>>> Jan >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel >>> > >
Sander Eikelenboom
2013-Feb-27 23:57 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Wednesday, February 27, 2013, 11:22:18 PM, you wrote:> On 2/27/2013 3:41 PM, Sander Eikelenboom wrote: >> Wednesday, February 27, 2013, 8:28:10 PM, you wrote: >> >>> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: >>>> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >>>> >>>>>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>>>>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >>>>> Which is -EINVAL. With nothing else printed, I''m afraid you need to >>>>> find the origin of this return value by instrumenting the involved >>>>> call tree. >>>> Just wondering, is multiple msi''s per device actually supported by xen ? >>> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs >>> use them and they work great with Xen. >>> BTW, this is merge: >>> ommit 5800700f66678ea5c85e7d62b138416070bf7f60 >>> Merge: 266d7ad af8d102 >>> Author: Linus Torvalds <torvalds@linux-foundation.org> >>> Date: Tue Feb 19 19:07:27 2013 -0800 >>> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip >>> >>> Pull x86/apic changes from Ingo Molnar: >>> "Main changes: >>> >>> - Multiple MSI support added to the APIC, PCI and AHCI code - acked >>> by all relevant maintainers, by Alexander Gordeev. >>> >>> The advantage is that multiple AHCI ports can have multiple MSI >>> irqs assigned, and can thus spread to multiple CPUs. >>> >>> [ Drivers can make use of this new facility via the >>> pci_enable_msi_block_auto() method ] >> >> >>> With MSI per device, the hypercall that ends up happening is: >>> PHYSDEVOP_map_pirq with: >>> map_irq.domid = domid; >>> map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; >>> map_irq.index = -1; >>> map_irq.pirq = -1; >>> map_irq.bus = dev->bus->number | >>> (pci_domain_nr(dev->bus) << 16); >>> map_irq.devfn = dev->devfn; >>> Which would imply that we are doing this call multiple times? >>> (This is xen_initdom_setup_msi_irqs). >>> It looks like pci_enable_msi_block_auto is the multiple MSI one >>> and it should perculate down to xen_initdom_setup_msi_irqs. >>> Granted the xen_init.. does not do anything with the ''nvec'' call. >>> So could I ask you try out your hunch by doing three things: >>> 1). Instrument xen_initdom_setup_msi_irqs to see if the >>> nvec has anything but 1 and in its loop instrument to >>> see if it has more than on MSI attribute? >>> 2). The ahci driver has ahci_init_interrupts which only does >>> the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. >>> If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) >>> to have AHCI_HFLAG_NO_MSI flag (you probably want to do this >>> seperatly from 1). >>> 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 >>> and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab? >> >> So of interest are commits: >> - 5ca72c4f7c412c2002363218901eba5516c476b1 >> - 08261d87f7d1b6253ab3223756625a5c74532293 >> - 51906e779f2b13b38f8153774c4c7163d412ffd9 >> >> Hmmm reading the commit message of 51906e779f2b13b38f8153774c4c7163d412ffd9: >> >> x86/MSI: Support multiple MSIs in presense of IRQ remapping >> >> The MSI specification has several constraints in comparison with >> MSI-X, most notable of them is the inability to configure MSIs >> independently. As a result, it is impossible to dispatch >> interrupts from different queues to different CPUs. This is >> largely devalues the support of multiple MSIs in SMP systems. >> >> Also, a necessity to allocate a contiguous block of vector >> numbers for devices capable of multiple MSIs might cause a >> considerable pressure on x86 interrupt vector allocator and >> could lead to fragmentation of the interrupt vectors space. >> >> This patch overcomes both drawbacks in presense of IRQ remapping >> and lets devices take advantage of multiple queues and per-IRQ >> affinity assignments. >> >> At least makes clear why baremetal does boot and xen doesn''t: >> >> Baremetal behaves differently and thus boots because interrupt remapping gets disabled on boot by the kernel iommu code due to the buggy bios iommu errata, so according to the commit message above it doesn''t even try the multiple MSI per device scenario. >> >> So the question is if it can be enabled in Xen (and if it actually could be beneficial because the commit messages seems to indicate that could be questionable). >> If not, the check in arch/x86/kernel/apic/io_apic.c:setup_msi_irqs should fail > Except that function in Xen is not run. that is b/c > x86_msi_ops.setup_msi_irqs end up pointing to xen_initdom_setup_irqs. > While if IOMMU is enabled it gets set to irq_remapping_setup_msi_irqs.> So a fix like this: > diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c > index 56ab749..47f8cca 100644 > --- a/arch/x86/pci/xen.c > +++ b/arch/x86/pci/xen.c > @@ -263,6 +263,9 @@ static int xen_initdom_setup_msi_irqs(struct pci_dev > *dev, int nvec, int type) > int ret = 0; > struct msi_desc *msidesc;> + if (type == PCI_CAP_ID_MSI && nvec > 1) > + return 1; > + > list_for_each_entry(msidesc, &dev->msi_list, list) { > struct physdev_map_pirq map_irq; > domid_t domid;> (sorry about the paste getting messed up here) - ought to do it? As for > example on one of my AMD machines there is no IOMMU, and this is where > AHCI does work under baremetal but not under Xen.Yes it boots again :-) [ 37.742109] SE | bus: ''pci'': really_probe: probing driver ahci with device 0000:00:11.0 [ 37.773491] really_probe: pinctrl_bind_pins(0000:00:11.0) ret: 0 [ 37.798862] ahci 0000:00:11.0: SE | ahci_init_one start [ 37.822040] ahci 0000:00:11.0: version 3.0 [ 37.841606] xen: registering gsi 19 triggering 0 polarity 1 [ 37.865577] xen: --> pirq=19 -> irq=19 (gsi=19) [ 37.913087] ahci 0000:00:11.0: SE | pcim_enable_device(pdev) rc:0 [ 37.938519] ahci 0000:00:11.0: SE pcim_iomap_regions_request_all(pdev, 1 << ahci_pci_bar, DRV_NAME) rc:0 [ 37.974447] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 4 type:5 [ 38.001806] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 1 type:5 [ 38.029026] ahci 0000:00:11.0: SE pci_enable_msi_block_auto(pdev, &maxvec) rc:1 [ 38.057960] ahci 0000:00:11.0: SE | n_msis: 1 [ 38.078065] ahci 0000:00:11.0: SE | ahci_configure_dma_masks(pdev, hpriv->cap & HOST_CAP_64) rc:0 [ 38.112045] ahci 0000:00:11.0: SE | ahci_pci_reset_controller(host) rc:0 [ 38.139426] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 38.170664] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 38.201684] ahci 0000:00:11.0: SE | me here 1 [ 38.221977] ahci 0000:00:11.0: SE | me here 2 [ 38.244756] scsi0 : ahci [ 38.259700] scsi1 : ahci [ 38.274411] scsi2 : ahci [ 38.289278] scsi3 : ahci [ 38.303718] ata1: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff100 irq 121 [ 38.332566] ata2: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff180 irq 121 [ 38.361366] ata3: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff200 irq 121 [ 38.390080] ata4: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff280 irq 121 [ 38.418787] really_probe: dev->bus->probe(0000:00:11.0) ret: 0 [ 38.442420] really_probe: 0000:00:11.0 done ret: 1> We can future wise implement a better version of this to deal with > multiple MSIs, but lets make sure to first get it booting. >> -- >> Sander >> >> >> >> >>>> -- >>>> Sander >>>> >>>>> Jan >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel >>>> >> >>
Jan Beulich
2013-Feb-28 07:51 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
>>> On 27.02.13 at 18:50, Sander Eikelenboom <linux@eikelenboom.it> wrote: > Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 > >> Which is -EINVAL. With nothing else printed, I''m afraid you need to >> find the origin of this return value by instrumenting the involved >> call tree. > > Just wondering, is multiple msi''s per device actually supported by xen ?No, it isn''t; only MSI-X is. But that should be irrelevant to your case, as iirc one of the quirk related messages your logs had said something about disabling MSI mode for this controller (or even a wider part of the system). Jan
Sander Eikelenboom
2013-Feb-28 08:15 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Thursday, February 28, 2013, 8:51:52 AM, you wrote:>>>> On 27.02.13 at 18:50, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >>>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >>>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >> >>> Which is -EINVAL. With nothing else printed, I''m afraid you need to >>> find the origin of this return value by instrumenting the involved >>> call tree. >> >> Just wondering, is multiple msi''s per device actually supported by xen ?> No, it isn''t; only MSI-X is. But that should be irrelevant to your > case, as iirc one of the quirk related messages your logs had said > something about disabling MSI mode for this controller (or even a > wider part of the system).Ah you seem to refer to the line: [ 24.957481] pci 0000:00:11.0: calling quirk_msi_intx_disable_ati_bug+0x0/0x50 That one threw me off as well, but it seems it''s only "Calling" the quirk, comments in the code make it clear it''s only for the SB700, i have a SB850. And the "no it isn''t" seems to be quite relevant in my case and in general, see later mails and the associated commits .. (all merged very earlier in this merge window and the ahci changes not via the ahci tree ...) -- Sander> Jan
Konrad Rzeszutek Wilk
2013-Feb-28 13:52 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
On Thu, Feb 28, 2013 at 12:57:24AM +0100, Sander Eikelenboom wrote:> > Wednesday, February 27, 2013, 11:22:18 PM, you wrote: > > > > On 2/27/2013 3:41 PM, Sander Eikelenboom wrote: > >> Wednesday, February 27, 2013, 8:28:10 PM, you wrote: > >> > >>> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: > >>>> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: > >>>> > >>>>>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: > >>>>>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 > >>>>> Which is -EINVAL. With nothing else printed, I''m afraid you need to > >>>>> find the origin of this return value by instrumenting the involved > >>>>> call tree. > >>>> Just wondering, is multiple msi''s per device actually supported by xen ? > >>> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs > >>> use them and they work great with Xen. > >>> BTW, this is merge: > >>> ommit 5800700f66678ea5c85e7d62b138416070bf7f60 > >>> Merge: 266d7ad af8d102 > >>> Author: Linus Torvalds <torvalds@linux-foundation.org> > >>> Date: Tue Feb 19 19:07:27 2013 -0800 > >>> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > >>> > >>> Pull x86/apic changes from Ingo Molnar: > >>> "Main changes: > >>> > >>> - Multiple MSI support added to the APIC, PCI and AHCI code - acked > >>> by all relevant maintainers, by Alexander Gordeev. > >>> > >>> The advantage is that multiple AHCI ports can have multiple MSI > >>> irqs assigned, and can thus spread to multiple CPUs. > >>> > >>> [ Drivers can make use of this new facility via the > >>> pci_enable_msi_block_auto() method ] > >> > >> > >>> With MSI per device, the hypercall that ends up happening is: > >>> PHYSDEVOP_map_pirq with: > >>> map_irq.domid = domid; > >>> map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; > >>> map_irq.index = -1; > >>> map_irq.pirq = -1; > >>> map_irq.bus = dev->bus->number | > >>> (pci_domain_nr(dev->bus) << 16); > >>> map_irq.devfn = dev->devfn; > >>> Which would imply that we are doing this call multiple times? > >>> (This is xen_initdom_setup_msi_irqs). > >>> It looks like pci_enable_msi_block_auto is the multiple MSI one > >>> and it should perculate down to xen_initdom_setup_msi_irqs. > >>> Granted the xen_init.. does not do anything with the ''nvec'' call. > >>> So could I ask you try out your hunch by doing three things: > >>> 1). Instrument xen_initdom_setup_msi_irqs to see if the > >>> nvec has anything but 1 and in its loop instrument to > >>> see if it has more than on MSI attribute? > >>> 2). The ahci driver has ahci_init_interrupts which only does > >>> the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. > >>> If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) > >>> to have AHCI_HFLAG_NO_MSI flag (you probably want to do this > >>> seperatly from 1). > >>> 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 > >>> and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab? > >> > >> So of interest are commits: > >> - 5ca72c4f7c412c2002363218901eba5516c476b1 > >> - 08261d87f7d1b6253ab3223756625a5c74532293 > >> - 51906e779f2b13b38f8153774c4c7163d412ffd9 > >> > >> Hmmm reading the commit message of 51906e779f2b13b38f8153774c4c7163d412ffd9: > >> > >> x86/MSI: Support multiple MSIs in presense of IRQ remapping > >> > >> The MSI specification has several constraints in comparison with > >> MSI-X, most notable of them is the inability to configure MSIs > >> independently. As a result, it is impossible to dispatch > >> interrupts from different queues to different CPUs. This is > >> largely devalues the support of multiple MSIs in SMP systems. > >> > >> Also, a necessity to allocate a contiguous block of vector > >> numbers for devices capable of multiple MSIs might cause a > >> considerable pressure on x86 interrupt vector allocator and > >> could lead to fragmentation of the interrupt vectors space. > >> > >> This patch overcomes both drawbacks in presense of IRQ remapping > >> and lets devices take advantage of multiple queues and per-IRQ > >> affinity assignments. > >> > >> At least makes clear why baremetal does boot and xen doesn''t: > >> > >> Baremetal behaves differently and thus boots because interrupt remapping gets disabled on boot by the kernel iommu code due to the buggy bios iommu errata, so according to the commit message above it doesn''t even try the multiple MSI per device scenario. > >> > >> So the question is if it can be enabled in Xen (and if it actually could be beneficial because the commit messages seems to indicate that could be questionable). > >> If not, the check in arch/x86/kernel/apic/io_apic.c:setup_msi_irqs should fail > > Except that function in Xen is not run. that is b/c > > x86_msi_ops.setup_msi_irqs end up pointing to xen_initdom_setup_irqs. > > While if IOMMU is enabled it gets set to irq_remapping_setup_msi_irqs. > > > So a fix like this: > > diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c > > index 56ab749..47f8cca 100644 > > --- a/arch/x86/pci/xen.c > > +++ b/arch/x86/pci/xen.c > > @@ -263,6 +263,9 @@ static int xen_initdom_setup_msi_irqs(struct pci_dev > > *dev, int nvec, int type) > > int ret = 0; > > struct msi_desc *msidesc; > > > + if (type == PCI_CAP_ID_MSI && nvec > 1) > > + return 1; > > + > > list_for_each_entry(msidesc, &dev->msi_list, list) { > > struct physdev_map_pirq map_irq; > > domid_t domid; > > > > (sorry about the paste getting messed up here) - ought to do it? As for > > example on one of my AMD machines there is no IOMMU, and this is where > > AHCI does work under baremetal but not under Xen. > > Yes it boots again :-)Great! Are you OK if I put ''Reported-and-Tested-by:" tag on the patch with your name for the above quick fix? Thanks!> > [ 37.742109] SE | bus: ''pci'': really_probe: probing driver ahci with device 0000:00:11.0 > [ 37.773491] really_probe: pinctrl_bind_pins(0000:00:11.0) ret: 0 > [ 37.798862] ahci 0000:00:11.0: SE | ahci_init_one start > [ 37.822040] ahci 0000:00:11.0: version 3.0 > [ 37.841606] xen: registering gsi 19 triggering 0 polarity 1 > [ 37.865577] xen: --> pirq=19 -> irq=19 (gsi=19) > [ 37.913087] ahci 0000:00:11.0: SE | pcim_enable_device(pdev) rc:0 > [ 37.938519] ahci 0000:00:11.0: SE pcim_iomap_regions_request_all(pdev, 1 << ahci_pci_bar, DRV_NAME) rc:0 > [ 37.974447] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 4 type:5 > [ 38.001806] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 1 type:5 > [ 38.029026] ahci 0000:00:11.0: SE pci_enable_msi_block_auto(pdev, &maxvec) rc:1 > [ 38.057960] ahci 0000:00:11.0: SE | n_msis: 1 > [ 38.078065] ahci 0000:00:11.0: SE | ahci_configure_dma_masks(pdev, hpriv->cap & HOST_CAP_64) rc:0 > [ 38.112045] ahci 0000:00:11.0: SE | ahci_pci_reset_controller(host) rc:0 > [ 38.139426] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode > [ 38.170664] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 38.201684] ahci 0000:00:11.0: SE | me here 1 > [ 38.221977] ahci 0000:00:11.0: SE | me here 2 > [ 38.244756] scsi0 : ahci > [ 38.259700] scsi1 : ahci > [ 38.274411] scsi2 : ahci > [ 38.289278] scsi3 : ahci > [ 38.303718] ata1: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff100 irq 121 > [ 38.332566] ata2: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff180 irq 121 > [ 38.361366] ata3: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff200 irq 121 > [ 38.390080] ata4: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff280 irq 121 > [ 38.418787] really_probe: dev->bus->probe(0000:00:11.0) ret: 0 > [ 38.442420] really_probe: 0000:00:11.0 done ret: 1 > > > > We can future wise implement a better version of this to deal with > > multiple MSIs, but lets make sure to first get it booting. > >> -- > >> Sander > >> > >> > >> > >> > >>>> -- > >>>> Sander > >>>> > >>>>> Jan > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Xen-devel mailing list > >>>> Xen-devel@lists.xen.org > >>>> http://lists.xen.org/xen-devel > >>>> > >> > >> > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Sander Eikelenboom
2013-Feb-28 13:57 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
Thursday, February 28, 2013, 2:52:20 PM, you wrote:> On Thu, Feb 28, 2013 at 12:57:24AM +0100, Sander Eikelenboom wrote: >> >> Wednesday, February 27, 2013, 11:22:18 PM, you wrote: >> >> >> > On 2/27/2013 3:41 PM, Sander Eikelenboom wrote: >> >> Wednesday, February 27, 2013, 8:28:10 PM, you wrote: >> >> >> >>> On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: >> >>>> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: >> >>>> >> >>>>>>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: >> >>>>>> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 >> >>>>> Which is -EINVAL. With nothing else printed, I''m afraid you need to >> >>>>> find the origin of this return value by instrumenting the involved >> >>>>> call tree. >> >>>> Just wondering, is multiple msi''s per device actually supported by xen ? >> >>> That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs >> >>> use them and they work great with Xen. >> >>> BTW, this is merge: >> >>> ommit 5800700f66678ea5c85e7d62b138416070bf7f60 >> >>> Merge: 266d7ad af8d102 >> >>> Author: Linus Torvalds <torvalds@linux-foundation.org> >> >>> Date: Tue Feb 19 19:07:27 2013 -0800 >> >>> Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip >> >>> >> >>> Pull x86/apic changes from Ingo Molnar: >> >>> "Main changes: >> >>> >> >>> - Multiple MSI support added to the APIC, PCI and AHCI code - acked >> >>> by all relevant maintainers, by Alexander Gordeev. >> >>> >> >>> The advantage is that multiple AHCI ports can have multiple MSI >> >>> irqs assigned, and can thus spread to multiple CPUs. >> >>> >> >>> [ Drivers can make use of this new facility via the >> >>> pci_enable_msi_block_auto() method ] >> >> >> >> >> >>> With MSI per device, the hypercall that ends up happening is: >> >>> PHYSDEVOP_map_pirq with: >> >>> map_irq.domid = domid; >> >>> map_irq.type = MAP_PIRQ_TYPE_MSI_SEG; >> >>> map_irq.index = -1; >> >>> map_irq.pirq = -1; >> >>> map_irq.bus = dev->bus->number | >> >>> (pci_domain_nr(dev->bus) << 16); >> >>> map_irq.devfn = dev->devfn; >> >>> Which would imply that we are doing this call multiple times? >> >>> (This is xen_initdom_setup_msi_irqs). >> >>> It looks like pci_enable_msi_block_auto is the multiple MSI one >> >>> and it should perculate down to xen_initdom_setup_msi_irqs. >> >>> Granted the xen_init.. does not do anything with the ''nvec'' call. >> >>> So could I ask you try out your hunch by doing three things: >> >>> 1). Instrument xen_initdom_setup_msi_irqs to see if the >> >>> nvec has anything but 1 and in its loop instrument to >> >>> see if it has more than on MSI attribute? >> >>> 2). The ahci driver has ahci_init_interrupts which only does >> >>> the multiple MSI thing if AHCI_HFLAG_NO_MSI is not set. >> >>> If you edit drivers/ata/ahci ahci_port_info for the SB600 (or 700?) >> >>> to have AHCI_HFLAG_NO_MSI flag (you probably want to do this >> >>> seperatly from 1). >> >>> 3). Checkout before merge 5800700f66678ea5c85e7d62b138416070bf7f60 >> >>> and try 266d7ad7f4fe2f44b91561f5b812115c1b3018ab? >> >> >> >> So of interest are commits: >> >> - 5ca72c4f7c412c2002363218901eba5516c476b1 >> >> - 08261d87f7d1b6253ab3223756625a5c74532293 >> >> - 51906e779f2b13b38f8153774c4c7163d412ffd9 >> >> >> >> Hmmm reading the commit message of 51906e779f2b13b38f8153774c4c7163d412ffd9: >> >> >> >> x86/MSI: Support multiple MSIs in presense of IRQ remapping >> >> >> >> The MSI specification has several constraints in comparison with >> >> MSI-X, most notable of them is the inability to configure MSIs >> >> independently. As a result, it is impossible to dispatch >> >> interrupts from different queues to different CPUs. This is >> >> largely devalues the support of multiple MSIs in SMP systems. >> >> >> >> Also, a necessity to allocate a contiguous block of vector >> >> numbers for devices capable of multiple MSIs might cause a >> >> considerable pressure on x86 interrupt vector allocator and >> >> could lead to fragmentation of the interrupt vectors space. >> >> >> >> This patch overcomes both drawbacks in presense of IRQ remapping >> >> and lets devices take advantage of multiple queues and per-IRQ >> >> affinity assignments. >> >> >> >> At least makes clear why baremetal does boot and xen doesn''t: >> >> >> >> Baremetal behaves differently and thus boots because interrupt remapping gets disabled on boot by the kernel iommu code due to the buggy bios iommu errata, so according to the commit message above it doesn''t even try the multiple MSI per device scenario. >> >> >> >> So the question is if it can be enabled in Xen (and if it actually could be beneficial because the commit messages seems to indicate that could be questionable). >> >> If not, the check in arch/x86/kernel/apic/io_apic.c:setup_msi_irqs should fail >> > Except that function in Xen is not run. that is b/c >> > x86_msi_ops.setup_msi_irqs end up pointing to xen_initdom_setup_irqs. >> > While if IOMMU is enabled it gets set to irq_remapping_setup_msi_irqs. >> >> > So a fix like this: >> > diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c >> > index 56ab749..47f8cca 100644 >> > --- a/arch/x86/pci/xen.c >> > +++ b/arch/x86/pci/xen.c >> > @@ -263,6 +263,9 @@ static int xen_initdom_setup_msi_irqs(struct pci_dev >> > *dev, int nvec, int type) >> > int ret = 0; >> > struct msi_desc *msidesc; >> >> > + if (type == PCI_CAP_ID_MSI && nvec > 1) >> > + return 1; >> > + >> > list_for_each_entry(msidesc, &dev->msi_list, list) { >> > struct physdev_map_pirq map_irq; >> > domid_t domid; >> >> >> > (sorry about the paste getting messed up here) - ought to do it? As for >> > example on one of my AMD machines there is no IOMMU, and this is where >> > AHCI does work under baremetal but not under Xen. >> >> Yes it boots again :-)> Great! Are you OK if I put ''Reported-and-Tested-by:" tag on the patch with your > name for the above quick fix?Sure !> Thanks! >> >> [ 37.742109] SE | bus: ''pci'': really_probe: probing driver ahci with device 0000:00:11.0 >> [ 37.773491] really_probe: pinctrl_bind_pins(0000:00:11.0) ret: 0 >> [ 37.798862] ahci 0000:00:11.0: SE | ahci_init_one start >> [ 37.822040] ahci 0000:00:11.0: version 3.0 >> [ 37.841606] xen: registering gsi 19 triggering 0 polarity 1 >> [ 37.865577] xen: --> pirq=19 -> irq=19 (gsi=19) >> [ 37.913087] ahci 0000:00:11.0: SE | pcim_enable_device(pdev) rc:0 >> [ 37.938519] ahci 0000:00:11.0: SE pcim_iomap_regions_request_all(pdev, 1 << ahci_pci_bar, DRV_NAME) rc:0 >> [ 37.974447] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 4 type:5 >> [ 38.001806] ahci 0000:00:11.0: xen_initdom_setup_msi_irqs nvec: 1 type:5 >> [ 38.029026] ahci 0000:00:11.0: SE pci_enable_msi_block_auto(pdev, &maxvec) rc:1 >> [ 38.057960] ahci 0000:00:11.0: SE | n_msis: 1 >> [ 38.078065] ahci 0000:00:11.0: SE | ahci_configure_dma_masks(pdev, hpriv->cap & HOST_CAP_64) rc:0 >> [ 38.112045] ahci 0000:00:11.0: SE | ahci_pci_reset_controller(host) rc:0 >> [ 38.139426] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode >> [ 38.170664] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part >> [ 38.201684] ahci 0000:00:11.0: SE | me here 1 >> [ 38.221977] ahci 0000:00:11.0: SE | me here 2 >> [ 38.244756] scsi0 : ahci >> [ 38.259700] scsi1 : ahci >> [ 38.274411] scsi2 : ahci >> [ 38.289278] scsi3 : ahci >> [ 38.303718] ata1: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff100 irq 121 >> [ 38.332566] ata2: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff180 irq 121 >> [ 38.361366] ata3: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff200 irq 121 >> [ 38.390080] ata4: SATA max UDMA/133 abar m1024@0xf96ff000 port 0xf96ff280 irq 121 >> [ 38.418787] really_probe: dev->bus->probe(0000:00:11.0) ret: 0 >> [ 38.442420] really_probe: 0000:00:11.0 done ret: 1 >> >> >> > We can future wise implement a better version of this to deal with >> > multiple MSIs, but lets make sure to first get it booting. >> >> -- >> >> Sander >> >> >> >> >> >> >> >> >> >>>> -- >> >>>> Sander >> >>>> >> >>>>> Jan >> >>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> Xen-devel mailing list >> >>>> Xen-devel@lists.xen.org >> >>>> http://lists.xen.org/xen-devel >> >>>> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>
Konrad Rzeszutek Wilk
2013-Feb-28 14:20 UTC
Re: linux-3.9-rc0 regression from 3.8 SATA controller not detected under xen
On Wed, Feb 27, 2013 at 08:56:07PM +0100, Sander Eikelenboom wrote:> > Wednesday, February 27, 2013, 8:28:10 PM, you wrote: > > > On Wed, Feb 27, 2013 at 06:50:59PM +0100, Sander Eikelenboom wrote: > >> > >> Wednesday, February 27, 2013, 1:54:31 PM, you wrote: > >> > >> >>>> On 27.02.13 at 12:46, Sander Eikelenboom <linux@eikelenboom.it> wrote: > >> >> [ 89.338827] ahci: probe of 0000:00:11.0 failed with error -22 > >> > >> > Which is -EINVAL. With nothing else printed, I''m afraid you need to > >> > find the origin of this return value by instrumenting the involved > >> > call tree. > >> > >> Just wondering, is multiple msi''s per device actually supported by xen ? > > > That is very good question. I know we support MSI-X b/c 1GB or 10GB NICs > > use them and they work great with Xen. > > > BTW, this is merge: > > ommit 5800700f66678ea5c85e7d62b138416070bf7f60 > > Merge: 266d7ad af8d102 > > Author: Linus Torvalds <torvalds@linux-foundation.org> > > Date: Tue Feb 19 19:07:27 2013 -0800 > > > Merge branch ''x86-apic-for-linus'' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > > > > Pull x86/apic changes from Ingo Molnar: > > "Main changes: > > > > - Multiple MSI support added to the APIC, PCI and AHCI code - acked > > by all relevant maintainers, by Alexander Gordeev. > > > > The advantage is that multiple AHCI ports can have multiple MSI > > irqs assigned, and can thus spread to multiple CPUs. > > > > [ Drivers can make use of this new facility via the > > pci_enable_msi_block_auto() method ] > > > Ahh yes, i have added some debug info to ahci.c: > > [ 36.778395] SE | bus: ''pci'': really_probe: probing driver ahci with device 0000:00:11.0 > [ 36.809777] really_probe: pinctrl_bind_pins(0000:00:11.0) ret: 0 > [ 36.835136] ahci 0000:00:11.0: SE | ahci_init_one start > [ 36.858284] ahci 0000:00:11.0: version 3.0 > [ 36.877840] xen: registering gsi 19 triggering 0 polarity 1 > [ 36.901791] xen: --> pirq=19 -> irq=19 (gsi=19) > (XEN) [2013-02-27 19:43:07] IOAPIC[0]: Set PCI routing entry (6-19 -> 0x99 -> IRQ 19 Mode:1 Active:1) > [ 36.949293] ahci 0000:00:11.0: SE | pcim_enable_device(pdev) rc:0 > [ 36.974714] ahci 0000:00:11.0: SE pcim_iomap_regions_request_all(pdev, 1 << ahci_pci_bar, DRV_NAME) rc:0 > [ 37.010706] ahci 0000:00:11.0: SE pci_enable_msi_block_auto(pdev, &maxvec) rc:4 > [ 37.039878] ahci 0000:00:11.0: SE | n_msis: 4 > [ 37.060115] ahci 0000:00:11.0: SE | ahci_configure_dma_masks(pdev, hpriv->cap & HOST_CAP_64) rc:0 > [ 37.094135] ahci 0000:00:11.0: SE | ahci_pci_reset_controller(host) rc:0 > [ 37.121658] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode > [ 37.153118] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part > [ 37.184265] ahci 0000:00:11.0: SE | me here 1 > [ 37.204568] ahci 0000:00:11.0: SE | n_msis(4) host->n_ports(4) irq:121 > [ 37.231748] ahci 0000:00:11.0: SE | ata_host_start(host) rc:0 > [ 37.256222] ahci 0000:00:11.0: SE | devm_request_threaded_irq i:0 rc:0 > [ 37.283023] ahci 0000:00:11.0: SE | devm_request_threaded_irq i:1 rc:-22 > [ 37.310344] really_probe: dev->bus->probe(0000:00:11.0) ret: -22 > [ 37.335467] ahci: probe of 0000:00:11.0 failed with error -22 > [ 37.359552] really_probe: 0000:00:11.0 done ret: 0 > > So it bails out at the second devm_request_threaded_irq in: > > int ahci_host_activate(struct ata_host *host, int irq, unsigned int n_msis) > { > int i, rc; > > dev_err(host->dev, "SE | n_msis(%d) host->n_ports(%d) irq:%d\n",n_msis , host->n_ports,irq); > /* Sharing Last Message among several ports is not supported */ > if (n_msis < host->n_ports){ > dev_err(host->dev, "SE | uhoh n_msis(%d) < host->n_ports(%d) irq:%d\n",n_msis , host->n_ports,irq); > return -EINVAL; > } > rc = ata_host_start(host); > dev_err(host->dev, "SE | ata_host_start(host) rc:%d\n",rc); > if (rc) > return rc; > > for (i = 0; i < host->n_ports; i++) { > rc = devm_request_threaded_irq(host->dev, > irq + i, ahci_hw_interrupt, ahci_thread_fn, IRQF_SHARED, > dev_driver_string(host->dev), host->ports[i]); > dev_err(host->dev, "SE | devm_request_threaded_irq i:%d rc:%d\n",i,rc); > if (rc) > goto out_free_irqs; > } >This is what I have for the patch, OK with me sending it tomorrow to Linus? From f4dce2c2114ef623dc6d931b5ea950a08e80057b Mon Sep 17 00:00:00 2001 From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Date: Thu, 28 Feb 2013 09:05:41 -0500 Subject: [PATCH] xen/pci: We don''t do multiple MSI''s. There is no hypercall to setup multiple MSI per PCI device. As such with these two new commits: - 08261d87f7d1b6253ab3223756625a5c74532293 PCI/MSI: Enable multiple MSIs with pci_enable_msi_block_auto() - 5ca72c4f7c412c2002363218901eba5516c476b1 AHCI: Support multiple MSIs we would call the PHYSDEVOP_map_pirq ''nvec'' times with the same contents of the PCI device. Sander discovered that we would get the same PIRQ value ''nvec'' times and return said values to the caller. That of course meant that the device was configured only with one MSI and AHCI would fail with: ahci 0000:00:11.0: version 3.0 xen: registering gsi 19 triggering 0 polarity 1 xen: --> pirq=19 -> irq=19 (gsi=19) (XEN) [2013-02-27 19:43:07] IOAPIC[0]: Set PCI routing entry (6-19 -> 0x99 -> IRQ 19 Mode:1 Active:1) ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ahci: probe of 0000:00:11.0 failed with error -22 That is b/c in ahci_host_activate the second call to devm_request_threaded_irq would return -EINVAL as we passed in (on the second run) an IRQ value that was never initialized. Reported-and-Tested-by: Sander Eikelenboom <linux@eikelenboom.it> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> --- arch/x86/pci/xen.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index 56ab749..94e7662 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -162,6 +162,9 @@ static int xen_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) struct msi_desc *msidesc; int *v; + if (type == PCI_CAP_ID_MSI && nvec > 1) + return 1; + v = kzalloc(sizeof(int) * max(1, nvec), GFP_KERNEL); if (!v) return -ENOMEM; @@ -220,6 +223,9 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) struct msi_desc *msidesc; struct msi_msg msg; + if (type == PCI_CAP_ID_MSI && nvec > 1) + return 1; + list_for_each_entry(msidesc, &dev->msi_list, list) { __read_msi_msg(msidesc, &msg); pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | @@ -263,6 +269,9 @@ static int xen_initdom_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) int ret = 0; struct msi_desc *msidesc; + if (type == PCI_CAP_ID_MSI && nvec > 1) + return 1; + list_for_each_entry(msidesc, &dev->msi_list, list) { struct physdev_map_pirq map_irq; domid_t domid; -- 1.8.0.2