Tom Goetz
2012-Mar-05 16:11 UTC
Lose XHCI Device on HP Ivybridge While Resuming on Battery
After resuming more than once on battery these HP Ivybridge laptops, the XHCI devices stop working. Have anyone seen this before? I wanted to check before diving in deeper. Let me know if you have any ideas. Thanks! Facts - Xen 4.0.3, Linux 3.2.7 PVOPs - Happens on HP Ivybridge. Doesn''t happen on very similar HP Sandybridge Clash system. - Happens on battery, but not on AC. - Doesn''t happen on first suspend/resume. First resume may be long. Don''t have enough sample to be sure of this. - CPU power governor doesn''t effect the issue. Happens irregardless of governor. - Seems to effect MSI devices only. - IRQ changes appear in good resume case. Not just bad. I''ve diffed lspci output across a working resume on AC and across a bad resume on battery. There seem to be the differences unique to the bad case: 00:14.0 USB controller: Intel Corporation Panther Point USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Hewlett-Packard Company Device 179b -Flags: bus master, medium devsel, latency 0, IRQ 326 +Flags: medium devsel, IRQ 327 <-------------- lose bus master, latency 0 Memory at 94720000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: xhci_hcd Kernel modules: xhci-hcd --- 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04) Subsystem: Hewlett-Packard Company Device 179b -Flags: bus master, fast devsel, latency 0, IRQ 327 -Memory at 94700000 (32-bit, non-prefetchable) [size=128K] -Memory at 9473a000 (32-bit, non-prefetchable) [size=4K] -I/O ports at 4060 [size=32] +Flags: fast devsel, IRQ 20 <-------------- lose bus master, latency 0 +Memory at 94700000 (32-bit, non-prefetchable) [disabled] [size=128K] <------ becomes disabled +Memory at 9473a000 (32-bit, non-prefetchable) [disabled] [size=4K] <------ becomes disabled +I/O ports at 4060 [disabled] [size=32] <------ becomes disabled Capabilities: [c8] Power Management version 2 -Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ +Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ <---- toggle enable polarity Capabilities: [e0] PCI Advanced Features Kernel driver in use: e1000e Kernel modules: e1000e --- 00:1a.0 USB controller: Intel Corporation Panther Point USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI]) Subsystem: Hewlett-Packard Company Device 179b -Flags: bus master, medium devsel, latency 0, IRQ 16 +Flags: medium devsel, IRQ 16 <-------------- lose bus master, latency 0 Memory at 94739000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci_hcd Kernel modules: ehci-hcd syslog looks like this for both good and bad resumes: Mar 5 15:48:51 newt kernel: [ 289.730303] xhci_hcd 0000:00:14.0: can''t derive routing for PCI INT A Mar 5 15:48:51 newt kernel: [ 289.730311] xhci_hcd 0000:00:14.0: PCI INT A: no GSI - using ISA IRQ 10 Mar 5 15:48:51 newt kernel: [ 289.730319] xen: registering gsi 10 triggering 0 polarity 1 Mar 5 15:48:51 newt kernel: [ 289.730327] xen_map_pirq_gsi: returning irq 10 for gsi 10 Mar 5 15:48:51 newt kernel: [ 289.730332] xen: --> pirq=10 -> irq=10 (gsi=10) Mar 5 15:48:51 newt kernel: [ 289.730337] Already setup the GSI :10 Mar 5 15:48:51 newt kernel: [ 289.730445] xhci_hcd 0000:00:14.0: setting latency timer to 64 Mar 5 15:48:51 newt kernel: [ 289.730455] xhci_hcd 0000:00:14.0: xHCI Host Controller Mar 5 15:48:51 newt kernel: [ 289.730687] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 Mar 5 15:48:51 newt kernel: [ 289.730826] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported Mar 5 15:48:51 newt kernel: [ 289.730858] xhci_hcd 0000:00:14.0: irq 10, io mem 0x94720000 Mar 5 15:48:51 newt kernel: [ 289.731470] xHCI xhci_add_endpoint called for root hub Mar 5 15:48:51 newt kernel: [ 289.731477] xHCI xhci_check_bandwidth called for root hub Mar 5 15:48:51 newt kernel: [ 289.731551] hub 3-0:1.0: USB hub found Mar 5 15:48:51 newt kernel: [ 289.731568] hub 3-0:1.0: 4 ports detected Mar 5 15:48:51 newt kernel: [ 289.843501] xhci_hcd 0000:00:14.0: xHCI Host Controller Mar 5 15:48:51 newt kernel: [ 289.843722] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 Mar 5 15:48:51 newt kernel: [ 289.844015] xHCI xhci_add_endpoint called for root hub Mar 5 15:48:51 newt kernel: [ 289.844021] xHCI xhci_check_bandwidth called for root hub Mar 5 15:48:51 newt kernel: [ 289.844084] hub 4-0:1.0: USB hub found Mar 5 15:48:51 newt kernel: [ 289.844104] hub 4-0:1.0: 4 ports detected
Tom Goetz
2012-Mar-09 20:24 UTC
Re: Lose XHCI Device on HP Ivybridge While Resuming on Battery
On Mar 5, 2012, at 12:09 PM, xen-devel-request@lists.xen.org wrote:> Send Xen-devel mailing list submissions to > xen-devel@lists.xen.org > > > > Today''s Topics: > > 2. Lose XHCI Device on HP Ivybridge While Resuming on Battery > (Tom Goetz) > > > > Message: 2 > Date: Mon, 5 Mar 2012 11:11:10 -0500 > From: Tom Goetz <tom.goetz@virtualcomputer.com> > To: xen-devel@lists.xen.org > Subject: [Xen-devel] Lose XHCI Device on HP Ivybridge While Resuming > on Battery > Message-ID: <5D6CE9D4-9859-47F7-9885-7A8CAD00AF64@virtualcomputer.com> > Content-Type: text/plain; charset=us-ascii > > After resuming more than once on battery these Ivybridge laptop, the XHCI devices stop working. Have anyone seen this before? I wanted to check before diving in deeper. Let me know if you have any ideas. Thanks! >I have more information on this. Here''s the sequence of events: 1. XHCI driver removed with SUSPEND_MODULES 2. suspend 3. resume 4. XHCI driver loaded by SUSPEND_MODULES 5. Device is enabled by driver. 6. Device disabled by runtime PM. 7. Plugging in a USB device does not result in the device being enabled. This works on the same machine in Ubuntu 12.04 Beta 1 (Linux 3.2.0) native which is very similar to our Ubuntu 12.04 based (Linux 3.2.5 & 3.2.9) build. So either this was broken between 3.2.0 and 3.2.5 or this is Xen related. I''ve included some tracing of resume and runtime pm suspend of the device. A USB flash stick was plugged in at the end of the log where nothing happens: Mar 8 21:57:47 newt kernel: [ 407.323444] __pm_runtime_resume[879] 0000:00:14.0 RPM_GET_PUT 2 Mar 8 21:57:47 newt kernel: [ 407.323451] rpm_resume[512] 0000:00:14.0 before Mar 8 21:57:47 newt kernel: [ 407.323461] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 1 Mar 8 21:57:47 newt kernel: [ 407.323477] __pm_runtime_resume[879] 0000:00:14.0 RPM_GET_PUT 2 Mar 8 21:57:47 newt kernel: [ 407.323487] rpm_resume[512] 0000:00:14.0 before Mar 8 21:57:47 newt kernel: [ 407.323489] xhci_hcd 0000:00:14.0: remove, state 4 Mar 8 21:57:47 newt kernel: [ 407.323530] rpm_resume[512] 0000:00:14.0 before Mar 8 21:57:47 newt kernel: [ 407.323532] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 2 Mar 8 21:57:47 newt kernel: [ 407.323731] xhci_hcd 0000:00:14.0: USB bus 4 deregistered Mar 8 21:57:47 newt kernel: [ 407.323820] xhci_hcd 0000:00:14.0: remove, state 4 Mar 8 21:57:47 newt kernel: [ 407.323862] rpm_resume[512] 0000:00:14.0 before Mar 8 21:57:47 newt kernel: [ 407.323873] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 3 Mar 8 21:57:47 newt kernel: [ 407.324123] xhci_hcd 0000:00:14.0: USB bus 3 deregistered Mar 8 21:57:47 newt kernel: [ 407.324232] pci_disable_device[1345] 0000:00:14.0 Mar 8 21:57:47 newt kernel: [ 407.324244] xhci_hcd 0000:00:14.0: can''t derive routing for PCI INT A Mar 8 21:57:47 newt kernel: [ 407.324246] pci_disable_device[1353] 0000:00:14.0 busmaster off Mar 8 21:57:57 newt kernel: [ 409.053506] rpm_resume[512] 0000:00:14.0 before Mar 8 21:57:57 newt kernel: [ 413.422310] pci 0000:00:14.0: restoring config space at offset 0xf (was 0x100, writing 0x10a) Mar 8 21:57:57 newt kernel: [ 413.422354] pci 0000:00:14.0: restoring config space at offset 0x4 (was 0x4, writing 0x94720004) Mar 8 21:57:57 newt kernel: [ 413.422368] pci 0000:00:14.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900002) Mar 8 21:57:57 newt kernel: [ 413.424862] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 2 Mar 8 21:57:58 newt kernel: [ 414.122464] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 1 Mar 8 21:57:59 newt kernel: [ 415.758316] __pci_enable_device_flags[1129] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.758323] __pci_enable_device_flags[1133] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.758342] xhci_hcd 0000:00:14.0: can''t derive routing for PCI INT A Mar 8 21:57:59 newt kernel: [ 415.758348] xhci_hcd 0000:00:14.0: PCI INT A: no GSI - using ISA IRQ 10 Mar 8 21:57:59 newt kernel: [ 415.758378] __pci_enable_device_flags[1145] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.758481] pci_set_master[2620] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.758485] __pci_set_master[2593] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.758510] __pci_set_master[2606] 0000:00:14.0 enabling pci_write_config_word ret 0 Mar 8 21:57:59 newt kernel: [ 415.758519] xhci_hcd 0000:00:14.0: setting latency timer to 64 Mar 8 21:57:59 newt kernel: [ 415.758529] xhci_hcd 0000:00:14.0: xHCI Host Controller Mar 8 21:57:59 newt kernel: [ 415.758771] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 Mar 8 21:57:59 newt kernel: [ 415.758924] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported Mar 8 21:57:59 newt kernel: [ 415.758957] xhci_hcd 0000:00:14.0: irq 10, io mem 0x94720000 Mar 8 21:57:59 newt kernel: [ 415.759002] pci_enable_msi_block[699] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.759027] pci_enable_msi_block[703] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.759036] pci_enable_msi_block[709] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.759062] pci_enable_msi_block[716] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.759066] pci_enable_msi_block[724] 0000:00:14.0 Mar 8 21:57:59 newt kernel: [ 415.821636] xhci_hcd 0000:00:14.0: xHCI Host Controller Mar 8 21:57:59 newt kernel: [ 415.821879] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 Mar 8 21:57:59 newt kernel: [ 415.971575] __pm_runtime_idle[811] 0000:00:14.0 RPM_GET_PUT 1 Mar 8 21:57:59 newt kernel: [ 416.403188] pm_runtime_allow[1156] 0000:00:14.0 1 Mar 8 21:57:59 newt kernel: [ 416.403195] pm_runtime_allow[1159] 0000:00:14.0 IDLE Mar 8 21:57:59 newt kernel: [ 416.403202] rpm_idle[206] 0000:00:14.0 before Mar 8 21:57:59 newt kernel: [ 416.403208] rpm_check_suspend_allowed[145] 0000:00:14.0 children Mar 8 21:57:59 newt kernel: [ 416.403214] rpm_idle[210] 0000:00:14.0 !conditions Mar 8 21:58:01 newt kernel: [ 418.441612] rpm_suspend[474] usb3 parent 0000:00:14.0 Mar 8 21:58:01 newt kernel: [ 418.441613] rpm_idle[206] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441615] rpm_check_suspend_allowed[145] 0000:00:14.0 children Mar 8 21:58:01 newt kernel: [ 418.441616] rpm_idle[210] 0000:00:14.0 !conditions Mar 8 21:58:01 newt kernel: [ 418.441630] rpm_suspend[474] usb4 parent 0000:00:14.0 Mar 8 21:58:01 newt kernel: [ 418.441631] rpm_idle[206] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441635] pm_runtime_work[708] 0000:00:14.0 RPM_REQ_IDLE Mar 8 21:58:01 newt kernel: [ 418.441636] rpm_idle[206] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441639] pci_pm_runtime_idle[1064] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441640] pci_pm_runtime_idle[1074] 0000:00:14.0 pm_runtime_suspend Mar 8 21:58:01 newt kernel: [ 418.441642] __pm_runtime_suspend[849] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441643] rpm_suspend[323] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441644] rpm_suspend[419] 0000:00:14.0 RPM_SUSPENDING Mar 8 21:58:01 newt kernel: [ 418.441646] pci_pm_runtime_suspend[1016] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441648] hcd_pci_runtime_suspend[564] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441649] suspend_common[389] 0000:00:14.0 before Mar 8 21:58:01 newt kernel: [ 418.441688] pci_disable_device[1345] 0000:00:14.0 Mar 8 21:58:01 newt kernel: [ 418.441697] pci_disable_device[1353] 0000:00:14.0 busmaster off Mar 8 21:58:01 newt kernel: [ 418.441850] suspend_common[436] 0000:00:14.0 after Mar 8 21:58:01 newt kernel: [ 418.441852] hcd_pci_runtime_suspend[569] 0000:00:14.0 afyer ret 0 Mar 8 21:58:01 newt kernel: [ 418.441853] pci_pm_runtime_suspend[1018] 0000:00:14.0 after Mar 8 21:58:01 newt kernel: [ 418.441917] xhci_hcd 0000:00:14.0: PME# enabled Mar 8 21:58:01 newt kernel: [ 418.461530] pci_pm_runtime_suspend[1037] 0000:00:14.0 done Mar 8 21:58:01 newt kernel: [ 418.461544] rpm_suspend[474] 0000:00:14.0 parent pci0000:00 Mar 8 21:58:01 newt kernel: [ 418.461553] __pm_runtime_suspend[853] 0000:00:14.0 after Mar 8 21:58:01 newt kernel: [ 418.461555] pci_pm_runtime_idle[1076] 0000:00:14.0 after
Konrad Rzeszutek Wilk
2012-Mar-13 16:25 UTC
Re: Lose XHCI Device on HP Ivybridge While Resuming on Battery
On Mon, Mar 05, 2012 at 11:11:10AM -0500, Tom Goetz wrote:> After resuming more than once on battery these HP Ivybridge laptops, the XHCI devices stop working. Have anyone seen this before? I wanted to check before diving in deeper. Let me know if you have any ideas. Thanks! > > Facts > - Xen 4.0.3, Linux 3.2.7 PVOPs > - Happens on HP Ivybridge. Doesn''t happen on very similar HP Sandybridge Clash system. > - Happens on battery, but not on AC.Did you guys try to limit the C-state? say max_cstate=2 ?> - Doesn''t happen on first suspend/resume. First resume may be long. Don''t have enough sample to be sure of this. > - CPU power governor doesn''t effect the issue. Happens irregardless of governor. > - Seems to effect MSI devices only. > - IRQ changes appear in good resume case. Not just bad. > > I''ve diffed lspci output across a working resume on AC and across a bad resume on battery. There seem to be the differences unique to the bad case: > > 00:14.0 USB controller: Intel Corporation Panther Point USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Hewlett-Packard Company Device 179b > -Flags: bus master, medium devsel, latency 0, IRQ 326 > +Flags: medium devsel, IRQ 327 <-------------- lose bus master, latency 0 > Memory at 94720000 (64-bit, non-prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > Kernel modules: xhci-hcd > > --- > > 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04) > Subsystem: Hewlett-Packard Company Device 179b > -Flags: bus master, fast devsel, latency 0, IRQ 327 > -Memory at 94700000 (32-bit, non-prefetchable) [size=128K] > -Memory at 9473a000 (32-bit, non-prefetchable) [size=4K] > -I/O ports at 4060 [size=32] > +Flags: fast devsel, IRQ 20 <-------------- lose bus master, latency 0 > +Memory at 94700000 (32-bit, non-prefetchable) [disabled] [size=128K] <------ becomes disabled > +Memory at 9473a000 (32-bit, non-prefetchable) [disabled] [size=4K] <------ becomes disabled > +I/O ports at 4060 [disabled] [size=32] <------ becomes disabled > Capabilities: [c8] Power Management version 2 > -Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > +Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ <---- toggle enable polarity > Capabilities: [e0] PCI Advanced Features > Kernel driver in use: e1000e > Kernel modules: e1000e > > --- > > 00:1a.0 USB controller: Intel Corporation Panther Point USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI]) > Subsystem: Hewlett-Packard Company Device 179b > -Flags: bus master, medium devsel, latency 0, IRQ 16 > +Flags: medium devsel, IRQ 16 <-------------- lose bus master, latency 0 > Memory at 94739000 (32-bit, non-prefetchable) [size=1K] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Debug port: BAR=1 offset=00a0 > Capabilities: [98] PCI Advanced Features > Kernel driver in use: ehci_hcd > Kernel modules: ehci-hcd > > syslog looks like this for both good and bad resumes: > > Mar 5 15:48:51 newt kernel: [ 289.730303] xhci_hcd 0000:00:14.0: can''t derive routing for PCI INT A > Mar 5 15:48:51 newt kernel: [ 289.730311] xhci_hcd 0000:00:14.0: PCI INT A: no GSI - using ISA IRQ 10 > Mar 5 15:48:51 newt kernel: [ 289.730319] xen: registering gsi 10 triggering 0 polarity 1 > Mar 5 15:48:51 newt kernel: [ 289.730327] xen_map_pirq_gsi: returning irq 10 for gsi 10 > Mar 5 15:48:51 newt kernel: [ 289.730332] xen: --> pirq=10 -> irq=10 (gsi=10) > Mar 5 15:48:51 newt kernel: [ 289.730337] Already setup the GSI :10 > Mar 5 15:48:51 newt kernel: [ 289.730445] xhci_hcd 0000:00:14.0: setting latency timer to 64 > Mar 5 15:48:51 newt kernel: [ 289.730455] xhci_hcd 0000:00:14.0: xHCI Host Controller > Mar 5 15:48:51 newt kernel: [ 289.730687] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 > Mar 5 15:48:51 newt kernel: [ 289.730826] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported > Mar 5 15:48:51 newt kernel: [ 289.730858] xhci_hcd 0000:00:14.0: irq 10, io mem 0x94720000 > Mar 5 15:48:51 newt kernel: [ 289.731470] xHCI xhci_add_endpoint called for root hub > Mar 5 15:48:51 newt kernel: [ 289.731477] xHCI xhci_check_bandwidth called for root hub > Mar 5 15:48:51 newt kernel: [ 289.731551] hub 3-0:1.0: USB hub found > Mar 5 15:48:51 newt kernel: [ 289.731568] hub 3-0:1.0: 4 ports detected > Mar 5 15:48:51 newt kernel: [ 289.843501] xhci_hcd 0000:00:14.0: xHCI Host Controller > Mar 5 15:48:51 newt kernel: [ 289.843722] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 > Mar 5 15:48:51 newt kernel: [ 289.844015] xHCI xhci_add_endpoint called for root hub > Mar 5 15:48:51 newt kernel: [ 289.844021] xHCI xhci_check_bandwidth called for root hub > Mar 5 15:48:51 newt kernel: [ 289.844084] hub 4-0:1.0: USB hub found > Mar 5 15:48:51 newt kernel: [ 289.844104] hub 4-0:1.0: 4 ports detected > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Tom Goetz
2012-Mar-13 17:36 UTC
Re: Lose XHCI Device on HP Ivybridge While Resuming on Battery
On Mar 13, 2012, at 12:25 PM, Konrad Rzeszutek Wilk wrote:> On Mon, Mar 05, 2012 at 11:11:10AM -0500, Tom Goetz wrote: >> After resuming more than once on battery these HP Ivybridge laptops, the XHCI devices stop working. Have anyone seen this before? I wanted to check before diving in deeper. Let me know if you have any ideas. Thanks! >> >> Facts >> - Xen 4.0.3, Linux 3.2.7 PVOPs >> - Happens on HP Ivybridge. Doesn''t happen on very similar HP Sandybridge Clash system. >> - Happens on battery, but not on AC. > > Did you guys try to limit the C-state? say max_cstate=2 ?Hi Konrad. Thanks for the reply. C-states don''t matter. Don''t know if you''ve seen my latest post. http://lists.xen.org/archives/html/xen-devel/2012-03/msg00904.html This seems to be something to do with not getting a wakeup interrupt for the xHCI device when it''s in D3 if it has previously been in S3 and resumed. I''m starting to debug the Xen -> Dom0 interrupt path for that now. Tom