I have been doing some experiments in upgrading the Xen version in a future version of XenClient Enterprise, and I''ve been running into a regression that I''m wondering if anyone else has seen. dom0 suspend/resume (S3) does not seem to be working for me. In swapping out components of the system, the common failure seems to be when I use Xen-4.2 (upgraded from Xen-4.0.3) The first suspend seems to mostly work...but subsequent ones always resume improperly. By "improperly" - I see I/O failures, and stalls of many processes. Below is a log excerpt of 2 S3 attempts. Has anyone else seen these failures? - Ben (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Breaking vcpu affinity for domain 0 vcpu 1 (XEN) Breaking vcpu affinity for domain 0 vcpu 2 (XEN) Breaking vcpu affinity for domain 0 vcpu 3 (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0 (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) Enabling non-boot CPUs ... (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 [ 36.440696] [drm:pch_irq_handler] *ERROR* PCH poison interrupt (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0 (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) Enabling non-boot CPUs ... (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 [ 65.893235] [drm:pch_irq_handler] *ERROR* PCH poison interrupt [ 66.508829] ata3.00: revalidation failed (errno=-5) [ 66.508861] ata1.00: revalidation failed (errno=-5) [ 76.858815] ata3.00: revalidation failed (errno=-5) [ 76.898807] ata1.00: revalidation failed (errno=-5) [ 107.208817] ata3.00: revalidation failed (errno=-5) [ 107.288807] ata1.00: revalidation failed (errno=-5) [ 107.718866] pm_op(): scsi_bus_resume_common+0x0/0x60 returns 262144 [ 107.718877] PM: Device 0:0:0:0 failed to resume async: error 262144 [ 107.718913] end_request: I/O error, dev sda, sector 35193296 [ 107.718919] Buffer I/O error on device dm-5, logical block 7690 [ 107.718947] end_request: I/O error, dev sda, sector 35657184 [ 107.718965] end_request: I/O error, dev sda, sector 246202760 [ 107.718968] Buffer I/O error on device dm-6, logical block 26252801 [ 107.718995] end_request: I/O error, dev sda, sector 254548368 [ 107.719009] Aborting journal on device dm-6-8. [ 107.719021] end_request: I/O error, dev sda, sector 35164192 [ 107.719023] Buffer I/O error on device dm-5, logical block 4052 [ 107.719063] Aborting journal on device dm-5-8. [ 107.719085] end_request: I/O error, dev sda, sector 254546304 [ 107.719097] Buffer I/O error on device dm-6, logical block 27295744 [ 107.719129] JBD2: I/O error detected when updating journal superblock for dm-6-8. [ 107.719141] end_request: I/O error, dev sda, sector 35656064 [ 107.719146] Buffer I/O error on device dm-5, logical block 65536 [ 107.719168] JBD2: I/O error detected when updating journal superblock for dm-5-8. [ 107.870082] end_request: I/O error, dev sda, sector 35131776 [ 107.875825] Buffer I/O error on device dm-5, logical block 0 [ 107.881805] end_request: I/O error, dev sda, sector 35131776 [ 107.887637] Buffer I/O error on device dm-5, logical block 0 [ 107.893573] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: [ 107.893579] EXT4-fs (dm-5): I/O error while writing superblock [ 107.893582] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: Detected aborted journal [ 107.893584] EXT4-fs (dm-5): Remounting filesystem read-only [ 107.893617] end_request: I/O error, dev sda, sector 35131776 [ 107.893620] Buffer I/O error on device dm-5, logical block 0 [ 107.893749] end_request: I/O error, dev sda, sector 36180352 [ 107.893752] Buffer I/O error on device dm-6, logical block 0 [ 107.893762] EXT4-fs error (device dm-6): ext4_journal_start_sb:327: Detected aborted journal [ 107.893765] EXT4-fs (dm-6): Remounting filesystem read-only [ 107.893766] EXT4-fs (dm-6): previous I/O error to superblock detected [ 107.893784] end_request: I/O error, dev sda, sector 36180352 [ 107.893787] Buffer I/O error on device dm-6, logical block 0 [ 107.894467] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: Detected aborted journal [ 108.669763] end_request: I/O error, dev sda, sector 25957784 [ 108.675555] Aborting journal on device dm-3-8. [ 108.680246] end_request: I/O error, dev sda, sector 25956736 [ 108.686099] JBD2: I/O error detected when updating journal superblock for dm-3-8. [ 108.693908] journal commit I/O error [ 108.755829] end_request: I/O error, dev sda, sector 17305984 [ 108.761600] EXT4-fs error (device dm-3): ext4_journal_start_sb:327: Detected aborted journal [ 108.770340] EXT4-fs (dm-3): Remounting filesystem read-only [ 108.776159] EXT4-fs (dm-3): previous I/O error to superblock detected [ 108.782904] end_request: I/O error, dev sda, sector 17305984 [ 109.660011] end_request: I/O error, dev sda, sector 358788 [ 109.665572] Buffer I/O error on device dm-1, logical block 46082 [ 109.682479] end_request: I/O error, dev sda, sector 18832256 [ 109.688246] end_request: I/O error, dev sda, sector 18832256 [ 109.709559] end_request: I/O error, dev sda, sector 357762 [ 109.715120] Buffer I/O error on device dm-1, logical block 45569 [ 109.721506] end_request: I/O error, dev sda, sector 358790 [ 109.727114] Buffer I/O error on device dm-1, logical block 46083 [ 109.743714] end_request: I/O error, dev sda, sector 18832256 [ 109.755555] end_request: I/O error, dev sda, sector 18832256 [ 109.886187] end_request: I/O error, dev sda, sector 357764 [ 109.891756] Buffer I/O error on device dm-1, logical block 45570 [ 109.908344] end_request: I/O error, dev sda, sector 18832256 [ 109.928369] end_request: I/O error, dev sda, sector 349574 [ 109.933938] Buffer I/O error on device dm-1, logical block 41475 [ 109.950336] end_request: I/O error, dev sda, sector 18832256 [ 115.378875] end_request: I/O error, dev sda, sector 365000 [ 115.384445] Aborting journal on device dm-1-8. [ 115.389120] end_request: I/O error, dev sda, sector 364930 [ 115.394798] Buffer I/O error on device dm-1, logical block 49153 [ 115.401101] JBD2: I/O error detected when updating journal superblock for dm-1-8. [ 207.207426] end_request: I/O error, dev sda, sector 246192376 [ 207.213313] end_request: I/O error, dev sda, sector 246192376 [ 207.903181] end_request: I/O error, dev sda, sector 246192376 [ 209.234399] end_request: I/O error, dev sda, sector 18518400 [ 209.240221] end_request: I/O error, dev sda, sector 18518400
It looks like this regression may be related to MSI handling. "pci=nomsi" on the kernel command line seems to bypass the issue. Clearly, legacy interrupts are not ideal. On Tue, Aug 7, 2012 at 11:04 AM, Ben Guthro <ben@guthro.net> wrote:> I have been doing some experiments in upgrading the Xen version in a > future version of XenClient Enterprise, and I''ve been running into a > regression that I''m wondering if anyone else has seen. > > dom0 suspend/resume (S3) does not seem to be working for me. > > In swapping out components of the system, the common failure seems to > be when I use Xen-4.2 (upgraded from Xen-4.0.3) > > The first suspend seems to mostly work...but subsequent ones always > resume improperly. > By "improperly" - I see I/O failures, and stalls of many processes. > > Below is a log excerpt of 2 S3 attempts. > > > Has anyone else seen these failures? > > - Ben > > > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Breaking vcpu affinity for domain 0 vcpu 1 > (XEN) Breaking vcpu affinity for domain 0 vcpu 2 > (XEN) Breaking vcpu affinity for domain 0 vcpu 3 > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank > 0 extended MCE MSR 0 > (XEN) CPU0 CMCI LVT vector (0xf1) already installed > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) Enabling non-boot CPUs ... > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > [ 36.440696] [drm:pch_irq_handler] *ERROR* PCH poison interrupt > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank > 0 extended MCE MSR 0 > (XEN) CPU0 CMCI LVT vector (0xf1) already installed > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) Enabling non-boot CPUs ... > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > [ 65.893235] [drm:pch_irq_handler] *ERROR* PCH poison interrupt > [ 66.508829] ata3.00: revalidation failed (errno=-5) > [ 66.508861] ata1.00: revalidation failed (errno=-5) > [ 76.858815] ata3.00: revalidation failed (errno=-5) > [ 76.898807] ata1.00: revalidation failed (errno=-5) > [ 107.208817] ata3.00: revalidation failed (errno=-5) > [ 107.288807] ata1.00: revalidation failed (errno=-5) > [ 107.718866] pm_op(): scsi_bus_resume_common+0x0/0x60 returns 262144 > [ 107.718877] PM: Device 0:0:0:0 failed to resume async: error 262144 > [ 107.718913] end_request: I/O error, dev sda, sector 35193296 > [ 107.718919] Buffer I/O error on device dm-5, logical block 7690 > [ 107.718947] end_request: I/O error, dev sda, sector 35657184 > [ 107.718965] end_request: I/O error, dev sda, sector 246202760 > [ 107.718968] Buffer I/O error on device dm-6, logical block 26252801 > [ 107.718995] end_request: I/O error, dev sda, sector 254548368 > [ 107.719009] Aborting journal on device dm-6-8. > [ 107.719021] end_request: I/O error, dev sda, sector 35164192 > [ 107.719023] Buffer I/O error on device dm-5, logical block 4052 > [ 107.719063] Aborting journal on device dm-5-8. > [ 107.719085] end_request: I/O error, dev sda, sector 254546304 > [ 107.719097] Buffer I/O error on device dm-6, logical block 27295744 > [ 107.719129] JBD2: I/O error detected when updating journal > superblock for dm-6-8. > [ 107.719141] end_request: I/O error, dev sda, sector 35656064 > [ 107.719146] Buffer I/O error on device dm-5, logical block 65536 > [ 107.719168] JBD2: I/O error detected when updating journal > superblock for dm-5-8. > [ 107.870082] end_request: I/O error, dev sda, sector 35131776 > [ 107.875825] Buffer I/O error on device dm-5, logical block 0 > [ 107.881805] end_request: I/O error, dev sda, sector 35131776 > [ 107.887637] Buffer I/O error on device dm-5, logical block 0 > [ 107.893573] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > [ 107.893579] EXT4-fs (dm-5): I/O error while writing superblock > [ 107.893582] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > Detected aborted journal > [ 107.893584] EXT4-fs (dm-5): Remounting filesystem read-only > [ 107.893617] end_request: I/O error, dev sda, sector 35131776 > [ 107.893620] Buffer I/O error on device dm-5, logical block 0 > [ 107.893749] end_request: I/O error, dev sda, sector 36180352 > [ 107.893752] Buffer I/O error on device dm-6, logical block 0 > [ 107.893762] EXT4-fs error (device dm-6): ext4_journal_start_sb:327: > Detected aborted journal > [ 107.893765] EXT4-fs (dm-6): Remounting filesystem read-only > [ 107.893766] EXT4-fs (dm-6): previous I/O error to superblock detected > [ 107.893784] end_request: I/O error, dev sda, sector 36180352 > [ 107.893787] Buffer I/O error on device dm-6, logical block 0 > [ 107.894467] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > Detected aborted journal > [ 108.669763] end_request: I/O error, dev sda, sector 25957784 > [ 108.675555] Aborting journal on device dm-3-8. > [ 108.680246] end_request: I/O error, dev sda, sector 25956736 > [ 108.686099] JBD2: I/O error detected when updating journal > superblock for dm-3-8. > [ 108.693908] journal commit I/O error > [ 108.755829] end_request: I/O error, dev sda, sector 17305984 > [ 108.761600] EXT4-fs error (device dm-3): ext4_journal_start_sb:327: > Detected aborted journal > [ 108.770340] EXT4-fs (dm-3): Remounting filesystem read-only > [ 108.776159] EXT4-fs (dm-3): previous I/O error to superblock detected > [ 108.782904] end_request: I/O error, dev sda, sector 17305984 > [ 109.660011] end_request: I/O error, dev sda, sector 358788 > [ 109.665572] Buffer I/O error on device dm-1, logical block 46082 > [ 109.682479] end_request: I/O error, dev sda, sector 18832256 > [ 109.688246] end_request: I/O error, dev sda, sector 18832256 > [ 109.709559] end_request: I/O error, dev sda, sector 357762 > [ 109.715120] Buffer I/O error on device dm-1, logical block 45569 > [ 109.721506] end_request: I/O error, dev sda, sector 358790 > [ 109.727114] Buffer I/O error on device dm-1, logical block 46083 > [ 109.743714] end_request: I/O error, dev sda, sector 18832256 > [ 109.755555] end_request: I/O error, dev sda, sector 18832256 > [ 109.886187] end_request: I/O error, dev sda, sector 357764 > [ 109.891756] Buffer I/O error on device dm-1, logical block 45570 > [ 109.908344] end_request: I/O error, dev sda, sector 18832256 > [ 109.928369] end_request: I/O error, dev sda, sector 349574 > [ 109.933938] Buffer I/O error on device dm-1, logical block 41475 > [ 109.950336] end_request: I/O error, dev sda, sector 18832256 > [ 115.378875] end_request: I/O error, dev sda, sector 365000 > [ 115.384445] Aborting journal on device dm-1-8. > [ 115.389120] end_request: I/O error, dev sda, sector 364930 > [ 115.394798] Buffer I/O error on device dm-1, logical block 49153 > [ 115.401101] JBD2: I/O error detected when updating journal > superblock for dm-1-8. > [ 207.207426] end_request: I/O error, dev sda, sector 246192376 > [ 207.213313] end_request: I/O error, dev sda, sector 246192376 > [ 207.903181] end_request: I/O error, dev sda, sector 246192376 > [ 209.234399] end_request: I/O error, dev sda, sector 18518400 > [ 209.240221] end_request: I/O error, dev sda, sector 18518400
On Tue, Aug 07, 2012 at 12:21:22PM -0400, Ben Guthro wrote:> It looks like this regression may be related to MSI handling. > > "pci=nomsi" on the kernel command line seems to bypass the issue. > > Clearly, legacy interrupts are not ideal.This is with v3.5 kernel right? With the earlier one you did not have this issue?> > > On Tue, Aug 7, 2012 at 11:04 AM, Ben Guthro <ben@guthro.net> wrote: > > I have been doing some experiments in upgrading the Xen version in a > > future version of XenClient Enterprise, and I''ve been running into a > > regression that I''m wondering if anyone else has seen. > > > > dom0 suspend/resume (S3) does not seem to be working for me. > > > > In swapping out components of the system, the common failure seems to > > be when I use Xen-4.2 (upgraded from Xen-4.0.3) > > > > The first suspend seems to mostly work...but subsequent ones always > > resume improperly. > > By "improperly" - I see I/O failures, and stalls of many processes. > > > > Below is a log excerpt of 2 S3 attempts. > > > > > > Has anyone else seen these failures? > > > > - Ben > > > > > > (XEN) Preparing system for ACPI S3 state. > > (XEN) Disabling non-boot CPUs ... > > (XEN) Breaking vcpu affinity for domain 0 vcpu 1 > > (XEN) Breaking vcpu affinity for domain 0 vcpu 2 > > (XEN) Breaking vcpu affinity for domain 0 vcpu 3 > > (XEN) Entering ACPI S3 state. > > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank > > 0 extended MCE MSR 0 > > (XEN) CPU0 CMCI LVT vector (0xf1) already installed > > (XEN) Finishing wakeup from ACPI S3 state. > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) Enabling non-boot CPUs ... > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > [ 36.440696] [drm:pch_irq_handler] *ERROR* PCH poison interrupt > > (XEN) Preparing system for ACPI S3 state. > > (XEN) Disabling non-boot CPUs ... > > (XEN) Entering ACPI S3 state. > > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank > > 0 extended MCE MSR 0 > > (XEN) CPU0 CMCI LVT vector (0xf1) already installed > > (XEN) Finishing wakeup from ACPI S3 state. > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) Enabling non-boot CPUs ... > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 > > [ 65.893235] [drm:pch_irq_handler] *ERROR* PCH poison interrupt > > [ 66.508829] ata3.00: revalidation failed (errno=-5) > > [ 66.508861] ata1.00: revalidation failed (errno=-5) > > [ 76.858815] ata3.00: revalidation failed (errno=-5) > > [ 76.898807] ata1.00: revalidation failed (errno=-5) > > [ 107.208817] ata3.00: revalidation failed (errno=-5) > > [ 107.288807] ata1.00: revalidation failed (errno=-5) > > [ 107.718866] pm_op(): scsi_bus_resume_common+0x0/0x60 returns 262144 > > [ 107.718877] PM: Device 0:0:0:0 failed to resume async: error 262144 > > [ 107.718913] end_request: I/O error, dev sda, sector 35193296 > > [ 107.718919] Buffer I/O error on device dm-5, logical block 7690 > > [ 107.718947] end_request: I/O error, dev sda, sector 35657184 > > [ 107.718965] end_request: I/O error, dev sda, sector 246202760 > > [ 107.718968] Buffer I/O error on device dm-6, logical block 26252801 > > [ 107.718995] end_request: I/O error, dev sda, sector 254548368 > > [ 107.719009] Aborting journal on device dm-6-8. > > [ 107.719021] end_request: I/O error, dev sda, sector 35164192 > > [ 107.719023] Buffer I/O error on device dm-5, logical block 4052 > > [ 107.719063] Aborting journal on device dm-5-8. > > [ 107.719085] end_request: I/O error, dev sda, sector 254546304 > > [ 107.719097] Buffer I/O error on device dm-6, logical block 27295744 > > [ 107.719129] JBD2: I/O error detected when updating journal > > superblock for dm-6-8. > > [ 107.719141] end_request: I/O error, dev sda, sector 35656064 > > [ 107.719146] Buffer I/O error on device dm-5, logical block 65536 > > [ 107.719168] JBD2: I/O error detected when updating journal > > superblock for dm-5-8. > > [ 107.870082] end_request: I/O error, dev sda, sector 35131776 > > [ 107.875825] Buffer I/O error on device dm-5, logical block 0 > > [ 107.881805] end_request: I/O error, dev sda, sector 35131776 > > [ 107.887637] Buffer I/O error on device dm-5, logical block 0 > > [ 107.893573] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > > [ 107.893579] EXT4-fs (dm-5): I/O error while writing superblock > > [ 107.893582] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > > Detected aborted journal > > [ 107.893584] EXT4-fs (dm-5): Remounting filesystem read-only > > [ 107.893617] end_request: I/O error, dev sda, sector 35131776 > > [ 107.893620] Buffer I/O error on device dm-5, logical block 0 > > [ 107.893749] end_request: I/O error, dev sda, sector 36180352 > > [ 107.893752] Buffer I/O error on device dm-6, logical block 0 > > [ 107.893762] EXT4-fs error (device dm-6): ext4_journal_start_sb:327: > > Detected aborted journal > > [ 107.893765] EXT4-fs (dm-6): Remounting filesystem read-only > > [ 107.893766] EXT4-fs (dm-6): previous I/O error to superblock detected > > [ 107.893784] end_request: I/O error, dev sda, sector 36180352 > > [ 107.893787] Buffer I/O error on device dm-6, logical block 0 > > [ 107.894467] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: > > Detected aborted journal > > [ 108.669763] end_request: I/O error, dev sda, sector 25957784 > > [ 108.675555] Aborting journal on device dm-3-8. > > [ 108.680246] end_request: I/O error, dev sda, sector 25956736 > > [ 108.686099] JBD2: I/O error detected when updating journal > > superblock for dm-3-8. > > [ 108.693908] journal commit I/O error > > [ 108.755829] end_request: I/O error, dev sda, sector 17305984 > > [ 108.761600] EXT4-fs error (device dm-3): ext4_journal_start_sb:327: > > Detected aborted journal > > [ 108.770340] EXT4-fs (dm-3): Remounting filesystem read-only > > [ 108.776159] EXT4-fs (dm-3): previous I/O error to superblock detected > > [ 108.782904] end_request: I/O error, dev sda, sector 17305984 > > [ 109.660011] end_request: I/O error, dev sda, sector 358788 > > [ 109.665572] Buffer I/O error on device dm-1, logical block 46082 > > [ 109.682479] end_request: I/O error, dev sda, sector 18832256 > > [ 109.688246] end_request: I/O error, dev sda, sector 18832256 > > [ 109.709559] end_request: I/O error, dev sda, sector 357762 > > [ 109.715120] Buffer I/O error on device dm-1, logical block 45569 > > [ 109.721506] end_request: I/O error, dev sda, sector 358790 > > [ 109.727114] Buffer I/O error on device dm-1, logical block 46083 > > [ 109.743714] end_request: I/O error, dev sda, sector 18832256 > > [ 109.755555] end_request: I/O error, dev sda, sector 18832256 > > [ 109.886187] end_request: I/O error, dev sda, sector 357764 > > [ 109.891756] Buffer I/O error on device dm-1, logical block 45570 > > [ 109.908344] end_request: I/O error, dev sda, sector 18832256 > > [ 109.928369] end_request: I/O error, dev sda, sector 349574 > > [ 109.933938] Buffer I/O error on device dm-1, logical block 41475 > > [ 109.950336] end_request: I/O error, dev sda, sector 18832256 > > [ 115.378875] end_request: I/O error, dev sda, sector 365000 > > [ 115.384445] Aborting journal on device dm-1-8. > > [ 115.389120] end_request: I/O error, dev sda, sector 364930 > > [ 115.394798] Buffer I/O error on device dm-1, logical block 49153 > > [ 115.401101] JBD2: I/O error detected when updating journal > > superblock for dm-1-8. > > [ 207.207426] end_request: I/O error, dev sda, sector 246192376 > > [ 207.213313] end_request: I/O error, dev sda, sector 246192376 > > [ 207.903181] end_request: I/O error, dev sda, sector 246192376 > > [ 209.234399] end_request: I/O error, dev sda, sector 18518400 > > [ 209.240221] end_request: I/O error, dev sda, sector 18518400 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
No - the issue seems to follow xen-4.2 my test matrix looks as such: Xen Linux S3 result 4.0.3 3.2.23 OK 4.0.3 3.5 OK 4.2 3.2.23 FAIL 4.2 3.5 FAIL 4.2 3.2.23 pci=nomsi OK 4.2 3.5 pci=nomsi (untested) On Tue, Aug 7, 2012 at 12:33 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Tue, Aug 07, 2012 at 12:21:22PM -0400, Ben Guthro wrote: >> It looks like this regression may be related to MSI handling. >> >> "pci=nomsi" on the kernel command line seems to bypass the issue. >> >> Clearly, legacy interrupts are not ideal. > > This is with v3.5 kernel right? With the earlier one you did not have > this issue? >> >> >> On Tue, Aug 7, 2012 at 11:04 AM, Ben Guthro <ben@guthro.net> wrote: >> > I have been doing some experiments in upgrading the Xen version in a >> > future version of XenClient Enterprise, and I''ve been running into a >> > regression that I''m wondering if anyone else has seen. >> > >> > dom0 suspend/resume (S3) does not seem to be working for me. >> > >> > In swapping out components of the system, the common failure seems to >> > be when I use Xen-4.2 (upgraded from Xen-4.0.3) >> > >> > The first suspend seems to mostly work...but subsequent ones always >> > resume improperly. >> > By "improperly" - I see I/O failures, and stalls of many processes. >> > >> > Below is a log excerpt of 2 S3 attempts. >> > >> > >> > Has anyone else seen these failures? >> > >> > - Ben >> > >> > >> > (XEN) Preparing system for ACPI S3 state. >> > (XEN) Disabling non-boot CPUs ... >> > (XEN) Breaking vcpu affinity for domain 0 vcpu 1 >> > (XEN) Breaking vcpu affinity for domain 0 vcpu 2 >> > (XEN) Breaking vcpu affinity for domain 0 vcpu 3 >> > (XEN) Entering ACPI S3 state. >> > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank >> > 0 extended MCE MSR 0 >> > (XEN) CPU0 CMCI LVT vector (0xf1) already installed >> > (XEN) Finishing wakeup from ACPI S3 state. >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) Enabling non-boot CPUs ... >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > [ 36.440696] [drm:pch_irq_handler] *ERROR* PCH poison interrupt >> > (XEN) Preparing system for ACPI S3 state. >> > (XEN) Disabling non-boot CPUs ... >> > (XEN) Entering ACPI S3 state. >> > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank >> > 0 extended MCE MSR 0 >> > (XEN) CPU0 CMCI LVT vector (0xf1) already installed >> > (XEN) Finishing wakeup from ACPI S3 state. >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) Enabling non-boot CPUs ... >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >> > [ 65.893235] [drm:pch_irq_handler] *ERROR* PCH poison interrupt >> > [ 66.508829] ata3.00: revalidation failed (errno=-5) >> > [ 66.508861] ata1.00: revalidation failed (errno=-5) >> > [ 76.858815] ata3.00: revalidation failed (errno=-5) >> > [ 76.898807] ata1.00: revalidation failed (errno=-5) >> > [ 107.208817] ata3.00: revalidation failed (errno=-5) >> > [ 107.288807] ata1.00: revalidation failed (errno=-5) >> > [ 107.718866] pm_op(): scsi_bus_resume_common+0x0/0x60 returns 262144 >> > [ 107.718877] PM: Device 0:0:0:0 failed to resume async: error 262144 >> > [ 107.718913] end_request: I/O error, dev sda, sector 35193296 >> > [ 107.718919] Buffer I/O error on device dm-5, logical block 7690 >> > [ 107.718947] end_request: I/O error, dev sda, sector 35657184 >> > [ 107.718965] end_request: I/O error, dev sda, sector 246202760 >> > [ 107.718968] Buffer I/O error on device dm-6, logical block 26252801 >> > [ 107.718995] end_request: I/O error, dev sda, sector 254548368 >> > [ 107.719009] Aborting journal on device dm-6-8. >> > [ 107.719021] end_request: I/O error, dev sda, sector 35164192 >> > [ 107.719023] Buffer I/O error on device dm-5, logical block 4052 >> > [ 107.719063] Aborting journal on device dm-5-8. >> > [ 107.719085] end_request: I/O error, dev sda, sector 254546304 >> > [ 107.719097] Buffer I/O error on device dm-6, logical block 27295744 >> > [ 107.719129] JBD2: I/O error detected when updating journal >> > superblock for dm-6-8. >> > [ 107.719141] end_request: I/O error, dev sda, sector 35656064 >> > [ 107.719146] Buffer I/O error on device dm-5, logical block 65536 >> > [ 107.719168] JBD2: I/O error detected when updating journal >> > superblock for dm-5-8. >> > [ 107.870082] end_request: I/O error, dev sda, sector 35131776 >> > [ 107.875825] Buffer I/O error on device dm-5, logical block 0 >> > [ 107.881805] end_request: I/O error, dev sda, sector 35131776 >> > [ 107.887637] Buffer I/O error on device dm-5, logical block 0 >> > [ 107.893573] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >> > [ 107.893579] EXT4-fs (dm-5): I/O error while writing superblock >> > [ 107.893582] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >> > Detected aborted journal >> > [ 107.893584] EXT4-fs (dm-5): Remounting filesystem read-only >> > [ 107.893617] end_request: I/O error, dev sda, sector 35131776 >> > [ 107.893620] Buffer I/O error on device dm-5, logical block 0 >> > [ 107.893749] end_request: I/O error, dev sda, sector 36180352 >> > [ 107.893752] Buffer I/O error on device dm-6, logical block 0 >> > [ 107.893762] EXT4-fs error (device dm-6): ext4_journal_start_sb:327: >> > Detected aborted journal >> > [ 107.893765] EXT4-fs (dm-6): Remounting filesystem read-only >> > [ 107.893766] EXT4-fs (dm-6): previous I/O error to superblock detected >> > [ 107.893784] end_request: I/O error, dev sda, sector 36180352 >> > [ 107.893787] Buffer I/O error on device dm-6, logical block 0 >> > [ 107.894467] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >> > Detected aborted journal >> > [ 108.669763] end_request: I/O error, dev sda, sector 25957784 >> > [ 108.675555] Aborting journal on device dm-3-8. >> > [ 108.680246] end_request: I/O error, dev sda, sector 25956736 >> > [ 108.686099] JBD2: I/O error detected when updating journal >> > superblock for dm-3-8. >> > [ 108.693908] journal commit I/O error >> > [ 108.755829] end_request: I/O error, dev sda, sector 17305984 >> > [ 108.761600] EXT4-fs error (device dm-3): ext4_journal_start_sb:327: >> > Detected aborted journal >> > [ 108.770340] EXT4-fs (dm-3): Remounting filesystem read-only >> > [ 108.776159] EXT4-fs (dm-3): previous I/O error to superblock detected >> > [ 108.782904] end_request: I/O error, dev sda, sector 17305984 >> > [ 109.660011] end_request: I/O error, dev sda, sector 358788 >> > [ 109.665572] Buffer I/O error on device dm-1, logical block 46082 >> > [ 109.682479] end_request: I/O error, dev sda, sector 18832256 >> > [ 109.688246] end_request: I/O error, dev sda, sector 18832256 >> > [ 109.709559] end_request: I/O error, dev sda, sector 357762 >> > [ 109.715120] Buffer I/O error on device dm-1, logical block 45569 >> > [ 109.721506] end_request: I/O error, dev sda, sector 358790 >> > [ 109.727114] Buffer I/O error on device dm-1, logical block 46083 >> > [ 109.743714] end_request: I/O error, dev sda, sector 18832256 >> > [ 109.755555] end_request: I/O error, dev sda, sector 18832256 >> > [ 109.886187] end_request: I/O error, dev sda, sector 357764 >> > [ 109.891756] Buffer I/O error on device dm-1, logical block 45570 >> > [ 109.908344] end_request: I/O error, dev sda, sector 18832256 >> > [ 109.928369] end_request: I/O error, dev sda, sector 349574 >> > [ 109.933938] Buffer I/O error on device dm-1, logical block 41475 >> > [ 109.950336] end_request: I/O error, dev sda, sector 18832256 >> > [ 115.378875] end_request: I/O error, dev sda, sector 365000 >> > [ 115.384445] Aborting journal on device dm-1-8. >> > [ 115.389120] end_request: I/O error, dev sda, sector 364930 >> > [ 115.394798] Buffer I/O error on device dm-1, logical block 49153 >> > [ 115.401101] JBD2: I/O error detected when updating journal >> > superblock for dm-1-8. >> > [ 207.207426] end_request: I/O error, dev sda, sector 246192376 >> > [ 207.213313] end_request: I/O error, dev sda, sector 246192376 >> > [ 207.903181] end_request: I/O error, dev sda, sector 246192376 >> > [ 209.234399] end_request: I/O error, dev sda, sector 18518400 >> > [ 209.240221] end_request: I/O error, dev sda, sector 18518400 >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel
Any suggestions on how best to chase this down? The first S3 suspend/resume cycle works, but the second does not. On the second try, I never get any interrupts delivered to ahci. (at least according to /proc/interrupts) syslog traces from the first (good) and the second (bad) are attached, as well as the output from the "*" debug Ctrl+a handler in both cases. On Tue, Aug 7, 2012 at 12:48 PM, Ben Guthro <ben@guthro.net> wrote:> No - the issue seems to follow xen-4.2 > > my test matrix looks as such: > > Xen Linux S3 result > 4.0.3 3.2.23 OK > 4.0.3 3.5 OK > 4.2 3.2.23 FAIL > 4.2 3.5 FAIL > 4.2 3.2.23 pci=nomsi OK > 4.2 3.5 pci=nomsi (untested) > > > > > On Tue, Aug 7, 2012 at 12:33 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> On Tue, Aug 07, 2012 at 12:21:22PM -0400, Ben Guthro wrote: >>> It looks like this regression may be related to MSI handling. >>> >>> "pci=nomsi" on the kernel command line seems to bypass the issue. >>> >>> Clearly, legacy interrupts are not ideal. >> >> This is with v3.5 kernel right? With the earlier one you did not have >> this issue? >>> >>> >>> On Tue, Aug 7, 2012 at 11:04 AM, Ben Guthro <ben@guthro.net> wrote: >>> > I have been doing some experiments in upgrading the Xen version in a >>> > future version of XenClient Enterprise, and I''ve been running into a >>> > regression that I''m wondering if anyone else has seen. >>> > >>> > dom0 suspend/resume (S3) does not seem to be working for me. >>> > >>> > In swapping out components of the system, the common failure seems to >>> > be when I use Xen-4.2 (upgraded from Xen-4.0.3) >>> > >>> > The first suspend seems to mostly work...but subsequent ones always >>> > resume improperly. >>> > By "improperly" - I see I/O failures, and stalls of many processes. >>> > >>> > Below is a log excerpt of 2 S3 attempts. >>> > >>> > >>> > Has anyone else seen these failures? >>> > >>> > - Ben >>> > >>> > >>> > (XEN) Preparing system for ACPI S3 state. >>> > (XEN) Disabling non-boot CPUs ... >>> > (XEN) Breaking vcpu affinity for domain 0 vcpu 1 >>> > (XEN) Breaking vcpu affinity for domain 0 vcpu 2 >>> > (XEN) Breaking vcpu affinity for domain 0 vcpu 3 >>> > (XEN) Entering ACPI S3 state. >>> > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank >>> > 0 extended MCE MSR 0 >>> > (XEN) CPU0 CMCI LVT vector (0xf1) already installed >>> > (XEN) Finishing wakeup from ACPI S3 state. >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) Enabling non-boot CPUs ... >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > [ 36.440696] [drm:pch_irq_handler] *ERROR* PCH poison interrupt >>> > (XEN) Preparing system for ACPI S3 state. >>> > (XEN) Disabling non-boot CPUs ... >>> > (XEN) Entering ACPI S3 state. >>> > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank >>> > 0 extended MCE MSR 0 >>> > (XEN) CPU0 CMCI LVT vector (0xf1) already installed >>> > (XEN) Finishing wakeup from ACPI S3 state. >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) Enabling non-boot CPUs ... >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > (XEN) microcode: collect_cpu_info : sig=0x306a4, pf=0x2, rev=0x7 >>> > [ 65.893235] [drm:pch_irq_handler] *ERROR* PCH poison interrupt >>> > [ 66.508829] ata3.00: revalidation failed (errno=-5) >>> > [ 66.508861] ata1.00: revalidation failed (errno=-5) >>> > [ 76.858815] ata3.00: revalidation failed (errno=-5) >>> > [ 76.898807] ata1.00: revalidation failed (errno=-5) >>> > [ 107.208817] ata3.00: revalidation failed (errno=-5) >>> > [ 107.288807] ata1.00: revalidation failed (errno=-5) >>> > [ 107.718866] pm_op(): scsi_bus_resume_common+0x0/0x60 returns 262144 >>> > [ 107.718877] PM: Device 0:0:0:0 failed to resume async: error 262144 >>> > [ 107.718913] end_request: I/O error, dev sda, sector 35193296 >>> > [ 107.718919] Buffer I/O error on device dm-5, logical block 7690 >>> > [ 107.718947] end_request: I/O error, dev sda, sector 35657184 >>> > [ 107.718965] end_request: I/O error, dev sda, sector 246202760 >>> > [ 107.718968] Buffer I/O error on device dm-6, logical block 26252801 >>> > [ 107.718995] end_request: I/O error, dev sda, sector 254548368 >>> > [ 107.719009] Aborting journal on device dm-6-8. >>> > [ 107.719021] end_request: I/O error, dev sda, sector 35164192 >>> > [ 107.719023] Buffer I/O error on device dm-5, logical block 4052 >>> > [ 107.719063] Aborting journal on device dm-5-8. >>> > [ 107.719085] end_request: I/O error, dev sda, sector 254546304 >>> > [ 107.719097] Buffer I/O error on device dm-6, logical block 27295744 >>> > [ 107.719129] JBD2: I/O error detected when updating journal >>> > superblock for dm-6-8. >>> > [ 107.719141] end_request: I/O error, dev sda, sector 35656064 >>> > [ 107.719146] Buffer I/O error on device dm-5, logical block 65536 >>> > [ 107.719168] JBD2: I/O error detected when updating journal >>> > superblock for dm-5-8. >>> > [ 107.870082] end_request: I/O error, dev sda, sector 35131776 >>> > [ 107.875825] Buffer I/O error on device dm-5, logical block 0 >>> > [ 107.881805] end_request: I/O error, dev sda, sector 35131776 >>> > [ 107.887637] Buffer I/O error on device dm-5, logical block 0 >>> > [ 107.893573] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >>> > [ 107.893579] EXT4-fs (dm-5): I/O error while writing superblock >>> > [ 107.893582] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >>> > Detected aborted journal >>> > [ 107.893584] EXT4-fs (dm-5): Remounting filesystem read-only >>> > [ 107.893617] end_request: I/O error, dev sda, sector 35131776 >>> > [ 107.893620] Buffer I/O error on device dm-5, logical block 0 >>> > [ 107.893749] end_request: I/O error, dev sda, sector 36180352 >>> > [ 107.893752] Buffer I/O error on device dm-6, logical block 0 >>> > [ 107.893762] EXT4-fs error (device dm-6): ext4_journal_start_sb:327: >>> > Detected aborted journal >>> > [ 107.893765] EXT4-fs (dm-6): Remounting filesystem read-only >>> > [ 107.893766] EXT4-fs (dm-6): previous I/O error to superblock detected >>> > [ 107.893784] end_request: I/O error, dev sda, sector 36180352 >>> > [ 107.893787] Buffer I/O error on device dm-6, logical block 0 >>> > [ 107.894467] EXT4-fs error (device dm-5): ext4_journal_start_sb:327: >>> > Detected aborted journal >>> > [ 108.669763] end_request: I/O error, dev sda, sector 25957784 >>> > [ 108.675555] Aborting journal on device dm-3-8. >>> > [ 108.680246] end_request: I/O error, dev sda, sector 25956736 >>> > [ 108.686099] JBD2: I/O error detected when updating journal >>> > superblock for dm-3-8. >>> > [ 108.693908] journal commit I/O error >>> > [ 108.755829] end_request: I/O error, dev sda, sector 17305984 >>> > [ 108.761600] EXT4-fs error (device dm-3): ext4_journal_start_sb:327: >>> > Detected aborted journal >>> > [ 108.770340] EXT4-fs (dm-3): Remounting filesystem read-only >>> > [ 108.776159] EXT4-fs (dm-3): previous I/O error to superblock detected >>> > [ 108.782904] end_request: I/O error, dev sda, sector 17305984 >>> > [ 109.660011] end_request: I/O error, dev sda, sector 358788 >>> > [ 109.665572] Buffer I/O error on device dm-1, logical block 46082 >>> > [ 109.682479] end_request: I/O error, dev sda, sector 18832256 >>> > [ 109.688246] end_request: I/O error, dev sda, sector 18832256 >>> > [ 109.709559] end_request: I/O error, dev sda, sector 357762 >>> > [ 109.715120] Buffer I/O error on device dm-1, logical block 45569 >>> > [ 109.721506] end_request: I/O error, dev sda, sector 358790 >>> > [ 109.727114] Buffer I/O error on device dm-1, logical block 46083 >>> > [ 109.743714] end_request: I/O error, dev sda, sector 18832256 >>> > [ 109.755555] end_request: I/O error, dev sda, sector 18832256 >>> > [ 109.886187] end_request: I/O error, dev sda, sector 357764 >>> > [ 109.891756] Buffer I/O error on device dm-1, logical block 45570 >>> > [ 109.908344] end_request: I/O error, dev sda, sector 18832256 >>> > [ 109.928369] end_request: I/O error, dev sda, sector 349574 >>> > [ 109.933938] Buffer I/O error on device dm-1, logical block 41475 >>> > [ 109.950336] end_request: I/O error, dev sda, sector 18832256 >>> > [ 115.378875] end_request: I/O error, dev sda, sector 365000 >>> > [ 115.384445] Aborting journal on device dm-1-8. >>> > [ 115.389120] end_request: I/O error, dev sda, sector 364930 >>> > [ 115.394798] Buffer I/O error on device dm-1, logical block 49153 >>> > [ 115.401101] JBD2: I/O error detected when updating journal >>> > superblock for dm-1-8. >>> > [ 207.207426] end_request: I/O error, dev sda, sector 246192376 >>> > [ 207.213313] end_request: I/O error, dev sda, sector 246192376 >>> > [ 207.903181] end_request: I/O error, dev sda, sector 246192376 >>> > [ 209.234399] end_request: I/O error, dev sda, sector 18518400 >>> > [ 209.240221] end_request: I/O error, dev sda, sector 18518400 >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 07.08.12 at 22:14, Ben Guthro <ben@guthro.net> wrote: > Any suggestions on how best to chase this down? > > The first S3 suspend/resume cycle works, but the second does not. > > On the second try, I never get any interrupts delivered to ahci. > (at least according to /proc/interrupts) > > > syslog traces from the first (good) and the second (bad) are attached, > as well as the output from the "*" debug Ctrl+a handler in both cases.You should have provided this also for the state before the first suspend. The state after the first resume already looks corrupted (presumably just not as badly): (XEN) PCI-MSI interrupt information: (XEN) MSI 26 vec=71 lowest edge assert log lowest dest=00000001 mask=0/1/-1 (XEN) MSI 27 vec=00 fixed edge deassert phys lowest dest=00000001 mask=0/1/-1 ^^ (XEN) MSI 28 vec=29 lowest edge assert log lowest dest=00000001 mask=0/1/-1 (XEN) MSI 29 vec=79 lowest edge assert log lowest dest=00000001 mask=0/1/-1 (XEN) MSI 30 vec=81 lowest edge assert log lowest dest=00000001 mask=0/1/-1 (XEN) MSI 31 vec=99 lowest edge assert log lowest dest=00000001 mask=0/1/-1 so this is likely the reason for thing falling apart on the second iteration: (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) SVT SQ SID DST V AVL DLM TM RH DM FPD P (XEN) 0000: 1 0 f0f8 00000001 38 0 1 0 1 1 0 1 ... (XEN) 0014: 1 0 00d8 00000001 a1 0 1 0 1 1 0 1 (XEN) 0015: 1 0 00fa 00000001 00 0 0 0 0 0 0 1 ^ ^ ^ (XEN) 0016: 1 0 f0f8 00000001 31 0 1 1 1 1 0 1 (XEN) 0017: 1 0 00a0 00000001 a9 0 1 0 1 1 0 1 (XEN) 0018: 1 0 0200 00000001 b1 0 1 0 1 1 0 1 (XEN) 0019: 1 0 00c8 00000001 c9 0 1 0 1 1 0 1 Surprisingly in both cases we get (with the other vector fields varying accordingly) (XEN) IRQ: 26 affinity:0001 vec:71 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:279(-S--), (XEN) IRQ: 27 affinity:0001 vec:21 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:278(-S--), ^^ (XEN) IRQ: 28 affinity:0001 vec:29 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:277(-S--), (XEN) IRQ: 29 affinity:0001 vec:79 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:276(-S--), (XEN) IRQ: 30 affinity:0001 vec:81 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(PS--), (XEN) IRQ: 31 affinity:0001 vec:99 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:274(PS--), The interrupt in question belongs to 0000:00:1f.2, i.e. the AHCI contoller. Unfortunately I can''t make sense of the kernel side config space restore messages - an offset of 1 gets reported for the device in question (and various other odd offsets exist), yet 3.5''s drivers/pci/pci.c:pci_restore_config_space_range() calls pci_restore_config_dword() with an offset that''s always divisible by 4. Could you clarify which kernel version you were using here? We first need to determine whether the kernel corrupts something (after all, config space isn''t protected from Dom0 modifications) - if that''s the case, we may need to understand why older Xen was immune against that. If that''s not the case, adding some extra logging to Xen''s pci_restore_msi_state() would seem the best first step, plus (maybe) logging of Dom0 post-resume config space accesses to the device in question. The most likely thing happening (though unclear where) is that the corresponding struct msi_msg instance gets cleared in the course of the first resume (but after the corresponding interrupt remapping entry already got restored). Jan
Thanks for taking the time to reply. I''m out of the office today, so don''t have direct access to the machine in question until tomorrow... but I''ll do my best to answer (inline below) and I''ll follow up tomorrow with concrete answers. On Wed, Aug 8, 2012 at 4:35 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 07.08.12 at 22:14, Ben Guthro <ben@guthro.net> wrote: >> Any suggestions on how best to chase this down? >> >> The first S3 suspend/resume cycle works, but the second does not. >> >> On the second try, I never get any interrupts delivered to ahci. >> (at least according to /proc/interrupts) >> >> >> syslog traces from the first (good) and the second (bad) are attached, >> as well as the output from the "*" debug Ctrl+a handler in both cases. > > You should have provided this also for the state before the > first suspend. The state after the first resume already looks > corrupted (presumably just not as badly):I''ll be able to send this tomorrow.> > (XEN) PCI-MSI interrupt information: > (XEN) MSI 26 vec=71 lowest edge assert log lowest dest=00000001 mask=0/1/-1 > (XEN) MSI 27 vec=00 fixed edge deassert phys lowest dest=00000001 mask=0/1/-1 > ^^ > (XEN) MSI 28 vec=29 lowest edge assert log lowest dest=00000001 mask=0/1/-1 > (XEN) MSI 29 vec=79 lowest edge assert log lowest dest=00000001 mask=0/1/-1 > (XEN) MSI 30 vec=81 lowest edge assert log lowest dest=00000001 mask=0/1/-1 > (XEN) MSI 31 vec=99 lowest edge assert log lowest dest=00000001 mask=0/1/-1 > > so this is likely the reason for thing falling apart on the second > iteration: > > (XEN) Interrupt Remapping: supported and enabled. > (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): > (XEN) SVT SQ SID DST V AVL DLM TM RH DM FPD P > (XEN) 0000: 1 0 f0f8 00000001 38 0 1 0 1 1 0 1 > ... > (XEN) 0014: 1 0 00d8 00000001 a1 0 1 0 1 1 0 1 > (XEN) 0015: 1 0 00fa 00000001 00 0 0 0 0 0 0 1 > ^ ^ ^ > (XEN) 0016: 1 0 f0f8 00000001 31 0 1 1 1 1 0 1 > (XEN) 0017: 1 0 00a0 00000001 a9 0 1 0 1 1 0 1 > (XEN) 0018: 1 0 0200 00000001 b1 0 1 0 1 1 0 1 > (XEN) 0019: 1 0 00c8 00000001 c9 0 1 0 1 1 0 1 > > Surprisingly in both cases we get (with the other vector fields varying > accordingly) > > (XEN) IRQ: 26 affinity:0001 vec:71 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:279(-S--), > (XEN) IRQ: 27 affinity:0001 vec:21 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:278(-S--), > ^^ > (XEN) IRQ: 28 affinity:0001 vec:29 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:277(-S--), > (XEN) IRQ: 29 affinity:0001 vec:79 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:276(-S--), > (XEN) IRQ: 30 affinity:0001 vec:81 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(PS--), > (XEN) IRQ: 31 affinity:0001 vec:99 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:274(PS--), > > The interrupt in question belongs to 0000:00:1f.2, i.e. the > AHCI contoller.This would be consistent with what I''ve observed.> > Unfortunately I can''t make sense of the kernel side config space > restore messages - an offset of 1 gets reported for the device in > question (and various other odd offsets exist), yet 3.5''s > drivers/pci/pci.c:pci_restore_config_space_range() calls > pci_restore_config_dword() with an offset that''s always divisible > by 4. Could you clarify which kernel version you were using here? > We first need to determine whether the kernel corrupts something > (after all, config space isn''t protected from Dom0 modifications) - > if that''s the case, we may need to understand why older Xen was > immune against that. If that''s not the case, adding some extra > logging to Xen''s pci_restore_msi_state() would seem the best > first step, plus (maybe) logging of Dom0 post-resume config space > accesses to the device in question.This particular failure is using linux-3.2.23 + some of Konrad''s branches that haven''t been merged into mainline (s3 branches, are probably the most appropriate here)> > The most likely thing happening (though unclear where) is that > the corresponding struct msi_msg instance gets cleared in the > course of the first resume (but after the corresponding interrupt > remapping entry already got restored). > > Jan >
Attached is a new run for new boot (pre-s3) first suspend / resume cycle (s3-first) second (failing) suspend / resume cycle (s3-second) To go into greater detail on the kernel used - It is a 3.2.23 kernel based off of the Ubuntu 12.04 git tree here http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-precise.git;a=summary To that, I also have some of Konrad''s branches - specifically /devel/ioperm /devel/acpi-s3.v7 /stable/misc (mostly for the microcode fixes) /stable/for-linus-fixes-3.3 /stable/for-linus-3.3 /devel/ttm.dma_pool.v2.9 /stable/for-x86 On top of that, are some more patches specific to our operations, not terribly interesting here, but I can provide them, if necessary. The 3.5 tree I tested with has a similar makeup - with some fewer branches from Konrad. On Wed, Aug 8, 2012 at 6:39 AM, Ben Guthro <ben@guthro.net> wrote:> Thanks for taking the time to reply. > > I''m out of the office today, so don''t have direct access to the > machine in question until tomorrow... but I''ll do my best to answer > (inline below) and I''ll follow up tomorrow with concrete answers. > > On Wed, Aug 8, 2012 at 4:35 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 07.08.12 at 22:14, Ben Guthro <ben@guthro.net> wrote: >>> Any suggestions on how best to chase this down? >>> >>> The first S3 suspend/resume cycle works, but the second does not. >>> >>> On the second try, I never get any interrupts delivered to ahci. >>> (at least according to /proc/interrupts) >>> >>> >>> syslog traces from the first (good) and the second (bad) are attached, >>> as well as the output from the "*" debug Ctrl+a handler in both cases. >> >> You should have provided this also for the state before the >> first suspend. The state after the first resume already looks >> corrupted (presumably just not as badly): > > I''ll be able to send this tomorrow. > >> >> (XEN) PCI-MSI interrupt information: >> (XEN) MSI 26 vec=71 lowest edge assert log lowest dest=00000001 mask=0/1/-1 >> (XEN) MSI 27 vec=00 fixed edge deassert phys lowest dest=00000001 mask=0/1/-1 >> ^^ >> (XEN) MSI 28 vec=29 lowest edge assert log lowest dest=00000001 mask=0/1/-1 >> (XEN) MSI 29 vec=79 lowest edge assert log lowest dest=00000001 mask=0/1/-1 >> (XEN) MSI 30 vec=81 lowest edge assert log lowest dest=00000001 mask=0/1/-1 >> (XEN) MSI 31 vec=99 lowest edge assert log lowest dest=00000001 mask=0/1/-1 >> >> so this is likely the reason for thing falling apart on the second >> iteration: >> >> (XEN) Interrupt Remapping: supported and enabled. >> (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): >> (XEN) SVT SQ SID DST V AVL DLM TM RH DM FPD P >> (XEN) 0000: 1 0 f0f8 00000001 38 0 1 0 1 1 0 1 >> ... >> (XEN) 0014: 1 0 00d8 00000001 a1 0 1 0 1 1 0 1 >> (XEN) 0015: 1 0 00fa 00000001 00 0 0 0 0 0 0 1 >> ^ ^ ^ >> (XEN) 0016: 1 0 f0f8 00000001 31 0 1 1 1 1 0 1 >> (XEN) 0017: 1 0 00a0 00000001 a9 0 1 0 1 1 0 1 >> (XEN) 0018: 1 0 0200 00000001 b1 0 1 0 1 1 0 1 >> (XEN) 0019: 1 0 00c8 00000001 c9 0 1 0 1 1 0 1 >> >> Surprisingly in both cases we get (with the other vector fields varying >> accordingly) >> >> (XEN) IRQ: 26 affinity:0001 vec:71 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:279(-S--), >> (XEN) IRQ: 27 affinity:0001 vec:21 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:278(-S--), >> ^^ >> (XEN) IRQ: 28 affinity:0001 vec:29 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:277(-S--), >> (XEN) IRQ: 29 affinity:0001 vec:79 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:276(-S--), >> (XEN) IRQ: 30 affinity:0001 vec:81 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(PS--), >> (XEN) IRQ: 31 affinity:0001 vec:99 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:274(PS--), >> >> The interrupt in question belongs to 0000:00:1f.2, i.e. the >> AHCI contoller. > > This would be consistent with what I''ve observed. > >> >> Unfortunately I can''t make sense of the kernel side config space >> restore messages - an offset of 1 gets reported for the device in >> question (and various other odd offsets exist), yet 3.5''s >> drivers/pci/pci.c:pci_restore_config_space_range() calls >> pci_restore_config_dword() with an offset that''s always divisible >> by 4. Could you clarify which kernel version you were using here? >> We first need to determine whether the kernel corrupts something >> (after all, config space isn''t protected from Dom0 modifications) - >> if that''s the case, we may need to understand why older Xen was >> immune against that. If that''s not the case, adding some extra >> logging to Xen''s pci_restore_msi_state() would seem the best >> first step, plus (maybe) logging of Dom0 post-resume config space >> accesses to the device in question. > > This particular failure is using linux-3.2.23 + some of Konrad''s > branches that haven''t been merged into mainline (s3 branches, are > probably the most appropriate here) > >> >> The most likely thing happening (though unclear where) is that >> the corresponding struct msi_msg instance gets cleared in the >> course of the first resume (but after the corresponding interrupt >> remapping entry already got restored). >> >> Jan >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 09.08.12 at 17:21, Ben Guthro <ben@guthro.net> wrote: > Attached is a new run for > new boot (pre-s3) > first suspend / resume cycle (s3-first) > second (failing) suspend / resume cycle (s3-second)That confirms that the corruption occurs during the first suspend, but presumably towards its end (where MSI and/or interrupt redirection stuff already got restored). There''s nothing I can add on top of my recommendations as to the debugging of this. One thing I think you didn''t tell us so far is whether without interrupt remapping (or the IOMMU turned off altogether) the problem would also be observed. Jan
On Thu, Aug 9, 2012 at 11:37 AM, Jan Beulich <JBeulich@suse.com> wrote:> One thing I think you didn''t tell us so far is whether without > interrupt remapping (or the IOMMU turned off altogether) the > problem would also be observed.I assume you mean the xen cli param iommu=off for the second test here. What parameter should I be flipping for the first?
>>> On 09.08.12 at 17:46, Ben Guthro <ben@guthro.net> wrote: > On Thu, Aug 9, 2012 at 11:37 AM, Jan Beulich <JBeulich@suse.com> wrote: >> One thing I think you didn''t tell us so far is whether without >> interrupt remapping (or the IOMMU turned off altogether) the >> problem would also be observed. > > I assume you mean the xen cli param iommu=off for the second test here. > What parameter should I be flipping for the first?iommu=no-intremap Jan
iommu=no-intremap This seems to work around the issue on this platform, performing multiple suspend/resume cycles, and ahci came back afterwards just fine. What is the downside to flipping this off? iommu=off This test behaved similarly to the above, also working around the issue. On Thu, Aug 9, 2012 at 11:51 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 09.08.12 at 17:46, Ben Guthro <ben@guthro.net> wrote: >> On Thu, Aug 9, 2012 at 11:37 AM, Jan Beulich <JBeulich@suse.com> wrote: >>> One thing I think you didn''t tell us so far is whether without >>> interrupt remapping (or the IOMMU turned off altogether) the >>> problem would also be observed. >> >> I assume you mean the xen cli param iommu=off for the second test here. >> What parameter should I be flipping for the first? > > iommu=no-intremap > > Jan >
>>> On 09.08.12 at 18:09, Ben Guthro <ben@guthro.net> wrote: > iommu=no-intremap > This seems to work around the issue on this platform, performing > multiple suspend/resume cycles, and ahci came back afterwards just > fine. > > What is the downside to flipping this off?Loss of security (against misbehaving/malicious guests). So we certainly want/need to get to the bottom of this (especially if this is not only one kind of system that''s affected).> iommu=off > This test behaved similarly to the above, also working around the issue.Of course, this is a superset of the former. This result, however, makes it more likely again to indeed be a Xen side problem, not Dom0 induced corruption. Jan
I''ll continue to investigate, as my schedule allows, but haven''t found any smoking gun just yet. This happens to be on an Ivybridge system - I have other Sandybridge systems that will go to sleep, but never wake up at all, forcing a hard power cycle. I tested these iommu= parameters on one of these machines, to no effect. Every time they go into S3, a hard reset is necessary to get them to come out of it. On Fri, Aug 10, 2012 at 2:50 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 09.08.12 at 18:09, Ben Guthro <ben@guthro.net> wrote: >> iommu=no-intremap >> This seems to work around the issue on this platform, performing >> multiple suspend/resume cycles, and ahci came back afterwards just >> fine. >> >> What is the downside to flipping this off? > > Loss of security (against misbehaving/malicious guests). So we > certainly want/need to get to the bottom of this (especially if > this is not only one kind of system that''s affected). > >> iommu=off >> This test behaved similarly to the above, also working around the issue. > > Of course, this is a superset of the former. > > This result, however, makes it more likely again to indeed be a > Xen side problem, not Dom0 induced corruption. > > Jan >
I''ve bisected this issue - and it looks like it is a rather old problem. The following changeset introduced it in the 4.1 development stream on Jun 17, 2010 http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 Since we (XenClient Enterprise / NxTop) skipped over Xen-4.1 - we never ran into this until now. Any thoughts as to a solution? -Ben On Fri, Aug 10, 2012 at 3:15 PM, Ben Guthro <ben@guthro.net> wrote:> I''ll continue to investigate, as my schedule allows, but haven''t found > any smoking gun just yet. > > This happens to be on an Ivybridge system - I have other Sandybridge > systems that will go to sleep, but never wake up at all, forcing a > hard power cycle. > > I tested these iommu= parameters on one of these machines, to no > effect. Every time they go into S3, a hard reset is necessary to get > them to come out of it. > > > > > On Fri, Aug 10, 2012 at 2:50 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 09.08.12 at 18:09, Ben Guthro <ben@guthro.net> wrote: >>> iommu=no-intremap >>> This seems to work around the issue on this platform, performing >>> multiple suspend/resume cycles, and ahci came back afterwards just >>> fine. >>> >>> What is the downside to flipping this off? >> >> Loss of security (against misbehaving/malicious guests). So we >> certainly want/need to get to the bottom of this (especially if >> this is not only one kind of system that''s affected). >> >>> iommu=off >>> This test behaved similarly to the above, also working around the issue. >> >> Of course, this is a superset of the former. >> >> This result, however, makes it more likely again to indeed be a >> Xen side problem, not Dom0 induced corruption. >> >> Jan >>
>>> On 14.08.12 at 19:31, Ben Guthro <ben@guthro.net> wrote: > I''ve bisected this issue - and it looks like it is a rather old problem. > > The following changeset introduced it in the 4.1 development stream on > Jun 17, 2010 > > http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42Interesting. That I wouldn''t have suspected at all.> Since we (XenClient Enterprise / NxTop) skipped over Xen-4.1 - we > never ran into this until now. > > Any thoughts as to a solution?First try collectively removing the three calls to evtchn_move_pirqs() in xen/common/schedule.c. If that helps, see whether any smaller set also does. From the result of this, I''ll have to think further, perhaps handing you a debugging patch. Jan
On Wed, Aug 15, 2012 at 4:11 AM, Jan Beulich <JBeulich@suse.com> wrote:> First try collectively removing the three calls to > evtchn_move_pirqs() in xen/common/schedule.c. If that helps,Sadly, I tried this yesterday (against the tip) with no success. I''m starting to question my bisecting results. It is, of course easy to know that it failed - but since it doesn''t fail until sometime after the first, second, or third suspend/resume cycle - it can be easy to misinterpret a failure that has not occurred yet as a success. I will retest this morning when I get to work with this changeset, and its parent, to better verify my results. I''ll also try the evtchn_move_pirqs() removal against the changeset above, to see if my results differ than when I did the same test against the tip.> see whether any smaller set also does. From the result of this, > I''ll have to think further, perhaps handing you a debugging > patch.I''m happy to test any debug patch. I will report results from the tests above later this morning. Ben
On Wed, Aug 15, 2012 at 6:32 AM, Ben Guthro <ben@guthro.net> wrote:> I will retest this morning when I get to work with this changeset, and > its parent, to better verify my results.I retested this, and am more convinced now that this introduced a failure. (see test results below)> I''ll also try the evtchn_move_pirqs() removal against the changeset > above, to see if my results differ than when I did the same test > against the tip.21624:b9c541d9c138 12 successful suspend / resume cycles 21625:0695a5cdcb42 2 successful suspend / resume cycles - failure on 3rd (ahci) 21625:0695a5cdcb42 + evtchn_move_pirqs() removal: 12 successsful suspend / resume cycles This was encouraging, so I tried the same change against the tree tip...unfortunately that didn''t go as well. tip + evtchn_move_pirqs() removal: did not resume from the first suspend.
>>> On 15.08.12 at 14:32, Ben Guthro <ben@guthro.net> wrote: > This was encouraging, so I tried the same change against the tree > tip...unfortunately that didn''t go as well. > > tip + evtchn_move_pirqs() removal: > did not resume from the first suspend.Any logs of this (i.e. indications of what''s going wrong - still the same AHCI not working, but else apparently fine)? Jan
On Wed, Aug 15, 2012 at 8:58 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 15.08.12 at 14:32, Ben Guthro <ben@guthro.net> wrote: >> This was encouraging, so I tried the same change against the tree >> tip...unfortunately that didn''t go as well. >> >> tip + evtchn_move_pirqs() removal: >> did not resume from the first suspend. > > Any logs of this (i.e. indications of what''s going wrong - still > the same AHCI not working, but else apparently fine)?This is a bit strange, in that the observed behavior changes when I am logging to the serial connection. When I am logging to serial, the failure is the same as before - The first suspend / resume works - The second fails with AHCI not working However, when I am not logging to serial - the system goes down, but never comes back up. I cannot ssh in, and no graphics are displayed on the screen. My only recourse is to hard power cycle the machine. This, of course makes collecting logs difficult.
>>> On 15.08.12 at 15:11, Ben Guthro <ben@guthro.net> wrote: > On Wed, Aug 15, 2012 at 8:58 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 15.08.12 at 14:32, Ben Guthro <ben@guthro.net> wrote: >>> This was encouraging, so I tried the same change against the tree >>> tip...unfortunately that didn''t go as well. >>> >>> tip + evtchn_move_pirqs() removal: >>> did not resume from the first suspend. >> >> Any logs of this (i.e. indications of what''s going wrong - still >> the same AHCI not working, but else apparently fine)? > > This is a bit strange, in that the observed behavior changes when I am > logging to the serial connection. > > When I am logging to serial, the failure is the same as before - > The first suspend / resume works - > The second fails with AHCI not working > > However, when I am not logging to serial - the system goes down, but > never comes back up. I cannot ssh in, and no graphics are displayed on > the screen. My only recourse is to hard power cycle the machine. > > This, of course makes collecting logs difficult.Indeed. Did you try using the serial driver in polling mode (without IRQ that is)? Jan
On Wed, Aug 15, 2012 at 10:50 AM, Jan Beulich <JBeulich@suse.com> wrote:> > > This, of course makes collecting logs difficult. > > Indeed. Did you try using the serial driver in polling mode > (without IRQ that is)? > >I''m not familiar with how to set this up, and a quick glance through xen/drivers/charr/ns16550.c didn''t really shed much light. Is there a README / wiki page, etc describing this? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 15/08/12 15:58, Ben Guthro wrote:> > > On Wed, Aug 15, 2012 at 10:50 AM, Jan Beulich <JBeulich@suse.com > <mailto:JBeulich@suse.com>> wrote: > > > > This, of course makes collecting logs difficult. > > Indeed. Did you try using the serial driver in polling mode > (without IRQ that is)? > > > I''m not familiar with how to set this up, and a quick glance through > xen/drivers/charr/ns16550.c didn''t really shed much light. > > Is there a README / wiki page, etc describing this?http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html You want the com1 entry -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 15.08.12 at 16:58, Ben Guthro <ben@guthro.net> wrote: > On Wed, Aug 15, 2012 at 10:50 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >> > This, of course makes collecting logs difficult. >> >> Indeed. Did you try using the serial driver in polling mode >> (without IRQ that is)? >> >> > I''m not familiar with how to set this up, and a quick glance through > xen/drivers/charr/ns16550.c didn''t really shed much light. > > Is there a README / wiki page, etc describing this?There''s docs/misc/xen-command-line.markdown, which describes this. It''s basically com1=<baud>,8n1,<port>,<irq> and you''d want to set <irq> to 0 (I have a patch pending for post-4.2 that allows omitting all the fields that you don''t really care to change from their default values, but for now you''ll have to specify them). Jan
OK, well, I tried it with the following boot config: With serial: multiboot /xen.gz com1=115200,8n1,pci,0 console=com1 dom0_mem=max:1024M cpufreq=xen cpuidle sync_console loglvl=all xsave=0 module /vmlinuz-3.2.23-orc root=/dev/mapper/NxVG--eb56f027--0aeb--4e9a--9233--678d57b9dc9e-NxDisk5 ro ignore_loglevel no_console_suspend xencons=tty console=hvc I''m not sure it matters, but I''m making use of the renamed "magic" patch to autodetect the PCI serial card. Without serial: multiboot /xen.gz console=null dom0_mem=max:1024M cpufreq=xen cpuidle xsave=0 module /vmlinuz-3.2.23-orc dummy root=/dev/mapper/NxVG--eb56f027--0aeb--4e9a--9233--678d57b9dc9e-NxDisk5 ro module /initrd.img-3.2.23-orc All of that said - this exhibited the same behavior as before, with the presence of the serial connection changing the test behavior. On Wed, Aug 15, 2012 at 11:06 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 15.08.12 at 16:58, Ben Guthro <ben@guthro.net> wrote: > > On Wed, Aug 15, 2012 at 10:50 AM, Jan Beulich <JBeulich@suse.com> wrote: > > > >> > >> > This, of course makes collecting logs difficult. > >> > >> Indeed. Did you try using the serial driver in polling mode > >> (without IRQ that is)? > >> > >> > > I''m not familiar with how to set this up, and a quick glance through > > xen/drivers/charr/ns16550.c didn''t really shed much light. > > > > Is there a README / wiki page, etc describing this? > > There''s docs/misc/xen-command-line.markdown, which describes > this. It''s basically > > com1=<baud>,8n1,<port>,<irq> > > and you''d want to set <irq> to 0 (I have a patch pending for > post-4.2 that allows omitting all the fields that you don''t really > care to change from their default values, but for now you''ll > have to specify them). > > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 15.08.12 at 15:11, Ben Guthro <ben@guthro.net> wrote: > On Wed, Aug 15, 2012 at 8:58 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 15.08.12 at 14:32, Ben Guthro <ben@guthro.net> wrote: >>> This was encouraging, so I tried the same change against the tree >>> tip...unfortunately that didn''t go as well. >>> >>> tip + evtchn_move_pirqs() removal: >>> did not resume from the first suspend. >> >> Any logs of this (i.e. indications of what''s going wrong - still >> the same AHCI not working, but else apparently fine)? > > This is a bit strange, in that the observed behavior changes when I am > logging to the serial connection. > > When I am logging to serial, the failure is the same as before - > The first suspend / resume works - > The second fails with AHCI not workingAnd this is with and/or without the evtchn_move_pirqs() calls removed? Otherwise, this might allow us at least debugging that part of the problem.> However, when I am not logging to serial - the system goes down, but > never comes back up. I cannot ssh in, and no graphics are displayed on > the screen. My only recourse is to hard power cycle the machine. > > This, of course makes collecting logs difficult.Did you try whether anything would make it out to the screen when you allow Xen to continue to access the video buffer even post-boot? Quite possibly for this it would be advantageous (or even required, as I don''t think the video mode would get restored after coming back out of S3) to also only use a (simpler) text mode console (requiring that you have this up on the screen when you invoke S3, i.e. you''d have to switch away from or not make use of the GUI). That would be "vga=text-80x25,keep" on the Xen command line (while 80x50 or 80x60 would allow for more output to remain visible, even those modes would - I think - need code to be added to get restored during resume from S3). Jan
On Thu, Aug 16, 2012 at 4:31 AM, Jan Beulich <JBeulich@suse.com> wrote:>> When I am logging to serial, the failure is the same as before - >> The first suspend / resume works - >> The second fails with AHCI not working > > And this is with and/or without the evtchn_move_pirqs() calls > removed? Otherwise, this might allow us at least debugging > that part of the problem.I am now convinced there is more than one problem: One is the MSI issue we are chasing here... the other seems to be a bit more insidious, where the system does not come back from S3 at all - as mentioned in the Intel bug report from the other thread. Running on serial to debug the former seems to at least mask the latter. Removing evtchn_move_pirqs() at the tip does not seem to have the same effect as removing them from the changeset that I bisected the problem to. At the tip, with these changes - I observe no change in behavior - AHCI still has problems after the 2nd suspend/resume cycle. At 21625:0695a5cdcb42, with evtchn_move_pirqs() - I am able to suspend / resume a dozen times, or more.> >> However, when I am not logging to serial - the system goes down, but >> never comes back up. I cannot ssh in, and no graphics are displayed on >> the screen. My only recourse is to hard power cycle the machine. >> >> This, of course makes collecting logs difficult. > > Did you try whether anything would make it out to the screen > when you allow Xen to continue to access the video buffer even > post-boot? Quite possibly for this it would be advantageous (or > even required, as I don''t think the video mode would get restored > after coming back out of S3) to also only use a (simpler) text mode > console (requiring that you have this up on the screen when you > invoke S3, i.e. you''d have to switch away from or not make use of > the GUI). That would be "vga=text-80x25,keep" on the Xen > command line (while 80x50 or 80x60 would allow for more output > to remain visible, even those modes would - I think - need code > to be added to get restored during resume from S3). >I did not try this, but will investigate when I get to work today. Ben
>>> On 16.08.12 at 12:37, Ben Guthro <ben@guthro.net> wrote: > On Thu, Aug 16, 2012 at 4:31 AM, Jan Beulich <JBeulich@suse.com> wrote: >>> When I am logging to serial, the failure is the same as before - >>> The first suspend / resume works - >>> The second fails with AHCI not working >> >> And this is with and/or without the evtchn_move_pirqs() calls >> removed? Otherwise, this might allow us at least debugging >> that part of the problem. > > I am now convinced there is more than one problem: > One is the MSI issue we are chasing here... the other seems to be a > bit more insidious, where the system does not come back from S3 at all > - as mentioned in the Intel bug report from the other thread. > > Running on serial to debug the former seems to at least mask the latter. > > Removing evtchn_move_pirqs() at the tip does not seem to have the same > effect as removing them from the changeset that I bisected the problem > to.Odd.> At the tip, with these changes - I observe no change in behavior - > AHCI still has problems after the 2nd suspend/resume cycle. > At 21625:0695a5cdcb42, with evtchn_move_pirqs() - I am able to suspend > / resume a dozen times, or more.As there ought to be at least some affinity break messages during the suspend part, and I don''t recall having seen any, could you - for starters - provide a full serial log of the suspend/resume process, with "loglvl=all guest_loglvl=all" in place? I''ll then try to get to produce a debugging patch for you to try. Jan
On Thu, Aug 16, 2012 at 7:07 AM, Jan Beulich <JBeulich@suse.com> wrote:> > >>> On 16.08.12 at 12:37, Ben Guthro <ben@guthro.net> wrote: > > On Thu, Aug 16, 2012 at 4:31 AM, Jan Beulich <JBeulich@suse.com> wrote: > >>> When I am logging to serial, the failure is the same as before - > >>> The first suspend / resume works - > >>> The second fails with AHCI not working > >> > >> And this is with and/or without the evtchn_move_pirqs() calls > >> removed? Otherwise, this might allow us at least debugging > >> that part of the problem. > > > > I am now convinced there is more than one problem: > > One is the MSI issue we are chasing here... the other seems to be a > > bit more insidious, where the system does not come back from S3 at all > > - as mentioned in the Intel bug report from the other thread. > > > > Running on serial to debug the former seems to at least mask the latter. > > > > Removing evtchn_move_pirqs() at the tip does not seem to have the same > > effect as removing them from the changeset that I bisected the problem > > to. > > Odd.Indeed. I can''t explain it either.> > > > At the tip, with these changes - I observe no change in behavior - > > AHCI still has problems after the 2nd suspend/resume cycle. > > At 21625:0695a5cdcb42, with evtchn_move_pirqs() - I am able to suspend > > / resume a dozen times, or more. > > As there ought to be at least some affinity break messages during > the suspend part, and I don''t recall having seen any, could you - > for starters - provide a full serial log of the suspend/resume process, > with "loglvl=all guest_loglvl=all" in place? I''ll then try to get to > produce a debugging patch for you to try.In order to not flood the list with large logs, I put the logs here: https://citrix.sharefile.com/d/sfab699024a54df39 I wasn''t sure if you wanted a log with, or without the calls to evtchn_move_pirqs() commented out - so I included both. Please let me know if there are anything else you''d like me to experiment with. Ben
On Thu, Aug 16, 2012 at 7:56 AM, Ben Guthro <ben@guthro.net> wrote:> In order to not flood the list with large logs, I put the logs here: > https://citrix.sharefile.com/d/sfab699024a54df39 > I wasn''t sure if you wanted a log with, or without the calls to > evtchn_move_pirqs() commented out - so I included both.I received notifications that this got downloaded yesterday by a couple people. Did you have an opportunity to review it? If so - did you glean any new knowledge? Thanks for your time Ben
>>> On 17.08.12 at 12:22, Ben Guthro <ben@guthro.net> wrote: > On Thu, Aug 16, 2012 at 7:56 AM, Ben Guthro <ben@guthro.net> wrote: >> In order to not flood the list with large logs, I put the logs here: >> https://citrix.sharefile.com/d/sfab699024a54df39 >> I wasn''t sure if you wanted a log with, or without the calls to >> evtchn_move_pirqs() commented out - so I included both. > > I received notifications that this got downloaded yesterday by a couple > people. > Did you have an opportunity to review it?Yes, I did.> If so - did you glean any new knowledge?Unfortunately not. Instead I was surprised that there were no IRQ fixup related messages at all, of which I still will need to make sense. In any case, I''m planning on putting together a debugging patch, but can''t immediately tell when this will be. Jan
I did some more bisecting here, and I came up with another changeset
that seems to be problematic, Re: IRQs
After bisecting the problem discussed earlier in this thread to the
changeset below,
http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42
I worked past that issue by the following hack:
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d)
void evtchn_move_pirqs(struct vcpu *v)
{
struct domain *d = v->domain;
- const cpumask_t *mask = cpumask_of(v->processor);
+ //const cpumask_t *mask = cpumask_of(v->processor);
unsigned int port;
struct evtchn *chn;
@@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v)
for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port
)
{
chn = evtchn_from_port(d, port);
+#if 0
pirq_set_affinity(d, chn->u.pirq.irq, mask);
+#endif
}
spin_unlock(&d->event_lock);
}
This seemed to work for this rather old changeset, but it was not
sufficient to fix it against the 4.1, or unstable trees.
I further bisected, in combination with this hack, and found the
following changeset to also be problematic:
http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365
That is, before this change I could resume reliably (with the hack
above) - and after I could not.
This was surprising to me, as this change also looks rather innocuous.
Naturally, backing out this change seems to be non-trivial against the
tip, since so much around it has changed.
On Fri, Aug 17, 2012 at 6:40 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 17.08.12 at 12:22, Ben Guthro <ben@guthro.net> wrote:
>> On Thu, Aug 16, 2012 at 7:56 AM, Ben Guthro <ben@guthro.net>
wrote:
>>> In order to not flood the list with large logs, I put the logs
here:
>>> https://citrix.sharefile.com/d/sfab699024a54df39
>>> I wasn''t sure if you wanted a log with, or without the
calls to
>>> evtchn_move_pirqs() commented out - so I included both.
>>
>> I received notifications that this got downloaded yesterday by a couple
>> people.
>> Did you have an opportunity to review it?
>
> Yes, I did.
>
>> If so - did you glean any new knowledge?
>
> Unfortunately not. Instead I was surprised that there were no
> IRQ fixup related messages at all, of which I still will need to
> make sense.
>
> In any case, I''m planning on putting together a debugging patch,
> but can''t immediately tell when this will be.
>
> Jan
>
On 23/08/12 19:03, Ben Guthro wrote:> I did some more bisecting here, and I came up with another changeset > that seems to be problematic, Re: IRQs > > After bisecting the problem discussed earlier in this thread to the > changeset below, > http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 > > > I worked past that issue by the following hack: > > --- a/xen/common/event_channel.c > +++ b/xen/common/event_channel.c > @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) > void evtchn_move_pirqs(struct vcpu *v) > { > struct domain *d = v->domain; > - const cpumask_t *mask = cpumask_of(v->processor); > + //const cpumask_t *mask = cpumask_of(v->processor); > unsigned int port; > struct evtchn *chn; > > @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) > for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) > { > chn = evtchn_from_port(d, port); > +#if 0 > pirq_set_affinity(d, chn->u.pirq.irq, mask); > +#endif > } > spin_unlock(&d->event_lock); > } > > > This seemed to work for this rather old changeset, but it was not > sufficient to fix it against the 4.1, or unstable trees. > > I further bisected, in combination with this hack, and found the > following changeset to also be problematic: > > http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 > > > That is, before this change I could resume reliably (with the hack > above) - and after I could not. > This was surprising to me, as this change also looks rather innocuous.And by the looks of that changeset, the logic in fixup_irqs() in irq.c was changed. Jan: The commit message says "simplify operations [in] a few cases". Was the change in fixup_irqs() deliberate? ~Andrew> > > Naturally, backing out this change seems to be non-trivial against the > tip, since so much around it has changed. >-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com
> On 23/08/12 19:03, Ben Guthro wrote: >> I did some more bisecting here, and I came up with another changeset >> that seems to be problematic, Re: IRQs >> >> After bisecting the problem discussed earlier in this thread to the >> changeset below, >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 >> >> >> I worked past that issue by the following hack: >> >> --- a/xen/common/event_channel.c >> +++ b/xen/common/event_channel.c >> @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) >> void evtchn_move_pirqs(struct vcpu *v) >> { >> struct domain *d = v->domain; >> - const cpumask_t *mask = cpumask_of(v->processor); >> + //const cpumask_t *mask = cpumask_of(v->processor); >> unsigned int port; >> struct evtchn *chn; >> >> @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) >> for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) >> { >> chn = evtchn_from_port(d, port); >> +#if 0 >> pirq_set_affinity(d, chn->u.pirq.irq, mask); >> +#endif >> } >> spin_unlock(&d->event_lock); >> } >> >> >> This seemed to work for this rather old changeset, but it was not >> sufficient to fix it against the 4.1, or unstable trees. >> >> I further bisected, in combination with this hack, and found the >> following changeset to also be problematic: >> >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 >> >> >> That is, before this change I could resume reliably (with the hack >> above) - and after I could not. >> This was surprising to me, as this change also looks rather innocuous. > And by the looks of that changeset, the logic in fixup_irqs() in irq.c > was changed. > > Jan: The commit message says "simplify operations [in] a few cases". > Was the change in fixup_irqs() deliberate? > > ~AndrewBen: Could you test the attached patch? It is for unstable and undoes the logical change to fixup_irqs() ~Andrew> >> Naturally, backing out this change seems to be non-trivial against the >> tip, since so much around it has changed. >> > -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 > (0)1223 225 900, http://www.citrix.com > _______________________________________________ Xen-devel mailing list > Xen-devel@lists.xen.org http://lists.xen.org/xen-devel-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
No such luck. Panic below: (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) ----[ Xen-4.2.0-rc3-pre x86_64 debug=y Tainted: C ]---- (XEN) CPU: 1 (XEN) RIP: e008:[<ffff82c48016773e>] irq_complete_move+0x42/0xb4 (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor (XEN) rax: 0000000000000000 rbx: ffff830148a81880 rcx: 0000000000000001 (XEN) rdx: ffff82c480301e48 rsi: 0000000000000028 rdi: ffff830148a81880 (XEN) rbp: ffff830148a77d80 rsp: ffff830148a77d50 r8: 0000000000000004 (XEN) r9: ffff82c3fffff000 r10: ffff82c4803027c0 r11: 0000000000000246 (XEN) r12: 0000000000000018 r13: ffff830148a818a4 r14: 0000000000000000 (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000001026b0 (XEN) cr3: 00000000aa2c5000 cr2: 000000000000007c (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff830148a77d50: (XEN) 0000000000000086 ffff830148a809a4 0000000000000000 0000000000000001 (XEN) ffff830148a77d80 ffff830148ae2620 ffff830148a77da0 ffff82c480144013 (XEN) ffff830148a81880 0000000000000018 ffff830148a77e10 ffff82c4801696b6 (XEN) 0000000000000086 00000001030d8ac4 0000000000000001 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000202 0000000000000010 (XEN) ffff82c480302938 0000000000000001 0000000000000001 ffff82c480302938 (XEN) ffff830148a77e70 ffff82c48017e132 0000001048a77e70 ffff82c480302930 (XEN) ffff82c480302938 0000000000000001 000000010115fa86 0000000000000003 (XEN) ffff830148a77f18 0000000000000001 ffffffffffffffff ffff830148ac3080 (XEN) ffff830148a77e80 ffff82c4801013e3 ffff830148a77ea0 ffff82c480125fb6 (XEN) ffff830148ac30c0 ffff830148ac30f0 ffff830148a77ec0 ffff82c48012753e (XEN) ffff82c4801256aa ffff830148ac3110 ffff830148a77ef0 ffff82c4801278a9 (XEN) 00000000ffffffff ffff830148a77f18 ffff830148a77f18 00000008030d8ac4 (XEN) ffff830148a77f10 ffff82c48015888d ffff8300aa303000 ffff8300aa303000 (XEN) ffff830148a77e10 0000000000000000 0000000000000000 0000000000000000 (XEN) ffffffff81aafda0 ffff88003976df10 ffff88003976dfd8 0000000000000246 (XEN) 00000000deadbeef 0000000000000000 aaaaaaaaaaaaaaaa 0000000000000000 (XEN) ffffffff8100130a 00000000deadbeef 00000000deadbeef 00000000deadbeef (XEN) 0000010000000000 ffffffff8100130a 000000000000e033 0000000000000246 (XEN) ffff88003976def8 000000000000e02b d0354dcf5bcd9824 9ea5d0ef618deca5 (XEN) Xen call trace: (XEN) [<ffff82c48016773e>] irq_complete_move+0x42/0xb4 (XEN) [<ffff82c480144013>] dma_msi_mask+0x1d/0x49 (XEN) [<ffff82c4801696b6>] fixup_irqs+0x19b/0x2ff (XEN) [<ffff82c48017e132>] __cpu_disable+0x337/0x37e (XEN) [<ffff82c4801013e3>] take_cpu_down+0x43/0x4a (XEN) [<ffff82c480125fb6>] stopmachine_action+0x8a/0xb3 (XEN) [<ffff82c48012753e>] do_tasklet_work+0x8d/0xc7 (XEN) [<ffff82c4801278a9>] do_tasklet+0x6b/0x9b (XEN) [<ffff82c48015888d>] idle_loop+0x67/0x71 (XEN) (XEN) Pagetable walk from 000000000000007c: (XEN) L4[0x000] = 0000000148adf063 ffffffffffffffff (XEN) L3[0x000] = 0000000148ade063 ffffffffffffffff (XEN) L2[0x000] = 0000000148add063 ffffffffffffffff (XEN) L1[0x000] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: 000000000000007c (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... On Thu, Aug 23, 2012 at 2:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:> On 23/08/12 19:03, Ben Guthro wrote: > > I did some more bisecting here, and I came up with another changeset > that seems to be problematic, Re: IRQs > > After bisecting the problem discussed earlier in this thread to the > changeset below, > http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 > > > I worked past that issue by the following hack: > > --- a/xen/common/event_channel.c > +++ b/xen/common/event_channel.c > @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) > void evtchn_move_pirqs(struct vcpu *v) > { > struct domain *d = v->domain; > - const cpumask_t *mask = cpumask_of(v->processor); > + //const cpumask_t *mask = cpumask_of(v->processor); > unsigned int port; > struct evtchn *chn; > > @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) > for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) > { > chn = evtchn_from_port(d, port); > +#if 0 > pirq_set_affinity(d, chn->u.pirq.irq, mask); > +#endif > } > spin_unlock(&d->event_lock); > } > > > This seemed to work for this rather old changeset, but it was not > sufficient to fix it against the 4.1, or unstable trees. > > I further bisected, in combination with this hack, and found the > following changeset to also be problematic: > > http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 > > > That is, before this change I could resume reliably (with the hack > above) - and after I could not. > This was surprising to me, as this change also looks rather innocuous. > > And by the looks of that changeset, the logic in fixup_irqs() in irq.c > was changed. > > Jan: The commit message says "simplify operations [in] a few cases". > Was the change in fixup_irqs() deliberate? > > ~Andrew > > > Ben: Could you test the attached patch? It is for unstable and undoes the > logical change to fixup_irqs() > > ~Andrew > > > Naturally, backing out this change seems to be non-trivial against the > tip, since so much around it has changed. > > -- > Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer > T: +44 (0)1223 225 900, http://www.citrix.com > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > > > -- > Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer > T: +44 (0)1223 225 900, http://www.citrix.com
Interestingly enough, just updating to the parent of this changeset, without the hack mentioned previously seems to be enough to suspend/resume multiple times on this machine. On Thu, Aug 23, 2012 at 3:06 PM, Ben Guthro <ben@guthro.net> wrote:> No such luck. > > Panic below: > > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) ----[ Xen-4.2.0-rc3-pre x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 1 > (XEN) RIP: e008:[<ffff82c48016773e>] irq_complete_move+0x42/0xb4 > (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor > (XEN) rax: 0000000000000000 rbx: ffff830148a81880 rcx: 0000000000000001 > (XEN) rdx: ffff82c480301e48 rsi: 0000000000000028 rdi: ffff830148a81880 > (XEN) rbp: ffff830148a77d80 rsp: ffff830148a77d50 r8: 0000000000000004 > (XEN) r9: ffff82c3fffff000 r10: ffff82c4803027c0 r11: 0000000000000246 > (XEN) r12: 0000000000000018 r13: ffff830148a818a4 r14: 0000000000000000 > (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000001026b0 > (XEN) cr3: 00000000aa2c5000 cr2: 000000000000007c > (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff830148a77d50: > (XEN) 0000000000000086 ffff830148a809a4 0000000000000000 0000000000000001 > (XEN) ffff830148a77d80 ffff830148ae2620 ffff830148a77da0 ffff82c480144013 > (XEN) ffff830148a81880 0000000000000018 ffff830148a77e10 ffff82c4801696b6 > (XEN) 0000000000000086 00000001030d8ac4 0000000000000001 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000202 0000000000000010 > (XEN) ffff82c480302938 0000000000000001 0000000000000001 ffff82c480302938 > (XEN) ffff830148a77e70 ffff82c48017e132 0000001048a77e70 ffff82c480302930 > (XEN) ffff82c480302938 0000000000000001 000000010115fa86 0000000000000003 > (XEN) ffff830148a77f18 0000000000000001 ffffffffffffffff ffff830148ac3080 > (XEN) ffff830148a77e80 ffff82c4801013e3 ffff830148a77ea0 ffff82c480125fb6 > (XEN) ffff830148ac30c0 ffff830148ac30f0 ffff830148a77ec0 ffff82c48012753e > (XEN) ffff82c4801256aa ffff830148ac3110 ffff830148a77ef0 ffff82c4801278a9 > (XEN) 00000000ffffffff ffff830148a77f18 ffff830148a77f18 00000008030d8ac4 > (XEN) ffff830148a77f10 ffff82c48015888d ffff8300aa303000 ffff8300aa303000 > (XEN) ffff830148a77e10 0000000000000000 0000000000000000 0000000000000000 > (XEN) ffffffff81aafda0 ffff88003976df10 ffff88003976dfd8 0000000000000246 > (XEN) 00000000deadbeef 0000000000000000 aaaaaaaaaaaaaaaa 0000000000000000 > (XEN) ffffffff8100130a 00000000deadbeef 00000000deadbeef 00000000deadbeef > (XEN) 0000010000000000 ffffffff8100130a 000000000000e033 0000000000000246 > (XEN) ffff88003976def8 000000000000e02b d0354dcf5bcd9824 9ea5d0ef618deca5 > (XEN) Xen call trace: > (XEN) [<ffff82c48016773e>] irq_complete_move+0x42/0xb4 > (XEN) [<ffff82c480144013>] dma_msi_mask+0x1d/0x49 > (XEN) [<ffff82c4801696b6>] fixup_irqs+0x19b/0x2ff > (XEN) [<ffff82c48017e132>] __cpu_disable+0x337/0x37e > (XEN) [<ffff82c4801013e3>] take_cpu_down+0x43/0x4a > (XEN) [<ffff82c480125fb6>] stopmachine_action+0x8a/0xb3 > (XEN) [<ffff82c48012753e>] do_tasklet_work+0x8d/0xc7 > (XEN) [<ffff82c4801278a9>] do_tasklet+0x6b/0x9b > (XEN) [<ffff82c48015888d>] idle_loop+0x67/0x71 > (XEN) > (XEN) Pagetable walk from 000000000000007c: > (XEN) L4[0x000] = 0000000148adf063 ffffffffffffffff > (XEN) L3[0x000] = 0000000148ade063 ffffffffffffffff > (XEN) L2[0x000] = 0000000148add063 ffffffffffffffff > (XEN) L1[0x000] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 1: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: 000000000000007c > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > > On Thu, Aug 23, 2012 at 2:54 PM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: >> On 23/08/12 19:03, Ben Guthro wrote: >> >> I did some more bisecting here, and I came up with another changeset >> that seems to be problematic, Re: IRQs >> >> After bisecting the problem discussed earlier in this thread to the >> changeset below, >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 >> >> >> I worked past that issue by the following hack: >> >> --- a/xen/common/event_channel.c >> +++ b/xen/common/event_channel.c >> @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) >> void evtchn_move_pirqs(struct vcpu *v) >> { >> struct domain *d = v->domain; >> - const cpumask_t *mask = cpumask_of(v->processor); >> + //const cpumask_t *mask = cpumask_of(v->processor); >> unsigned int port; >> struct evtchn *chn; >> >> @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) >> for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) >> { >> chn = evtchn_from_port(d, port); >> +#if 0 >> pirq_set_affinity(d, chn->u.pirq.irq, mask); >> +#endif >> } >> spin_unlock(&d->event_lock); >> } >> >> >> This seemed to work for this rather old changeset, but it was not >> sufficient to fix it against the 4.1, or unstable trees. >> >> I further bisected, in combination with this hack, and found the >> following changeset to also be problematic: >> >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 >> >> >> That is, before this change I could resume reliably (with the hack >> above) - and after I could not. >> This was surprising to me, as this change also looks rather innocuous. >> >> And by the looks of that changeset, the logic in fixup_irqs() in irq.c >> was changed. >> >> Jan: The commit message says "simplify operations [in] a few cases". >> Was the change in fixup_irqs() deliberate? >> >> ~Andrew >> >> >> Ben: Could you test the attached patch? It is for unstable and undoes the >> logical change to fixup_irqs() >> >> ~Andrew >> >> >> Naturally, backing out this change seems to be non-trivial against the >> tip, since so much around it has changed. >> >> -- >> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer >> T: +44 (0)1223 225 900, http://www.citrix.com >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >> >> >> -- >> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer >> T: +44 (0)1223 225 900, http://www.citrix.com
On 23/08/12 20:06, Ben Guthro wrote:> No such luck.Huh. It was a shot in the dark, but I was really not expecting this.> > Panic below: > > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) ----[ Xen-4.2.0-rc3-pre x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 1 > (XEN) RIP: e008:[<ffff82c48016773e>] irq_complete_move+0x42/0xb4 > (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor > (XEN) rax: 0000000000000000 rbx: ffff830148a81880 rcx: 0000000000000001 > (XEN) rdx: ffff82c480301e48 rsi: 0000000000000028 rdi: ffff830148a81880 > (XEN) rbp: ffff830148a77d80 rsp: ffff830148a77d50 r8: 0000000000000004 > (XEN) r9: ffff82c3fffff000 r10: ffff82c4803027c0 r11: 0000000000000246 > (XEN) r12: 0000000000000018 r13: ffff830148a818a4 r14: 0000000000000000 > (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000001026b0 > (XEN) cr3: 00000000aa2c5000 cr2: 000000000000007c > (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff830148a77d50: > (XEN) 0000000000000086 ffff830148a809a4 0000000000000000 0000000000000001 > (XEN) ffff830148a77d80 ffff830148ae2620 ffff830148a77da0 ffff82c480144013 > (XEN) ffff830148a81880 0000000000000018 ffff830148a77e10 ffff82c4801696b6 > (XEN) 0000000000000086 00000001030d8ac4 0000000000000001 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000202 0000000000000010 > (XEN) ffff82c480302938 0000000000000001 0000000000000001 ffff82c480302938 > (XEN) ffff830148a77e70 ffff82c48017e132 0000001048a77e70 ffff82c480302930 > (XEN) ffff82c480302938 0000000000000001 000000010115fa86 0000000000000003 > (XEN) ffff830148a77f18 0000000000000001 ffffffffffffffff ffff830148ac3080 > (XEN) ffff830148a77e80 ffff82c4801013e3 ffff830148a77ea0 ffff82c480125fb6 > (XEN) ffff830148ac30c0 ffff830148ac30f0 ffff830148a77ec0 ffff82c48012753e > (XEN) ffff82c4801256aa ffff830148ac3110 ffff830148a77ef0 ffff82c4801278a9 > (XEN) 00000000ffffffff ffff830148a77f18 ffff830148a77f18 00000008030d8ac4 > (XEN) ffff830148a77f10 ffff82c48015888d ffff8300aa303000 ffff8300aa303000 > (XEN) ffff830148a77e10 0000000000000000 0000000000000000 0000000000000000 > (XEN) ffffffff81aafda0 ffff88003976df10 ffff88003976dfd8 0000000000000246 > (XEN) 00000000deadbeef 0000000000000000 aaaaaaaaaaaaaaaa 0000000000000000 > (XEN) ffffffff8100130a 00000000deadbeef 00000000deadbeef 00000000deadbeef > (XEN) 0000010000000000 ffffffff8100130a 000000000000e033 0000000000000246 > (XEN) ffff88003976def8 000000000000e02b d0354dcf5bcd9824 9ea5d0ef618deca5 > (XEN) Xen call trace: > (XEN) [<ffff82c48016773e>] irq_complete_move+0x42/0xb4 > (XEN) [<ffff82c480144013>] dma_msi_mask+0x1d/0x49 > (XEN) [<ffff82c4801696b6>] fixup_irqs+0x19b/0x2ff > (XEN) [<ffff82c48017e132>] __cpu_disable+0x337/0x37e > (XEN) [<ffff82c4801013e3>] take_cpu_down+0x43/0x4a > (XEN) [<ffff82c480125fb6>] stopmachine_action+0x8a/0xb3 > (XEN) [<ffff82c48012753e>] do_tasklet_work+0x8d/0xc7 > (XEN) [<ffff82c4801278a9>] do_tasklet+0x6b/0x9b > (XEN) [<ffff82c48015888d>] idle_loop+0x67/0x71 > (XEN) > (XEN) Pagetable walk from 000000000000007c: > (XEN) L4[0x000] = 0000000148adf063 ffffffffffffffff > (XEN) L3[0x000] = 0000000148ade063 ffffffffffffffff > (XEN) L2[0x000] = 0000000148add063 ffffffffffffffff > (XEN) L1[0x000] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 1: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: 000000000000007c > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds...Looks like some pointer has turned to NULL, although I cant identify exactly which. Either way, I would not pay it too much heed. ~Andrew> > > On Thu, Aug 23, 2012 at 2:54 PM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: >> On 23/08/12 19:03, Ben Guthro wrote: >> >> I did some more bisecting here, and I came up with another changeset >> that seems to be problematic, Re: IRQs >> >> After bisecting the problem discussed earlier in this thread to the >> changeset below, >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 >> >> >> I worked past that issue by the following hack: >> >> --- a/xen/common/event_channel.c >> +++ b/xen/common/event_channel.c >> @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) >> void evtchn_move_pirqs(struct vcpu *v) >> { >> struct domain *d = v->domain; >> - const cpumask_t *mask = cpumask_of(v->processor); >> + //const cpumask_t *mask = cpumask_of(v->processor); >> unsigned int port; >> struct evtchn *chn; >> >> @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) >> for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) >> { >> chn = evtchn_from_port(d, port); >> +#if 0 >> pirq_set_affinity(d, chn->u.pirq.irq, mask); >> +#endif >> } >> spin_unlock(&d->event_lock); >> } >> >> >> This seemed to work for this rather old changeset, but it was not >> sufficient to fix it against the 4.1, or unstable trees. >> >> I further bisected, in combination with this hack, and found the >> following changeset to also be problematic: >> >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 >> >> >> That is, before this change I could resume reliably (with the hack >> above) - and after I could not. >> This was surprising to me, as this change also looks rather innocuous. >> >> And by the looks of that changeset, the logic in fixup_irqs() in irq.c >> was changed. >> >> Jan: The commit message says "simplify operations [in] a few cases". >> Was the change in fixup_irqs() deliberate? >> >> ~Andrew >> >> >> Ben: Could you test the attached patch? It is for unstable and undoes the >> logical change to fixup_irqs() >> >> ~Andrew >> >> >> Naturally, backing out this change seems to be non-trivial against the >> tip, since so much around it has changed. >> >> -- >> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer >> T: +44 (0)1223 225 900, http://www.citrix.com >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >> >> >> -- >> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer >> T: +44 (0)1223 225 900, http://www.citrix.com-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com
On Thu, Aug 23, 2012 at 3:38 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:> > On 23/08/12 20:06, Ben Guthro wrote: >> No such luck. > > Huh. It was a shot in the dark, but I was really not expecting this.Thanks for taking a look. I tested the equivalent change against the offending changeset, and it did, in fact solve the S3 issue back then (but not against the 4.1.3 tag) I guess I''ll do some more bisecting, with this change in place. Ben
The attached patch is essentially the change you suggested, plus a check for NULL in irq_complete_move() This patch seems to fix some of the issues I was seeing, but not all. MSI''s are now delivered, after a handful of suspend / resumes, which is the issue I was setting out to solve here. However, once I get out of my debugging mode, and run without a serial console - things start to go bad. I am unable to get the machine to resume from S3. The fan comes on, but I never see any graphics, or HDD activity. Of course, without a console, I have no clues to go on, as to what is wrong. This is similar to the other S3 problem I have observed on laptops that never get out of the "pulsing power button" state. I suspect they are related, but with the behavior changing so drastically while running with the serial connection enabled - it is rather difficult to narrow down where it might be... On Thu, Aug 23, 2012 at 4:38 PM, Ben Guthro <ben@guthro.net> wrote:> On Thu, Aug 23, 2012 at 3:38 PM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: >> >> On 23/08/12 20:06, Ben Guthro wrote: >>> No such luck. >> >> Huh. It was a shot in the dark, but I was really not expecting this. > > Thanks for taking a look. > > I tested the equivalent change against the offending changeset, and it > did, in fact solve the S3 issue back then (but not against the 4.1.3 > tag) > > I guess I''ll do some more bisecting, with this change in place. > > Ben_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 23.08.12 at 20:37, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > On 23/08/12 19:03, Ben Guthro wrote: >> I did some more bisecting here, and I came up with another changeset >> that seems to be problematic, Re: IRQs >> >> After bisecting the problem discussed earlier in this thread to the >> changeset below, >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 >> >> >> I worked past that issue by the following hack: >> >> --- a/xen/common/event_channel.c >> +++ b/xen/common/event_channel.c >> @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) >> void evtchn_move_pirqs(struct vcpu *v) >> { >> struct domain *d = v->domain; >> - const cpumask_t *mask = cpumask_of(v->processor); >> + //const cpumask_t *mask = cpumask_of(v->processor); >> unsigned int port; >> struct evtchn *chn; >> >> @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) >> for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) >> { >> chn = evtchn_from_port(d, port); >> +#if 0 >> pirq_set_affinity(d, chn->u.pirq.irq, mask); >> +#endif >> } >> spin_unlock(&d->event_lock); >> } >> >> >> This seemed to work for this rather old changeset, but it was not >> sufficient to fix it against the 4.1, or unstable trees. >> >> I further bisected, in combination with this hack, and found the >> following changeset to also be problematic: >> >> http://xenbits.xen.org/hg/xen-unstable.hg/rev/c2cb776a5365 >> >> >> That is, before this change I could resume reliably (with the hack >> above) - and after I could not. >> This was surprising to me, as this change also looks rather innocuous. > > And by the looks of that changeset, the logic in fixup_irqs() in irq.c > was changed. > > Jan: The commit message says "simplify operations [in] a few cases". > Was the change in fixup_irqs() deliberate?Yes, it was: There''s no need to break/adjust the affinity if it continues to be a subset of cpu_online_map (i.e. there''s no need for the to match exactly). A similar change was also done to Linux''es fixup_irqs() later one, without any problems that I''m aware of. Jan
>>> On 24.08.12 at 17:10, Ben Guthro <ben@guthro.net> wrote: > The attached patch is essentially the change you suggested, plus a > check for NULL in irq_complete_move() > > This patch seems to fix some of the issues I was seeing, but not all. > MSI''s are now delivered, after a handful of suspend / resumes, which > is the issue I was setting out to solve here.But I''m afraid this is just masking some other problem (see my response to Andrew''s mail that I just sent). Further more, the NULL check here seems pretty odd - I''d be very curious what code path could get us there with the IRQ regs pointer being NULL. It would likely be that code path that needs fixing, not irq_complete_move(). Could you add a call to dump_execution_state() to the early return path that you added? Jan
>>> On 23.08.12 at 20:03, Ben Guthro <ben@guthro.net> wrote: > I did some more bisecting here, and I came up with another changeset > that seems to be problematic, Re: IRQs > > After bisecting the problem discussed earlier in this thread to the > changeset below, > http://xenbits.xen.org/hg/xen-unstable.hg/rev/0695a5cdcb42 > > > I worked past that issue by the following hack: > > --- a/xen/common/event_channel.c > +++ b/xen/common/event_channel.c > @@ -1103,7 +1103,7 @@ void evtchn_destroy_final(struct domain *d) > void evtchn_move_pirqs(struct vcpu *v) > { > struct domain *d = v->domain; > - const cpumask_t *mask = cpumask_of(v->processor); > + //const cpumask_t *mask = cpumask_of(v->processor); > unsigned int port; > struct evtchn *chn; > > @@ -1111,7 +1111,9 @@ void evtchn_move_pirqs(struct vcpu *v) > for ( port = v->pirq_evtchn_head; port; port = chn->u.pirq.next_port ) > { > chn = evtchn_from_port(d, port); > +#if 0 > pirq_set_affinity(d, chn->u.pirq.irq, mask); > +#endif > } > spin_unlock(&d->event_lock); > }Did you also make an attempt at figuring out which of the three calls to evtchn_move_pirqs() is the actual problematic one? Jan
On Fri, Aug 24, 2012 at 6:55 PM, Jan Beulich <JBeulich@suse.com> wrote:> Did you also make an attempt at figuring out which of the three calls > to evtchn_move_pirqs() is the actual problematic one? >I did a lot of experimentation...I think this was one of the tests that I did - though I''ve slept, and rebooted that machine so many times over the past few days, a lot of the tests are starting to run together. If I recall correctly, I was unable to isolate this issue - commenting out just one of the three calls didn''t seem to fix the problem. I''ll be traveling all next week (part of which is the Xen summit) - but I will be able to give a more definitive answer to this when I return to the office. Since this is such an old changeset, I moved past it, since the same fix at the tip didn''t have the same effect.
>>> On 16.08.12 at 13:56, Ben Guthro <ben@guthro.net> wrote: > Please let me know if there are anything else you''d like me to experiment > with.Attached a first take at a debugging patch. While putting this together I realized that the situation gets complicated by the fact that plain Linux 3.2.23 doesn''t even have support for post-S3 MSI restoration, so much would depend on how exactly this is implemented in the kernel you''re running (i.e. whether you have a _complete_ backport in place). I suppose you must be having some support for this patched in, otherwise I wouldn''t understand why things work without IOMMU/interrupt remapping. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
I''ve put the console log of this test run here: https://citrix.sharefile.com/d/sdc383e252fb41c5a (again, so as not to clog everyone''s inbox) I have not yet gone through the log in its entirety yet, but thought I would first send it to you to see if you had something in particular you were looking for. The file name is console-S3-MSI.txt On Mon, Sep 3, 2012 at 5:31 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 16.08.12 at 13:56, Ben Guthro <ben@guthro.net> wrote: >> Please let me know if there are anything else you''d like me to experiment >> with. > > Attached a first take at a debugging patch. > > While putting this together I realized that the situation gets > complicated by the fact that plain Linux 3.2.23 doesn''t even > have support for post-S3 MSI restoration, so much would > depend on how exactly this is implemented in the kernel > you''re running (i.e. whether you have a _complete_ backport > in place). I suppose you must be having some support for this > patched in, otherwise I wouldn''t understand why things work > without IOMMU/interrupt remapping. > > Jan >
I forgot to address your dom0 kernel concern - All of these tests have been run with Konrad Wilk''s acpi-s3.vX branches. The later kernels took more recent branches, while the 3.2.yy kernels used an older (but still functional) branch (acpi-s3.v7) For reference: http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=shortlog;h=refs/heads/devel/acpi-s3.v7 On Tue, Sep 4, 2012 at 8:27 AM, Ben Guthro <ben@guthro.net> wrote:> I''ve put the console log of this test run here: > https://citrix.sharefile.com/d/sdc383e252fb41c5a > > (again, so as not to clog everyone''s inbox) > > I have not yet gone through the log in its entirety yet, but thought I > would first send it to you to see if you had something in particular > you were looking for. > > The file name is console-S3-MSI.txt > > > On Mon, Sep 3, 2012 at 5:31 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 16.08.12 at 13:56, Ben Guthro <ben@guthro.net> wrote: >>> Please let me know if there are anything else you''d like me to experiment >>> with. >> >> Attached a first take at a debugging patch. >> >> While putting this together I realized that the situation gets >> complicated by the fact that plain Linux 3.2.23 doesn''t even >> have support for post-S3 MSI restoration, so much would >> depend on how exactly this is implemented in the kernel >> you''re running (i.e. whether you have a _complete_ backport >> in place). I suppose you must be having some support for this >> patched in, otherwise I wouldn''t understand why things work >> without IOMMU/interrupt remapping. >> >> Jan >>
>>> On 04.09.12 at 14:49, Ben Guthro <ben@guthro.net> wrote: > I forgot to address your dom0 kernel concern - > > All of these tests have been run with Konrad Wilk''s acpi-s3.vX branches. > > The later kernels took more recent branches, while the 3.2.yy kernels > used an older (but still functional) branch (acpi-s3.v7)This and the log you provided suggest that your kernel is lacking the MSI restore code altogether (or it is not getting called as intended). With that, there''s pretty little point in continuing the investigation on the hypervisor side. Jan
How is it that it works with an older hypervisor, then? Konrad - is the v7 code functionally different than the v10 code, or is it more stylistic changes? On Sep 4, 2012, at 10:25 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 04.09.12 at 14:49, Ben Guthro <ben@guthro.net> wrote: >> I forgot to address your dom0 kernel concern - >> >> All of these tests have been run with Konrad Wilk''s acpi-s3.vX branches. >> >> The later kernels took more recent branches, while the 3.2.yy kernels >> used an older (but still functional) branch (acpi-s3.v7) > > This and the log you provided suggest that your kernel is lacking > the MSI restore code altogether (or it is not getting called as > intended). With that, there''s pretty little point in continuing > the investigation on the hypervisor side. > > Jan >
On Tue, Sep 04, 2012 at 10:28:37AM -0400, Ben Guthro wrote:> How is it that it works with an older hypervisor, then? > > Konrad - is the v7 code functionally different than the v10 code, or > is it more stylistic changes?It is that v3.4 (and up) already has the restore_MSI hypervisor call: commit 8605c6844fb9bdf55471bb87c3ac62d44eb34e04 Author: Tang Liang <liang.tang@oracle.com> Date: Thu Dec 8 17:36:39 2011 +0800 xen: Utilize the restore_msi_irqs hook. to make a hypercall to restore the vectors in the MSI/MSI-X configuration space. Signed-off-by: Tang Liang <liang.tang@oracle.com> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> The only part that is left out of mainline is the part where the hypercall is done properly. That is what the acpi/s3.v10 does.. but after talking to Len he pointed out a different way of doing this.> > On Sep 4, 2012, at 10:25 AM, Jan Beulich <JBeulich@suse.com> wrote: > > >>>> On 04.09.12 at 14:49, Ben Guthro <ben@guthro.net> wrote: > >> I forgot to address your dom0 kernel concern - > >> > >> All of these tests have been run with Konrad Wilk''s acpi-s3.vX branches. > >> > >> The later kernels took more recent branches, while the 3.2.yy kernels > >> used an older (but still functional) branch (acpi-s3.v7) > > > > This and the log you provided suggest that your kernel is lacking > > the MSI restore code altogether (or it is not getting called as > > intended). With that, there''s pretty little point in continuing > > the investigation on the hypervisor side. > > > > Jan > >
>>> On 04.09.12 at 16:28, Ben Guthro <ben.guthro@gmail.com> wrote: > How is it that it works with an older hypervisor, then?Actually, checking again I see that I overlooked that specific log entry (I expected them to be all close together, but they aren''t). It is obvious that on the second restore the kernel passes bad data to the hypervisor, but I can''t tell yet whether that''s the kernel''s fault - will need to look more closely. Jan> Konrad - is the v7 code functionally different than the v10 code, or > is it more stylistic changes? > > On Sep 4, 2012, at 10:25 AM, Jan Beulich <JBeulich@suse.com> wrote: > >>>>> On 04.09.12 at 14:49, Ben Guthro <ben@guthro.net> wrote: >>> I forgot to address your dom0 kernel concern - >>> >>> All of these tests have been run with Konrad Wilk''s acpi-s3.vX branches. >>> >>> The later kernels took more recent branches, while the 3.2.yy kernels >>> used an older (but still functional) branch (acpi-s3.v7) >> >> This and the log you provided suggest that your kernel is lacking >> the MSI restore code altogether (or it is not getting called as >> intended). With that, there''s pretty little point in continuing >> the investigation on the hypervisor side. >> >> Jan >>
On Tue, Sep 4, 2012 at 12:33 PM, Ben Guthro <ben@guthro.net> wrote:> This is getting reasonably in depth to be off-list. > > You really should CC xen-devel for these kinds of things.> <snip>>> Damn it. It is a 3.3 tree and I need drivers introduced in 3.4 and I >> also use bcache >> which right now has a 3.2 stable version, a 3.5 not so stable and the >> devel branch.> Take the top 3 commits by Konrad & apply them as a patch. It should be > sufficient....but you''ll still run into issues with S3 on Xen, related > to MSI delivery...if it resumes at all. There are clearly still > hypervisor issues - so you seem to be jumping to some conclusions here > with workarounds that I''m not sure are going to get you anywhere.I''ve tried the top two commits from Konrad''s acpi-s3.v10 branch merged on top of kernel 3.5.3. The third commit was already merged. I''ve tried it from work and I believe it suspends fine, but there is some problem during resume since the machine responded to a ping for a second and then rebooted. The only error I have on the logs is: (XEN) memory.c:131:d0 Could not allocate order=18 extent: id=5 memflags=0 (0 of 1) (XEN) sh error: sh_remove_all_mappings(): can''t find all mappings of mfn 270d05: c=c000000000000002 t=7400000000000001 I don''t know whether that''s enough to make resume from S3 fail or it''s something else. When I''m at home I''ll see if I can dig further. -- Javier Marcet <jmarcet@gmail.com>
>>> On 04.09.12 at 20:34, Ben Guthro <ben@guthro.net> wrote: > On Fri, Aug 24, 2012 at 6:16 PM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 24.08.12 at 17:10, Ben Guthro <ben@guthro.net> wrote: >>> The attached patch is essentially the change you suggested, plus a >>> check for NULL in irq_complete_move() >>> >>> This patch seems to fix some of the issues I was seeing, but not all. >>> MSI''s are now delivered, after a handful of suspend / resumes, which >>> is the issue I was setting out to solve here. >> >> But I''m afraid this is just masking some other problem (see my >> response to Andrew''s mail that I just sent). >> >> Further more, the NULL check here seems pretty odd - I''d be very >> curious what code path could get us there with the IRQ regs >> pointer being NULL. It would likely be that code path that needs >> fixing, not irq_complete_move(). Could you add a call to >> dump_execution_state() to the early return path that you added? > > Apologies that I''m just getting back to this. > > The call trace in this early return patch looks like this: > > (XEN) Xen call trace: > (XEN) [<ffff82c480167d1a>] irq_complete_move+0x3e/0xd9 > (XEN) [<ffff82c48014424a>] dma_msi_mask+0x1d/0x49 > (XEN) [<ffff82c480169cc2>] fixup_irqs+0x19c/0x300 > (XEN) [<ffff82c48017e762>] __cpu_disable+0x337/0x37e > (XEN) [<ffff82c4801013e3>] take_cpu_down+0x43/0x4a > (XEN) [<ffff82c480125fe6>] stopmachine_action+0x8a/0xb3 > (XEN) [<ffff82c48012756e>] do_tasklet_work+0x8d/0xc7 > (XEN) [<ffff82c4801278d9>] do_tasklet+0x6b/0x9b > (XEN) [<ffff82c480158cbd>] idle_loop+0x67/0x71 > > > This seems to get printed 4 times - twice on CPU1, and CPU2This one appears to be relatively clear, and I''ll add a tentative fix to the next version of debugging patch that I''m in the process of preparing: irq_complete_move() is supposed to be getting called from the hw_irq_controllers'' .ack methods, yet VT-d currently uses the same handler for .ack and .disable (and calling irq_complete_move() in the context of .disable is certainly wrong) - this appears to have been the case forever, but a flaw like this may of course get uncovered with completely unrelated changes. Jan (also restored the Cc list)
>>> On 04.09.12 at 14:27, Ben Guthro <ben@guthro.net> wrote: > I''ve put the console log of this test run here: > https://citrix.sharefile.com/d/sdc383e252fb41c5a > > (again, so as not to clog everyone''s inbox) > > I have not yet gone through the log in its entirety yet, but thought I > would first send it to you to see if you had something in particular > you were looking for. > > The file name is console-S3-MSI.txtI think that nailed it: pci_restore_msi_state() passed a pointer to the stored entry->msg to write_msi_msg(), but with interrupt remapping enabled that function''s call to iommu_update_ire_from_msi() alters the passed in struct msi_msg instance. As the stored value is used for nothing but a subsequent (second) restore, a problem would only arise if between the two saves to further writing of the stored entry would occur (i.e. no intermediate call to set_msi_affinity()). Attached the advertised next version of the debugging patch (printks - slightly altered - left in to catch eventual further problems or to deal with my analysis being wrong; none of the "bogus!" ones should now trigger anymore). If this works, I''d be curious to see how much of your other workaround code you could then remove without breaking things again. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Fantastic! Initial tests are very good, with this dma_msi_ack modification. I''ve been able to get past the ahci stall, and run through ~10 suspend / resume cycles with this fix. Additional tests are warranted, and I''ll run through an automated sleep / wake script I have to make sure this fix holds over time. Thanks for this. Ben On Thu, Sep 6, 2012 at 6:22 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 04.09.12 at 14:27, Ben Guthro <ben@guthro.net> wrote: >> I''ve put the console log of this test run here: >> https://citrix.sharefile.com/d/sdc383e252fb41c5a >> >> (again, so as not to clog everyone''s inbox) >> >> I have not yet gone through the log in its entirety yet, but thought I >> would first send it to you to see if you had something in particular >> you were looking for. >> >> The file name is console-S3-MSI.txt > > I think that nailed it: pci_restore_msi_state() passed a pointer > to the stored entry->msg to write_msi_msg(), but with interrupt > remapping enabled that function''s call to > iommu_update_ire_from_msi() alters the passed in struct > msi_msg instance. As the stored value is used for nothing but > a subsequent (second) restore, a problem would only arise if > between the two saves to further writing of the stored entry > would occur (i.e. no intermediate call to set_msi_affinity()). > > Attached the advertised next version of the debugging patch > (printks - slightly altered - left in to catch eventual further > problems or to deal with my analysis being wrong; none of the > "bogus!" ones should now trigger anymore). If this works, I''d > be curious to see how much of your other workaround code > you could then remove without breaking things again. > > Jan >
Additionally - I was able to use this patch on its own, without any of the prior debug patch - which, as you pointed out before, ended up being unnecessary. On Thu, Sep 6, 2012 at 7:48 AM, Ben Guthro <ben@guthro.net> wrote:> Fantastic! > > Initial tests are very good, with this dma_msi_ack modification. > > I''ve been able to get past the ahci stall, and run through ~10 suspend > / resume cycles with this fix. > Additional tests are warranted, and I''ll run through an automated > sleep / wake script I have to make sure this fix holds over time. > > Thanks for this. > > Ben > > On Thu, Sep 6, 2012 at 6:22 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 04.09.12 at 14:27, Ben Guthro <ben@guthro.net> wrote: >>> I''ve put the console log of this test run here: >>> https://citrix.sharefile.com/d/sdc383e252fb41c5a >>> >>> (again, so as not to clog everyone''s inbox) >>> >>> I have not yet gone through the log in its entirety yet, but thought I >>> would first send it to you to see if you had something in particular >>> you were looking for. >>> >>> The file name is console-S3-MSI.txt >> >> I think that nailed it: pci_restore_msi_state() passed a pointer >> to the stored entry->msg to write_msi_msg(), but with interrupt >> remapping enabled that function''s call to >> iommu_update_ire_from_msi() alters the passed in struct >> msi_msg instance. As the stored value is used for nothing but >> a subsequent (second) restore, a problem would only arise if >> between the two saves to further writing of the stored entry >> would occur (i.e. no intermediate call to set_msi_affinity()). >> >> Attached the advertised next version of the debugging patch >> (printks - slightly altered - left in to catch eventual further >> problems or to deal with my analysis being wrong; none of the >> "bogus!" ones should now trigger anymore). If this works, I''d >> be curious to see how much of your other workaround code >> you could then remove without breaking things again. >> >> Jan >>
On Thu, Sep 06, 2012 at 07:48:26AM -0400, Ben Guthro wrote:> Fantastic! > > Initial tests are very good, with this dma_msi_ack modification. > > I''ve been able to get past the ahci stall, and run through ~10 suspend > / resume cycles with this fix. > Additional tests are warranted, and I''ll run through an automated > sleep / wake script I have to make sure this fix holds over time.How are you automating this? (/me imagines some robotic arm pressing the power button every couple of minutes).
We have a cycle_wakeup script (attached) that programs the RTC
This in itsself is not sufficient, due to interaction with the HPET.
We have an additional patch (see below) to deal with that.
The patch is not the cleanest solution ever, but it works...
diff -r fc9dd79fe91a xen/arch/x86/hpet.c
--- a/xen/arch/x86/hpet.c Thu Mar 03 17:21:16 2011 -0500
+++ b/xen/arch/x86/hpet.c Thu Mar 03 17:35:57 2011 -0500
@@ -520,13 +520,28 @@
if ( index != RTC_REG_B )
return;
-
+
+ /*
+ * Wake on RTC alarm does not conflict with hpet.
+ * The HW will wake up the CPU when the RTC alarm kicks off
+ * even thought the interrupt is not routed to the APIC.
+ *
+ * In our distro, xen will always own the interrupt so
+ * that we never disable deep cstates. We do not run
+ * dom0 apps/drivers that require RTC interrupts other
+ * than wakeup functionality.
+ */
+#if 0
/* RTC Reg B, contain PIE/AIE/UIE */
if ( value & (RTC_PIE | RTC_AIE | RTC_UIE ) )
{
cpuidle_disable_deep_cstate();
pv_rtc_handler = NULL;
}
+#else
+ if ( value & (RTC_PIE | RTC_UIE ) )
+ printk("WARNING: dom0 attempting to use RTC interrupts!\n");
+#endif
}
void hpet_broadcast_init(void)
On Thu, Sep 6, 2012 at 9:05 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Thu, Sep 06, 2012 at 07:48:26AM -0400, Ben Guthro wrote:
>> Fantastic!
>>
>> Initial tests are very good, with this dma_msi_ack modification.
>>
>> I''ve been able to get past the ahci stall, and run through ~10
suspend
>> / resume cycles with this fix.
>> Additional tests are warranted, and I''ll run through an
automated
>> sleep / wake script I have to make sure this fix holds over time.
>
> How are you automating this? (/me imagines some robotic arm pressing
> the power button every couple of minutes).
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Hmm - that script uses some stuff that is specific to our distro, but could be adapted to do do something more standard, like echo mem > /sys/power/state On Thu, Sep 6, 2012 at 9:27 AM, Ben Guthro <ben@guthro.net> wrote:> We have a cycle_wakeup script (attached) that programs the RTC > > This in itsself is not sufficient, due to interaction with the HPET. > We have an additional patch (see below) to deal with that. > > The patch is not the cleanest solution ever, but it works... > > > diff -r fc9dd79fe91a xen/arch/x86/hpet.c > --- a/xen/arch/x86/hpet.c Thu Mar 03 17:21:16 2011 -0500 > +++ b/xen/arch/x86/hpet.c Thu Mar 03 17:35:57 2011 -0500 > @@ -520,13 +520,28 @@ > > if ( index != RTC_REG_B ) > return; > - > + > + /* > + * Wake on RTC alarm does not conflict with hpet. > + * The HW will wake up the CPU when the RTC alarm kicks off > + * even thought the interrupt is not routed to the APIC. > + * > + * In our distro, xen will always own the interrupt so > + * that we never disable deep cstates. We do not run > + * dom0 apps/drivers that require RTC interrupts other > + * than wakeup functionality. > + */ > +#if 0 > /* RTC Reg B, contain PIE/AIE/UIE */ > if ( value & (RTC_PIE | RTC_AIE | RTC_UIE ) ) > { > cpuidle_disable_deep_cstate(); > pv_rtc_handler = NULL; > } > +#else > + if ( value & (RTC_PIE | RTC_UIE ) ) > + printk("WARNING: dom0 attempting to use RTC interrupts!\n"); > +#endif > } > > void hpet_broadcast_init(void) > > > On Thu, Sep 6, 2012 at 9:05 AM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> On Thu, Sep 06, 2012 at 07:48:26AM -0400, Ben Guthro wrote: >>> Fantastic! >>> >>> Initial tests are very good, with this dma_msi_ack modification. >>> >>> I''ve been able to get past the ahci stall, and run through ~10 suspend >>> / resume cycles with this fix. >>> Additional tests are warranted, and I''ll run through an automated >>> sleep / wake script I have to make sure this fix holds over time. >> >> How are you automating this? (/me imagines some robotic arm pressing >> the power button every couple of minutes).
Odd. The behavior seems to change, when not running with serial output. On this desktop system that worked OK, once I go back to not running the console output over serial - things fail on resume. It is uncertain at what point things stop working, though - as networking is unavailable. I''ll continue to look into it, but will be traveling next week. On Thu, Sep 6, 2012 at 7:48 AM, Ben Guthro <ben@guthro.net> wrote:> Fantastic! > > Initial tests are very good, with this dma_msi_ack modification. > > I''ve been able to get past the ahci stall, and run through ~10 suspend > / resume cycles with this fix. > Additional tests are warranted, and I''ll run through an automated > sleep / wake script I have to make sure this fix holds over time. > > Thanks for this. > > Ben > > On Thu, Sep 6, 2012 at 6:22 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 04.09.12 at 14:27, Ben Guthro <ben@guthro.net> wrote: >>> I''ve put the console log of this test run here: >>> https://citrix.sharefile.com/d/sdc383e252fb41c5a >>> >>> (again, so as not to clog everyone''s inbox) >>> >>> I have not yet gone through the log in its entirety yet, but thought I >>> would first send it to you to see if you had something in particular >>> you were looking for. >>> >>> The file name is console-S3-MSI.txt >> >> I think that nailed it: pci_restore_msi_state() passed a pointer >> to the stored entry->msg to write_msi_msg(), but with interrupt >> remapping enabled that function''s call to >> iommu_update_ire_from_msi() alters the passed in struct >> msi_msg instance. As the stored value is used for nothing but >> a subsequent (second) restore, a problem would only arise if >> between the two saves to further writing of the stored entry >> would occur (i.e. no intermediate call to set_msi_affinity()). >> >> Attached the advertised next version of the debugging patch >> (printks - slightly altered - left in to catch eventual further >> problems or to deal with my analysis being wrong; none of the >> "bogus!" ones should now trigger anymore). If this works, I''d >> be curious to see how much of your other workaround code >> you could then remove without breaking things again. >> >> Jan >>
>>> On 06.09.12 at 18:42, Ben Guthro <ben@guthro.net> wrote: > Odd. > The behavior seems to change, when not running with serial output.Odd indeed. Could you (unless you are already) try running the serial console in polling mode (i.e. without IRQ), to see whether the IRQs coming from it somehow keep the system alive at a certain point? Jan
On Fri, Sep 7, 2012 at 4:38 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 06.09.12 at 18:42, Ben Guthro <ben@guthro.net> wrote: > > Odd. > > The behavior seems to change, when not running with serial output. > > Odd indeed. Could you (unless you are already) try running the > serial console in polling mode (i.e. without IRQ), to see whether > the IRQs coming from it somehow keep the system alive at a > certain point? > >I tried to do this at the end of yesterday, since you had suggested it previously, in this thread. I did so by adding a ",0" to my com1 line, per the documentation - However, I am running with a PCI serial card, and not an "on-board" one - so my parameter looks like: com1=115200,8n1,pci,0 I am not totally convinced that it is actually running in polling mode, and started to investigate ns16550.c to verify it was. I plan on resuming this investigation this morning. If you have any pointers on what I should be looking for - I''d appreciate any suggestions. /btg _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 07.09.12 at 12:37, Ben Guthro <ben@guthro.net> wrote: > On Fri, Sep 7, 2012 at 4:38 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >>> On 06.09.12 at 18:42, Ben Guthro <ben@guthro.net> wrote: >> > Odd. >> > The behavior seems to change, when not running with serial output. >> >> Odd indeed. Could you (unless you are already) try running the >> serial console in polling mode (i.e. without IRQ), to see whether >> the IRQs coming from it somehow keep the system alive at a >> certain point? >> >> > I tried to do this at the end of yesterday, since you had suggested it > previously, in this thread. > > I did so by adding a ",0" to my com1 line, per the documentation - However, > I am running with a PCI serial card, and not an "on-board" one - so my > parameter looks like: > > com1=115200,8n1,pci,0 > > I am not totally convinced that it is actually running in polling mode, and > started to investigate ns16550.c to verify it was. I plan on resuming this > investigation this morning. > If you have any pointers on what I should be looking for - I''d appreciate > any suggestions. >The only thing you need to make sure is that at the end of ns16550_parse_port_config() uart->irq is zero. But for PCI that should be the default right now in -unstable (but I think I have a patch pending to alter this in certain cases). Jan
I have verified it is running with uart->irq == 0 when running on serial. However, when I run with console=none, the observed behavior is very different. The system seems to go to sleep successfully - but when I press the power button to wake it up - the power comes on - the fans spin up - but the system is unresponsive. No video No network keyboard LEDs (Caps,Numlock) do not light up. Alternate debugging strategies welcome. /btg On Fri, Sep 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 07.09.12 at 12:37, Ben Guthro <ben@guthro.net> wrote: >> On Fri, Sep 7, 2012 at 4:38 AM, Jan Beulich <JBeulich@suse.com> wrote: >> >>> >>> On 06.09.12 at 18:42, Ben Guthro <ben@guthro.net> wrote: >>> > Odd. >>> > The behavior seems to change, when not running with serial output. >>> >>> Odd indeed. Could you (unless you are already) try running the >>> serial console in polling mode (i.e. without IRQ), to see whether >>> the IRQs coming from it somehow keep the system alive at a >>> certain point? >>> >>> >> I tried to do this at the end of yesterday, since you had suggested it >> previously, in this thread. >> >> I did so by adding a ",0" to my com1 line, per the documentation - However, >> I am running with a PCI serial card, and not an "on-board" one - so my >> parameter looks like: >> >> com1=115200,8n1,pci,0 >> >> I am not totally convinced that it is actually running in polling mode, and >> started to investigate ns16550.c to verify it was. I plan on resuming this >> investigation this morning. >> If you have any pointers on what I should be looking for - I''d appreciate >> any suggestions. >> > > The only thing you need to make sure is that at the end of > ns16550_parse_port_config() uart->irq is zero. But for PCI > that should be the default right now in -unstable (but I > think I have a patch pending to alter this in certain cases). > > Jan >
>>> On 07.09.12 at 13:51, Ben Guthro <ben@guthro.net> wrote: > However, when I run with console=none, the observed behavior is very > different. > The system seems to go to sleep successfully - but when I press the > power button to wake it up - the power comes on - the fans spin up - > but the system is unresponsive. > No video > No network > keyboard LEDs (Caps,Numlock) do not light up. > > > Alternate debugging strategies welcome.I''m afraid other than being lucky to spot something via code inspection, the only alternative is an ITP/ICE. Maybe Intel folks could help out debugging this if it''s reproducible for them. Jan
I''ll work on getting a JTAG, ICE, or something else - it is on an Intel SDP - so it should have the ports for it. My current suspicion on this is that the hardware registers are not being programmed the same way as they were in 4.0.x (Since the "pulsing power button LED" on the laptops, and the behavior of the Desktop SDP are now similar) Once again - I don''t have a lot of evidence to back this up - however, if I ifdef out the register writes that actually start the low level suspend - in xen/arch/x86/acpi/power.c acpi_enter_sleep_state() - the rest of the suspend process completes as though the machine suspended, and then immediately resumed. In this case - the system seems to be functioning properly. Hack to prevent low level S3 attached. On Fri, Sep 7, 2012 at 8:18 AM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 07.09.12 at 13:51, Ben Guthro <ben@guthro.net> wrote: >> However, when I run with console=none, the observed behavior is very >> different. >> The system seems to go to sleep successfully - but when I press the >> power button to wake it up - the power comes on - the fans spin up - >> but the system is unresponsive. >> No video >> No network >> keyboard LEDs (Caps,Numlock) do not light up. >> >> >> Alternate debugging strategies welcome. > > I''m afraid other than being lucky to spot something via code > inspection, the only alternative is an ITP/ICE. Maybe Intel folks > could help out debugging this if it''s reproducible for them. > > Jan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
No hardware debugger just yet - but I''ve moved to another machine (Lenovo T400 laptop) - and am now seeing the following stack trace when I resume (this is using the tip of the 4.2-testing tree) It looks like either the vcpu, or the runstate is NULL, at this point in the resume process... (XEN) Finishing wakeup from ACPI S3 state. (XEN) Enabling non-boot CPUs ... (XEN) CPU#1 already initialized! (XEN) Stuck ?? (XEN) Error taking CPU1 up: -5 [ 38.570054] ACPI: Low-level resume complete [ 38.570054] PM: Restoring platform NVS memory [ 38.570054] Enabling non-boot CPUs ... (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: e008:[<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 (XEN) RFLAGS: 0000000000010006 CONTEXT: hypervisor (XEN) rax: 00007d3b7fd17180 rbx: ffff8300bd2fe000 rcx: 0000000000000000 (XEN) rdx: ffff08003fc8bd80 rsi: ffff82c48029fe28 rdi: ffff8300bd2fe000 (XEN) rbp: ffff82c48029fe28 rsp: ffff82c48029fdf8 r8: 0000000000000008 (XEN) r9: 00000000000001c0 r10: ffff82c48021f4a0 r11: 0000000000000282 (XEN) r12: ffff82c4802e8ee0 r13: ffff880039762da0 r14: ffff82c4802d3140 (XEN) r15: fffffffffffffff2 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000139ee4000 cr2: 0000000000000060 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff82c48029fdf8: (XEN) ffff8300bd2fe000 ffff82c48029ff18 ffff880037481d40 ffff880039762da0 (XEN) 0000000000000001 ffff82c480157df4 0000000000000070 ffff82f6016db300 (XEN) 00000000000b6d98 ffff8301355d8000 0000000000000070 ffff82c4801702ab (XEN) ffff88003fc8bd80 0000000000000000 0000000000000020 ffff8300bd2fe000 (XEN) ffff8301355d8000 ffff880037481d40 ffff880039762da0 0000000000000001 (XEN) 0000000000000003 ffff82c4801058df ffff82c48029ff18 ffff82c48011462e (XEN) 0000000000000000 0000000000000000 0000000400000004 ffff82c48029ff18 (XEN) 0000000000000010 ffff8300bd6a0000 ffff8800374819a8 ffff8300bd6a0000 (XEN) ffff880037481d48 0000000000000001 ffff880039762da0 ffff82c480214288 (XEN) 0000000000000003 0000000000000001 ffff880039762da0 0000000000000001 (XEN) ffff880037481d48 0000000000000001 0000000000000282 ffff880002dc4240 (XEN) 00000000000001c0 00000000000001c0 0000000000000018 ffffffff8100130a (XEN) ffff880037481d40 0000000000000001 0000000000000005 0000010000000000 (XEN) ffffffff8100130a 000000000000e033 0000000000000282 ffff880037481d20 (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffff8300bd6a0000 0000000000000000 (XEN) 0000000000000000 (XEN) Xen call trace: (XEN) [<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 (XEN) [<ffff82c480157df4>] arch_do_vcpu_op+0x134/0x5d0 (XEN) [<ffff82c4801702ab>] do_update_descriptor+0x1db/0x220 (XEN) [<ffff82c4801058df>] do_vcpu_op+0x6f/0x4a0 (XEN) [<ffff82c48011462e>] do_multicall+0x13e/0x330 (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d (XEN) (XEN) Pagetable walk from 0000000000000060: (XEN) L4[0x000] = 00000001004a5067 0000000000038c9d (XEN) L3[0x000] = 000000013a703067 0000000000003094 (XEN) L2[0x000] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: 0000000000000060 (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... On Fri, Sep 7, 2012 at 12:06 PM, Ben Guthro <ben@guthro.net> wrote:> I''ll work on getting a JTAG, ICE, or something else - it is on an > Intel SDP - so it should have the ports for it. > > My current suspicion on this is that the hardware registers are not > being programmed the same way as they were in 4.0.x > (Since the "pulsing power button LED" on the laptops, and the behavior > of the Desktop SDP are now similar) > > Once again - I don''t have a lot of evidence to back this up - however, > if I ifdef out the register writes that actually start the low level > suspend - in > xen/arch/x86/acpi/power.c acpi_enter_sleep_state() - the rest of the > suspend process completes as though the machine suspended, and then > immediately resumed. > > In this case - the system seems to be functioning properly. > > > > > > Hack to prevent low level S3 attached. > > > > On Fri, Sep 7, 2012 at 8:18 AM, Jan Beulich <JBeulich@suse.com> wrote: > >>>> On 07.09.12 at 13:51, Ben Guthro <ben@guthro.net> wrote: > >> However, when I run with console=none, the observed behavior is very > >> different. > >> The system seems to go to sleep successfully - but when I press the > >> power button to wake it up - the power comes on - the fans spin up - > >> but the system is unresponsive. > >> No video > >> No network > >> keyboard LEDs (Caps,Numlock) do not light up. > >> > >> > >> Alternate debugging strategies welcome. > > > > I''m afraid other than being lucky to spot something via code > > inspection, the only alternative is an ITP/ICE. Maybe Intel folks > > could help out debugging this if it''s reproducible for them. > > > > Jan > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
CPU#1 got stuck in loop in cpu_init() as it appears to be already initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and carries on, but the resume code assumes all CPUs are brought back online and crashes later. I wonder how long this has been broken. I recall reworking the CPU bringup code a lot early during 4.1.0 development... And I didn¹t test S3. -- Keir On 19/09/2012 22:07, "Ben Guthro" <ben@guthro.net> wrote:> No hardware debugger just yet - but I''ve moved to another machine (Lenovo T400 > laptop) - and am now seeing the following stack trace when I resume > (this is using the tip of the 4.2-testing tree) > > It looks like either the vcpu, or the runstate is NULL, at this point in the > resume process... > > > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) CPU#1 already initialized! > (XEN) Stuck ?? > (XEN) Error taking CPU1 up: -5 > [ 38.570054] ACPI: Low-level resume complete > [ 38.570054] PM: Restoring platform NVS memory > [ 38.570054] Enabling non-boot CPUs ... > (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- > (XEN) CPU: 0 > (XEN) RIP: e008:[<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 > (XEN) RFLAGS: 0000000000010006 CONTEXT: hypervisor > (XEN) rax: 00007d3b7fd17180 rbx: ffff8300bd2fe000 rcx: 0000000000000000 > (XEN) rdx: ffff08003fc8bd80 rsi: ffff82c48029fe28 rdi: ffff8300bd2fe000 > (XEN) rbp: ffff82c48029fe28 rsp: ffff82c48029fdf8 r8: 0000000000000008 > (XEN) r9: 00000000000001c0 r10: ffff82c48021f4a0 r11: 0000000000000282 > (XEN) r12: ffff82c4802e8ee0 r13: ffff880039762da0 r14: ffff82c4802d3140 > (XEN) r15: fffffffffffffff2 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 0000000139ee4000 cr2: 0000000000000060 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff82c48029fdf8: > (XEN) ffff8300bd2fe000 ffff82c48029ff18 ffff880037481d40 ffff880039762da0 > (XEN) 0000000000000001 ffff82c480157df4 0000000000000070 ffff82f6016db300 > (XEN) 00000000000b6d98 ffff8301355d8000 0000000000000070 ffff82c4801702ab > (XEN) ffff88003fc8bd80 0000000000000000 0000000000000020 ffff8300bd2fe000 > (XEN) ffff8301355d8000 ffff880037481d40 ffff880039762da0 0000000000000001 > (XEN) 0000000000000003 ffff82c4801058df ffff82c48029ff18 ffff82c48011462e > (XEN) 0000000000000000 0000000000000000 0000000400000004 ffff82c48029ff18 > (XEN) 0000000000000010 ffff8300bd6a0000 ffff8800374819a8 ffff8300bd6a0000 > (XEN) ffff880037481d48 0000000000000001 ffff880039762da0 ffff82c480214288 > (XEN) 0000000000000003 0000000000000001 ffff880039762da0 0000000000000001 > (XEN) ffff880037481d48 0000000000000001 0000000000000282 ffff880002dc4240 > (XEN) 00000000000001c0 00000000000001c0 0000000000000018 ffffffff8100130a > (XEN) ffff880037481d40 0000000000000001 0000000000000005 0000010000000000 > (XEN) ffffffff8100130a 000000000000e033 0000000000000282 ffff880037481d20 > (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffff8300bd6a0000 0000000000000000 > (XEN) 0000000000000000 > (XEN) Xen call trace: > (XEN) [<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 > (XEN) [<ffff82c480157df4>] arch_do_vcpu_op+0x134/0x5d0 > (XEN) [<ffff82c4801702ab>] do_update_descriptor+0x1db/0x220 > (XEN) [<ffff82c4801058df>] do_vcpu_op+0x6f/0x4a0 > (XEN) [<ffff82c48011462e>] do_multicall+0x13e/0x330 > (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d > (XEN) > (XEN) Pagetable walk from 0000000000000060: > (XEN) L4[0x000] = 00000001004a5067 0000000000038c9d > (XEN) L3[0x000] = 000000013a703067 0000000000003094 > (XEN) L2[0x000] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: 0000000000000060 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > > On Fri, Sep 7, 2012 at 12:06 PM, Ben Guthro <ben@guthro.net> wrote: >> I''ll work on getting a JTAG, ICE, or something else - it is on an >> Intel SDP - so it should have the ports for it. >> >> My current suspicion on this is that the hardware registers are not >> being programmed the same way as they were in 4.0.x >> (Since the "pulsing power button LED" on the laptops, and the behavior >> of the Desktop SDP are now similar) >> >> Once again - I don''t have a lot of evidence to back this up - however, >> if I ifdef out the register writes that actually start the low level >> suspend - in >> xen/arch/x86/acpi/power.c acpi_enter_sleep_state() - the rest of the >> suspend process completes as though the machine suspended, and then >> immediately resumed. >> >> In this case - the system seems to be functioning properly. >> >> >> >> >> >> Hack to prevent low level S3 attached. >> >> >> >> On Fri, Sep 7, 2012 at 8:18 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>> >>>> On 07.09.12 at 13:51, Ben Guthro <ben@guthro.net> wrote: >>>> >> However, when I run with console=none, the observed behavior is very >>>> >> different. >>>> >> The system seems to go to sleep successfully - but when I press the >>>> >> power button to wake it up - the power comes on - the fans spin up - >>>> >> but the system is unresponsive. >>>> >> No video >>>> >> No network >>>> >> keyboard LEDs (Caps,Numlock) do not light up. >>>> >> >>>> >> >>>> >> Alternate debugging strategies welcome. >>> > >>> > I''m afraid other than being lucky to spot something via code >>> > inspection, the only alternative is an ITP/ICE. Maybe Intel folks >>> > could help out debugging this if it''s reproducible for them. >>> > >>> > Jan >>> > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 20/09/2012 07:13, "Keir Fraser" <keir.xen@gmail.com> wrote:> CPU#1 got stuck in loop in cpu_init() as it appears to be already > initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and carries > on, but the resume code assumes all CPUs are brought back online and crashes > later. > > I wonder how long this has been broken. I recall reworking the CPU bringup > code a lot early during 4.1.0 development... And I didn¹t test S3. > > -- KeirHowever, I did test CPU hotplug a lot, and S3 uses the hotplug logic to take down and bring up CPUs. So I don''t think I can have broken this. Are you able to hotplug physical CPUs from dom0 using the tools/misc/xen-hptool utility? If not, at least this might be a friendlier test method and environment than a full S3. -- Keir> On 19/09/2012 22:07, "Ben Guthro" <ben@guthro.net> wrote: > >> No hardware debugger just yet - but I''ve moved to another machine (Lenovo >> T400 laptop) - and am now seeing the following stack trace when I resume >> (this is using the tip of the 4.2-testing tree) >> >> It looks like either the vcpu, or the runstate is NULL, at this point in the >> resume process... >> >> >> (XEN) Finishing wakeup from ACPI S3 state. >> (XEN) Enabling non-boot CPUs ... >> (XEN) CPU#1 already initialized! >> (XEN) Stuck ?? >> (XEN) Error taking CPU1 up: -5 >> [ 38.570054] ACPI: Low-level resume complete >> [ 38.570054] PM: Restoring platform NVS memory >> [ 38.570054] Enabling non-boot CPUs ... >> (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- >> (XEN) CPU: 0 >> (XEN) RIP: e008:[<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 >> (XEN) RFLAGS: 0000000000010006 CONTEXT: hypervisor >> (XEN) rax: 00007d3b7fd17180 rbx: ffff8300bd2fe000 rcx: 0000000000000000 >> (XEN) rdx: ffff08003fc8bd80 rsi: ffff82c48029fe28 rdi: ffff8300bd2fe000 >> (XEN) rbp: ffff82c48029fe28 rsp: ffff82c48029fdf8 r8: 0000000000000008 >> (XEN) r9: 00000000000001c0 r10: ffff82c48021f4a0 r11: 0000000000000282 >> (XEN) r12: ffff82c4802e8ee0 r13: ffff880039762da0 r14: ffff82c4802d3140 >> (XEN) r15: fffffffffffffff2 cr0: 000000008005003b cr4: 00000000000026f0 >> (XEN) cr3: 0000000139ee4000 cr2: 0000000000000060 >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 >> (XEN) Xen stack trace from rsp=ffff82c48029fdf8: >> (XEN) ffff8300bd2fe000 ffff82c48029ff18 ffff880037481d40 ffff880039762da0 >> (XEN) 0000000000000001 ffff82c480157df4 0000000000000070 ffff82f6016db300 >> (XEN) 00000000000b6d98 ffff8301355d8000 0000000000000070 ffff82c4801702ab >> (XEN) ffff88003fc8bd80 0000000000000000 0000000000000020 ffff8300bd2fe000 >> (XEN) ffff8301355d8000 ffff880037481d40 ffff880039762da0 0000000000000001 >> (XEN) 0000000000000003 ffff82c4801058df ffff82c48029ff18 ffff82c48011462e >> (XEN) 0000000000000000 0000000000000000 0000000400000004 ffff82c48029ff18 >> (XEN) 0000000000000010 ffff8300bd6a0000 ffff8800374819a8 ffff8300bd6a0000 >> (XEN) ffff880037481d48 0000000000000001 ffff880039762da0 ffff82c480214288 >> (XEN) 0000000000000003 0000000000000001 ffff880039762da0 0000000000000001 >> (XEN) ffff880037481d48 0000000000000001 0000000000000282 ffff880002dc4240 >> (XEN) 00000000000001c0 00000000000001c0 0000000000000018 ffffffff8100130a >> (XEN) ffff880037481d40 0000000000000001 0000000000000005 0000010000000000 >> (XEN) ffffffff8100130a 000000000000e033 0000000000000282 ffff880037481d20 >> (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 ffff8300bd6a0000 0000000000000000 >> (XEN) 0000000000000000 >> (XEN) Xen call trace: >> (XEN) [<ffff82c480120585>] vcpu_runstate_get+0xe5/0x130 >> (XEN) [<ffff82c480157df4>] arch_do_vcpu_op+0x134/0x5d0 >> (XEN) [<ffff82c4801702ab>] do_update_descriptor+0x1db/0x220 >> (XEN) [<ffff82c4801058df>] do_vcpu_op+0x6f/0x4a0 >> (XEN) [<ffff82c48011462e>] do_multicall+0x13e/0x330 >> (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d >> (XEN) >> (XEN) Pagetable walk from 0000000000000060: >> (XEN) L4[0x000] = 00000001004a5067 0000000000038c9d >> (XEN) L3[0x000] = 000000013a703067 0000000000003094 >> (XEN) L2[0x000] = 0000000000000000 ffffffffffffffff >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) FATAL PAGE FAULT >> (XEN) [error_code=0000] >> (XEN) Faulting linear address: 0000000000000060 >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> >> On Fri, Sep 7, 2012 at 12:06 PM, Ben Guthro <ben@guthro.net> wrote: >>> I''ll work on getting a JTAG, ICE, or something else - it is on an >>> Intel SDP - so it should have the ports for it. >>> >>> My current suspicion on this is that the hardware registers are not >>> being programmed the same way as they were in 4.0.x >>> (Since the "pulsing power button LED" on the laptops, and the behavior >>> of the Desktop SDP are now similar) >>> >>> Once again - I don''t have a lot of evidence to back this up - however, >>> if I ifdef out the register writes that actually start the low level >>> suspend - in >>> xen/arch/x86/acpi/power.c acpi_enter_sleep_state() - the rest of the >>> suspend process completes as though the machine suspended, and then >>> immediately resumed. >>> >>> In this case - the system seems to be functioning properly. >>> >>> >>> >>> >>> >>> Hack to prevent low level S3 attached. >>> >>> >>> >>> On Fri, Sep 7, 2012 at 8:18 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>> On 07.09.12 at 13:51, Ben Guthro <ben@guthro.net> wrote: >>>>> However, when I run with console=none, the observed behavior is very >>>>> different. >>>>> The system seems to go to sleep successfully - but when I press the >>>>> power button to wake it up - the power comes on - the fans spin up - >>>>> but the system is unresponsive. >>>>> No video >>>>> No network >>>>> keyboard LEDs (Caps,Numlock) do not light up. >>>>> >>>>> >>>>> Alternate debugging strategies welcome. >>>> >>>> I''m afraid other than being lucky to spot something via code >>>> inspection, the only alternative is an ITP/ICE. Maybe Intel folks >>>> could help out debugging this if it''s reproducible for them. >>>> >>>> Jan >>>> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >
>>> On 19.09.12 at 23:07, Ben Guthro <ben@guthro.net> wrote: > No hardware debugger just yet - but I''ve moved to another machine (Lenovo > T400 laptop) - and am now seeing the following stack trace when I resume > (this is using the tip of the 4.2-testing tree)Considering that the situation is similar to the one with very early boot problems - as long as the system either reboots on its own (albeit I don''t think you ever said it does) or has a reset button (or other way to initiate reset without turning off power), an alternative debugging method would be to write data into I/O ports the values of which persist across reset, and read them out during the following boot. I have found the 0x008x range to be a candidate for this, as well as certain DMA controller ones. And of course, if the CMOS RAM has its upper 128 bytes present and (part of it) usable (i.e. otherwise unused), that would even be an option permitting power cycling the system without losing the information. Jan
>>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: > CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready > initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and > carries on, but the resume code assumes all CPUs are brought back online and > crashes later.So this would suggest play_dead() (-> cpu_exit_clear() -> cpu_uninit()) not getting reached during the suspend cycle. That should be fairly easy to verify, as the serial console ought to still work when the secondary CPUs get offlined. That might imply cpumask_clear_cpu(cpu, &cpu_online_map) not getting reached in __cpu_disable(), which would be in line with the observation that none of the logs provided so far showed anything being done by fixup_irqs() (called right after clearing the online bit). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 20/09/2012 09:03, "Jan Beulich" <JBeulich@suse.com> wrote:>>>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: >> CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready >> initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and >> carries on, but the resume code assumes all CPUs are brought back online and >> crashes later. > > So this would suggest play_dead() (-> cpu_exit_clear() -> > cpu_uninit()) not getting reached during the suspend cycle. > That should be fairly easy to verify, as the serial console > ought to still work when the secondary CPUs get offlined.Yes.> That might imply cpumask_clear_cpu(cpu, &cpu_online_map) > not getting reached in __cpu_disable(), which would be in line > with the observation that none of the logs provided so far > showed anything being done by fixup_irqs() (called right > after clearing the online bit).I did just test cpu offline/online via xen-hptool myself, and that does work. So perhaps this is platform specific, or S3 specific... -- Keir> Jan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
It appears __cpu_disable() is not getting reached at all, for CPU1 I put a cpu id conditional BUG() call in there, to verify - and while it is reached when using xen-hptool cpu-offline 1 It never seems to be reached from the S3 path. What is the expected call chain to get into this code during S3? On Thu, Sep 20, 2012 at 4:03 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: > > CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready > > initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and > > carries on, but the resume code assumes all CPUs are brought back online > and > > crashes later. > > So this would suggest play_dead() (-> cpu_exit_clear() -> > cpu_uninit()) not getting reached during the suspend cycle. > That should be fairly easy to verify, as the serial console > ought to still work when the secondary CPUs get offlined. > > That might imply cpumask_clear_cpu(cpu, &cpu_online_map) > not getting reached in __cpu_disable(), which would be in line > with the observation that none of the logs provided so far > showed anything being done by fixup_irqs() (called right > after clearing the online bit). > > Jan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
disable_nonboot_cpus() -> cpu_down(1) -> ... ...and from there it is same as the xen-hptool case: arch_do_sysctl() -> cpu_down_helper() -> cpu_down(1) -> stop_machine_run(take_cpu_down, 1) -> [on CPU#1] take_cpu_down() -> __cpu_disable() On 20/09/2012 13:56, "Ben Guthro" <ben@guthro.net> wrote:> It appears __cpu_disable() is not getting reached at all, for CPU1 > > I put a cpu id conditional BUG() call in there, to verify - and while it is > reached when using > xen-hptool cpu-offline 1 > It never seems to be reached from the S3 path. > > > What is the expected call chain to get into this code during S3? > > > On Thu, Sep 20, 2012 at 4:03 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> >>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: >>> > CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready >>> > initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and >>> > carries on, but the resume code assumes all CPUs are brought back online >>> and >>> > crashes later. >> >> So this would suggest play_dead() (-> cpu_exit_clear() -> >> cpu_uninit()) not getting reached during the suspend cycle. >> That should be fairly easy to verify, as the serial console >> ought to still work when the secondary CPUs get offlined. >> >> That might imply cpumask_clear_cpu(cpu, &cpu_online_map) >> not getting reached in __cpu_disable(), which would be in line >> with the observation that none of the logs provided so far >> showed anything being done by fixup_irqs() (called right >> after clearing the online bit). >> >> Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, Sep 20, 2012 at 8:56 AM, Ben Guthro <ben@guthro.net> wrote:> It appears __cpu_disable() is not getting reached at all, for CPU1 > >I was incorrect about this, after messing around with various serial configs to properly get all output. I have traced this through to verify that the sequence in question, does, in fact seem to be getting executed. disable_nonboot_cpus() cpu_down() __cpu_disable() play_dead() cpu_exit_clear() cpu_uninit() __cpu_die() do_suspend_lowlevel() I also enabled the printk''s in smpboot.c [ 32.145824] ACPI: Preparing to enter system sleep state S3 [ 32.600118] PM: Saving platform NVS memory [ 32.671666] Disabling non-boot CPUs ... (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Bringing CPU1 down (XEN) Disabling CPU1 (XEN) Disabled CPU1 (XEN) play_dead: CPU1 (XEN) cpu_exit_clear: CPU1 (XEN) cpu_uninit: CPU1 (XEN) CPU1 dead (XEN) Entering ACPI S3 state. (XEN) Finishing wakeup from ACPI S3 state. (XEN) Enabling non-boot CPUs ... (XEN) Bringing CPU1 up (XEN) Setting warm reset code and vector. (XEN) Asserting INIT. (XEN) Waiting for send to finish... (XEN) +Deasserting INIT. (XEN) Waiting for send to finish... (XEN) +#startup loops: 2. (XEN) Sending STARTUP #1. (XEN) After apic_write. (XEN) CPU#1 already initialized! (XEN) Startup point 1. (XEN) Waiting for send to finish... (XEN) +Sending STARTUP #2. (XEN) After apic_write. (XEN) Startup point 1. (XEN) Waiting for send to finish... (XEN) +After Startup. (XEN) After Callout 1. (XEN) Stuck ?? (XEN) cpu_exit_clear: CPU1 (XEN) cpu_uninit: CPU1 (XEN) __cpu_up - do_boot_cpu error (XEN) cpu_up CPU1 CPU not up (XEN) cpu_up CPU1 fail (XEN) Error taking CPU1 up: -5 [ 32.780055] ACPI: Low-level resume complete [ 32.780055] PM: Restoring platform NVS memory [ 32.780055] Enabling non-boot CPUs ... then it crashes. It seems that it is always falling through into the "else" clause of the do_boot_cpu() function when attempting to bring it back up, seemingly stuck in CPU_STATE_CALLOUT Any ideas as to what might be causing it to get stuck in that state?> I put a cpu id conditional BUG() call in there, to verify - and while it > is reached when using > xen-hptool cpu-offline 1 > It never seems to be reached from the S3 path. > > > What is the expected call chain to get into this code during S3? > > > On Thu, Sep 20, 2012 at 4:03 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: >> > CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready >> > initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and >> > carries on, but the resume code assumes all CPUs are brought back >> online and >> > crashes later. >> >> So this would suggest play_dead() (-> cpu_exit_clear() -> >> cpu_uninit()) not getting reached during the suspend cycle. >> That should be fairly easy to verify, as the serial console >> ought to still work when the secondary CPUs get offlined. >> >> That might imply cpumask_clear_cpu(cpu, &cpu_online_map) >> not getting reached in __cpu_disable(), which would be in line >> with the observation that none of the logs provided so far >> showed anything being done by fixup_irqs() (called right >> after clearing the online bit). >> >> Jan >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 20/09/2012 21:30, "Ben Guthro" <ben@guthro.net> wrote:> (XEN) Bringing CPU1 down > (XEN) Disabling CPU1 > (XEN) Disabled CPU1 > (XEN) play_dead: CPU1 > (XEN) cpu_exit_clear: CPU1 > (XEN) cpu_uninit: CPU1 > (XEN) CPU1 deadSo CPU1 is taken down properly, apparently...> (XEN) Entering ACPI S3 state.... During S3 suspend.> (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Bringing CPU1 up > (XEN) Setting warm reset code and vector. > (XEN) Asserting INIT. > (XEN) Waiting for send to finish... > (XEN) +Deasserting INIT. > (XEN) Waiting for send to finish... > (XEN) +#startup loops: 2. > (XEN) Sending STARTUP #1. > (XEN) After apic_write. > (XEN) CPU#1 already initialized!But here CPU1 thinks it is already initialised! *This* is the bug you need to go look at. CPU1 will spin at this point...> (XEN) Startup point 1. > (XEN) Waiting for send to finish... > (XEN) +Sending STARTUP #2. > (XEN) After apic_write. > (XEN) Startup point 1. > (XEN) Waiting for send to finish... > (XEN) +After Startup. > (XEN) After Callout 1. > (XEN) Stuck ??...Causing CPU0 to think CPU1 is stuck (which is fair, because it is).> (XEN) cpu_exit_clear: CPU1 > (XEN) cpu_uninit: CPU1 > (XEN) __cpu_up - do_boot_cpu error > (XEN) cpu_up CPU1 CPU not up > (XEN) cpu_up CPU1 fail > (XEN) Error taking CPU1 up: -5 > [ 32.780055] ACPI: Low-level resume complete > [ 32.780055] PM: Restoring platform NVS memory > [ 32.780055] Enabling non-boot CPUs ... > > then it crashes. > > It seems that it is always falling through into the "else" clause of > the do_boot_cpu() function when attempting to bring it back up, seemingly > stuck in CPU_STATE_CALLOUT > > Any ideas as to what might be causing it to get stuck in that state?Yes, see explanation above, which is actually the same explanation I gave you before. You need to go investigate why CPU1 is getting confused in cpu_init(). -- Keir> > > > >> I put a cpu id conditional BUG() call in there, to verify - and while it is >> reached when using >> xen-hptool cpu-offline 1 >> It never seems to be reached from the S3 path. >> >> >> What is the expected call chain to get into this code during S3? >> >> >> On Thu, Sep 20, 2012 at 4:03 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>> On 20.09.12 at 08:13, Keir Fraser <keir.xen@gmail.com> wrote: >>>> CPU#1 got stuck in loop in cpu_init() as it appears to be Œalready >>>> initialised¹ in cpu_initialized bitmap. CPU#0 detects it is stuck and >>>> carries on, but the resume code assumes all CPUs are brought back online >>>> and >>>> crashes later. >>> >>> So this would suggest play_dead() (-> cpu_exit_clear() -> >>> cpu_uninit()) not getting reached during the suspend cycle. >>> That should be fairly easy to verify, as the serial console >>> ought to still work when the secondary CPUs get offlined. >>> >>> That might imply cpumask_clear_cpu(cpu, &cpu_online_map) >>> not getting reached in __cpu_disable(), which would be in line >>> with the observation that none of the logs provided so far >>> showed anything being done by fixup_irqs() (called right >>> after clearing the online bit). >>> >>> Jan >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 20.09.12 at 22:30, Ben Guthro <ben@guthro.net> wrote: > On Thu, Sep 20, 2012 at 8:56 AM, Ben Guthro <ben@guthro.net> wrote: > [ 32.145824] ACPI: Preparing to enter system sleep state S3 > [ 32.600118] PM: Saving platform NVS memory > [ 32.671666] Disabling non-boot CPUs ... > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Bringing CPU1 down > (XEN) Disabling CPU1 > (XEN) Disabled CPU1 > (XEN) play_dead: CPU1 > (XEN) cpu_exit_clear: CPU1 > (XEN) cpu_uninit: CPU1 > (XEN) CPU1 deadSo how about inserting printout of cpu_initialized here ...> (XEN) Entering ACPI S3 state. > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ...... and here? Plus adding "cpuinfo" to the Xen command line, which would allow us to see whether CPU1 gets into cpu_init() twice. Perhaps tracking cpu_state throughout the below sequence might also be useful.> (XEN) Bringing CPU1 up > (XEN) Setting warm reset code and vector. > (XEN) Asserting INIT. > (XEN) Waiting for send to finish... > (XEN) +Deasserting INIT. > (XEN) Waiting for send to finish... > (XEN) +#startup loops: 2. > (XEN) Sending STARTUP #1. > (XEN) After apic_write. > (XEN) CPU#1 already initialized! > (XEN) Startup point 1. > (XEN) Waiting for send to finish... > (XEN) +Sending STARTUP #2. > (XEN) After apic_write. > (XEN) Startup point 1. > (XEN) Waiting for send to finish... > (XEN) +After Startup. > (XEN) After Callout 1. > (XEN) Stuck ?? > (XEN) cpu_exit_clear: CPU1 > (XEN) cpu_uninit: CPU1 > (XEN) __cpu_up - do_boot_cpu error > (XEN) cpu_up CPU1 CPU not up > (XEN) cpu_up CPU1 fail > (XEN) Error taking CPU1 up: -5 > [ 32.780055] ACPI: Low-level resume complete > [ 32.780055] PM: Restoring platform NVS memory > [ 32.780055] Enabling non-boot CPUs ... > > then it crashes. > > It seems that it is always falling through into the "else" clause of > the do_boot_cpu() function when attempting to bring it back up, seemingly > stuck in CPU_STATE_CALLOUT > > > Any ideas as to what might be causing it to get stuck in that state?That''s because CPU1 is stuck in cpu_init() (in the infinite loop after printing "CPU#1 already initialized!"), as Keir pointed out yesterday. Jan
On Fri, Sep 21, 2012 at 2:47 AM, Jan Beulich <JBeulich@suse.com> wrote:> > That''s because CPU1 is stuck in cpu_init() (in the infinite loop after > printing "CPU#1 already initialized!"), as Keir pointed out yesterday. > >I''ve done some more tracing on this, and instrumented cpu_init(), cpu_uninit() - and found something I cannot quite explain. I was most interested in the cpu_initialized mask, set just above these two functions (and only used in those two functions) I convert cpu_initialized to a string, using cpumask_scnprintf - and print it out when it is read, or written in these two functions. When CPU1 is being torn down, the cpumask bit gets cleared for CPU1, and I am able to print this to the console to verify. However, when the machine is returning from S3, and going through cpu_init - the bit is set again. Could this be an issue of caches not being flushed? I see that the last thing done before acpi_enter_sleep_state actually writes PM1A_CONTROL / PM1B_CONTROL to enter S3 is a ACPI_FLUSH_CPU_CACHE() This analysis seems unlikely, at this point...but I''m not sure what to make of the data other than a cache issue. Am I "barking up the wrong tree" here? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 21/09/2012 19:20, "Ben Guthro" <ben@guthro.net> wrote:> > > On Fri, Sep 21, 2012 at 2:47 AM, Jan Beulich <JBeulich@suse.com> wrote: >> >> That''s because CPU1 is stuck in cpu_init() (in the infinite loop after >> printing "CPU#1 already initialized!"), as Keir pointed out yesterday. >> > > I''ve done some more tracing on this, and instrumented cpu_init(), cpu_uninit() > - and found something I cannot quite explain. > I was most interested in the cpu_initialized mask, set just above these two > functions (and only used in those two functions) > > I convert cpu_initialized to a string, using cpumask_scnprintf - and print it > out when it is read, or written in these two functions. > > When CPU1 is being torn down, the cpumask bit gets cleared for CPU1, and I am > able to print this to the console to verify. > However, when the machine is returning from S3, and going through cpu_init - > the bit is set again. > > Could this be an issue of caches not being flushed? > > I see that the last thing done before acpi_enter_sleep_state actually > writes PM1A_CONTROL / PM1B_CONTROL to enter S3 is a ACPI_FLUSH_CPU_CACHE() > > This analysis seems unlikely, at this point...but I''m not sure what to make of > the data other than a cache issue. > > Am I "barking up the wrong tree" here?Perhaps not. Try dumping it immediately before and after the actual S3 sleep. Since you probably can''t print to serial line at that point, you could just take a copy of the bitmap and print them both shortly after S3 resume. Then if it still looks bad, or the problem magically resolves with the extra printing, you can suspect cache flush a bit more strongly. However, WBINVD (which is what ACPI_FLUSH_CPU_CACHE() is) should be enough. -- Keir
>>> On 21.09.12 at 20:42, Keir Fraser <keir@xen.org> wrote: > On 21/09/2012 19:20, "Ben Guthro" <ben@guthro.net> wrote: > >> >> >> On Fri, Sep 21, 2012 at 2:47 AM, Jan Beulich <JBeulich@suse.com> wrote: >>> >>> That''s because CPU1 is stuck in cpu_init() (in the infinite loop after >>> printing "CPU#1 already initialized!"), as Keir pointed out yesterday. >>> >> >> I''ve done some more tracing on this, and instrumented cpu_init(), > cpu_uninit() >> - and found something I cannot quite explain. >> I was most interested in the cpu_initialized mask, set just above these two >> functions (and only used in those two functions) >> >> I convert cpu_initialized to a string, using cpumask_scnprintf - and print > it >> out when it is read, or written in these two functions. >> >> When CPU1 is being torn down, the cpumask bit gets cleared for CPU1, and I > am >> able to print this to the console to verify. >> However, when the machine is returning from S3, and going through cpu_init - >> the bit is set again. >> >> Could this be an issue of caches not being flushed? >> >> I see that the last thing done before acpi_enter_sleep_state actually >> writes PM1A_CONTROL / PM1B_CONTROL to enter S3 is a ACPI_FLUSH_CPU_CACHE() >> >> This analysis seems unlikely, at this point...but I''m not sure what to make > of >> the data other than a cache issue. >> >> Am I "barking up the wrong tree" here? > > Perhaps not. Try dumping it immediately before and after the actual S3 > sleep. Since you probably can''t print to serial line at that point, you > could just take a copy of the bitmap and print them both shortly after S3 > resume. Then if it still looks bad, or the problem magically resolves with > the extra printing, you can suspect cache flush a bit more strongly. > However, WBINVD (which is what ACPI_FLUSH_CPU_CACHE() is) should be enough.CPU0 issuing WBINVD might not be enough; other CPUs should probably also do so unconditionally (currently they do this only when using one of the advanced halt forms in acpi_dead_idle()). While one would think that a halted CPU would not only continue to keep its cache up-to-date, but also eventually write back its dirty cache lines, I don''t think the latter is actually guaranteed, so if the CPU ends up getting the INIT before the line was written back, the modification could get lost. But of course this theory depends on Ben''s system actually using the default halt mechanism rather than one of the advanced ones. Jan
It is an older system (Core2) - so it may be using the default mechanism, but I''d have to go dig up some processor docs to be sure. I''m still seeing some behavior in my experiments that I can''t entirely explain (yet) I''ll be spending some time today looking into it more closely. On Mon, Sep 24, 2012 at 7:22 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 21.09.12 at 20:42, Keir Fraser <keir@xen.org> wrote: > > On 21/09/2012 19:20, "Ben Guthro" <ben@guthro.net> wrote: > > > >> > >> > >> On Fri, Sep 21, 2012 at 2:47 AM, Jan Beulich <JBeulich@suse.com> wrote: > >>> > >>> That''s because CPU1 is stuck in cpu_init() (in the infinite loop after > >>> printing "CPU#1 already initialized!"), as Keir pointed out yesterday. > >>> > >> > >> I''ve done some more tracing on this, and instrumented cpu_init(), > > cpu_uninit() > >> - and found something I cannot quite explain. > >> I was most interested in the cpu_initialized mask, set just above these > two > >> functions (and only used in those two functions) > >> > >> I convert cpu_initialized to a string, using cpumask_scnprintf - and > print > > it > >> out when it is read, or written in these two functions. > >> > >> When CPU1 is being torn down, the cpumask bit gets cleared for CPU1, > and I > > am > >> able to print this to the console to verify. > >> However, when the machine is returning from S3, and going through > cpu_init - > >> the bit is set again. > >> > >> Could this be an issue of caches not being flushed? > >> > >> I see that the last thing done before acpi_enter_sleep_state actually > >> writes PM1A_CONTROL / PM1B_CONTROL to enter S3 is a > ACPI_FLUSH_CPU_CACHE() > >> > >> This analysis seems unlikely, at this point...but I''m not sure what to > make > > of > >> the data other than a cache issue. > >> > >> Am I "barking up the wrong tree" here? > > > > Perhaps not. Try dumping it immediately before and after the actual S3 > > sleep. Since you probably can''t print to serial line at that point, you > > could just take a copy of the bitmap and print them both shortly after S3 > > resume. Then if it still looks bad, or the problem magically resolves > with > > the extra printing, you can suspect cache flush a bit more strongly. > > However, WBINVD (which is what ACPI_FLUSH_CPU_CACHE() is) should be > enough. > > CPU0 issuing WBINVD might not be enough; other CPUs should > probably also do so unconditionally (currently they do this only > when using one of the advanced halt forms in acpi_dead_idle()). > > While one would think that a halted CPU would not only continue > to keep its cache up-to-date, but also eventually write back its > dirty cache lines, I don''t think the latter is actually guaranteed, > so if the CPU ends up getting the INIT before the line was written > back, the modification could get lost. > > But of course this theory depends on Ben''s system actually using > the default halt mechanism rather than one of the advanced ones. > > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 24.09.12 at 13:25, Ben Guthro <ben@guthro.net> wrote: > It is an older system (Core2) - so it may be using the default mechanism, > but I''d have to go dig up some processor docs to be sure.This is not just a matter of what the CPU supports, but also what info gets passed down from Dom0 - if e.g. your kernel doesn''t have Konrad''s P-/C-state patches, then there''s no way for Xen to use any of the advanced methods. Jan
I see - Konrad - do you have a changeset I can look for with the features Jan describes? On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 24.09.12 at 13:25, Ben Guthro <ben@guthro.net> wrote: > > It is an older system (Core2) - so it may be using the default mechanism, > > but I''d have to go dig up some processor docs to be sure. > > This is not just a matter of what the CPU supports, but also what > info gets passed down from Dom0 - if e.g. your kernel doesn''t > have Konrad''s P-/C-state patches, then there''s no way for Xen to > use any of the advanced methods. > > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 24.09.12 at 13:54, Ben Guthro <ben@guthro.net> wrote: > I see - Konrad - do you have a changeset I can look for with the features > Jan describes?Alternatively you could also stick a wbinvd() in the two possibly relevant places... Jan> On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >>> On 24.09.12 at 13:25, Ben Guthro <ben@guthro.net> wrote: >> > It is an older system (Core2) - so it may be using the default mechanism, >> > but I''d have to go dig up some processor docs to be sure. >> >> This is not just a matter of what the CPU supports, but also what >> info gets passed down from Dom0 - if e.g. your kernel doesn''t >> have Konrad''s P-/C-state patches, then there''s no way for Xen to >> use any of the advanced methods. >> >> Jan >> >>
On Mon, Sep 24, 2012 at 07:54:05AM -0400, Ben Guthro wrote:> I see - Konrad - do you have a changeset I can look for with the features > Jan describes?xen-acpi-processor driver was merged to upstream Linux kernel v3.4. -- Pasi> On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <[1]JBeulich@suse.com> wrote: > > >>> On 24.09.12 at 13:25, Ben Guthro <[2]ben@guthro.net> wrote: > > It is an older system (Core2) - so it may be using the default > mechanism, > > but I''d have to go dig up some processor docs to be sure. > > This is not just a matter of what the CPU supports, but also what > info gets passed down from Dom0 - if e.g. your kernel doesn''t > have Konrad''s P-/C-state patches, then there''s no way for Xen to > use any of the advanced methods. > Jan > > References > > Visible links > 1. mailto:JBeulich@suse.com > 2. mailto:ben@guthro.net> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 8:05 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 24.09.12 at 13:54, Ben Guthro <ben@guthro.net> wrote: > > I see - Konrad - do you have a changeset I can look for with the features > > Jan describes? > > Alternatively you could also stick a wbinvd() in the two possibly > relevant places... > >I tried doing this right before disable_nonboot_cpus() in power.c, to no effect. The behavior that I''m currently failing to understand seems to be a heisenbug, exhibited by the 2 attached patches, which are essentially the same, with a cpumask_copy in 2 different places. When I have the debug copy just before do_suspend_lowlevel() - (debug1.patch) the output changes to what I would expect: (XEN) XXX: CACHE issue? before=1 after=1 That is, only CPU0 is initialized both before, and after the S3 cycle. However, when that same copy is moved up to before disable_nonboot_cpus() - as in debug2.patch - it seems to have an effect on the state afterward. I would have expected this to be something like (XEN) XXX: CACHE issue? before=3 after=1 But that is not what is actually printed. I get (XEN) XXX: CACHE issue? before=3 after=3 That is, the initialized bit is set for CPU1, when it shouldn''t be. It should be noted that even with the debug in that causes the value to be what I expect, this does not in itself is not sufficient to get things working entirely, as I then get stalls in the kernel rcu_sched> Jan > > > On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <JBeulich@suse.com> wrote: > > > >> >>> On 24.09.12 at 13:25, Ben Guthro <ben@guthro.net> wrote: > >> > It is an older system (Core2) - so it may be using the default > mechanism, > >> > but I''d have to go dig up some processor docs to be sure. > >> > >> This is not just a matter of what the CPU supports, but also what > >> info gets passed down from Dom0 - if e.g. your kernel doesn''t > >> have Konrad''s P-/C-state patches, then there''s no way for Xen to > >> use any of the advanced methods. > >> > >> Jan > >> > >> > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 8:22 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Mon, Sep 24, 2012 at 07:54:05AM -0400, Ben Guthro wrote: > > I see - Konrad - do you have a changeset I can look for with the > features > > Jan describes? > > xen-acpi-processor driver was merged to upstream Linux kernel v3.4. > >I see - so the advanced methods are not being used then This same kernel does work with Xen-4.0.x though. Is there an assumption in the code that ties Xen-4.2 to a newer kernel?> -- Pasi > > > On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <[1]JBeulich@suse.com> > wrote: > > > > >>> On 24.09.12 at 13:25, Ben Guthro <[2]ben@guthro.net> wrote: > > > It is an older system (Core2) - so it may be using the default > > mechanism, > > > but I''d have to go dig up some processor docs to be sure. > > > > This is not just a matter of what the CPU supports, but also what > > info gets passed down from Dom0 - if e.g. your kernel doesn''t > > have Konrad''s P-/C-state patches, then there''s no way for Xen to > > use any of the advanced methods. > > Jan > > > > References > > > > Visible links > > 1. mailto:JBeulich@suse.com > > 2. mailto:ben@guthro.net > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 24.09.12 at 14:24, Ben Guthro <ben@guthro.net> wrote: > On Mon, Sep 24, 2012 at 8:05 AM, Jan Beulich <JBeulich@suse.com> wrote: >> Alternatively you could also stick a wbinvd() in the two possibly >> relevant places... >> > I tried doing this right before disable_nonboot_cpus() in power.c, to no > effect.That''s too early - this must be done the last thing before entering the halt loops (which is why I referred to two places). Jan
On Mon, Sep 24, 2012 at 2:27 PM, Ben Guthro <ben@guthro.net> wrote: Hi Ben,>> > I see - Konrad - do you have a changeset I can look for with the >> > features >> > Jan describes? >> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. >> > > I see - so the advanced methods are not being used then > This same kernel does work with Xen-4.0.x though. Is there an assumption in > the code that ties Xen-4.2 to a newer kernel?I''m being bitten by this bug with a 3.5 kernel and Xen 4.2 -- Javier Marcet <jmarcet@gmail.com>
>>> On 24.09.12 at 14:27, Ben Guthro <ben@guthro.net> wrote: > On Mon, Sep 24, 2012 at 8:22 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: > >> On Mon, Sep 24, 2012 at 07:54:05AM -0400, Ben Guthro wrote: >> > I see - Konrad - do you have a changeset I can look for with the >> features >> > Jan describes? >> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. >> >> > I see - so the advanced methods are not being used then > This same kernel does work with Xen-4.0.x though. Is there an assumption in > the code that ties Xen-4.2 to a newer kernel?No, but if e.g. you merely look at the differences between both cpu_uninit() versions, you'll already see that there's quite a bit more writing of memory being done in 4.0.x than in 4.2.0 or -unstable, which increases the chances of hiding an eventual cache flushing problem. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 07:54:05AM -0400, Ben Guthro wrote:> I see - Konrad - do you have a changeset I can look for with the features > Jan describes?konrad@phenom:~/work/linux$ git log --oneline drivers/xen/xen-acpi-processor.c 17f9b89 xen/acpi: Fix potential memory leak. 323f90a xen-acpi-processor: Add missing #include <xen/xen.h> b930fe5 xen/acpi: Workaround broken BIOSes exporting non-existing C-states. 27257fc xen/acpi: Remove the WARN''s as they just create noise. 59a5680 xen/acpi-processor: C and P-state driver that uploads said data to hypervisor.> > > On Mon, Sep 24, 2012 at 7:45 AM, Jan Beulich <JBeulich@suse.com> wrote: > > > >>> On 24.09.12 at 13:25, Ben Guthro <ben@guthro.net> wrote: > > > It is an older system (Core2) - so it may be using the default mechanism, > > > but I''d have to go dig up some processor docs to be sure. > > > > This is not just a matter of what the CPU supports, but also what > > info gets passed down from Dom0 - if e.g. your kernel doesn''t > > have Konrad''s P-/C-state patches, then there''s no way for Xen to > > use any of the advanced methods. > > > > Jan > > > >
On Mon, Sep 24, 2012 at 02:37:22PM +0200, Javier Marcet wrote:> On Mon, Sep 24, 2012 at 2:27 PM, Ben Guthro <ben@guthro.net> wrote: > > Hi Ben, > > >> > I see - Konrad - do you have a changeset I can look for with the > >> > features > >> > Jan describes? > >> > >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. > >> > > > > I see - so the advanced methods are not being used thenYou mean with the v3.5 kernel? I would think they would be used..> > This same kernel does work with Xen-4.0.x though. Is there an assumption in > > the code that ties Xen-4.2 to a newer kernel? > > I''m being bitten by this bug with a 3.5 kernel and Xen 4.2Huh? How so?
>>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> wrote: >> ...; the interesting ones are >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() >> - xen/arch/x86/domain.c:default_dead_idle() > > > Thanks! This fixes the issue on this machine!Hooray!> Is this a reasonable long-term solution - or are there reasons not to > call wbinvd() here?That''s a perfectly valid adjustment (see my earlier reply where I originally suggested it and explained why it may be necessary). Jan
Would you prefer a separate [PATCH] email for this fix, or will you apply it as-is? On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: > > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> ...; the interesting ones are > >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() > >> - xen/arch/x86/domain.c:default_dead_idle() > > > > > > Thanks! This fixes the issue on this machine! > > Hooray! > > > Is this a reasonable long-term solution - or are there reasons not to > > call wbinvd() here? > > That''s a perfectly valid adjustment (see my earlier reply where > I originally suggested it and explained why it may be necessary). > > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 24.09.12 at 16:16, Ben Guthro <ben@guthro.net> wrote: > Would you prefer a separate [PATCH] email for this fix, or will you apply > it as-is?I''ll put something together - the most important thing here obviously is having a proper description. Plus I''d like to slightly extend this and have acpi_dead_idle() actually use default_dead_idle(), just to have things consolidated in one place. I assume I can put your S-o-b on what you sent... Jan> On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: >> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> wrote: >> >> ...; the interesting ones are >> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() >> >> - xen/arch/x86/domain.c:default_dead_idle() >> > >> > >> > Thanks! This fixes the issue on this machine! >> >> Hooray! >> >> > Is this a reasonable long-term solution - or are there reasons not to >> > call wbinvd() here? >> >> That''s a perfectly valid adjustment (see my earlier reply where >> I originally suggested it and explained why it may be necessary). >> >> Jan >> >>
I would like to see it myself and Ack it. Although I think I can guess what it does and I will be happy with it if I¹m right. :) -- Keir On 24/09/2012 15:16, "Ben Guthro" <ben@guthro.net> wrote:> Would you prefer a separate [PATCH] email for this fix, or will you apply it > as-is? > > > > On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: >>> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>> >> ...; the interesting ones are >>>> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() >>>> >> - xen/arch/x86/domain.c:default_dead_idle() >>> > >>> > >>> > Thanks! This fixes the issue on this machine! >> >> Hooray! >> >>> > Is this a reasonable long-term solution - or are there reasons not to >>> > call wbinvd() here? >> >> That''s a perfectly valid adjustment (see my earlier reply where >> I originally suggested it and explained why it may be necessary). >> >> Jan >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 4:04 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> >> > I see - Konrad - do you have a changeset I can look for with the >> >> > features >> >> > Jan describes? >> >> >> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. >> >> >> > >> > I see - so the advanced methods are not being used then > > You mean with the v3.5 kernel? I would think they would be used.. > >> > This same kernel does work with Xen-4.0.x though. Is there an assumption in >> > the code that ties Xen-4.2 to a newer kernel? >> >> I''m being bitten by this bug with a 3.5 kernel and Xen 4.2 > > Huh? How so?Hmmm, yes, upon resuming from S3 I have this exact same bug. -- Javier Marcet <jmarcet@gmail.com>
Well...knock one bug down - and another crops up. It appears that dom0_vcpu_pin is incompatible with S3. I''ll start digging into why, but if you have any thoughts from the stack below, I''d welcome any pointers. /btg (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Thermal monitoring enabled (TM2) (XEN) Finishing wakeup from ACPI S3 state. (XEN) Enabling non-boot CPUs ... (XEN) Booting processor 1/1 eip 8a000 (XEN) Initializing CPU#1 (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K (XEN) CPU: L2 cache: 3072K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU: Processor Core ID: 1 (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Thermal monitoring enabled (TM2) (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date 2010-09-29 [ 60.100054] ACPI: Low-level resume complete [ 60.100054] PM: Restoring platform NVS memory [ 60.100054] Enabling non-boot CPUs ... [ 60.100054] installing Xen timer for CPU 1 [ 60.100054] cpu 1 spinlock event irq 279 (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- (XEN) CPU: 1 (XEN) RIP: e008:[<ffff82c480121562>] vcpu_migrate+0x172/0x360 (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor (XEN) rax: 00007d3b7fd17180 rbx: ffff82c4802e8ee0 rcx: ffff82c4802e8ee0 (XEN) rdx: ffff83013a3c5068 rsi: 0000000000000004 rdi: ffff8301300b7d68 (XEN) rbp: 0000000000000001 rsp: ffff8301300b7e28 r8: 0000000000000000 (XEN) r9: 000000000000003e r10: 000000000000003e r11: 0000000000000246 (XEN) r12: ffff83013a3c5068 r13: ffff83013a3c5068 r14: ffff82c4802d3140 (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000131a05000 cr2: 0000000000000060 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff8301300b7e28: (XEN) ffff82c4802d3140 ffff83013a3c5068 0000000000000246 0000000000000004 (XEN) ffff8300bd2fe000 ffff82c4802e8ee0 00000004012d3140 ffff82c4802e8ee0 (XEN) ffff88003fc8e820 ffff8300bd2fe000 ffff8301355d8000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffff88003fc8e820 ffff82c480105a50 (XEN) 0000000000000000 ffff82c4801805ec 0000060f00000000 ffff82c480184f16 (XEN) 0000000000000032 78a20f6e65780b0f ffff88003976fdc8 ffff8300bd2fe000 (XEN) ffff88003976fe50 ffff8300bd2fe000 ffff88003976fda0 0000000000000001 (XEN) 0000000000000000 ffff82c480214288 ffff88003fc8e820 0000000000000000 (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 ffff88003fc8bdc0 (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff 0000000000000000 (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 0000000000000001 (XEN) 0000000000000007 0000010000000000 ffffffff8100130a 000000000000e033 (XEN) 0000000000000246 ffff88003976fd88 000000000000e02b d43d5f3fedaef5e7 (XEN) d3b2ddaeed5038ff 270adb813ad76c9b ddfd6ff5f85e6775 b5881cbf00000001 (XEN) ffff8300bd2fe000 0000003cba0dc180 0a109ac649c118a1 (XEN) Xen call trace: (XEN) [<ffff82c480121562>] vcpu_migrate+0x172/0x360 (XEN) [<ffff82c480105a50>] do_vcpu_op+0x1e0/0x4a0 (XEN) [<ffff82c4801805ec>] do_invalid_op+0x19c/0x3f0 (XEN) [<ffff82c480184f16>] copy_from_user+0x26/0x90 (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d (XEN) (XEN) Pagetable walk from 0000000000000060: (XEN) L4[0x000] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: 0000000000000060 (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... On Mon, Sep 24, 2012 at 10:28 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 24.09.12 at 16:16, Ben Guthro <ben@guthro.net> wrote: > > Would you prefer a separate [PATCH] email for this fix, or will you apply > > it as-is? > > I''ll put something together - the most important thing here obviously > is having a proper description. Plus I''d like to slightly extend this and > have acpi_dead_idle() actually use default_dead_idle(), just to have > things consolidated in one place. I assume I can put your S-o-b on > what you sent... > > Jan > > > On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote: > > > >> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: > >> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> > wrote: > >> >> ...; the interesting ones are > >> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() > >> >> - xen/arch/x86/domain.c:default_dead_idle() > >> > > >> > > >> > Thanks! This fixes the issue on this machine! > >> > >> Hooray! > >> > >> > Is this a reasonable long-term solution - or are there reasons not to > >> > call wbinvd() here? > >> > >> That''s a perfectly valid adjustment (see my earlier reply where > >> I originally suggested it and explained why it may be necessary). > >> > >> Jan > >> > >> > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Do a debug build so the backtrace can be trusted. It¹s a NULL pointer dereference so shouldn¹t be too tricky to make some headway on this one. Easier than the previous bug. :) -- Keir On 24/09/2012 20:02, "Ben Guthro" <ben@guthro.net> wrote:> Well...knock one bug down - and another crops up. > > It appears that dom0_vcpu_pin is incompatible with S3. > I''ll start digging into why, but if you have any thoughts from the stack > below, I''d welcome any pointers. > > /btg > > > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 > extended MCE MSR 0 > (XEN) CMCI: CPU0 has no CMCI support > (XEN) CPU0: Thermal monitoring enabled (TM2) > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date = 2010-09-29 > [ 60.100054] ACPI: Low-level resume complete > [ 60.100054] PM: Restoring platform NVS memory > [ 60.100054] Enabling non-boot CPUs ... > [ 60.100054] installing Xen timer for CPU 1 > [ 60.100054] cpu 1 spinlock event irq 279 > (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- > (XEN) CPU: 1 > (XEN) RIP: e008:[<ffff82c480121562>] vcpu_migrate+0x172/0x360 > (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor > (XEN) rax: 00007d3b7fd17180 rbx: ffff82c4802e8ee0 rcx: ffff82c4802e8ee0 > (XEN) rdx: ffff83013a3c5068 rsi: 0000000000000004 rdi: ffff8301300b7d68 > (XEN) rbp: 0000000000000001 rsp: ffff8301300b7e28 r8: 0000000000000000 > (XEN) r9: 000000000000003e r10: 000000000000003e r11: 0000000000000246 > (XEN) r12: ffff83013a3c5068 r13: ffff83013a3c5068 r14: ffff82c4802d3140 > (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 0000000131a05000 cr2: 0000000000000060 > (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff8301300b7e28: > (XEN) ffff82c4802d3140 ffff83013a3c5068 0000000000000246 0000000000000004 > (XEN) ffff8300bd2fe000 ffff82c4802e8ee0 00000004012d3140 ffff82c4802e8ee0 > (XEN) ffff88003fc8e820 ffff8300bd2fe000 ffff8301355d8000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffff88003fc8e820 ffff82c480105a50 > (XEN) 0000000000000000 ffff82c4801805ec 0000060f00000000 ffff82c480184f16 > (XEN) 0000000000000032 78a20f6e65780b0f ffff88003976fdc8 ffff8300bd2fe000 > (XEN) ffff88003976fe50 ffff8300bd2fe000 ffff88003976fda0 0000000000000001 > (XEN) 0000000000000000 ffff82c480214288 ffff88003fc8e820 0000000000000000 > (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 ffff88003fc8bdc0 > (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff 0000000000000000 > (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 0000000000000001 > (XEN) 0000000000000007 0000010000000000 ffffffff8100130a 000000000000e033 > (XEN) 0000000000000246 ffff88003976fd88 000000000000e02b d43d5f3fedaef5e7 > (XEN) d3b2ddaeed5038ff 270adb813ad76c9b ddfd6ff5f85e6775 b5881cbf00000001 > (XEN) ffff8300bd2fe000 0000003cba0dc180 0a109ac649c118a1 > (XEN) Xen call trace: > (XEN) [<ffff82c480121562>] vcpu_migrate+0x172/0x360 > (XEN) [<ffff82c480105a50>] do_vcpu_op+0x1e0/0x4a0 > (XEN) [<ffff82c4801805ec>] do_invalid_op+0x19c/0x3f0 > (XEN) [<ffff82c480184f16>] copy_from_user+0x26/0x90 > (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d > (XEN) > (XEN) Pagetable walk from 0000000000000060: > (XEN) L4[0x000] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 1: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: 0000000000000060 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > > On Mon, Sep 24, 2012 at 10:28 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> >>> On 24.09.12 at 16:16, Ben Guthro <ben@guthro.net> wrote: >>> > Would you prefer a separate [PATCH] email for this fix, or will you apply >>> > it as-is? >> >> I''ll put something together - the most important thing here obviously >> is having a proper description. Plus I''d like to slightly extend this and >> have acpi_dead_idle() actually use default_dead_idle(), just to have >> things consolidated in one place. I assume I can put your S-o-b on >> what you sent... >> >> Jan >> >>> > On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote: >>> > >>>>>>> >> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: >>>>> >> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> >>>>> wrote: >>>>>> >> >> ...; the interesting ones are >>>>>> >> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() >>>>>> >> >> - xen/arch/x86/domain.c:default_dead_idle() >>>>> >> > >>>>> >> > >>>>> >> > Thanks! This fixes the issue on this machine! >>>> >> >>>> >> Hooray! >>>> >> >>>>> >> > Is this a reasonable long-term solution - or are there reasons not to >>>>> >> > call wbinvd() here? >>>> >> >>>> >> That''s a perfectly valid adjustment (see my earlier reply where >>>> >> I originally suggested it and explained why it may be necessary). >>>> >> >>>> >> Jan >>>> >> >>>> >> >> >> >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
I''ve managed to determine that _csched_cpu_pick is, for reasons not yet clear, picking a cpu id outside of the range of cpus that are valid for this system (in this case cpu id 4, on a 2 core machine) On Mon, Sep 24, 2012 at 4:30 PM, Keir Fraser <keir.xen@gmail.com> wrote:> Do a debug build so the backtrace can be trusted. It’s a NULL pointer > dereference so shouldn’t be too tricky to make some headway on this one. > Easier than the previous bug. :) > > -- Keir > > > On 24/09/2012 20:02, "Ben Guthro" <ben@guthro.net> wrote: > > Well...knock one bug down - and another crops up. > > It appears that dom0_vcpu_pin is incompatible with S3. > I''ll start digging into why, but if you have any thoughts from the stack > below, I''d welcome any pointers. > > /btg > > > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 > extended MCE MSR 0 > (XEN) CMCI: CPU0 has no CMCI support > (XEN) CPU0: Thermal monitoring enabled (TM2) > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date > 2010-09-29 > [ 60.100054] ACPI: Low-level resume complete > [ 60.100054] PM: Restoring platform NVS memory > [ 60.100054] Enabling non-boot CPUs ... > [ 60.100054] installing Xen timer for CPU 1 > [ 60.100054] cpu 1 spinlock event irq 279 > (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- > (XEN) CPU: 1 > (XEN) RIP: e008:[<ffff82c480121562>] vcpu_migrate+0x172/0x360 > (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor > (XEN) rax: 00007d3b7fd17180 rbx: ffff82c4802e8ee0 rcx: ffff82c4802e8ee0 > (XEN) rdx: ffff83013a3c5068 rsi: 0000000000000004 rdi: ffff8301300b7d68 > (XEN) rbp: 0000000000000001 rsp: ffff8301300b7e28 r8: 0000000000000000 > (XEN) r9: 000000000000003e r10: 000000000000003e r11: 0000000000000246 > (XEN) r12: ffff83013a3c5068 r13: ffff83013a3c5068 r14: ffff82c4802d3140 > (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 0000000131a05000 cr2: 0000000000000060 > (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff8301300b7e28: > (XEN) ffff82c4802d3140 ffff83013a3c5068 0000000000000246 > 0000000000000004 > (XEN) ffff8300bd2fe000 ffff82c4802e8ee0 00000004012d3140 > ffff82c4802e8ee0 > (XEN) ffff88003fc8e820 ffff8300bd2fe000 ffff8301355d8000 > 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffff88003fc8e820 > ffff82c480105a50 > (XEN) 0000000000000000 ffff82c4801805ec 0000060f00000000 > ffff82c480184f16 > (XEN) 0000000000000032 78a20f6e65780b0f ffff88003976fdc8 > ffff8300bd2fe000 > (XEN) ffff88003976fe50 ffff8300bd2fe000 ffff88003976fda0 > 0000000000000001 > (XEN) 0000000000000000 ffff82c480214288 ffff88003fc8e820 > 0000000000000000 > (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 > ffff88003fc8bdc0 > (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff > 0000000000000000 > (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 > 0000000000000001 > (XEN) 0000000000000007 0000010000000000 ffffffff8100130a > 000000000000e033 > (XEN) 0000000000000246 ffff88003976fd88 000000000000e02b > d43d5f3fedaef5e7 > (XEN) d3b2ddaeed5038ff 270adb813ad76c9b ddfd6ff5f85e6775 > b5881cbf00000001 > (XEN) ffff8300bd2fe000 0000003cba0dc180 0a109ac649c118a1 > (XEN) Xen call trace: > (XEN) [<ffff82c480121562>] vcpu_migrate+0x172/0x360 > (XEN) [<ffff82c480105a50>] do_vcpu_op+0x1e0/0x4a0 > (XEN) [<ffff82c4801805ec>] do_invalid_op+0x19c/0x3f0 > (XEN) [<ffff82c480184f16>] copy_from_user+0x26/0x90 > (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d > (XEN) > (XEN) Pagetable walk from 0000000000000060: > (XEN) L4[0x000] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 1: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: 0000000000000060 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > > On Mon, Sep 24, 2012 at 10:28 AM, Jan Beulich <JBeulich@suse.com> wrote: > > >>> On 24.09.12 at 16:16, Ben Guthro <ben@guthro.net> wrote: > > Would you prefer a separate [PATCH] email for this fix, or will you apply > > it as-is? > > I''ll put something together - the most important thing here obviously > is having a proper description. Plus I''d like to slightly extend this and > have acpi_dead_idle() actually use default_dead_idle(), just to have > things consolidated in one place. I assume I can put your S-o-b on > what you sent... > > Jan > > > On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> wrote: > > > >> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: > >> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> > wrote: > >> >> ...; the interesting ones are > >> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() > >> >> - xen/arch/x86/domain.c:default_dead_idle() > >> > > >> > > >> > Thanks! This fixes the issue on this machine! > >> > >> Hooray! > >> > >> > Is this a reasonable long-term solution - or are there reasons not to > >> > call wbinvd() here? > >> > >> That''s a perfectly valid adjustment (see my earlier reply where > >> I originally suggested it and explained why it may be necessary). > >> > >> Jan > >> > >> > > > > > > ------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Here''s my "Big hammer" debugging patch. If I force the cpu to be scheduled on CPU0 when the appropriate cpu is not online, I can resume properly. Clearly this is not the proper solution, and I''m sure the fix is subtle. I''m not seeing it right now though. Perhaps tomorrow morning. If you have any ideas, I''m happy to run tests then. /btg On Mon, Sep 24, 2012 at 4:46 PM, Ben Guthro <ben@guthro.net> wrote:> I''ve managed to determine that _csched_cpu_pick is, for reasons not yet > clear, picking a cpu id outside of the range of cpus that are valid for > this system > (in this case cpu id 4, on a 2 core machine) > > > > On Mon, Sep 24, 2012 at 4:30 PM, Keir Fraser <keir.xen@gmail.com> wrote: > >> Do a debug build so the backtrace can be trusted. It’s a NULL pointer >> dereference so shouldn’t be too tricky to make some headway on this one. >> Easier than the previous bug. :) >> >> -- Keir >> >> >> On 24/09/2012 20:02, "Ben Guthro" <ben@guthro.net> wrote: >> >> Well...knock one bug down - and another crops up. >> >> It appears that dom0_vcpu_pin is incompatible with S3. >> I''ll start digging into why, but if you have any thoughts from the stack >> below, I''d welcome any pointers. >> >> /btg >> >> >> (XEN) Preparing system for ACPI S3 state. >> (XEN) Disabling non-boot CPUs ... >> (XEN) Entering ACPI S3 state. >> (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 >> extended MCE MSR 0 >> (XEN) CMCI: CPU0 has no CMCI support >> (XEN) CPU0: Thermal monitoring enabled (TM2) >> (XEN) Finishing wakeup from ACPI S3 state. >> (XEN) Enabling non-boot CPUs ... >> (XEN) Booting processor 1/1 eip 8a000 >> (XEN) Initializing CPU#1 >> (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K >> (XEN) CPU: L2 cache: 3072K >> (XEN) CPU: Physical Processor ID: 0 >> (XEN) CPU: Processor Core ID: 1 >> (XEN) CMCI: CPU1 has no CMCI support >> (XEN) CPU1: Thermal monitoring enabled (TM2) >> (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 >> (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date >> 2010-09-29 >> [ 60.100054] ACPI: Low-level resume complete >> [ 60.100054] PM: Restoring platform NVS memory >> [ 60.100054] Enabling non-boot CPUs ... >> [ 60.100054] installing Xen timer for CPU 1 >> [ 60.100054] cpu 1 spinlock event irq 279 >> (XEN) ----[ Xen-4.2.1-pre x86_64 debug=n Tainted: C ]---- >> (XEN) CPU: 1 >> (XEN) RIP: e008:[<ffff82c480121562>] vcpu_migrate+0x172/0x360 >> (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor >> (XEN) rax: 00007d3b7fd17180 rbx: ffff82c4802e8ee0 rcx: >> ffff82c4802e8ee0 >> (XEN) rdx: ffff83013a3c5068 rsi: 0000000000000004 rdi: >> ffff8301300b7d68 >> (XEN) rbp: 0000000000000001 rsp: ffff8301300b7e28 r8: >> 0000000000000000 >> (XEN) r9: 000000000000003e r10: 000000000000003e r11: >> 0000000000000246 >> (XEN) r12: ffff83013a3c5068 r13: ffff83013a3c5068 r14: >> ffff82c4802d3140 >> (XEN) r15: 0000000000000001 cr0: 000000008005003b cr4: >> 00000000000026f0 >> (XEN) cr3: 0000000131a05000 cr2: 0000000000000060 >> (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 >> (XEN) Xen stack trace from rsp=ffff8301300b7e28: >> (XEN) ffff82c4802d3140 ffff83013a3c5068 0000000000000246 >> 0000000000000004 >> (XEN) ffff8300bd2fe000 ffff82c4802e8ee0 00000004012d3140 >> ffff82c4802e8ee0 >> (XEN) ffff88003fc8e820 ffff8300bd2fe000 ffff8301355d8000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 ffff88003fc8e820 >> ffff82c480105a50 >> (XEN) 0000000000000000 ffff82c4801805ec 0000060f00000000 >> ffff82c480184f16 >> (XEN) 0000000000000032 78a20f6e65780b0f ffff88003976fdc8 >> ffff8300bd2fe000 >> (XEN) ffff88003976fe50 ffff8300bd2fe000 ffff88003976fda0 >> 0000000000000001 >> (XEN) 0000000000000000 ffff82c480214288 ffff88003fc8e820 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 >> ffff88003fc8bdc0 >> (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff >> 0000000000000000 >> (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 >> 0000000000000001 >> (XEN) 0000000000000007 0000010000000000 ffffffff8100130a >> 000000000000e033 >> (XEN) 0000000000000246 ffff88003976fd88 000000000000e02b >> d43d5f3fedaef5e7 >> (XEN) d3b2ddaeed5038ff 270adb813ad76c9b ddfd6ff5f85e6775 >> b5881cbf00000001 >> (XEN) ffff8300bd2fe000 0000003cba0dc180 0a109ac649c118a1 >> (XEN) Xen call trace: >> (XEN) [<ffff82c480121562>] vcpu_migrate+0x172/0x360 >> (XEN) [<ffff82c480105a50>] do_vcpu_op+0x1e0/0x4a0 >> (XEN) [<ffff82c4801805ec>] do_invalid_op+0x19c/0x3f0 >> (XEN) [<ffff82c480184f16>] copy_from_user+0x26/0x90 >> (XEN) [<ffff82c480214288>] syscall_enter+0x88/0x8d >> (XEN) >> (XEN) Pagetable walk from 0000000000000060: >> (XEN) L4[0x000] = 0000000000000000 ffffffffffffffff >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 1: >> (XEN) FATAL PAGE FAULT >> (XEN) [error_code=0000] >> (XEN) Faulting linear address: 0000000000000060 >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> >> On Mon, Sep 24, 2012 at 10:28 AM, Jan Beulich <JBeulich@suse.com> wrote: >> >> >>> On 24.09.12 at 16:16, Ben Guthro <ben@guthro.net> wrote: >> > Would you prefer a separate [PATCH] email for this fix, or will you >> apply >> > it as-is? >> >> I''ll put something together - the most important thing here obviously >> is having a proper description. Plus I''d like to slightly extend this and >> have acpi_dead_idle() actually use default_dead_idle(), just to have >> things consolidated in one place. I assume I can put your S-o-b on >> what you sent... >> >> Jan >> >> > On Mon, Sep 24, 2012 at 10:10 AM, Jan Beulich <JBeulich@suse.com> >> wrote: >> > >> >> >>> On 24.09.12 at 15:56, Ben Guthro <ben@guthro.net> wrote: >> >> > On Mon, Sep 24, 2012 at 9:34 AM, Jan Beulich <JBeulich@suse.com> >> wrote: >> >> >> ...; the interesting ones are >> >> >> - at the end of xen/arch/x86/acpu/cpu_idle.c:acpi_dead_idle() >> >> >> - xen/arch/x86/domain.c:default_dead_idle() >> >> > >> >> > >> >> > Thanks! This fixes the issue on this machine! >> >> >> >> Hooray! >> >> >> >> > Is this a reasonable long-term solution - or are there reasons not to >> >> > call wbinvd() here? >> >> >> >> That''s a perfectly valid adjustment (see my earlier reply where >> >> I originally suggested it and explained why it may be necessary). >> >> >> >> Jan >> >> >> >> >> >> >> >> >> >> ------------------------------ >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 4:04 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. >> >> >> > >> > I see - so the advanced methods are not being used then > > You mean with the v3.5 kernel? I would think they would be used.. > >> > This same kernel does work with Xen-4.0.x though. Is there an assumption in >> > the code that ties Xen-4.2 to a newer kernel? >> >> I''m being bitten by this bug with a 3.5 kernel and Xen 4.2 > > Huh? How so?I''ve tried adding a wbinvd() call before the halts in xen/arch/x86/domain.c:default_idle and I could do full suspend and resume cycle once, but a reboot later I couldn''t resume anymore. Here is the trace I get: [ 142.322778] ACPI: Preparing to enter system sleep state S3 [ 142.723286] PM: Saving platform NVS memory [ 142.736453] Disabling non-boot CPUs ... [ 142.851387] ------------[ cut here ]------------ [ 142.851397] WARNING: at /home/storage/src/ubuntu-precise/kernel/rcutree.c:1550 rcu_do_batch.isra.41+0x5f/0x213() [ 142.851401] Hardware name: To Be Filled By O.E.M. [ 142.851404] Modules linked in: blktap(O) ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables [last unloaded: cx25840] [ 142.851466] Pid: 32, comm: migration/7 Tainted: G O 3.5.0-14-i7 #18~precise1 [ 142.851469] Call Trace: [ 142.851473] <IRQ> [<ffffffff8106a9bd>] warn_slowpath_common+0x7e/0x96 [ 142.851488] [<ffffffff8106a9ea>] warn_slowpath_null+0x15/0x17 [ 142.851496] [<ffffffff810c6e87>] rcu_do_batch.isra.41+0x5f/0x213 [ 142.851504] [<ffffffff810c687f>] ? check_for_new_grace_period.isra.35+0x38/0x44 [ 142.851512] [<ffffffff810c7101>] __rcu_process_callbacks+0xc6/0xee [ 142.851520] [<ffffffff810c7149>] rcu_process_callbacks+0x20/0x3e [ 142.851527] [<ffffffff81070c27>] __do_softirq+0x87/0x113 [ 142.851536] [<ffffffff812e7b22>] ? __xen_evtchn_do_upcall+0x1a0/0x1dd [ 142.851546] [<ffffffff8162f05c>] call_softirq+0x1c/0x30 [ 142.851557] [<ffffffff810343a3>] do_softirq+0x41/0x7e [ 142.851566] [<ffffffff81070e66>] irq_exit+0x3f/0x9a [ 142.851573] [<ffffffff812e95c9>] xen_evtchn_do_upcall+0x2c/0x39 [ 142.851580] [<ffffffff8162f0ae>] xen_do_hypervisor_callback+0x1e/0x30 [ 142.851588] <EOI> [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 [ 142.851601] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 [ 142.851609] [<ffffffff8102eee9>] ? xen_force_evtchn_callback+0xd/0xf [ 142.851618] [<ffffffff8102f552>] ? check_events+0x12/0x20 [ 142.851627] [<ffffffff8102f53f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 142.851635] [<ffffffff810b6ff4>] ? arch_local_irq_restore+0xb/0xd [ 142.851644] [<ffffffff810b73f4>] ? stop_machine_cpu_stop+0xc1/0xd3 [ 142.851652] [<ffffffff810b7333>] ? queue_stop_cpus_work+0xb5/0xb5 [ 142.851660] [<ffffffff810b7152>] ? cpu_stopper_thread+0xf7/0x187 [ 142.851667] [<ffffffff8108a927>] ? finish_task_switch+0x82/0xc1 [ 142.851676] [<ffffffff8162c50b>] ? __schedule+0x428/0x454 [ 142.851685] [<ffffffff8162cff0>] ? _raw_spin_unlock_irqrestore+0x15/0x18 [ 142.851693] [<ffffffff810b705b>] ? cpu_stop_signal_done+0x30/0x30 [ 142.851701] [<ffffffff81082627>] ? kthread+0x86/0x8e [ 142.851710] [<ffffffff8162ef64>] ? kernel_thread_helper+0x4/0x10 [ 142.851718] [<ffffffff8162d338>] ? retint_restore_args+0x5/0x6 [ 142.851726] [<ffffffff8162ef60>] ? gs_change+0x13/0x13 [ 142.851736] ---[ end trace 3722a99bcc5ae37a ]--- [ 144.257229] ACPI: Low-level resume complete [ 144.257322] PM: Restoring platform NVS memory -- Javier Marcet <jmarcet@gmail.com>
>>> On 24.09.12 at 23:12, Ben Guthro <ben@guthro.net> wrote: > Here''s my "Big hammer" debugging patch. > > If I force the cpu to be scheduled on CPU0 when the appropriate cpu is not > online, I can resume properly. > > Clearly this is not the proper solution, and I''m sure the fix is subtle. > I''m not seeing it right now though. Perhaps tomorrow morning. > If you have any ideas, I''m happy to run tests then.I can''t see how the printk() you add in the patch would ever get reached with the other adjustment you do there. A debug build, as Keir suggested, would not only get the stack trace right, but would also result in the ASSERT() right after your first modification to _csched_cpu_pick() to actually do something (and likely trigger). Anyway, this might be connected to cpu_disable_scheduler() not having a counterpart to restore the affinity it broke for pinned domains (for non-pinned ones I believe this behavior is intentional, albeit not ideal). Jan
On Tue, Sep 25, 2012 at 3:00 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 24.09.12 at 23:12, Ben Guthro <ben@guthro.net> wrote: > > Here''s my "Big hammer" debugging patch. > > > > If I force the cpu to be scheduled on CPU0 when the appropriate cpu is > not > > online, I can resume properly. > > > > Clearly this is not the proper solution, and I''m sure the fix is subtle. > > I''m not seeing it right now though. Perhaps tomorrow morning. > > If you have any ideas, I''m happy to run tests then. > > I can''t see how the printk() you add in the patch would ever get > reached with the other adjustment you do there.Apologies. I failed to separate prior debugging in this patch from the "big hammer" fix> A debug build, > as Keir suggested, would not only get the stack trace right, but > would also result in the ASSERT() right after your first modification > to _csched_cpu_pick() to actually do something (and likely trigger). >Indeed. I was using non-debug builds for 2 reasons that, in hindsight may not be the best of reasons. 1. It was the default 2. Mukesh''s kdb debugger requires debug to be off, which I was making use of previously, and had not disabled. The stack from a debug build can be found below. It did, indeed trigger the ASSERT, as you predicted. (XEN) Finishing wakeup from ACPI S3 state. (XEN) Enabling non-boot CPUs ... (XEN) Booting processor 1/1 eip 8a000 (XEN) Initializing CPU#1 (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K (XEN) CPU: L2 cache: 3072K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU: Processor Core ID: 1 (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Thermal monitoring enabled (TM2) (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date 2010-09-29 [ 82.310025] ACPI: Low-level resume complete [ 82.310025] PM: Restoring platform NVS memory [ 82.310025] Enabling non-boot CPUs ... [ 82.310025] installing Xen timer for CPU 1 [ 82.310025] cpu 1 spinlock event irq 279 (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' failed at sched_credit.c:477 (XEN) ----[ Xen-4.2.1-pre x86_64 debug=y Tainted: C ]---- (XEN) CPU: 1 (XEN) RIP: e008:[<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor (XEN) rax: 0000000000000001 rbx: 0000000000000004 rcx: 0000000000000004 (XEN) rdx: 000000000000000f rsi: 0000000000000004 rdi: 0000000000000000 (XEN) rbp: ffff8301355d7dd8 rsp: ffff8301355d7d08 r8: 0000000000000000 (XEN) r9: 000000000000003e r10: ffff82c480231700 r11: 0000000000000246 (XEN) r12: ffff82c480261b20 r13: 0000000000000001 r14: ffff82c480301a60 (XEN) r15: ffff83013a542068 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000131a05000 cr2: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff8301355d7d08: (XEN) 0100000131a05000 ffff8301355d7d40 0000000000000082 0000000000000002 (XEN) ffff8300bd503000 0000000000000001 0000000000000297 ffff8301355d7d58 (XEN) ffff82c480125499 ffff830138216000 ffff8301355d7d98 5400000000000002 (XEN) 0000000000000286 ffff8301355d7d88 ffff82c480125499 ffff830138216000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff830134ca6a50 ffff83013a542068 ffff83013a542068 0000000000000001 (XEN) ffff82c480301a60 ffff83013a542068 ffff8301355d7de8 ffff82c48011a785 (XEN) ffff8301355d7e58 ffff82c480123519 ffff82c480301a60 ffff82c480301a60 (XEN) ffff82c480301a60 ffff8300bd503000 0000000000503060 0000000000000246 (XEN) ffff82c480127c31 ffff8300bd503000 ffff82c480301a60 ffff82c4802ebd40 (XEN) ffff83013a542068 ffff88003fc8e820 ffff8301355d7e88 ffff82c4801237d3 (XEN) fffffffffffffffe ffff8301355ca000 ffff8300bd503000 0000000000000000 (XEN) ffff8301355d7ef8 ffff82c480106335 ffff8301355d7f18 ffffffff810030e1 (XEN) ffff8300bd503000 0000000000000000 ffff8301355d7f08 ffff82c480185390 (XEN) ffffffff81aafd32 ffff8300bd503000 0000000000000001 0000000000000000 (XEN) 0000000000000000 ffff88003fc8e820 00007cfecaa280c7 ffff82c480227348 (XEN) ffffffff8100130a 0000000000000018 ffff88003fc8e820 0000000000000000 (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 ffff88003fc8bdc0 (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff 0000000000000000 (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 0000000000000001 (XEN) Xen call trace: (XEN) [<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 (XEN) [<ffff82c48011a785>] csched_cpu_pick+0xe/0x10 (XEN) [<ffff82c480123519>] vcpu_migrate+0x19f/0x346 (XEN) [<ffff82c4801237d3>] vcpu_force_reschedule+0xa4/0xb6 (XEN) [<ffff82c480106335>] do_vcpu_op+0x2c9/0x452 (XEN) [<ffff82c480227348>] syscall_enter+0xc8/0x122 (XEN) (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' failed at sched_credit.c:477 (XEN) **************************************** (XEN) (XEN) Reboot in five seconds...> Anyway, this might be connected to cpu_disable_scheduler() not > having a counterpart to restore the affinity it broke for pinned > domains (for non-pinned ones I believe this behavior is intentional, > albeit not ideal). > > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Sep 24, 2012 at 11:36:23PM +0200, Javier Marcet wrote:> On Mon, Sep 24, 2012 at 4:04 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > > >> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. > >> >> > >> > > >> > I see - so the advanced methods are not being used then > > > > You mean with the v3.5 kernel? I would think they would be used.. > > > >> > This same kernel does work with Xen-4.0.x though. Is there an assumption in > >> > the code that ties Xen-4.2 to a newer kernel? > >> > >> I''m being bitten by this bug with a 3.5 kernel and Xen 4.2 > > > > Huh? How so? > > I''ve tried adding a wbinvd() call before the halts in > xen/arch/x86/domain.c:default_idle and I could do full suspend and > resume cycle once, but a reboot later I couldn''t resume anymore. HereDid you use the patch that Jan posted?> is the trace I get:That is a different issue. That is the kernel, while the outstanding issue in regards to suspend/resume is in the hypervisor.> > [ 142.322778] ACPI: Preparing to enter system sleep state S3 > [ 142.723286] PM: Saving platform NVS memory > [ 142.736453] Disabling non-boot CPUs ... > [ 142.851387] ------------[ cut here ]------------ > [ 142.851397] WARNING: at > /home/storage/src/ubuntu-precise/kernel/rcutree.c:1550 > rcu_do_batch.isra.41+0x5f/0x213()which ought to be fixed, but lets concentrate on one thing at a time. If you use the patch that Jan posted, does it work? And I presume you also have the two out-of-tree patches to make resume work in the dom0?> [ 142.851401] Hardware name: To Be Filled By O.E.M. > [ 142.851404] Modules linked in: blktap(O) ip6table_filter ip6_tables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 > xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp > iptable_filter ip_tables x_tables [last unloaded: cx25840] > [ 142.851466] Pid: 32, comm: migration/7 Tainted: G O > 3.5.0-14-i7 #18~precise1 > [ 142.851469] Call Trace: > [ 142.851473] <IRQ> [<ffffffff8106a9bd>] warn_slowpath_common+0x7e/0x96 > [ 142.851488] [<ffffffff8106a9ea>] warn_slowpath_null+0x15/0x17 > [ 142.851496] [<ffffffff810c6e87>] rcu_do_batch.isra.41+0x5f/0x213 > [ 142.851504] [<ffffffff810c687f>] ? > check_for_new_grace_period.isra.35+0x38/0x44 > [ 142.851512] [<ffffffff810c7101>] __rcu_process_callbacks+0xc6/0xee > [ 142.851520] [<ffffffff810c7149>] rcu_process_callbacks+0x20/0x3e > [ 142.851527] [<ffffffff81070c27>] __do_softirq+0x87/0x113 > [ 142.851536] [<ffffffff812e7b22>] ? __xen_evtchn_do_upcall+0x1a0/0x1dd > [ 142.851546] [<ffffffff8162f05c>] call_softirq+0x1c/0x30 > [ 142.851557] [<ffffffff810343a3>] do_softirq+0x41/0x7e > [ 142.851566] [<ffffffff81070e66>] irq_exit+0x3f/0x9a > [ 142.851573] [<ffffffff812e95c9>] xen_evtchn_do_upcall+0x2c/0x39 > [ 142.851580] [<ffffffff8162f0ae>] xen_do_hypervisor_callback+0x1e/0x30 > [ 142.851588] <EOI> [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 > [ 142.851601] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 > [ 142.851609] [<ffffffff8102eee9>] ? xen_force_evtchn_callback+0xd/0xf > [ 142.851618] [<ffffffff8102f552>] ? check_events+0x12/0x20 > [ 142.851627] [<ffffffff8102f53f>] ? xen_restore_fl_direct_reloc+0x4/0x4 > [ 142.851635] [<ffffffff810b6ff4>] ? arch_local_irq_restore+0xb/0xd > [ 142.851644] [<ffffffff810b73f4>] ? stop_machine_cpu_stop+0xc1/0xd3 > [ 142.851652] [<ffffffff810b7333>] ? queue_stop_cpus_work+0xb5/0xb5 > [ 142.851660] [<ffffffff810b7152>] ? cpu_stopper_thread+0xf7/0x187 > [ 142.851667] [<ffffffff8108a927>] ? finish_task_switch+0x82/0xc1 > [ 142.851676] [<ffffffff8162c50b>] ? __schedule+0x428/0x454 > [ 142.851685] [<ffffffff8162cff0>] ? _raw_spin_unlock_irqrestore+0x15/0x18 > [ 142.851693] [<ffffffff810b705b>] ? cpu_stop_signal_done+0x30/0x30 > [ 142.851701] [<ffffffff81082627>] ? kthread+0x86/0x8e > [ 142.851710] [<ffffffff8162ef64>] ? kernel_thread_helper+0x4/0x10 > [ 142.851718] [<ffffffff8162d338>] ? retint_restore_args+0x5/0x6 > [ 142.851726] [<ffffffff8162ef60>] ? gs_change+0x13/0x13 > [ 142.851736] ---[ end trace 3722a99bcc5ae37a ]--- > [ 144.257229] ACPI: Low-level resume complete > [ 144.257322] PM: Restoring platform NVS memory > > > -- > Javier Marcet <jmarcet@gmail.com> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
I went back to an old patch that had, since it was in this same function that you made reference to: http://markmail.org/message/qpnmiqzt5bngeejk I noticed that I was not seeing the "Breaking vcpu affinity" printk - so I tried to get that The change proposed in that thread seems to work around this pinning problem. However, I''m not sure that it is the "right thing" to be doing. Do you have any thoughts on this? On Tue, Sep 25, 2012 at 7:56 AM, Ben Guthro <ben@guthro.net> wrote:> > On Tue, Sep 25, 2012 at 3:00 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> >>> On 24.09.12 at 23:12, Ben Guthro <ben@guthro.net> wrote: >> > Here''s my "Big hammer" debugging patch. >> > >> > If I force the cpu to be scheduled on CPU0 when the appropriate cpu is >> not >> > online, I can resume properly. >> > >> > Clearly this is not the proper solution, and I''m sure the fix is subtle. >> > I''m not seeing it right now though. Perhaps tomorrow morning. >> > If you have any ideas, I''m happy to run tests then. >> >> I can''t see how the printk() you add in the patch would ever get >> reached with the other adjustment you do there. > > > Apologies. I failed to separate prior debugging in this patch from the > "big hammer" fix > > >> A debug build, >> as Keir suggested, would not only get the stack trace right, but >> would also result in the ASSERT() right after your first modification >> to _csched_cpu_pick() to actually do something (and likely trigger). >> > > Indeed. I was using non-debug builds for 2 reasons that, in hindsight may > not be the best of reasons. > 1. It was the default > 2. Mukesh''s kdb debugger requires debug to be off, which I was making use > of previously, and had not disabled. > > The stack from a debug build can be found below. > It did, indeed trigger the ASSERT, as you predicted. > > > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date > 2010-09-29 > [ 82.310025] ACPI: Low-level resume complete > [ 82.310025] PM: Restoring platform NVS memory > [ 82.310025] Enabling non-boot CPUs ... > [ 82.310025] installing Xen timer for CPU 1 > [ 82.310025] cpu 1 spinlock event irq 279 > (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' > failed at sched_credit.c:477 > (XEN) ----[ Xen-4.2.1-pre x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 1 > (XEN) RIP: e008:[<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 > (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor > (XEN) rax: 0000000000000001 rbx: 0000000000000004 rcx: 0000000000000004 > (XEN) rdx: 000000000000000f rsi: 0000000000000004 rdi: 0000000000000000 > (XEN) rbp: ffff8301355d7dd8 rsp: ffff8301355d7d08 r8: 0000000000000000 > (XEN) r9: 000000000000003e r10: ffff82c480231700 r11: 0000000000000246 > (XEN) r12: ffff82c480261b20 r13: 0000000000000001 r14: ffff82c480301a60 > (XEN) r15: ffff83013a542068 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 0000000131a05000 cr2: 0000000000000000 > (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff8301355d7d08: > (XEN) 0100000131a05000 ffff8301355d7d40 0000000000000082 > 0000000000000002 > (XEN) ffff8300bd503000 0000000000000001 0000000000000297 > ffff8301355d7d58 > (XEN) ffff82c480125499 ffff830138216000 ffff8301355d7d98 > 5400000000000002 > (XEN) 0000000000000286 ffff8301355d7d88 ffff82c480125499 > ffff830138216000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 > (XEN) ffff830134ca6a50 ffff83013a542068 ffff83013a542068 > 0000000000000001 > (XEN) ffff82c480301a60 ffff83013a542068 ffff8301355d7de8 > ffff82c48011a785 > (XEN) ffff8301355d7e58 ffff82c480123519 ffff82c480301a60 > ffff82c480301a60 > (XEN) ffff82c480301a60 ffff8300bd503000 0000000000503060 > 0000000000000246 > (XEN) ffff82c480127c31 ffff8300bd503000 ffff82c480301a60 > ffff82c4802ebd40 > (XEN) ffff83013a542068 ffff88003fc8e820 ffff8301355d7e88 > ffff82c4801237d3 > (XEN) fffffffffffffffe ffff8301355ca000 ffff8300bd503000 > 0000000000000000 > (XEN) ffff8301355d7ef8 ffff82c480106335 ffff8301355d7f18 > ffffffff810030e1 > (XEN) ffff8300bd503000 0000000000000000 ffff8301355d7f08 > ffff82c480185390 > (XEN) ffffffff81aafd32 ffff8300bd503000 0000000000000001 > 0000000000000000 > (XEN) 0000000000000000 ffff88003fc8e820 00007cfecaa280c7 > ffff82c480227348 > (XEN) ffffffff8100130a 0000000000000018 ffff88003fc8e820 > 0000000000000000 > (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 > ffff88003fc8bdc0 > (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff > 0000000000000000 > (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 > 0000000000000001 > (XEN) Xen call trace: > (XEN) [<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 > (XEN) [<ffff82c48011a785>] csched_cpu_pick+0xe/0x10 > (XEN) [<ffff82c480123519>] vcpu_migrate+0x19f/0x346 > (XEN) [<ffff82c4801237d3>] vcpu_force_reschedule+0xa4/0xb6 > (XEN) [<ffff82c480106335>] do_vcpu_op+0x2c9/0x452 > (XEN) [<ffff82c480227348>] syscall_enter+0xc8/0x122 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 1: > (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' > failed at sched_credit.c:477 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > >> Anyway, this might be connected to cpu_disable_scheduler() not >> having a counterpart to restore the affinity it broke for pinned >> domains (for non-pinned ones I believe this behavior is intentional, >> albeit not ideal). >> >> Jan >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, Sep 25, 2012 at 4:06 PM, Konrad Rzeszutek Wilk <konrad@kernel.org> wrote:>> >> >> xen-acpi-processor driver was merged to upstream Linux kernel v3.4. >> >> >> >> >> > >> >> > I see - so the advanced methods are not being used then >> > >> > You mean with the v3.5 kernel? I would think they would be used.. >> > >> >> > This same kernel does work with Xen-4.0.x though. Is there an assumption in >> >> > the code that ties Xen-4.2 to a newer kernel? >> >> >> >> I''m being bitten by this bug with a 3.5 kernel and Xen 4.2 >> > >> > Huh? How so? >> >> I''ve tried adding a wbinvd() call before the halts in >> xen/arch/x86/domain.c:default_idle and I could do full suspend and >> resume cycle once, but a reboot later I couldn''t resume anymore. Here > > Did you use the patch that Jan posted? >> is the trace I get: > > That is a different issue. That is the kernel, while the outstanding > issue in regards to suspend/resume is in the hypervisor. >> >> [ 142.322778] ACPI: Preparing to enter system sleep state S3 >> [ 142.723286] PM: Saving platform NVS memory >> [ 142.736453] Disabling non-boot CPUs ... >> [ 142.851387] ------------[ cut here ]------------ >> [ 142.851397] WARNING: at >> /home/storage/src/ubuntu-precise/kernel/rcutree.c:1550 >> rcu_do_batch.isra.41+0x5f/0x213() > > > which ought to be fixed, but lets concentrate on one thing at a time. > If you use the patch that Jan posted, does it work? And I presume > you also have the two out-of-tree patches to make resume work in the > dom0?I haven''t been able to see the actual patch, I only read a description of what it should do. Two possible places where a wbinvd() call should be added. On the first of these two places there were already calls to wbinvd() just before the halts. I added it to the other one, within xen/arch/x86/domain.c:default_idle and I could complete a suspend/resume cycle without the back trace but after rebooting I tried it again and it failed, more than once. I do have the other two out-of-tree patches applied, otherwise it would break way before than now.>> [ 142.851401] Hardware name: To Be Filled By O.E.M. >> [ 142.851404] Modules linked in: blktap(O) ip6table_filter ip6_tables >> ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 >> xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp >> iptable_filter ip_tables x_tables [last unloaded: cx25840] >> [ 142.851466] Pid: 32, comm: migration/7 Tainted: G O >> 3.5.0-14-i7 #18~precise1 >> [ 142.851469] Call Trace: >> [ 142.851473] <IRQ> [<ffffffff8106a9bd>] warn_slowpath_common+0x7e/0x96 >> [ 142.851488] [<ffffffff8106a9ea>] warn_slowpath_null+0x15/0x17 >> [ 142.851496] [<ffffffff810c6e87>] rcu_do_batch.isra.41+0x5f/0x213 >> [ 142.851504] [<ffffffff810c687f>] ? >> check_for_new_grace_period.isra.35+0x38/0x44 >> [ 142.851512] [<ffffffff810c7101>] __rcu_process_callbacks+0xc6/0xee >> [ 142.851520] [<ffffffff810c7149>] rcu_process_callbacks+0x20/0x3e >> [ 142.851527] [<ffffffff81070c27>] __do_softirq+0x87/0x113 >> [ 142.851536] [<ffffffff812e7b22>] ? __xen_evtchn_do_upcall+0x1a0/0x1dd >> [ 142.851546] [<ffffffff8162f05c>] call_softirq+0x1c/0x30 >> [ 142.851557] [<ffffffff810343a3>] do_softirq+0x41/0x7e >> [ 142.851566] [<ffffffff81070e66>] irq_exit+0x3f/0x9a >> [ 142.851573] [<ffffffff812e95c9>] xen_evtchn_do_upcall+0x2c/0x39 >> [ 142.851580] [<ffffffff8162f0ae>] xen_do_hypervisor_callback+0x1e/0x30 >> [ 142.851588] <EOI> [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 >> [ 142.851601] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000 >> [ 142.851609] [<ffffffff8102eee9>] ? xen_force_evtchn_callback+0xd/0xf >> [ 142.851618] [<ffffffff8102f552>] ? check_events+0x12/0x20 >> [ 142.851627] [<ffffffff8102f53f>] ? xen_restore_fl_direct_reloc+0x4/0x4 >> [ 142.851635] [<ffffffff810b6ff4>] ? arch_local_irq_restore+0xb/0xd >> [ 142.851644] [<ffffffff810b73f4>] ? stop_machine_cpu_stop+0xc1/0xd3 >> [ 142.851652] [<ffffffff810b7333>] ? queue_stop_cpus_work+0xb5/0xb5 >> [ 142.851660] [<ffffffff810b7152>] ? cpu_stopper_thread+0xf7/0x187 >> [ 142.851667] [<ffffffff8108a927>] ? finish_task_switch+0x82/0xc1 >> [ 142.851676] [<ffffffff8162c50b>] ? __schedule+0x428/0x454 >> [ 142.851685] [<ffffffff8162cff0>] ? _raw_spin_unlock_irqrestore+0x15/0x18 >> [ 142.851693] [<ffffffff810b705b>] ? cpu_stop_signal_done+0x30/0x30 >> [ 142.851701] [<ffffffff81082627>] ? kthread+0x86/0x8e >> [ 142.851710] [<ffffffff8162ef64>] ? kernel_thread_helper+0x4/0x10 >> [ 142.851718] [<ffffffff8162d338>] ? retint_restore_args+0x5/0x6 >> [ 142.851726] [<ffffffff8162ef60>] ? gs_change+0x13/0x13 >> [ 142.851736] ---[ end trace 3722a99bcc5ae37a ]--- >> [ 144.257229] ACPI: Low-level resume complete >> [ 144.257322] PM: Restoring platform NVS memory-- Javier Marcet <jmarcet@gmail.com>
This was introduced as part of a patch to avoid losing cpu and cpupool affinities/memberships across S3. Looks like it breaks some assumptions in the scheduler though, probably because all CPUs are not taken offline atomically, nor brought back online atomically. Hence some other running CPU can execute hypervisor code that observes VCPUs in this bad can¹t-run-anywhere state. I guess this is what is happening. I¹m not immediately sure of the best fix. :( -- Keir On 25/09/2012 15:22, "Ben Guthro" <ben@guthro.net> wrote:> I went back to an old patch that had, since it was in this same function that > you made reference to: > http://markmail.org/message/qpnmiqzt5bngeejk > > I noticed that I was not seeing the "Breaking vcpu affinity" printk - so I > tried to get that > > The change proposed in that thread seems to work around this pinning problem. > However, I''m not sure that it is the "right thing" to be doing. > > Do you have any thoughts on this? > > > On Tue, Sep 25, 2012 at 7:56 AM, Ben Guthro <ben@guthro.net> wrote: >> >> On Tue, Sep 25, 2012 at 3:00 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>> >>> On 24.09.12 at 23:12, Ben Guthro <ben@guthro.net> wrote: >>>> > Here''s my "Big hammer" debugging patch. >>>> > >>>> > If I force the cpu to be scheduled on CPU0 when the appropriate cpu is >>>> not >>>> > online, I can resume properly. >>>> > >>>> > Clearly this is not the proper solution, and I''m sure the fix is subtle. >>>> > I''m not seeing it right now though. Perhaps tomorrow morning. >>>> > If you have any ideas, I''m happy to run tests then. >>> >>> I can''t see how the printk() you add in the patch would ever get >>> reached with the other adjustment you do there. >> >> Apologies. I failed to separate prior debugging in this patch from the "big >> hammer" fix >> >>> A debug build, >>> as Keir suggested, would not only get the stack trace right, but >>> would also result in the ASSERT() right after your first modification >>> to _csched_cpu_pick() to actually do something (and likely trigger). >> >> Indeed. I was using non-debug builds for 2 reasons that, in hindsight may not >> be the best of reasons. >> 1. It was the default >> 2. Mukesh''s kdb debugger requires debug to be off, which I was making use of >> previously, and had not disabled. >> >> The stack from a debug build can be found below. >> It did, indeed trigger the ASSERT, as you predicted. >> >> >> (XEN) Finishing wakeup from ACPI S3 state. >> (XEN) Enabling non-boot CPUs ... >> (XEN) Booting processor 1/1 eip 8a000 >> (XEN) Initializing CPU#1 >> (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K >> (XEN) CPU: L2 cache: 3072K >> (XEN) CPU: Physical Processor ID: 0 >> (XEN) CPU: Processor Core ID: 1 >> (XEN) CMCI: CPU1 has no CMCI support >> (XEN) CPU1: Thermal monitoring enabled (TM2) >> (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 >> (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date >> 2010-09-29 >> [ 82.310025] ACPI: Low-level resume complete >> [ 82.310025] PM: Restoring platform NVS memory >> [ 82.310025] Enabling non-boot CPUs ... >> [ 82.310025] installing Xen timer for CPU 1 >> [ 82.310025] cpu 1 spinlock event irq 279 >> (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' >> failed at sched_credit.c:477 >> (XEN) ----[ Xen-4.2.1-pre x86_64 debug=y Tainted: C ]---- >> (XEN) CPU: 1 >> (XEN) RIP: e008:[<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 >> (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor >> (XEN) rax: 0000000000000001 rbx: 0000000000000004 rcx: 0000000000000004 >> (XEN) rdx: 000000000000000f rsi: 0000000000000004 rdi: 0000000000000000 >> (XEN) rbp: ffff8301355d7dd8 rsp: ffff8301355d7d08 r8: 0000000000000000 >> (XEN) r9: 000000000000003e r10: ffff82c480231700 r11: 0000000000000246 >> (XEN) r12: ffff82c480261b20 r13: 0000000000000001 r14: ffff82c480301a60 >> (XEN) r15: ffff83013a542068 cr0: 000000008005003b cr4: 00000000000026f0 >> (XEN) cr3: 0000000131a05000 cr2: 0000000000000000 >> (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 >> (XEN) Xen stack trace from rsp=ffff8301355d7d08: >> (XEN) 0100000131a05000 ffff8301355d7d40 0000000000000082 0000000000000002 >> (XEN) ffff8300bd503000 0000000000000001 0000000000000297 ffff8301355d7d58 >> (XEN) ffff82c480125499 ffff830138216000 ffff8301355d7d98 5400000000000002 >> (XEN) 0000000000000286 ffff8301355d7d88 ffff82c480125499 ffff830138216000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 >> (XEN) ffff830134ca6a50 ffff83013a542068 ffff83013a542068 0000000000000001 >> (XEN) ffff82c480301a60 ffff83013a542068 ffff8301355d7de8 ffff82c48011a785 >> (XEN) ffff8301355d7e58 ffff82c480123519 ffff82c480301a60 ffff82c480301a60 >> (XEN) ffff82c480301a60 ffff8300bd503000 0000000000503060 0000000000000246 >> (XEN) ffff82c480127c31 ffff8300bd503000 ffff82c480301a60 ffff82c4802ebd40 >> (XEN) ffff83013a542068 ffff88003fc8e820 ffff8301355d7e88 ffff82c4801237d3 >> (XEN) fffffffffffffffe ffff8301355ca000 ffff8300bd503000 0000000000000000 >> (XEN) ffff8301355d7ef8 ffff82c480106335 ffff8301355d7f18 ffffffff810030e1 >> (XEN) ffff8300bd503000 0000000000000000 ffff8301355d7f08 ffff82c480185390 >> (XEN) ffffffff81aafd32 ffff8300bd503000 0000000000000001 0000000000000000 >> (XEN) 0000000000000000 ffff88003fc8e820 00007cfecaa280c7 ffff82c480227348 >> (XEN) ffffffff8100130a 0000000000000018 ffff88003fc8e820 0000000000000000 >> (XEN) 0000000000000000 0000000000000001 ffff88003976fda0 ffff88003fc8bdc0 >> (XEN) 0000000000000246 ffff88003976fe60 00000000ffffffff 0000000000000000 >> (XEN) 0000000000000018 ffffffff8100130a 0000000000000000 0000000000000001 >> (XEN) Xen call trace: >> (XEN) [<ffff82c48011a35a>] _csched_cpu_pick+0x135/0x552 >> (XEN) [<ffff82c48011a785>] csched_cpu_pick+0xe/0x10 >> (XEN) [<ffff82c480123519>] vcpu_migrate+0x19f/0x346 >> (XEN) [<ffff82c4801237d3>] vcpu_force_reschedule+0xa4/0xb6 >> (XEN) [<ffff82c480106335>] do_vcpu_op+0x2c9/0x452 >> (XEN) [<ffff82c480227348>] syscall_enter+0xc8/0x122 >> (XEN) >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 1: >> (XEN) Assertion ''!cpumask_empty(&cpus) && cpumask_test_cpu(cpu, &cpus)'' >> failed at sched_credit.c:477 >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >>> >>> Anyway, this might be connected to cpu_disable_scheduler() not >>> having a counterpart to restore the affinity it broke for pinned >>> domains (for non-pinned ones I believe this behavior is intentional, >>> albeit not ideal). >>> >>> Jan >>> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 25.09.12 at 16:22, Ben Guthro <ben@guthro.net> wrote: > I went back to an old patch that had, since it was in this same function > that you made reference to: > http://markmail.org/message/qpnmiqzt5bngeejk > > I noticed that I was not seeing the "Breaking vcpu affinity" printk - so I > tried to get that > > The change proposed in that thread seems to work around this pinning > problem. > However, I''m not sure that it is the "right thing" to be doing.As said back then, the original change looks a little suspicious, and hence reverting it is certainly to be considered. However, I don''t see yet how this is connected to you having a problem only with pinned vCPU-s. But - with S3 working again (with the change here) or originally, did you ever check whether post-resume Dom0 vCPU-s would still be pinned? I suspect they aren''t, and hence addressing the crash may likely be achieved by properly restoring the pinning. Jan
>>> On 25.09.12 at 16:47, Javier Marcet <jmarcet@gmail.com> wrote: > I haven''t been able to see the actual patch, I only read a description > of what it should do. Two possible places where a wbinvd() call should > be added. On the first of these two places there were already calls to > wbinvd() just before the halts. I added it to the other one, within > xen/arch/x86/domain.c:default_idle and I could complete a > suspend/resume cycle without the back trace but after rebooting I > tried it again and it failed, more than once.default_idle() isn''t the right place, default_dead_idle() is. See http://xenbits.xen.org/hg/staging/xen-unstable.hg/rev/c8d65d91a6f2 Jan
On Tue, Sep 25, 2012 at 5:21 PM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 25.09.12 at 16:47, Javier Marcet <jmarcet@gmail.com> wrote: >> I haven''t been able to see the actual patch, I only read a description >> of what it should do. Two possible places where a wbinvd() call should >> be added. On the first of these two places there were already calls to >> wbinvd() just before the halts. I added it to the other one, within >> xen/arch/x86/domain.c:default_idle and I could complete a >> suspend/resume cycle without the back trace but after rebooting I >> tried it again and it failed, more than once. > > default_idle() isn''t the right place, default_dead_idle() is. > > See > http://xenbits.xen.org/hg/staging/xen-unstable.hg/rev/c8d65d91a6f2Thanks for the pointer Jan. This evening I''ll try it out and report back. -- Javier Marcet <jmarcet@gmail.com>
oops. I didn''t reply-all on this - re-replying. On Tue, Sep 25, 2012 at 11:10 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 25.09.12 at 16:22, Ben Guthro <ben@guthro.net> wrote: > > I went back to an old patch that had, since it was in this same function > > that you made reference to: > > http://markmail.org/message/qpnmiqzt5bngeejk > > > > I noticed that I was not seeing the "Breaking vcpu affinity" printk - so > I > > tried to get that > > > > The change proposed in that thread seems to work around this pinning > > problem. > > However, I''m not sure that it is the "right thing" to be doing. > > As said back then, the original change looks a little suspicious, > and hence reverting it is certainly to be considered. >Which is one of the reasons I brought it up again.> > However, I don''t see yet how this is connected to you having > a problem only with pinned vCPU-s. >Well - when the command line parameter dom0_vcpu_pin exists - it crashes without this change. It does not crash with it, or without the command line parameter. Perhaps they are unrelated.> > But - with S3 working again (with the change here) or originally, > did you ever check whether post-resume Dom0 vCPU-s would > still be pinned? I suspect they aren''t, and hence addressing the > crash may likely be achieved by properly restoring the pinning. >No, I don''t believe they are pinned after resume...so while I suppose it is not entirely correct, it is also better than crashing. (it is also the observed behavior in Xen-4.0.x)> > Jan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 25/09/2012 16:45, "Ben Guthro" <ben@guthro.net> wrote:> Which is one of the reasons I brought it up again. > >> >> However, I don''t see yet how this is connected to you having >> a problem only with pinned vCPU-s. > > Well - when the command line parameter dom0_vcpu_pin exists - > it crashes without this change. It does not crash with it, or without > the command line parameter. > > Perhaps they are unrelated.Of course they''re related. Dom0 kernel is fixing itself up after S3, while hypervisor is still bringing CPUs back online. It''s a race for one of dom0 kernel''s running VCPUs to do a hypercall that observes that one of its other VCPUs is unschedulable (because there are no currently-online CPUs in its cpu affinity mask!). -- Keir
On Tue, Sep 25, 2012 at 5:21 PM, Jan Beulich <JBeulich@suse.com> wrote:>>>> On 25.09.12 at 16:47, Javier Marcet <jmarcet@gmail.com> wrote: >> I haven''t been able to see the actual patch, I only read a description >> of what it should do. Two possible places where a wbinvd() call should >> be added. On the first of these two places there were already calls to >> wbinvd() just before the halts. I added it to the other one, within >> xen/arch/x86/domain.c:default_idle and I could complete a >> suspend/resume cycle without the back trace but after rebooting I >> tried it again and it failed, more than once. > > default_idle() isn''t the right place, default_dead_idle() is. > > See > http://xenbits.xen.org/hg/staging/xen-unstable.hg/rev/c8d65d91a6f2I''m sorry to say it doesn''t fix it on my system, i.e., kernel 3.5.3 and xen 4.2.1-pre (the head from git://xenbits.xen.org/xen.git#stable-4.2) It really was basically what I had tried yesterday, just with duplicated code. Adding the wbinvd() only in xen/arch/x86/domain.c was how I was able to suspend and resume once without the back trace. Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an instant reboot on resume. -- Javier Marcet <jmarcet@gmail.com>
You still haven''t said if you have the out-of-tree acpi-s3 patches from Konrad. Without these patches applied to linux - this is a non-starter. On Tue, Sep 25, 2012 at 3:55 PM, Javier Marcet <jmarcet@gmail.com> wrote:> On Tue, Sep 25, 2012 at 5:21 PM, Jan Beulich <JBeulich@suse.com> wrote: > > >>>> On 25.09.12 at 16:47, Javier Marcet <jmarcet@gmail.com> wrote: > >> I haven''t been able to see the actual patch, I only read a description > >> of what it should do. Two possible places where a wbinvd() call should > >> be added. On the first of these two places there were already calls to > >> wbinvd() just before the halts. I added it to the other one, within > >> xen/arch/x86/domain.c:default_idle and I could complete a > >> suspend/resume cycle without the back trace but after rebooting I > >> tried it again and it failed, more than once. > > > > default_idle() isn''t the right place, default_dead_idle() is. > > > > See > > http://xenbits.xen.org/hg/staging/xen-unstable.hg/rev/c8d65d91a6f2 > > I''m sorry to say it doesn''t fix it on my system, i.e., kernel 3.5.3 > and xen 4.2.1-pre (the head from > git://xenbits.xen.org/xen.git#stable-4.2) > > It really was basically what I had tried yesterday, just with duplicated > code. > > Adding the wbinvd() only in xen/arch/x86/domain.c was how I was able > to suspend and resume once without the back trace. > > Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an > instant reboot on resume. > > > -- > Javier Marcet <jmarcet@gmail.com> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, Sep 25, 2012 at 9:57 PM, Ben Guthro <ben@guthro.net> wrote:> You still haven''t said if you have the out-of-tree acpi-s3 patches from > Konrad. > Without these patches applied to linux - this is a non-starter.Yes, I _do_ have them applied. If you re-read my previous messages you''ll see I already said it before. Just to be clear, the patches are: - x86/acpi/sleep: Provide registration for acpi_suspend_lowlevel - xen/acpi/sleep: Register to the acpi_suspend_lowlevel a callback. Both from Konrad from a patch series posted last December on lklm. If there is any other patch I need for xen, then I don''t have it nor I know which patch. -- Javier Marcet <jmarcet@gmail.com>
>>> On 25.09.12 at 21:55, Javier Marcet <jmarcet@gmail.com> wrote: > Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an > instant reboot on resume.That addition can hardly be responsible for a reboot. Did you have "noreboot" (or "reboot=no") in place on the Xen command line? "sync_console"? And then again, for the other failure case, iirc it was the kernel that died, not the hypervisor, so the problem there isn''t directly related to the problems here I would guess. Jan
On Wed, Sep 26, 2012 at 9:17 AM, Jan Beulich <JBeulich@suse.com> wrote:>> Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an >> instant reboot on resume. > > That addition can hardly be responsible for a reboot. Did youJust like I could perform a full cycle without issues, the instant reboot might as well be another way the race ends.> have "noreboot" (or "reboot=no") in place on the Xen command > line? "sync_console"?Neither of them. I have used the sync_console parameter to check whether it changed anything but I removed it afterwards.> And then again, for the other failure case, iirc it was the kernel > that died, not the hypervisor, so the problem there isn''t directly > related to the problems here I would guess.All I know is that I''m using that same kernel without hypervisor, with lots of suspend/resume and not a single issue. With the two kernel patches from Konrad added I can also suspend and resume fine under the hypervisor but there is always a cpu which receives an irq while offline. -- Javier Marcet <jmarcet@gmail.com>
On Wed, Sep 26, 2012 at 9:17 AM, Jan Beulich <JBeulich@suse.com> wrote:>> Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an >> instant reboot on resume. > > That addition can hardly be responsible for a reboot. Did you > have "noreboot" (or "reboot=no") in place on the Xen command > line? "sync_console"? > > And then again, for the other failure case, iirc it was the kernel > that died, not the hypervisor, so the problem there isn''t directly > related to the problems here I would guess.By the way, I still don''t have a serial console ready. I should have it soon, one-two weeks tops. In the meantime, although I have very little time right now due to what I haven''t followed all the steps Ben has taken, I''m willing to try anything you deem worth it. -- Javier Marcet <jmarcet@gmail.com>
>>> On 25.09.12 at 13:56, Ben Guthro <ben@guthro.net> wrote: > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date = 2010-09-29Btw., I''m also puzzled by only seeing a ucode update message here for CPU1 - I just went through the logic again and can''t see why this would not also be done for CPU0. Could you make the pr_debug() in microcode_intel.c actually print something, so we can see whether at least collect_cpu_info() would get called for it, hopefully allowing to deduce what happens subsequently? Jan
On Wed, Sep 26, 2012 at 6:43 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 25.09.12 at 13:56, Ben Guthro <ben@guthro.net> wrote: > > (XEN) Finishing wakeup from ACPI S3 state. > > (XEN) Enabling non-boot CPUs ... > > (XEN) Booting processor 1/1 eip 8a000 > > (XEN) Initializing CPU#1 > > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > > (XEN) CPU: L2 cache: 3072K > > (XEN) CPU: Physical Processor ID: 0 > > (XEN) CPU: Processor Core ID: 1 > > (XEN) CMCI: CPU1 has no CMCI support > > (XEN) CPU1: Thermal monitoring enabled (TM2) > > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date > 2010-09-29 > > Btw., I''m also puzzled by only seeing a ucode update message > here for CPU1 - I just went through the logic again and can''t see > why this would not also be done for CPU0. Could you make the > pr_debug() in microcode_intel.c actually print something, so we > can see whether at least collect_cpu_info() would get called > for it, hopefully allowing to deduce what happens subsequently? >Happy to do so - but I may not be able to do this test until tomorrow. Ben _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 25.09.12 at 17:45, Ben Guthro <ben@guthro.net> wrote: > On Tue, Sep 25, 2012 at 11:10 AM, Jan Beulich <JBeulich@suse.com> wrote: >> >>> On 25.09.12 at 16:22, Ben Guthro <ben@guthro.net> wrote: >> > I went back to an old patch that had, since it was in this same function >> > that you made reference to: >> > http://markmail.org/message/qpnmiqzt5bngeejk >> > >> > I noticed that I was not seeing the "Breaking vcpu affinity" printk - so >> > I tried to get that >> > >> > The change proposed in that thread seems to work around this pinning >> > problem. >> > However, I''m not sure that it is the "right thing" to be doing. >> >> As said back then, the original change looks a little suspicious, >> and hence reverting it is certainly to be considered. > > Which is one of the reasons I brought it up again.So could you then do what I suggested back then - make a copy of the two involved CPU masks (*online and *vc->cpu_affinity) plus vc->domain->cpupool, vc->processor, and vc->vcpu_id into on-stack variables (ensuring they don''t get eliminated by the compiler, e.g. by adding a read access anywhere after the triggering ASSERT())? Then the source change (i.e. the data layout) available, either together with the xen-syms, or put them into a structure together with some magic ID to identify where in the stack dump the interesting bits are. Jan
On Wed, Sep 26, 2012 at 09:59:07AM +0200, Javier Marcet wrote:> On Wed, Sep 26, 2012 at 9:17 AM, Jan Beulich <JBeulich@suse.com> wrote: > > >> Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an > >> instant reboot on resume. > > > > That addition can hardly be responsible for a reboot. Did you > > Just like I could perform a full cycle without issues, the instant > reboot might as well be another way the race ends. > > > have "noreboot" (or "reboot=no") in place on the Xen command > > line? "sync_console"? > > Neither of them. I have used the sync_console parameter to check > whether it changed anything but I removed it afterwards. > > > And then again, for the other failure case, iirc it was the kernel > > that died, not the hypervisor, so the problem there isn''t directly > > related to the problems here I would guess. > > All I know is that I''m using that same kernel without hypervisor, with > lots of suspend/resume and not a single issue. > > With the two kernel patches from Konrad added I can also suspend and > resume fine under the hypervisor but there is always a cpu which > receives an irq while offline.And that error you get - can you reproduce it without going to suspend/resume? Meaning if you offline/online a VCPU in the guest by manipulating the /sys/../cpuX/online attribute? And actually - we should move that discussion to a completlty different thread as it has nothing to do with the hypervisor. That is a PV kernel issue.
On Wed, Sep 26, 2012 at 2:43 PM, Konrad Rzeszutek Wilk <konrad@kernel.org>wrote:> >> Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an > > >> instant reboot on resume. > > > > > > That addition can hardly be responsible for a reboot. Did you > > > > Just like I could perform a full cycle without issues, the instant > > reboot might as well be another way the race ends. > > > > > have "noreboot" (or "reboot=no") in place on the Xen command > > > line? "sync_console"? > > > > Neither of them. I have used the sync_console parameter to check > > whether it changed anything but I removed it afterwards. > > > > > And then again, for the other failure case, iirc it was the kernel > > > that died, not the hypervisor, so the problem there isn''t directly > > > related to the problems here I would guess. > > > > All I know is that I''m using that same kernel without hypervisor, with > > lots of suspend/resume and not a single issue. > > > > With the two kernel patches from Konrad added I can also suspend and > > resume fine under the hypervisor but there is always a cpu which > > receives an irq while offline. > > And that error you get - can you reproduce it without going to > suspend/resume? Meaning if you offline/online a VCPU in the > guest by manipulating the /sys/../cpuX/online attribute? > > And actually - we should move that discussion to a completlty different > thread as it has nothing to do with the hypervisor. That is a PV kernel > issue. >Konrad, this is on a dom0 with no guest running. -- Javier Marcet <jmarcet@gmail.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Yes. Pvops PV dom0 kernel. It is really more appropriate to start a new thread, than having 2 disparate conversations in this one. It makes it confusing to follow. On Sep 26, 2012, at 10:15 AM, Javier Marcet <jmarcet@gmail.com> wrote: On Wed, Sep 26, 2012 at 2:43 PM, Konrad Rzeszutek Wilk <konrad@kernel.org>wrote:> >> Adding the additional call in xen/arch/x86/acpi/cpu_idle.c causes an > > >> instant reboot on resume. > > > > > > That addition can hardly be responsible for a reboot. Did you > > > > Just like I could perform a full cycle without issues, the instant > > reboot might as well be another way the race ends. > > > > > have "noreboot" (or "reboot=no") in place on the Xen command > > > line? "sync_console"? > > > > Neither of them. I have used the sync_console parameter to check > > whether it changed anything but I removed it afterwards. > > > > > And then again, for the other failure case, iirc it was the kernel > > > that died, not the hypervisor, so the problem there isn''t directly > > > related to the problems here I would guess. > > > > All I know is that I''m using that same kernel without hypervisor, with > > lots of suspend/resume and not a single issue. > > > > With the two kernel patches from Konrad added I can also suspend and > > resume fine under the hypervisor but there is always a cpu which > > receives an irq while offline. > > And that error you get - can you reproduce it without going to > suspend/resume? Meaning if you offline/online a VCPU in the > guest by manipulating the /sys/../cpuX/online attribute? > > And actually - we should move that discussion to a completlty different > thread as it has nothing to do with the hypervisor. That is a PV kernel > issue. >Konrad, this is on a dom0 with no guest running. -- Javier Marcet <jmarcet@gmail.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Sep 26, 2012 at 4:26 PM, Ben Guthro <ben.guthro@gmail.com> wrote: Yes. Pvops PV dom0 kernel.> > It is really more appropriate to start a new thread, than having 2 > disparate conversations in this one. It makes it confusing to follow. >OK. I started it a few days ago, with the back trace et all but I have got no response so far. I''ll try what you suggest and continue on the other thread. -- Javier Marcet <jmarcet@gmail.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Sep 26, 2012 at 6:43 AM, Jan Beulich <JBeulich@suse.com> wrote:> > Btw., I''m also puzzled by only seeing a ucode update message > here for CPU1 - I just went through the logic again and can''t see > why this would not also be done for CPU0. Could you make the > pr_debug() in microcode_intel.c actually print something, so we > can see whether at least collect_cpu_info() would get called > for it, hopefully allowing to deduce what happens subsequently? > >I put a pr_debug at the top of collect_cpu_info - and from the trace below - it appears everything is as it should be, dom0 and Xen serial output get interspersed there for a few lines, but the microcode is, in fact being loaded for both CPUs. [ 36.787452] ACPI: Preparing to enter system sleep state S3 [ 37.240118] PM: Saving platform NVS memory [ 37.313160] Disabling non-boot CPUs ... (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Breaking vcpu affinity for domain 0 vcpu 1 (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Thermal monitoring enabled (TM2) (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: collect_cpu_info for CPU0 (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c (XEN) Enabling non-boot CPUs ... (XEN) Booting processor 1/1 eip 8a000 (XEN) Initializing CPU#1 (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K (XEN) CPU: L2 cache: 3072K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU: Processor Core ID: 1 (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Thermal monitoring enabled (TM2) (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 (XEN) microcode: collect_cpu_info for CPU1 [ 37.420030] A(XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c CPI: Low-level r(XEN) microcode: CPU1 found a matching microcode update with version 0x60f (current=0x60c) esume complete (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date 2010-09-29 [ 37.420030] PM: Restoring platform NVS memory (XEN) microcode: collect_cpu_info for CPU0 (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c (XEN) microcode: collect_cpu_info for CPU1 (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f [ 37.431008] Enabling non-boot CPUs ... [ 37.434851] installing Xen timer for CPU 1 [ 37.439031] cpu 1 spinlock event irq 279 [ 37.327214] Disabled fast string operations [ 37.458040] CPU1 is up [ 37.465417] ACPI: Waking up from system sleep state S3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 26.09.12 at 20:21, Ben Guthro <ben@guthro.net> wrote: > I put a pr_debug at the top of collect_cpu_info - and from the trace below > - it appears everything is as it should be, > dom0 and Xen serial output get interspersed there for a few lines, but the > microcode is, in fact being loaded for both CPUs.Definitely not:> [ 36.787452] ACPI: Preparing to enter system sleep state S3 > [ 37.240118] PM: Saving platform NVS memory > [ 37.313160] Disabling non-boot CPUs ... > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Breaking vcpu affinity for domain 0 vcpu 1 > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 > extended MCE MSR 0 > (XEN) CMCI: CPU0 has no CMCI support > (XEN) CPU0: Thermal monitoring enabled (TM2) > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) microcode: collect_cpu_info for CPU0 > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60cRev 0x60c is being found installed, but no action is being done.> (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: collect_cpu_info for CPU1 > [ 37.420030] A(XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c > CPI: Low-level r(XEN) microcode: CPU1 found a matching microcode update > with version 0x60f (current=0x60c) > esume complete > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date = 2010-09-29Whereas here an update is being done.> [ 37.420030] PM: Restoring platform NVS memory > (XEN) microcode: collect_cpu_info for CPU0 > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c > (XEN) microcode: collect_cpu_info for CPU1 > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60fAnd in the end we see that both cores run on different revisions. The mixup of Xen and Dom0 messages puzzles me too - in my understanding, all APs should be brought back up before domains get unpaused. Is that perhaps part of your problem with pinned vCPU-s? Or is the mixup not indicative of things actually happening in parallel? Jan> [ 37.431008] Enabling non-boot CPUs ... > [ 37.434851] installing Xen timer for CPU 1 > [ 37.439031] cpu 1 spinlock event irq 279 > [ 37.327214] Disabled fast string operations > [ 37.458040] CPU1 is up > [ 37.465417] ACPI: Waking up from system sleep state S3
On 27/09/2012 08:38, "Jan Beulich" <JBeulich@suse.com> wrote:>> [ 37.420030] PM: Restoring platform NVS memory >> (XEN) microcode: collect_cpu_info for CPU0 >> (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c >> (XEN) microcode: collect_cpu_info for CPU1 >> (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f > > And in the end we see that both cores run on different revisions. > > The mixup of Xen and Dom0 messages puzzles me too - in my > understanding, all APs should be brought back up before > domains get unpaused.Ah, of course! This makes a nonsense of my explanation of the observed crash in _csched_cpu_pick -- dom0 cannot be running while Aps are still being brought back online, because domains are not unpaused until all CPUs are back online. So it''s a mystery again. :) -- Keir> Is that perhaps part of your problem > with pinned vCPU-s? Or is the mixup not indicative of things > actually happening in parallel? > > Jan > >> [ 37.431008] Enabling non-boot CPUs ... >> [ 37.434851] installing Xen timer for CPU 1 >> [ 37.439031] cpu 1 spinlock event irq 279 >> [ 37.327214] Disabled fast string operations >> [ 37.458040] CPU1 is up >> [ 37.465417] ACPI: Waking up from system sleep state S3 > > >
On Thu, Sep 27, 2012 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote:> > > > [ 37.420030] PM: Restoring platform NVS memory > > (XEN) microcode: collect_cpu_info for CPU0 > > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c > > (XEN) microcode: collect_cpu_info for CPU1 > > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f > > And in the end we see that both cores run on different revisions. > >very interesting. I had overlooked that.> The mixup of Xen and Dom0 messages puzzles me too - in my > understanding, all APs should be brought back up before > domains get unpaused. Is that perhaps part of your problem > with pinned vCPU-s? Or is the mixup not indicative of things > actually happening in parallel? >It seems like it may be related, but I''m unsure how. When I run without the dom0_vcpu_pin option, I don''t see the collect_cpu_info messages, nor do I see the microcode revision printout for CPU0 nor do I see the dom0 / Xen messages mised up. [ 41.567458] ACPI: Preparing to enter system sleep state S3 [ 42.020120] PM: Saving platform NVS memory [ 42.092643] Disabling non-boot CPUs ... (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Thermal monitoring enabled (TM2) (XEN) Finishing wakeup from ACPI S3 state. (XEN) Enabling non-boot CPUs ... (XEN) Booting processor 1/1 eip 8a000 (XEN) Initializing CPU#1 (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K (XEN) CPU: L2 cache: 3072K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU: Processor Core ID: 1 (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Thermal monitoring enabled (TM2) (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date 2010-09-29 [ 42.200055] ACPI: Low-level resume complete [ 42.200055] PM: Restoring platform NVS memory [ 42.200055] Enabling non-boot CPUs ... Any pointers are appreciated. /btg _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 27.09.12 at 14:12, Ben Guthro <ben@guthro.net> wrote: > On Thu, Sep 27, 2012 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote: >> > (XEN) microcode: collect_cpu_info for CPU0 >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c >> > (XEN) microcode: collect_cpu_info for CPU1 >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f >> >> And in the end we see that both cores run on different revisions. >> >> > very interesting. I had overlooked that. > > >> The mixup of Xen and Dom0 messages puzzles me too - in my >> understanding, all APs should be brought back up before >> domains get unpaused. Is that perhaps part of your problem >> with pinned vCPU-s? Or is the mixup not indicative of things >> actually happening in parallel? >> > > It seems like it may be related, but I''m unsure how. > When I run without the dom0_vcpu_pin option, I don''t see the > collect_cpu_info messages, > nor do I see the microcode revision printout for CPU0That''s more likely because of either running different (unpatched) code, or at a lower log level, because ...> nor do I see the dom0 / Xen messages mised up. > > [ 41.567458] ACPI: Preparing to enter system sleep state S3 > [ 42.020120] PM: Saving platform NVS memory > [ 42.092643] Disabling non-boot CPUs ... > (XEN) Preparing system for ACPI S3 state. > (XEN) Disabling non-boot CPUs ... > (XEN) Entering ACPI S3 state. > (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended MCE MSR 0 > (XEN) CMCI: CPU0 has no CMCI support > (XEN) CPU0: Thermal monitoring enabled (TM2) > (XEN) Finishing wakeup from ACPI S3 state. > (XEN) Enabling non-boot CPUs ... > (XEN) Booting processor 1/1 eip 8a000 > (XEN) Initializing CPU#1 > (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K > (XEN) CPU: L2 cache: 3072K > (XEN) CPU: Physical Processor ID: 0 > (XEN) CPU: Processor Core ID: 1 > (XEN) CMCI: CPU1 has no CMCI support > (XEN) CPU1: Thermal monitoring enabled (TM2) > (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 > (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date = 2010-09-29... you can''t get here without going through collect_cpu_info().> [ 42.200055] ACPI: Low-level resume complete > [ 42.200055] PM: Restoring platform NVS memory > [ 42.200055] Enabling non-boot CPUs ... > > > Any pointers are appreciated.Same for me (regarding the mixup of messages). Maybe widening the console_{start,end}_sync() window from right after freezing domains until right before thawing them could make clear whether this is an artifact. Jan
>>> On 27.09.12 at 14:12, Ben Guthro <ben@guthro.net> wrote: > On Thu, Sep 27, 2012 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote: >> > (XEN) microcode: collect_cpu_info for CPU0 >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c >> > (XEN) microcode: collect_cpu_info for CPU1 >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f >> >> And in the end we see that both cores run on different revisions. >> >> > very interesting. I had overlooked that.Could you give the below patch a try? Jan x86/ucode: fix Intel case of resume handling on boot CPU Checking the stored version doesn''t tell us anything about the need to apply the update (during resume, what is stored doesn''t necessarily match what is loaded). Note that the check can be removed altogether because once switched to use what was read from the CPU (uci->cpu_sig.rev, as used in the subsequent pr_debug()), it would become redundant with the checks that lead to microcode_update_match() returning the indication that an update should be applied. Note further that this was not an issue on APs since they start with uci->mc.mc_intel being NULL. Signed-off-by: Jan Beulich <jbeulich@suse.com> --- a/xen/arch/x86/microcode_intel.c +++ b/xen/arch/x86/microcode_intel.c @@ -261,8 +261,6 @@ static int get_matching_microcode(const } return 0; find: - if ( uci->mc.mc_intel && uci->mc.mc_intel->hdr.rev >= mc_header->rev ) - return 0; pr_debug("microcode: CPU%d found a matching microcode update with" " version %#x (current=%#x)\n", cpu, mc_header->rev, uci->cpu_sig.rev);
ACK. I now see microcode updates for both CPUs. [ 47.612719] Disabling non-boot CPUs ... (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Breaking vcpu affinity for domain 0 vcpu 1 (XEN) Entering ACPI S3 state. (XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Thermal monitoring enabled (TM2) (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x60c to 0x60f, date 2010-09-29 (XEN) Enabling non-boot CPUs ... (XEN) Booting processor 1/1 eip 8a000 (XEN) Initializing CPU#1 (XEN) CPU: L1 I cache: 32K, L1 D cache: 32K (XEN) CPU: L2 cache: 3072K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU: Processor Core ID: 1 (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Thermal monitoring enabled (TM2) (XEN) CPU1: Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz stepping 06 (XEN) microcode: CPU1 updated from revision 0x60c to 0x60f, date 2010-09-29 [ 47.720054] ACPI: Low-level resume complete On Thu, Sep 27, 2012 at 11:25 AM, Jan Beulich <JBeulich@suse.com> wrote:> >>> On 27.09.12 at 14:12, Ben Guthro <ben@guthro.net> wrote: > > On Thu, Sep 27, 2012 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote: > >> > (XEN) microcode: collect_cpu_info for CPU0 > >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60c > >> > (XEN) microcode: collect_cpu_info for CPU1 > >> > (XEN) microcode: collect_cpu_info : sig=0x10676, pf=0x80, rev=0x60f > >> > >> And in the end we see that both cores run on different revisions. > >> > >> > > very interesting. I had overlooked that. > > Could you give the below patch a try? > > Jan > > x86/ucode: fix Intel case of resume handling on boot CPU > > Checking the stored version doesn''t tell us anything about the need to > apply the update (during resume, what is stored doesn''t necessarily > match what is loaded). > > Note that the check can be removed altogether because once switched to > use what was read from the CPU (uci->cpu_sig.rev, as used in the > subsequent pr_debug()), it would become redundant with the checks that > lead to microcode_update_match() returning the indication that an > update should be applied. > > Note further that this was not an issue on APs since they start with > uci->mc.mc_intel being NULL. > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > > --- a/xen/arch/x86/microcode_intel.c > +++ b/xen/arch/x86/microcode_intel.c > @@ -261,8 +261,6 @@ static int get_matching_microcode(const > } > return 0; > find: > - if ( uci->mc.mc_intel && uci->mc.mc_intel->hdr.rev >= mc_header->rev ) > - return 0; > pr_debug("microcode: CPU%d found a matching microcode update with" > " version %#x (current=%#x)\n", > cpu, mc_header->rev, uci->cpu_sig.rev); > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Jan Beulich
2012-Sep-27 15:59 UTC
[PATCH] x86/ucode: fix Intel case of resume handling on boot CPU
Checking the stored version doesn''t tell us anything about the need to
apply the update (during resume, what is stored doesn''t necessarily
match what is loaded).
Note that the check can be removed altogether because once switched to
use what was read from the CPU (uci->cpu_sig.rev, as used in the
subsequent pr_debug()), it would become redundant with the checks that
lead to microcode_update_match() returning the indication that an
update should be applied.
Note further that this was not an issue on APs since they start with
uci->mc.mc_intel being NULL.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Tested-by: Ben Guthro <ben@guthro.net>
--- a/xen/arch/x86/microcode_intel.c
+++ b/xen/arch/x86/microcode_intel.c
@@ -261,8 +261,6 @@ static int get_matching_microcode(const
}
return 0;
find:
- if ( uci->mc.mc_intel && uci->mc.mc_intel->hdr.rev >=
mc_header->rev )
- return 0;
pr_debug("microcode: CPU%d found a matching microcode update
with"
" version %#x (current=%#x)\n",
cpu, mc_header->rev, uci->cpu_sig.rev);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Keir Fraser
2012-Sep-27 16:06 UTC
Re: [PATCH] x86/ucode: fix Intel case of resume handling on boot CPU
On 27/09/2012 16:59, "Jan Beulich" <JBeulich@suse.com> wrote:> Checking the stored version doesn''t tell us anything about the need to > apply the update (during resume, what is stored doesn''t necessarily > match what is loaded). > > Note that the check can be removed altogether because once switched to > use what was read from the CPU (uci->cpu_sig.rev, as used in the > subsequent pr_debug()), it would become redundant with the checks that > lead to microcode_update_match() returning the indication that an > update should be applied. > > Note further that this was not an issue on APs since they start with > uci->mc.mc_intel being NULL. > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > Tested-by: Ben Guthro <ben@guthro.net>Acked-by: Keir Fraser <keir@xen.org>> --- a/xen/arch/x86/microcode_intel.c > +++ b/xen/arch/x86/microcode_intel.c > @@ -261,8 +261,6 @@ static int get_matching_microcode(const > } > return 0; > find: > - if ( uci->mc.mc_intel && uci->mc.mc_intel->hdr.rev >= mc_header->rev ) > - return 0; > pr_debug("microcode: CPU%d found a matching microcode update with" > " version %#x (current=%#x)\n", > cpu, mc_header->rev, uci->cpu_sig.rev); > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel