Jeremy Fitzhardinge
2009-Dec-10 00:19 UTC
[Xen-devel] Hang on boot when using IPMI 2.0 SoL console
With a current xen-unstable, I''m seeing a hang on boot just after
bringing up all the cpus:
0 __ __ _____ ____ _ _ _
\ \/ /___ _ __ |___ / | ___| _ _ _ __ ___| |_ __ _| |__ | | ___
\ // _ \ ''_ \ |_ \ |___ \ __| | | | ''_ \/ __| __/ _` |
''_ \| |/ _ \
/ \ __/ | | | ___) | ___) |__| |_| | | | \__ \ || (_| | |_) | | __/
/_/\_\___|_| |_| |____(_)____/ \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
(XEN) Xen version 3.5-unstable (jeremy@) (gcc version 4.4.2 20091027 (Red Hat
4.4.2-7) (GCC) ) Wed Dec 9 14:36:31 PST 2009
(XEN) Latest ChangeSet: Wed Dec 09 10:59:31 2009 +0000 20605:8f304c003af4
(XEN) Command line: com2=115200,8n1,0x3e8,5 console=com2,vga
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 0 MBR signatures
(XEN) Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009d800 (usable)
(XEN) 000000000009d800 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf790000 (usable)
(XEN) 00000000bf79e000 - 00000000bf7a0000 type 9
(XEN) 00000000bf7a0000 - 00000000bf7ae000 (ACPI data)
(XEN) 00000000bf7ae000 - 00000000bf7d0000 (ACPI NVS)
(XEN) 00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN) 00000000bf7ed000 - 00000000c0000000 (reserved)
(XEN) 00000000e0000000 - 00000000f0000000 (reserved)
(XEN) 00000000fed20000 - 00000000fed40000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 0000000100000000 - 0000000140000000 (usable)
(XEN) ACPI: RSDP 000F9FA0, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF7A0100, 008C (r1 SMCI 20090914 MSFT 97)
(XEN) ACPI: FACP BF7A0290, 00F4 (r3 091409 FACP2006 20090914 MSFT 97)
(XEN) ACPI: DSDT BF7A05E0, 6BB4 (r1 10605 10605000 0 INTL 20051117)
(XEN) ACPI: FACS BF7AE000, 0040
(XEN) ACPI: APIC BF7A0390, 008C (r1 091409 APIC2006 20090914 MSFT 97)
(XEN) ACPI: MCFG BF7A0420, 003C (r1 091409 OEMMCFG 20090914 MSFT 97)
(XEN) ACPI: SLIC BF7A0460, 0176 (r1 SMCI 20090914 MSFT 97)
(XEN) ACPI: OEMB BF7AE040, 0073 (r1 091409 OEMB2006 20090914 MSFT 97)
(XEN) ACPI: HPET BF7AA5E0, 0038 (r1 091409 OEMHPET 20090914 MSFT 97)
(XEN) ACPI: GSCI BF7AE0C0, 2024 (r1 091409 GMCHSCI 20090914 MSFT 97)
(XEN) ACPI: DMAR BF7B00F0, 0090 (r1 AMI OEMDMAR 1 MSFT 97)
(XEN) ACPI: SSDT BF7B10A0, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117)
(XEN) ACPI: EINJ BF7AA620, 0130 (r1 AMIER AMI_EINJ 20090914 MSFT 97)
(XEN) ACPI: BERT BF7AA7B0, 0030 (r1 AMIER AMI_BERT 20090914 MSFT 97)
(XEN) ACPI: ERST BF7AA7E0, 01B0 (r1 AMIER AMI_ERST 20090914 MSFT 97)
(XEN) ACPI: HEST BF7AA990, 00A8 (r1 AMIER ABC_HEST 20090914 MSFT 97)
(XEN) System RAM: 4051MB (4149168kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000140000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI: wakeup_vec[bf7ae00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
(XEN) ACPI: IOAPIC (id[0x07] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 7, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2400.058 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Thermal monitoring enabled (TM1)
(XEN) I/O virtualisation disabled
(XEN) CPU0: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping 05
(XEN) Booting processor 1/2 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Thermal monitoring enabled (TM1)
(XEN) CPU1: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping 05
(XEN) Booting processor 2/4 eip 8c000
(XEN) Initializing CPU#2
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 2
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Thermal monitoring enabled (TM1)
(XEN) CPU2: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping 05
(XEN) Booting processor 3/6 eip 8c000
(XEN) Initializing CPU#3
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 3
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Thermal monitoring enabled (TM1)
(XEN) CPU3: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping 05
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer is 14.318MHz HPET
(
The console is an IPMI SoL console. It appears as com3 (0x3e8 irq 5),
so I''ve faked out com2 to point to it, since Xen seems to deny the
existence of ports beyond com2.
Linux works fine with ttyS2 as its serial console.
I wonder if the (presumably emulated) serial device has some quirk that
causes it to disagree with Xen''s serial console here. Is there
anything
significant about that particular moment? Does it try to start using
interrupts there?
When I add "sync_console", it gets past here as expected and loads
dom0:
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) Brought up 4 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU2 resumed
(XEN) microcode.c:73:d32767 microcode: CPU3 resumed
(XEN) HPET: 8 timers in total, 8 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x80d000
(XEN) elf_parse_binary: phdr: paddr=0x180d000 memsz=0xe5638
(XEN) elf_parse_binary: phdr: paddr=0x18f3000 memsz=0x888
(XEN) elf_parse_binary: phdr: paddr=0x18f4000 memsz=0x15860
(XEN) elf_parse_binary: phdr: paddr=0x1909860 memsz=0x1a27a0
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1aac000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81909a60
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81009000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0x0
(XEN) virt_offset = 0xffffffff80000000
(XEN) virt_kstart = 0xffffffff81000000
(XEN) virt_kend = 0xffffffff81aac000
(XEN) virt_entry = 0xffffffff81909a60
(XEN) p2m_base = 0xffffffffffffffff
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1aac000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000138000000->000000013c000000 (984452 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff81aac000
(XEN) Init. ramdisk: ffffffff81aac000->ffffffff82778e00
(XEN) Phys-Mach map: ffffffff82779000->ffffffff82f1bc20
(XEN) Start info: ffffffff82f1c000->ffffffff82f1c4b4
(XEN) Page tables: ffffffff82f1d000->ffffffff82f38000
(XEN) Boot stack: ffffffff82f38000->ffffffff82f39000
(XEN) TOTAL: ffffffff80000000->ffffffff83000000
(XEN) ENTRY ADDRESS: ffffffff81909a60
(XEN) Dom0 has maximum 4 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff8180d000
(XEN) elf_load_binary: phdr 1 at 0xffffffff8180d000 -> 0xffffffff818f2638
(XEN) elf_load_binary: phdr 2 at 0xffffffff818f3000 -> 0xffffffff818f3888
(XEN) elf_load_binary: phdr 3 at 0xffffffff818f4000 -> 0xffffffff81909860
(XEN) elf_load_binary: phdr 4 at 0xffffffff81909860 -> 0xffffffff81981000
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **********************************************
(XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) ******* This option is intended to aid debugging of Xen by ensuring
(XEN) ******* that all output is synchronously delivered on the serial line.
(XEN) ******* However it can introduce SIGNIFICANT latencies and affect
(XEN) ******* timekeeping. It is NOT recommended for production use!
(XEN) **********************************************
(XEN) 3... 2... 1...
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen)
(XEN) Freed 156kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
[...]
J
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-10 01:38 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 12/09/09 16:19, Jeremy Fitzhardinge wrote:> When I add "sync_console", it gets past here as expected and loads dom0:But input doesn''t work... J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-10 01:42 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 12/09/09 17:38, Jeremy Fitzhardinge wrote:> On 12/09/09 16:19, Jeremy Fitzhardinge wrote: >> When I add "sync_console", it gets past here as expected and loads dom0: > > But input doesn''t work...But making irq=0 does work. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2009-Dec-10 05:57 UTC
[Xen-devel] Re: Hang on boot when using IPMI 2.0 SoL console
On 10/12/2009 00:19, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:> The console is an IPMI SoL console. It appears as com3 (0x3e8 irq 5), > so I''ve faked out com2 to point to it, since Xen seems to deny the > existence of ports beyond com2. > > Linux works fine with ttyS2 as its serial console. > > I wonder if the (presumably emulated) serial device has some quirk that > causes it to disagree with Xen''s serial console here. Is there anything > significant about that particular moment? Does it try to start using > interrupts there?Yes, that''s probably it. Asynchronous interrupt-based serial I/O gets turned on soon after the time subsystem is initialised. Also try adding boot parameter ''sync_console''. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2009-Dec-10 05:59 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 10/12/2009 01:42, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:> On 12/09/09 17:38, Jeremy Fitzhardinge wrote: >> On 12/09/09 16:19, Jeremy Fitzhardinge wrote: >>> When I add "sync_console", it gets past here as expected and loads dom0: >> >> But input doesn''t work... > > But making irq=0 does work.So the interrupt line isn''t working for your com3, for some reason. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Lyon
2009-Dec-10 08:10 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On Thu, Dec 10, 2009 at 5:59 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> On 10/12/2009 01:42, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote: > >> On 12/09/09 17:38, Jeremy Fitzhardinge wrote: >>> On 12/09/09 16:19, Jeremy Fitzhardinge wrote: >>>> When I add "sync_console", it gets past here as expected and loads dom0: >>> >>> But input doesn''t work... >> >> But making irq=0 does work. > > So the interrupt line isn''t working for your com3, for some reason.I had to use irq=0 with intel amt sol on a dell optiplex 755, I found this documentation which suggested it http://manpages.ubuntu.com/manpages/jaunty/man7/amt-howto.7.html Andy> > -- Keir > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-10 18:21 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 12/09/09 21:59, Keir Fraser wrote:> On 10/12/2009 01:42, "Jeremy Fitzhardinge"<jeremy@goop.org> wrote: > > >> On 12/09/09 17:38, Jeremy Fitzhardinge wrote: >> >>> On 12/09/09 16:19, Jeremy Fitzhardinge wrote: >>> >>>> When I add "sync_console", it gets past here as expected and loads dom0: >>>> >>> But input doesn''t work... >>> >> But making irq=0 does work. >> > So the interrupt line isn''t working for your com3, for some reason. >Not for Xen, at least. In Linux it appears fine: 5: 0 915 0 554 IO-APIC-edge serial J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-10 18:25 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 12/10/09 00:10, Andrew Lyon wrote:>> So the interrupt line isn''t working for your com3, for some reason. >> > I had to use irq=0 with intel amt sol on a dell optiplex 755, I found > this documentation which suggested it > http://manpages.ubuntu.com/manpages/jaunty/man7/amt-howto.7.html >Yeah, same deal. Disable interrupts for Xen, but not Linux. Konrad''s patch looks like it''s a good workaround, but it would be nice to work out how Xen and Linux''s serial drivers differ here (though I''m guessing its really a bug in the underlying 16550 emulation). J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-10 19:34 UTC
Re: [Xen-devel] Hang on boot when using IPMI 2.0 SoL console
On 12/09/09 21:59, Keir Fraser wrote:> So the interrupt line isn''t working for your com3, for some reason. >The Linux driver has two SoL-specific hacks (below). But I don''t think either of these are coming into play in this case; the first only applies with a serial break, and the second is only enabled if it detects specific Intel PCI devices (which don''t exist on this particular machine; the SoL is part of the integrated IPMI BMC, not an AMT ethernet device). Presumably the Linux driver is managing to work around the problems - either accidentally or deliberately - in some more subtle way. J static void receive_chars(struct uart_8250_port *up, unsigned int *status) { struct tty_struct *tty = up->port.info->port.tty; unsigned char ch, lsr = *status; int max_count = 256; char flag; do { if (likely(lsr& UART_LSR_DR)) ch = serial_inp(up, UART_RX); else /* * Intel 82571 has a Serial Over Lan device that will * set UART_LSR_BI without setting UART_LSR_DR when * it receives a break. To avoid reading from the * receive buffer without UART_LSR_DR bit set, we * just force the read character to be 0 */ ch = 0; and static int serial8250_startup(struct uart_port *port) { ... /* Serial over Lan (SoL) hack: Intel 8257x Gigabit ethernet chips have a 16550 emulation, to be used for Serial Over Lan. Those chips take a longer time than a normal serial device to signalize that a transmission data was queued. Due to that, the above test generally fails. One solution would be to delay the reading of iir. However, this is not reliable, since the timeout is variable. So, let''s just don''t test if we receive TX irq. This way, we''ll never enable UART_BUG_TXEN. */ if (up->port.flags& UPF_NO_TXEN_TEST) goto dont_test_tx_en; ... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel