I''m working on getting IOMMU support enabled
for AMD64 in Xen.
The current aperture.c code in dom0 requires
several calls to read_pci_config() to find
the AMD northbridge. These calls are causing
my system to crash.
read_pci_config() is defined in
linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h
as
static inline u32 read_pci_config(u8 bus, u8 slot, u8 func, u8 offset)
{
u32 v;
outl(0x80000000 | (bus<<16) | (slot<<11) | (func<<8) |
offset,
0xcf8);
v = inl(0xcfc);
if (v != 0xffffffff)
PDprintk("%x reading 4 from %x: %x\n", slot, offset, v);
return v;
}
This a specialized function intended for PCI access
early in the boot before the PCI subsystem starts
working.
How can I find the AMD northbridge(s) early in the
dom0 boot sequence without crashing the system?
-Mark Langsdorf
AMD, Inc.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Langsdorf, Mark wrote:> I''m working on getting IOMMU support enabled > for AMD64 in Xen. > > The current aperture.c code in dom0 requires > several calls to read_pci_config() to find > the AMD northbridge. These calls are causing > my system to crash.Can you be more specific (output from serial connection, for example)? The outl instruction causes #GP, and the emulator in Xen handle it. So, it''s possible the emulator has a bug?> > read_pci_config() is defined in > linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h > as > > static inline u32 read_pci_config(u8 bus, u8 slot, u8 func, u8 offset) > { > u32 v; > outl(0x80000000 | (bus<<16) | (slot<<11) | (func<<8) | offset, > 0xcf8); > v = inl(0xcfc); > if (v != 0xffffffff) > PDprintk("%x reading 4 from %x: %x\n", slot, offset, v); > return v; > } > > This a specialized function intended for PCI access > early in the boot before the PCI subsystem starts > working. > > How can I find the AMD northbridge(s) early in the > dom0 boot sequence without crashing the system? > > -Mark Langsdorf > AMD, Inc. >Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Langsdorf, Mark wrote: > > I''m working on getting IOMMU support enabled > > for AMD64 in Xen. > > > > The current aperture.c code in dom0 requires > > several calls to read_pci_config() to find > > the AMD northbridge. These calls are causing > > my system to crash. > > Can you be more specific (output from serial connection, for > example)? The outl instruction causes #GP, and the emulator > in Xen handle it. So, it''s possible the emulator has a bug?The serial console is pretty broken on AMD64. I was hoping to not have to debug the serial console in order to fix the IOMMU problem. Here''s the last page or two of messages on my screen: (all lines prefixed with (XEN)) I hand copied it but all the values are correct. *** LOADING DOMAIN 0 *** Xen-ELF header found: ''GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L OADER=generic'' PHYSICAL MEMORY ARRANGEMENT: Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be allocated) VIRTUAL MEMORY ARRANGEMNET: Loaded kernel: ffffffff80100000->ffffffff8063e086 Init. ramdisk: ffffffff8063f000->ffffffff8063f000 Phys-Mach map: ffffffff8063f000->ffffffff8063f000 Start info: ffffffff8063f000->ffffffff80840000 Page tables: ffffffff80840000->ffffffff80849000 Boot stack: ffffffff80849000->ffffffff8084a000 TOTAL: ffffffff80000000->ffffffff80c00000 ENTRY ADDRESS: ffffffff80100000 Scrubbing Free RAM: .................................done. *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to Xen). Domain 0 (vcpu#0) crashed on cpu#0: CPU: 0 RIP: e033:[<fffffff80600cc2>] RFLAGS: 000000000000286 CONTEXT: guest rax: 000000008000c300 rbx: 000000000000c000 rcx: 0000000000007fff rdx: 0000000000000cf8 rsi: 00000000000007e5 rdi: ffffffff804d2be0 rbp: 0000000000000018 rsp: ffffffff805f5ea0 r8: 000000o000000034 r9: 0000000000000000 r10: 00000000ffffffff r11: 0000000o00000000 r12: 0000000000000cf8 r13: 0000000000000cfc r14: 00000000o00d2000 r15: ffffffff804ca200 cr0: 000000008005003b cr3: 0000000007101000 Guest stack trace from rsp=ffffffff805f5ea0: (guest stack omitted because it''s basically the above registers) Domain 0 shutdown: rebooting machine. Is there a way to turn on more debug information inside the hypervisor itself? -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2005-09-29 at 15:26 -0500, Langsdorf, Mark wrote:> > Langsdorf, Mark wrote: > > > I''m working on getting IOMMU support enabled > > > for AMD64 in Xen. > > > > > > The current aperture.c code in dom0 requires > > > several calls to read_pci_config() to find > > > the AMD northbridge. These calls are causing > > > my system to crash. > > > > Can you be more specific (output from serial connection, for > > example)? The outl instruction causes #GP, and the emulator > > in Xen handle it. So, it''s possible the emulator has a bug? > > The serial console is pretty broken on AMD64. I was > hoping to not have to debug the serial console in > order to fix the IOMMU problem.Do you mean on your machine? Serial console works fine on my AMD64 boxes. Try this in your grub.conf: title Xen kernel /boot/xen.gz dom0_mem=512000 acpi=off com1=19200,8n1,0x3f8,4 console=com1 verbose=y sync_console module /boot/vmlinuz-2.6.12-xen0 root=/dev/sda1 debug=y module /boot/initrd-2.6.12-xen0.img> > Here''s the last page or two of messages on my screen: > (all lines prefixed with (XEN)) I hand copied it but > all the values are correct. > > *** LOADING DOMAIN 0 *** > Xen-ELF header found: > ''GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L > OADER=generic'' > PHYSICAL MEMORY ARRANGEMENT: > Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be > allocated) > VIRTUAL MEMORY ARRANGEMNET: > Loaded kernel: ffffffff80100000->ffffffff8063e086 > Init. ramdisk: ffffffff8063f000->ffffffff8063f000 > Phys-Mach map: ffffffff8063f000->ffffffff8063f000 > Start info: ffffffff8063f000->ffffffff80840000 > Page tables: ffffffff80840000->ffffffff80849000 > Boot stack: ffffffff80849000->ffffffff8084a000 > TOTAL: ffffffff80000000->ffffffff80c00000 > ENTRY ADDRESS: ffffffff80100000 > Scrubbing Free RAM: .................................done. > *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to > Xen). > Domain 0 (vcpu#0) crashed on cpu#0: > CPU: 0 > RIP: e033:[<fffffff80600cc2>] > RFLAGS: 000000000000286 CONTEXT: guest > rax: 000000008000c300 rbx: 000000000000c000 rcx: 0000000000007fff > rdx: 0000000000000cf8 rsi: 00000000000007e5 rdi: ffffffff804d2be0 > rbp: 0000000000000018 rsp: ffffffff805f5ea0 r8: 000000o000000034 > r9: 0000000000000000 r10: 00000000ffffffff r11: 0000000o00000000 > r12: 0000000000000cf8 r13: 0000000000000cfc r14: 00000000o00d2000 > r15: ffffffff804ca200 cr0: 000000008005003b cr3: 0000000007101000 > Guest stack trace from rsp=ffffffff805f5ea0: > (guest stack omitted because it''s basically the above registers) > Domain 0 shutdown: rebooting machine. > > Is there a way to turn on more debug information inside the hypervisor > itself? > > -Mark Langsdorf > AMD, Inc. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- Jerone Young IBM Linux Technology Center jyoung5@us.ibm.com 512-838-1157 (T/L: 678-1157) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2005-09-29 at 15:26 -0500, Langsdorf, Mark wrote:> > > Langsdorf, Mark wrote: > > > > I''m working on getting IOMMU support enabled > > > > for AMD64 in Xen. > > > > > > > > The current aperture.c code in dom0 requires > > > > several calls to read_pci_config() to find > > > > the AMD northbridge. These calls are causing > > > > my system to crash. > > > > > > Can you be more specific (output from serial connection, for > > > example)? The outl instruction causes #GP, and the emulator > > > in Xen handle it. So, it''s possible the emulator has a bug? > > > > The serial console is pretty broken on AMD64. I was > > hoping to not have to debug the serial console in > > order to fix the IOMMU problem. > > Do you mean on your machine? Serial console works fine on my > AMD64 boxes. > > Try this in your grub.conf: > > title Xen > kernel /boot/xen.gz dom0_mem=512000 acpi=off > com1=19200,8n1,0x3f8,4 console=com1 verbose=y sync_console > module /boot/vmlinuz-2.6.12-xen0 root=/dev/sda1 debug=y > module /boot/initrd-2.6.12-xen0.imgThanks! That works. -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Langsdorf, Mark wrote:>> Langsdorf, Mark wrote: >>> I''m working on getting IOMMU support enabled >>> for AMD64 in Xen. >>> >>> The current aperture.c code in dom0 requires >>> several calls to read_pci_config() to find >>> the AMD northbridge. These calls are causing >>> my system to crash. >> >> Can you be more specific (output from serial connection, for >> example)? The outl instruction causes #GP, and the emulator >> in Xen handle it. So, it''s possible the emulator has a bug? > > The serial console is pretty broken on AMD64. I was > hoping to not have to debug the serial console in > order to fix the IOMMU problem. > > Here''s the last page or two of messages on my screen: > (all lines prefixed with (XEN)) I hand copied it but > all the values are correct. > > *** LOADING DOMAIN 0 *** > Xen-ELF header found: >''GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L> OADER=generic'' > PHYSICAL MEMORY ARRANGEMENT: > Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be > allocated)I see you assigned just 1GB memory for dom0?> VIRTUAL MEMORY ARRANGEMNET: > Loaded kernel: ffffffff80100000->ffffffff8063e086 > Init. ramdisk: ffffffff8063f000->ffffffff8063f000 > Phys-Mach map: ffffffff8063f000->ffffffff8063f000 > Start info: ffffffff8063f000->ffffffff80840000 > Page tables: ffffffff80840000->ffffffff80849000 > Boot stack: ffffffff80849000->ffffffff8084a000 > TOTAL: ffffffff80000000->ffffffff80c00000 > ENTRY ADDRESS: ffffffff80100000 > Scrubbing Free RAM: .................................done.Looks like you are getting the failure at very early initialization stage. In this case, I think it''s better to add prink in Xen, for example, in emulate_privileged_op(struct cpu_user_regs *regs) to check it''s failing or not.> *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to > Xen). > Domain 0 (vcpu#0) crashed on cpu#0: > CPU: 0 > RIP: e033:[<fffffff80600cc2>] > RFLAGS: 000000000000286 CONTEXT: guest > rax: 000000008000c300 rbx: 000000000000c000 rcx: 0000000000007fff > rdx: 0000000000000cf8 rsi: 00000000000007e5 rdi: ffffffff804d2be0 > rbp: 0000000000000018 rsp: ffffffff805f5ea0 r8: 000000o000000034 > r9: 0000000000000000 r10: 00000000ffffffff r11: 0000000o00000000 > r12: 0000000000000cf8 r13: 0000000000000cfc r14: 00000000o00d2000 > r15: ffffffff804ca200 cr0: 000000008005003b cr3: 0000000007101000 > Guest stack trace from rsp=ffffffff805f5ea0: > (guest stack omitted because it''s basically the above registers) > Domain 0 shutdown: rebooting machine. > > Is there a way to turn on more debug information inside the hypervisor > itself? > > -Mark Langsdorf > AMD, Inc.Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Sep 2005, at 19:41, Langsdorf, Mark wrote:> The current aperture.c code in dom0 requires > several calls to read_pci_config() to find > the AMD northbridge. These calls are causing > my system to crash. > > read_pci_config() is defined in > linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h > asYou probably need to set your IOPL to 1. There''s a physdevop hypercall for doing that. We execute it during boot but you may be accessing teh northbridge earlier than that. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > The current aperture.c code in dom0 requires > > several calls to read_pci_config() to find > > the AMD northbridge. These calls are causing > > my system to crash. > > > > read_pci_config() is defined in > > linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h > > as > > You probably need to set your IOPL to 1. There''s a physdevop > hypercall for doing that. We execute it during boot but > you may be accessing the northbridge earlier than that.I''d just figured out the IOPL was too low and was about to ask about that. Thanks for the tip. -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel