I''m still working on getting IOMMU support for Opterons under Xen. During the boot sequence, Dom0 is finding the aperture through the AGP controller at address 0xe8000000 like it should. During AGP controller initialization, though, it fails to reserve the address space because the necessary pages are PageReserved in the mem_map. I''m not sure where the pages are being reserved, or even if mem_map for dom0 is completely valid, or if I should be handling this through the hypervisor. Any comments or suggestions? -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m still working on getting IOMMU support for Opterons under Xen. > > During the boot sequence, Dom0 is finding the aperture > through the AGP controller at address 0xe8000000 like it > should. During AGP controller initialization, though, it > fails to reserve the address space because the necessary > pages are PageReserved in the mem_map.Trying to looks a bus (machine) address up in dom0''s mem_map (pseudo-physical) won''t yield anything sensible. Presumably the driver isn''t expecting to find memory behind the aperture? Is it just trying to reserve the bus address range for purposes of ensuring that other PCI devices don''t get allocated to it? Ian> I''m not sure where the pages are being reserved, or even if > mem_map for dom0 is completely valid, or if I should be > handling this through the hypervisor. Any comments or suggestions? > > -Mark Langsdorf > AMD, Inc. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > I''m still working on getting IOMMU support for Opterons under Xen. > > > > During the boot sequence, Dom0 is finding the aperture > > through the AGP controller at address 0xe8000000 like it > > should. During AGP controller initialization, though, it > > fails to reserve the address space because the necessary > > pages are PageReserved in the mem_map. > > Trying to looks a bus (machine) address up in dom0''s mem_map > (pseudo-physical) won''t yield anything sensible.So how do I do this?> Presumably the driver isn''t expecting to find memory behind > the aperture? Is it just trying to reserve the bus address > range for purposes of ensuring that other PCI devices don''t > get allocated to it?The aperture is an address space that is not supported by DRAM. Writes to the aperture get shifted to other physical addresses through the magic of the GART. So no, the driver doesn''t expect memory to actually be there and I''m just trying to reserve the bus address range to make sure nothing else gets allocated to it. Normally, the address range would be initially reserved by the BIOS and the e820 map would reflect that, but I suppose that isn''t happening with Xen. -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > I''m still working on getting IOMMU support for Opterons under Xen. > > > > > > During the boot sequence, Dom0 is finding the aperture > through the > > > AGP controller at address 0xe8000000 like it should. During AGP > > > controller initialization, though, it fails to reserve > the address > > > space because the necessary pages are PageReserved in the mem_map. > > > > Trying to looks a bus (machine) address up in dom0''s mem_map > > (pseudo-physical) won''t yield anything sensible. > > So how do I do this?Don''t! Bus addresses should never be looked up in mem_map. The mem_map array is indexed by pseudo-physical address, and refers just to the DRAM the domain has. If the existing driver is trying to lookup a io bus address in mem_map, that''s a bug and you''ll need to fix it.> > Presumably the driver isn''t expecting to find memory behind the > > aperture? Is it just trying to reserve the bus address range for > > purposes of ensuring that other PCI devices don''t get > allocated to it? > > The aperture is an address space that is not supported by > DRAM. Writes to the aperture get shifted to other physical > addresses through the magic of the GART.Exactly -- the adress shouldn''t be looked up in mem_map. I suspect what the driver should be doing is just bumping pci_mem_start to avoid the aperture clashing with other pci resources. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''m still working on getting IOMMU support for Opterons under Xen.> > > > AGP controller initialization, fails to reserve > > > > the GART address space because the necessary > > > > pages are PageReserved in the mem_map. > > > > > > Trying to looks a bus (machine) address up in > > > dom0''s mem_map (pseudo-physical) won''t yield > > > anything sensible. > > > > So how do I do this? > > Don''t! Bus addresses should never be looked up in mem_map. > The mem_map array is indexed by pseudo-physical address, and > refers just to the DRAM the domain has. > > If the existing driver is trying to lookup a io bus address > in mem_map, that''s a bug and you''ll need to fix it.It''s trying to make sure that the page range isn''t being used by something else, and for no reason that I can determine, it''s failing. e820_reserve_resources() is creating a "reserved" memory gap from d000000 to ff700000 or so. When I request the e0000000 address to put the 256 MB GART there, I get collision with something.> > The aperture is an address space that is not supported by > > DRAM. Writes to the aperture get shifted to other physical > > addresses through the magic of the GART. > > Exactly -- the adress shouldn''t be looked up in mem_map. > > I suspect what the driver should be doing is just bumping > pci_mem_start to avoid the aperture clashing with other > pci resources.I''ve tried that, but it doesn''t seem to help. At some point I need to be able to remap the actual, physical e0000000 address range into something locally addressable. ioremap_cache is failing, though that might be because of the page reserve collision above. -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> At some point I need to be able to remap the actual, physical > e0000000 address range into something locally addressable. > ioremap_cache is failing, though that might be because of the > page reserve collision above.I haven''t looked at the code, but why would you want to map the GART aperture into kernel virtual address space? The CPU shouldn''t need to access stuff through the remapped region -- its PCI DMAs that should be using it. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > At some point I need to be able to remap the actual, physical > > e0000000 address range into something locally addressable. > > ioremap_cache is failing, though that might be because of the > > page reserve collision above. > > I haven''t looked at the code, but why would you want to map > the GART aperture into kernel virtual address space? The CPU > shouldn''t need to access stuff through the remapped region -- > its PCI DMAs that should be using it.You''re right, I don''t really want to do that, but I got inadvertantly sidetracked into thinking I needed AGP support working. Fortunately, I don''t. However, if I don''t have AGP support working, I need to program the Northbridge registers with the physical address of the GATT (see init_k8_gatt() in arch/xen/x86_64/kernel/pci-gart.c). Just doing a __pa() obviously doesn''t return meaningful data, which is to be expected. How should dom0 go about getting it? -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> You''re right, I don''t really want to do that, but I got > inadvertantly sidetracked into thinking I needed AGP support > working. Fortunately, I don''t. > > However, if I don''t have AGP support working, I need to > program the Northbridge registers with the physical address > of the GATT (see init_k8_gatt() in arch/xen/x86_64/kernel/pci-gart.c). > Just doing a __pa() obviously doesn''t return meaningful data, > which is to be expected. How should dom0 go about getting it?You need to allocate the gatt mapping table using alloc_gatt_pages (dma_alloc_coherent) rather than get_free_pages. You then want to use virt_to_gart on the address returned. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > You''re right, I don''t really want to do that, but I got > > inadvertantly sidetracked into thinking I needed AGP support > > working. Fortunately, I don''t. > > > > However, if I don''t have AGP support working, I need to > > program the Northbridge registers with the physical address > > of the GATT (see init_k8_gatt() in > arch/xen/x86_64/kernel/pci-gart.c). > > Just doing a __pa() obviously doesn''t return meaningful data, > > which is to be expected. How should dom0 go about getting it? > > You need to allocate the gatt mapping table using alloc_gatt_pages > (dma_alloc_coherent) rather than get_free_pages. > > You then want to use virt_to_gart on the address returned.Are you sure? I''ve been trying this for days, and it isn''t working. Here''s the original code in question, from pci-gart.c gatt = (void *) __get_free_pages(GFP_KERNEL, get_order(gatt_size)); for_all_nb(dev) { u32 gatt_reg; gatt_reg = __pa(gatt) >> 12; gatt_reg <= 4; pci_write_config_dword(dev, 0x98, gatt_reg); } I''ve changed it to: gatt = (void *) alloc_gatt_pages(get_order(gatt_size)); for_all_nb(dev) { u32 gatt_reg; gatt_reg = phys_to_gart(virt_to_phys(gatt)) >> 12; gatt_reg <= 4; pci_write_config_dword(dev, 0x98, gatt_reg); } I don''t think the northbridge is looking for the gart value of the gatt, even in a virtualized environment. -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > However, if I don''t have AGP support working, I need to > program the > > > Northbridge registers with the physical address of the GATT (see > > > init_k8_gatt() in > > arch/xen/x86_64/kernel/pci-gart.c). > > > Just doing a __pa() obviously doesn''t return meaningful data, > > > which is to be expected. How should dom0 go about getting it? > > > > You need to allocate the gatt mapping table using alloc_gatt_pages > > (dma_alloc_coherent) rather than get_free_pages. > > > > You then want to use virt_to_gart on the address returned. > > Are you sure? I''ve been trying this for days, and it > isn''t working.Never mind. There were a bunch of references to virt_to_phys() that should have been virt_to_bus(). Moving on to the next problem... -Mark Langsdorf AMD, Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel