Folks -- I''m looking through a PAE guest''s address map and I''m trying to make sense out of it. Is there some documentation about these address regions: 0xc0000000 - 0xc0800000: 8MB x/8x 0xc0000000 0xfde00000 - 0xfde04000: 16KB x/8x 0xfde00000 0xfdfac000 - 0xfdfae000: 8KB NOT READABLE 0xfdfb4000 - 0xfdfc0000: 48KB NOT READABLE 0xfdfe4000 - 0xfdfe6000: 8KB NOT READABLE 0xfdfec000 - 0xfdff0000: 16KB x/8x 0xfdfec000 0xfdff4000 - 0xfe000000: 48KB NOT READABLE Is the above map documented in the general case somewhere? And, where are they setup? Inside Xen? Inside Xend? And, in what files? It looks like Xen builds a (set of) self referential page table pointers (i.e., page table entries that are pointers to the page table itself. Where is that documented? Last, where''s the mfn to pfn array? -- Randy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Randy Thelen wrote:> Folks -- > > I''m looking through a PAE guest''s address map and I''m trying to > make sense out of it. Is there some documentation about these > address regions:I meant to annotate the following table a bit more:> 0xc0000000 - 0xc0800000: 8MB <-- My kernel > 0xfde00000 - 0xfde04000: 16KB <-- pfn to mfn ptes > 0xfdfac000 - 0xfdfae000: 8KB NOT READABLE > 0xfdfb4000 - 0xfdfc0000: 48KB NOT READABLE > 0xfdfe4000 - 0xfdfe6000: 8KB NOT READABLE > 0xfdfec000 - 0xfdff0000: 16KB <-- ??? > 0xfdff4000 - 0xfe000000: 48KB NOT READABLEFor example, the second entry (that which I think is the pfn to mfn ptes): (gdb) x/32x xen_phys_machine 0xc028c000: 0x000d16ee 0x000d160e 0xc028c008: 0x000d161b 0x000d161e 0xc028c010: 0x000d1620 0x000fb1fa 0xc028c018: 0x000d1603 0x000d1604 (gdb) x/32g 0xfde00000 0xfde00000: 0x00000000d16ee023 0x00000000d160e023 0xfde00010: 0x00000000d161b023 0x00000000d161e023 0xfde00020: 0x00000000d1620023 0x00000000fb1fa023 0xfde00030: 0x00000000d1603023 0x00000000d1604023 (gdb) The third (readable) entry appears to be the recursive page table mapping region: # ./xenctrl 1 -l3 0xfb046 dump 0xfdfec000 16384 0xfdfec000: -- page of zeros -- Address space 0x0 - 0x4000.0000 0xfdfed000: -- page of zeros -- Address space 0x4 - 0x8000.0000 0xfdfee000: -- page of zeros -- Address space 0x8 - 0xC000.0000 Following are the entries for my kernel''s Level 2 Page Directory pages 0xfdfef000: 0xfb041067 0x00000000 0xfb040067 0x00000000 0xfdfef010: 0xfb03f067 0x00000000 0xfb03e067 0x00000000 I.e.: (gdb) x/4x xen_start_info->pt_base 0xc03bb000: 0x00000000fb045001 0x00000000fb044001 0xc03bb010: 0x00000000fb043001 0x00000000fb042001 Below are two page entries to something I can''t read: 0xfdfefd60: 0xfbc000e1 0x00000000 0xfba000e1 0x00000000 Below are mappings to: ??? 0xfdfefda0: 0x018001e3 0x00000000 0x01a001e3 0x00000000 0xfdfefdb0: 0x01c001e3 0x00000000 0x01e001e3 0x00000000 0xfdfefdc0: 0x020001e3 0x00000000 0x022001e3 0x00000000 0xfdfefdd0: 0x024001e3 0x00000000 0x026001e3 0x00000000 0xfdfefde0: 0x028001e3 0x00000000 0x02a001e3 0x00000000 0xfdfefdf0: 0x02c001e3 0x00000000 0x02e001e3 0x00000000 Below are mappings to the same region above (that I can''t read): 0xfdfeff20: 0xfbc001e3 0x00000000 0xfba001e3 0x00000000 Again, more mappings to my page directories: 0xfdfeff60: 0xfb045063 0x00000000 0xfb044063 0x00000000 0xfdfeff70: 0xfb043063 0x00000000 0xfb042063 0x00000000 Mappings to something, but I know not what: 0xfdfeffa0: 0x00224063 0x00000000 0x00225063 0x00000000 0xfdfeffb0: 0x00226063 0x00000000 0x00227063 0x00000000 0xfdfeffc0: 0x00211063 0x00000000 0x00212063 0x00000000 0xfdfeffd0: 0x00213063 0x00000000 0x00214063 0x00000000 0xfdfeffe0: 0x00215063 0x00000000 0x00216063 0x00000000 0xfdfefff0: 0x00217063 0x00000000 0x00bfe063 0x00000000 # But, I''m looking for specific documentation regarding these regions. -- Randy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Randy Thelen wrote:> Randy Thelen wrote: > >> Folks -- >> >> I''m looking through a PAE guest''s address map and I''m trying to >> make sense out of it. Is there some documentation about these >> address regions: > > I meant to annotate the following table a bit more: > >> 0xc0000000 - 0xc0800000: 8MB <-- My kernel >> 0xfde00000 - 0xfde04000: 16KB <-- pfn to mfn ptes >> 0xfdfac000 - 0xfdfae000: 8KB NOT READABLE >> 0xfdfb4000 - 0xfdfc0000: 48KB NOT READABLE >> 0xfdfe4000 - 0xfdfe6000: 8KB NOT READABLE >> 0xfdfec000 - 0xfdff0000: 16KB <-- ??? >> 0xfdff4000 - 0xfe000000: 48KB NOT READABLEAlright, I now believe that in a PAE guest (with a PAE Xen, i.e., we''re using real MMU tables), the recursive virtual mapping base address is 0xFD80_0000. And, the basic virtual address to virtual address of the PTE is: 0xFD80_0000 | (VA & 0xFFFF_F000) >> 0. There''s a large drawing on the white board behind me to explain this. I apologize for not actually carrying it into words in this email, but my wife and child are at home, hungry, waiting for me so we can eat dinner. I''ll elaborate on the mechanics of the above function at a later date. It took me a while, but I was able to perform a series of transactions that properly mapped virtual addresses to the Page Table Entries that map them. If you''re interested, follow this: First, I modified Xend (privately) to always allocate 4 L2 page tables. I now believe I can revert that (local) change. At any rate, in Xend when I''m allocating page table pages, I create an L3 page and four L2 pages, consecutively: (gdb) x/4x xen_start_info->pt_base 0xc03bb000: 0x00000000fb045001 0x00000000fb044001 0xc03bb010: 0x00000000fb043001 0x00000000fb042001 (gdb) x/8x 0xc03bc000 0xc03bc000: 0x0000000000000000 0x0000000000000000 0xc03bc010: 0x0000000000000000 0x0000000000000000 0xc03bc020: 0x0000000000000000 0x0000000000000000 0xc03bc030: 0x0000000000000000 0x0000000000000000 (gdb) x/8x 0xc03bd000 0xc03bd000: 0x0000000000000000 0x0000000000000000 0xc03bd010: 0x0000000000000000 0x0000000000000000 0xc03bd020: 0x0000000000000000 0x0000000000000000 0xc03bd030: 0x0000000000000000 0x0000000000000000 (gdb) x/8x 0xc03be000 0xc03be000: 0x0000000000000000 0x0000000000000000 0xc03be010: 0x0000000000000000 0x0000000000000000 0xc03be020: 0x0000000000000000 0x0000000000000000 0xc03be030: 0x0000000000000000 0x0000000000000000 (gdb) x/8x 0xc03bf000 0xc03bf000: 0x00000000fb041067 0x00000000fb040067 0xc03bf010: 0x00000000fb03f067 0x00000000fb03e067 0xc03bf020: 0x0000000000000000 0x0000000000000000 0xc03bf030: 0x0000000000000000 0x0000000000000000 At address 0xc03bb000 you see the four mappings for the four L2 page tables. At address 0xc03bc000 .. 0xc03befff you (almost) see three pages of zeros. There are no valid virtual addresses below 0xC000_0000. My kernel is virtually mapped at address 0xC000_0000 and consume 8MB, resulting in four L1 pages of page table entries. Fine. Now, let''s see if we can find the PTE that maps the virtual address (0xC03BF000) for the L0 page of interest. In other words, address 0xC03BF000 is the virtual address of a Level 0 block (a data block). We can find the PTE which maps that block by applying the above algorithm on the virtual address. You know, I''m not sure you all want this level of detail. I''m going home for dinner. -- Randy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>Folks -- > >I''m looking through a PAE guest''s address map and I''m trying to make >sense out of it. Is there some documentation about these address >regions: > >0xc0000000 - 0xc0800000: 8MB x/8x 0xc0000000 >0xfde00000 - 0xfde04000: 16KB x/8x 0xfde00000 >0xfdfac000 - 0xfdfae000: 8KB NOT READABLE >0xfdfb4000 - 0xfdfc0000: 48KB NOT READABLE >0xfdfe4000 - 0xfdfe6000: 8KB NOT READABLE >0xfdfec000 - 0xfdff0000: 16KB x/8x 0xfdfec000 >0xfdff4000 - 0xfe000000: 48KB NOT READABLE > >Is the above map documented in the general case somewhere?Take a look at xen/include/asm-x86/config.h ; this defines the overall memory layout for x86_32, PAE & x86_64 systems, as well as including some comments that should help you. cheers, S. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Steven Hand wrote:> Take a look at xen/include/asm-x86/config.h ; this defines the > overall memory layout for x86_32, PAE & x86_64 systems, as well > as including some comments that should help you.Thanks, Steven. I hadn''t seen that file before. However, I''m still confused (or frustrated, I''m not entirely sure which!). Here''s the memory layout table from that file: /* * Memory layout (high to low): SIZE PAE-SIZE * ------ ------ * I/O remapping area ( 4MB) * Direct-map (1:1) area [Xen code/data/heap] (12MB) * Per-domain mappings (inc. 4MB map_domain_page cache) ( 8MB) * Shadow linear pagetable ( 4MB) ( 8MB) * Guest linear pagetable ( 4MB) ( 8MB) * Machine-to-physical translation table [writable] ( 4MB) (16MB) * Frame-info table (24MB) (96MB) * * Start of guest inaccessible area * Machine-to-physical translation table [read-only] ( 4MB) (16MB) * * Start of guest unmodifiable area */ When my guest domain starts (FreeBSD in PAE mode), I begin looking at the page tables. I note that I''ve modified Xend (the libxc code) to build 4 pages of Page Directories because FreeBSD uses the model of assuming that there are four Page Directories linearly mapped. This way it can just forget about the 4 PTD pointers in the L3 page. It means that 16KB are consumed for every address space, but it greatly simplifies the code. (gdb) x/4x xen_start_info->pt_base 0xc03ae000: 0x00000000fb152001 0x00000000fb151001 0xc03ae010: 0x00000000fb150001 0x00000000fb14f001 And, my kernel starts at virtual address 0xC000.0000, so only page 3 (in 0..3) of the 4 pages needs to have anything. And sure enough, that''s what I find: # ./xenctrl 4 -l3 0xfb153 dump64 0xc03af000 0x4000 # I''m dumping 64 bit values 0xc03af000: -- page of zeros -- 0xc03b0000: -- page of zeros -- 0xc03b1000: -- page of zeros -- 0xc03b2000: 0x00fb14e067 0x00fb14d067 0x00fb14c067 0x00fb14b067 ... 0xc03b2d60: 0x00fbc000e1 0x00fba000e1 0x0000000000 0x0000000000 ... 0xc03b2da0: 0x00018001e3 0x0001a001e3 0x0001c001e3 0x0001e001e3 0xc03b2dc0: 0x00020001e3 0x00022001e3 0x00024001e3 0x00026001e3 0xc03b2de0: 0x00028001e3 0x0002a001e3 0x0002c001e3 0x0002e001e3 ... 0xc03b2f20: 0x00fbc001e3 0x00fba001e3 0x0000000000 0x0000000000 ... 0xc03b2f60: 0x00fb152063 0x00fb151063 0x00fb150063 0x00fb14f063 ... 0xc03b2fa0: 0x0000224063 0x0000225063 0x0000226063 0x0000227063 0xc03b2fc0: 0x0000211063 0x0000212063 0x0000213063 0x0000214063 0xc03b2fe0: 0x0000215063 0x0000216063 0x0000217063 0x0000bfe063 # The four page table entries at 0xc03b2000 map my kernel (base address 0xC000.0000 means the first entries in page 3 of 0..3). And, my kernel is 8MB, so all is well. The following entries are broken-out as follows: 0xC03B2D60 has two entries which map to base address 0xF580.0000 for 4MB. This is 168 MB below 4GB. So, this should be the machine-to- physical translation table. However, I can''t map it from a tool in Dom0, nor can I see it in GDB: (gdb) x/32w 0xf5800000 0xf5800000: Cannot access memory at address 0xf5800000 Nor can I map the physical pages (of the L1 page table!): # ./xenctrl 4 -l3 0xfb153 dump64-mfn 0xfbc00000 0x1000 Error: Couldn''t map mfn fbc00000 This tells me I can''t determine the L0 pages used. That''s too bad. So, I''m not sure how I''m suppose to use this. My guest domain -can- see the F58 address region. And, so I can bcopy() the data into a visible space for gdb. In the following snippet, I am using two tools: GDB and a homebrew tool to view domain 0 pages, xenctrl: (gdb) call bzero(0xc0000000, 0x1000) # Just to certify that I can affect 0xC0000000 $22 = 0 # ./xenctrl 4 -l3 0xfb153 dump 0xc0000000 0x1000 # Verify that I can view it 0xc0000000: -- page of zeros -- (gdb) call bcopy(0xf5800000, 0xc0000000, 0x1000) # See what''s in the machine-to-phys table $23 = -897581056 # ./xenctrl 4 -l3 0xfb153 dump 0xc0000000 0x100 0xc0000000: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc0000020: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc0000040: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc0000060: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc0000080: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc00000a0: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc00000c0: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xc00000e0: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff I''m not sure what I''m suppose to make of those numbers. If I view the whole page (all 4K) I get long sequences of 0xffff''s and 0x5555''s. Is there any documentation on the machine-to-physical region of memory? Any sample code on its usage? I tried looking in Linux, but I wasn''t able to identify the machine-to-physical conversion macros or functions for x86-32 on PAE. Moving on to the next address space, I find Page Directory Entries for the address space 0xFD800000. I have learned that this address space is the recursive page table address space. I.e., in theory my guest domain could perform virtual to machine page translations here with the algorithm I stated last Wednesday: ((virtual-address & 0xFFFF.F000) >> 9) +0xFD80.0000 would generate the address of the PTE for the current virtual page (assuming the virtual address is valid in the first place!). But, the comment in the config.h file says this is beyond the "Start of guest inaccessible area." Which is a damn shame: (gdb) call bcopy(0xfde00000, 0xc0000000, 8) <hang> Because I can get access to the data from Dom0 (I actually executed the following steps -before- the steps above, which caused a hang): (gdb) p/x ((0xC0000000 & 0xFFFFF000) >> 9) +0xFD800000 $26 = 0xfde00000 (gdb) x/1g $26 0xfde00000: 0x00000000d1620063 (gdb) It would be -very- useful to my program if I could use the existing recursive page table. This all comes to pass because I''m trying to build the recursive address table for FreeBSD and I need information I can''t find; i.e., the machine to physical table. -- Randy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30/10/06 8:46 pm, "Randy Thelen" <rthelen@netapp.com> wrote:> This all comes to pass because I''m trying to build the recursive > address table for FreeBSD and I need information I can''t find; i.e., > the machine to physical table.Steal the function xc_linux_save.c:xc_map_m2p() to map the mach-to-phys table. That function could be moved somewhere more sensible, made non-static, and added to xenctrl.h. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel