Joel Fernandes
2025-Oct-21 18:26 UTC
[PATCH 7/7] nova-core: mm: Add data structures for page table management
Hi John, On 10/20/2025 4:59 PM, John Hubbard wrote:> On 10/20/25 11:55 AM, Joel Fernandes wrote: >> Add data structures and helpers for page table management. Uses >> bitfield for cleanly representing and accessing the bitfields in the >> structures. >> > ... >> +bitfield! { >> + pub(crate) struct Pte(u64), "Page Table Entry (PTE) to map virtual pages to physical frames." { >> + 0:0 valid as bool; // (1 = valid for PTEs) >> + 1:1 privilege as bool; // P - Privileged/kernel-only access >> + 2:2 read_only as bool; // RO - Write protection >> + 3:3 atomic_disable as bool; // AD - Disable atomic ops >> + 4:4 encrypted as bool; // E - Encryption enabled >> + 39:8 frame_number as u64; // PA[39:8] - Physical frame number (32 bits) >> + 41:40 aperture as u8 => AperturePte; // Memory aperture type. >> + 42:42 volatile as bool; // VOL - Volatile flag >> + 50:43 kind as u8; // K[7:0] - Compression/tiling kind >> + 63:51 comptag_line as u16; // CTL[12:0] - Compression tag line >> + } >> +} > > Hi Joel, > > What GPUs is this good for? I ask because I seem to recall that > the format has changed over the years, on a per-GPU-architecture > basis...Yes, there's different format versions. This patch supports all Turing and later GPUs because all GPUs including Hopper+ are backward compatible with version 1. However they wont be able to use the version 2 and 3 features with only this patch. I kind of intentionally did this for a first cut. But yes, I could split it into versions. The 3 MMU structures (PTE, PDE and Dual PDE) are versioned. Version 2 is Turing and later. Hopper+ is when Version 3 got introduced and it is also backward compatible with Version 2. We could eventually support versions 2 and 3 (instead of just version 1 as I am doing), however my working MMU translation prototype is based on version 1 (I did not have to configure anything in the MMU to switch versions, this was default). There are a couple of options: 1. For starters, support only version 1. Drawback is, when/if we want to use version 2 and 3 features, it may require some rework. 2. Have the following hierarchy: mm/types.rs - all common structures (stuff that is generic like Pfn). mm/types/ver1.rs - Version 1 specific types. mm/types/ver2.rs - Version 2 specific types. mm/types/ver3.rs - Version 3 specific types. The advantage of this is it keeps the structs namespaced. So it'd be nova_core::mm:types::ver2::Pte or nova_core::mm:types::ver3::Pte. And the nova-core MMU code can pick the correct version. 3. Keep the single file types.rs and just suffix the structs with version numbers. This is attractive because there are only 3 types that have version flavors (pte, pde and dual pde). So instead of Pte, we would have PteVersion1, PteVersion2 etc, and a helper abstraction can pick the correct struct. 4. Any of the options 1-3, but dropping version 1 since Turing+ supports version 2 and later. I do have to figure out how to configure the MMU to use a specific version (which is reasonable). 5. Your options here. Btw, I used Nouveau as a reference as well, so likely Nouveau doesn't support version 2 and 3 features. Not that that matters (we should support newer features in nova-core), but just thought I'd mention it. Other thoughts? thanks, - Joel
John Hubbard
2025-Oct-21 20:30 UTC
[PATCH 7/7] nova-core: mm: Add data structures for page table management
On 10/21/25 11:26 AM, Joel Fernandes wrote:> On 10/20/2025 4:59 PM, John Hubbard wrote: >> On 10/20/25 11:55 AM, Joel Fernandes wrote: >> ... > Yes, there's different format versions. > > This patch supports all Turing and later GPUs because all GPUs including Hopper+ > are backward compatible with version 1. However they wont be able to use the > version 2 and 3 features with only this patch. > > I kind of intentionally did this for a first cut. But yes, I could split it into > versions. The 3 MMU structures (PTE, PDE and Dual PDE) are versioned. Version 2 > is Turing and later. Hopper+ is when Version 3 got introduced and it is alsoAh, then we shouldn't even do version 1. We should take full advantage of the fact that Nova's earliest GPU is Turing.> backward compatible with Version 2. > > We could eventually support versions 2 and 3 (instead of just version 1 as I am > doing), however my working MMU translation prototype is based on version 1 (I > did not have to configure anything in the MMU to switch versions, this was default). > > There are a couple of options: > > 1. For starters, support only version 1. Drawback is, when/if we want to use > version 2 and 3 features, it may require some rework. > > 2. Have the following hierarchy: > mm/types.rs - all common structures (stuff that is generic like Pfn). > mm/types/ver1.rs - Version 1 specific types. > mm/types/ver2.rs - Version 2 specific types. > mm/types/ver3.rs - Version 3 specific types.Maybe a file/directory structure that more directly indicates page table formats. "mm/types" could be quite a few things.> The advantage of this is it keeps the structs namespaced. So it'd be > nova_core::mm:types::ver2::Pte or nova_core::mm:types::ver3::Pte. And the > nova-core MMU code can pick the correct version. > > 3. Keep the single file types.rs and just suffix the structs with version > numbers. This is attractive because there are only 3 types that have version > flavors (pte, pde and dual pde). So instead of Pte, we would have PteVersion1, > PteVersion2 etc, and a helper abstraction can pick the correct struct. > > 4. Any of the options 1-3, but dropping version 1 since Turing+ supports version > 2 and later. I do have to figure out how to configure the MMU to use a specificRight, I see you already noticed that we can start with Turing. Good.> version (which is reasonable). > > 5. Your options here. > > Btw, I used Nouveau as a reference as well, so likely Nouveau doesn't support > version 2 and 3 features. Not that that matters (we should support newer > features in nova-core), but just thought I'd mention it. > > Other thoughts?Two things: 1) Danilo is working on writing down locking requirements for Nova page tables, based on earlier experience with Nouveau page table locking problems, which were apparently very serious. 2) Maybe it would be good to start with versions 2 and 3, so that we can see how to do >1 version? thanks, -- John Hubbard