Joel Fernandes
2025-Oct-20 18:55 UTC
[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
While not terribly complicated, a little bit of documentation will help
augment the code for this very important mechanism.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/pramin.rst | 113 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 114 insertions(+)
create mode 100644 Documentation/gpu/nova/core/pramin.rst
diff --git a/Documentation/gpu/nova/core/pramin.rst
b/Documentation/gpu/nova/core/pramin.rst
new file mode 100644
index 000000000000..19615e504db9
--- /dev/null
+++ b/Documentation/gpu/nova/core/pramin.rst
@@ -0,0 +1,113 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========================+PRAMIN aperture mechanism
+========================+
+.. note::
+ The following description is approximate and current as of the Ampere
family.
+ It may change for future generations and is intended to assist in
understanding
+ the driver code.
+
+Introduction
+===========+
+PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video
RAM (VRAM) before
+the GPU's Memory Management Unit (MMU) and page tables are initialized.
This 1MB sliding window,
+located at a fixed offset within BAR0, is essential for setting up page tables
and other critical
+GPU data structures without relying on the GPU's MMU.
+
+Architecture Overview
+====================+
+Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS
(PCIe Bus Controller Unit)
+and provides a CPU-accessible window into VRAM through the PCIe interface::
+
+ +-----------------+ PCIe +------------------------------+
+ | CPU |<----------->| GPU |
+ +-----------------+ | |
+ | +----------------------+ |
+ | | PBUS | |
+ | | (Bus Controller) | |
+ | | | |
+ | | +--------------.<------------
(window always starts at
+ | | | PRAMIN | | | BAR0 +
0x700000)
+ | | | Window | | |
+ | | | (1MB) | | |
+ | | +--------------+ | |
+ | | | | |
+ | +---------|------------+ |
+ | | |
+ | v |
+ | .----------------------.<------------
(Program PRAMIN to any
+ | | VRAM | | 64KB
VRAM physical boundary)
+ | | (Several GBs) | |
+ | | | |
+ | | FB[0x000000000000] | |
+ | | ... | |
+ | | FB[0x7FFFFFFFFFF] | |
+ | +----------------------+ |
+ +------------------------------+
+
+PBUS (PCIe Bus Controller) among other things is responsible in the GPU for
handling MMIO
+accesses to the BAR registers.
+
+PRAMIN Window Operation
+======================+
+The PRAMIN window provides a 1MB sliding aperture that can be repositioned over
+the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register.
+
+Window Control Mechanism
+-------------------------
+
+The window position is controlled via the PBUS BAR0_WINDOW register::
+
+ NV_PBUS_BAR0_WINDOW Register
+ +-----+-----+--------------------------------------+
+ |31-26|25-24| 23-0 |
+ | |TARG | BASE_ADDR |
+ | | ET | (bits 39:16 of VRAM address) |
+ +-----+-----+--------------------------------------+
+
+ TARGET field values:
+ - 0x0: VID_MEM (Video Memory / VRAM)
+ - 0x1: SYS_MEM_COHERENT (Coherent system memory)
+ - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)
+
+64KB Alignment Requirement
+---------------------------
+
+The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced
+by the BASE_ADDR field representing bits [39:16] of the target address::
+
+ VRAM Address Calculation:
+ actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
+ Where:
+ - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
+ - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]
+ Example Window Positioning:
+ +---------------------------------------------------------+
+ | VRAM Space |
+ | |
+ | 0x000000000 +-----------------+ <-- 64KB aligned |
+ | | PRAMIN Window | |
+ | | (1MB) | |
+ | 0x0000FFFFF +-----------------+ |
+ | |
+ | | ^ |
+ | | | Window can slide |
+ | v | to any 64KB boundary |
+ | |
+ | 0x123400000 +-----------------+ <-- 64KB aligned |
+ | | PRAMIN Window | |
+ | | (1MB) | |
+ | 0x1234FFFFF +-----------------+ |
+ | |
+ | ... |
+ | |
+ | 0x7FFFF0000 +-----------------+ <-- 64KB aligned |
+ | | PRAMIN Window | |
+ | | (1MB) | |
+ | 0x7FFFFFFFF +-----------------+ |
+ +---------------------------------------------------------+
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 46302daace34..e77d3ee336a4 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/fwsec
core/falcon
core/msgq
+ core/pramin
--
2.34.1
John Hubbard
2025-Oct-20 19:36 UTC
[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
On 10/20/25 11:55 AM, Joel Fernandes wrote: ...> +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)> +and provides a CPU-accessible window into VRAM through the PCIe interface:: > + > + +-----------------+ PCIe +------------------------------+ > + | CPU |<----------->| GPU | > + +-----------------+ | | > + | +----------------------+ | > + | | PBUS | | > + | | (Bus Controller) | | > + | | | | > + | | +--------------.<------------ (window always starts at > + | | | PRAMIN | | | BAR0 + 0x700000)Quick question: does "window always starts at" actually mean "windows is always initialized to" ? Or something else? thanks, -- John Hubbard> + | | | Window | | | > + | | | (1MB) | | | > + | | +--------------+ | | > + | | | | | > + | +---------|------------+ | > + | | | > + | v | > + | .----------------------.<------------ (Program PRAMIN to any > + | | VRAM | | 64KB VRAM physical boundary) > + | | (Several GBs) | | > + | | | | > + | | FB[0x000000000000] | | > + | | ... | | > + | | FB[0x7FFFFFFFFFF] | | > + | +----------------------+ | > + +------------------------------+ > + > +PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIO > +accesses to the BAR registers. > + > +PRAMIN Window Operation > +======================> + > +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over > +the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register. > + > +Window Control Mechanism > +------------------------- > + > +The window position is controlled via the PBUS BAR0_WINDOW register:: > + > + NV_PBUS_BAR0_WINDOW Register > + +-----+-----+--------------------------------------+ > + |31-26|25-24| 23-0 | > + | |TARG | BASE_ADDR | > + | | ET | (bits 39:16 of VRAM address) | > + +-----+-----+--------------------------------------+ > + > + TARGET field values: > + - 0x0: VID_MEM (Video Memory / VRAM) > + - 0x1: SYS_MEM_COHERENT (Coherent system memory) > + - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory) > + > +64KB Alignment Requirement > +--------------------------- > + > +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced > +by the BASE_ADDR field representing bits [39:16] of the target address:: > + > + VRAM Address Calculation: > + actual_vram_addr = (BASE_ADDR << 16) + pramin_offset > + Where: > + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] > + - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF] > + Example Window Positioning: > + +---------------------------------------------------------+ > + | VRAM Space | > + | | > + | 0x000000000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x0000FFFFF +-----------------+ | > + | | > + | | ^ | > + | | | Window can slide | > + | v | to any 64KB boundary | > + | | > + | 0x123400000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x1234FFFFF +-----------------+ | > + | | > + | ... | > + | | > + | 0x7FFFF0000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x7FFFFFFFF +-----------------+ | > + +---------------------------------------------------------+ > diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst > index 46302daace34..e77d3ee336a4 100644 > --- a/Documentation/gpu/nova/index.rst > +++ b/Documentation/gpu/nova/index.rst > @@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver. > core/fwsec > core/falcon > core/msgq > + core/pramin
John Hubbard
2025-Oct-20 22:08 UTC
[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
On 10/20/25 11:55 AM, Joel Fernandes wrote:> While not terribly complicated, a little bit of documentation will help > augment the code for this very important mechanism. > > Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> > --- > Documentation/gpu/nova/core/pramin.rst | 113 +++++++++++++++++++++++++ > Documentation/gpu/nova/index.rst | 1 + > 2 files changed, 114 insertions(+) > create mode 100644 Documentation/gpu/nova/core/pramin.rstHi Joel, Here's a more thorough review.> > diff --git a/Documentation/gpu/nova/core/pramin.rst b/Documentation/gpu/nova/core/pramin.rst > new file mode 100644 > index 000000000000..19615e504db9 > --- /dev/null > +++ b/Documentation/gpu/nova/core/pramin.rst > @@ -0,0 +1,113 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +========================> +PRAMIN aperture mechanism > +========================> + > +.. note:: > + The following description is approximate and current as of the Ampere family. > + It may change for future generations and is intended to assist in understanding > + the driver code. > + > +Introduction > +===========> + > +PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video RAM (VRAM) before > +the GPU's Memory Management Unit (MMU) and page tables are initialized. This 1MB sliding window, > +located at a fixed offset within BAR0, is essential for setting up page tables and other critical > +GPU data structures without relying on the GPU's MMU. > + > +Architecture Overview > +====================> + > +Logically, the PRAMIN aperture mechanism is implemented by the GPU's PBUS (PCIe Bus Controller Unit)How about: The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe Bus Controller Unit)> +and provides a CPU-accessible window into VRAM through the PCIe interface:: > + > + +-----------------+ PCIe +------------------------------+ > + | CPU |<----------->| GPU | > + +-----------------+ | | > + | +----------------------+ | > + | | PBUS | | > + | | (Bus Controller) | | > + | | | | > + | | +--------------.<------------ (window always starts at > + | | | PRAMIN | | | BAR0 + 0x700000) > + | | | Window | | | > + | | | (1MB) | | | > + | | +--------------+ | | > + | | | | | > + | +---------|------------+ | > + | | | > + | v | > + | .----------------------.<------------ (Program PRAMIN to any > + | | VRAM | | 64KB VRAM physical boundary) > + | | (Several GBs) | | > + | | | | > + | | FB[0x000000000000] | | > + | | ... | | > + | | FB[0x7FFFFFFFFFF] | | > + | +----------------------+ | > + +------------------------------+ > + > +PBUS (PCIe Bus Controller) among other things is responsible in the GPU for handling MMIOHow about: PBUS (PCIe Bus Controller) is responsible for, among other things, handling MMIO> +accesses to the BAR registers. > + > +PRAMIN Window Operation > +======================> + > +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over > +the entire VRAM address space using the NV_PBUS_BAR0_WINDOW register. > + > +Window Control Mechanism > +------------------------- > + > +The window position is controlled via the PBUS BAR0_WINDOW register:: > + > + NV_PBUS_BAR0_WINDOW Register > + +-----+-----+--------------------------------------+ > + |31-26|25-24| 23-0 |Should we use ":" notation here? 31:26 | 25:24 | 23:0> + | |TARG | BASE_ADDR | > + | | ET | (bits 39:16 of VRAM address) | > + +-----+-----+--------------------------------------+ > + > + TARGET field values: > + - 0x0: VID_MEM (Video Memory / VRAM) > + - 0x1: SYS_MEM_COHERENT (Coherent system memory) > + - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory)These SYS_MEM_* items imply to the reader that PRAMIN can target sysmem. However, pramin.rs in this series hard-codes this field to VID_MEM: let window_reg = regs::NV_PBUS_BAR0_WINDOW::default().set_window_addr(fb_offset); It's not clear why that is done (I seem to recall a potential deadlock in older chips, in similar situations, but that's probably stale info). If, however, there is some limitation that prevents using SYS_MEM*, then we should document it, probably here.> + > +64KB Alignment Requirement > +--------------------------- > + > +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced > +by the BASE_ADDR field representing bits [39:16] of the target address:: > + > + VRAM Address Calculation: > + actual_vram_addr = (BASE_ADDR << 16) + pramin_offset > + Where: > + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] > + - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF]How about: - pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF]> + Example Window Positioning: > + +---------------------------------------------------------+ > + | VRAM Space | > + | | > + | 0x000000000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x0000FFFFF +-----------------+ | > + | | > + | | ^ | > + | | | Window can slide | > + | v | to any 64KB boundary |Maybe: s/any 64KB boundary/any 64KB-aligned boundary/> + | | > + | 0x123400000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x1234FFFFF +-----------------+ | > + | | > + | ... | > + | | > + | 0x7FFFF0000 +-----------------+ <-- 64KB aligned | > + | | PRAMIN Window | | > + | | (1MB) | | > + | 0x7FFFFFFFF +-----------------+ | > + +---------------------------------------------------------+ > diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst > index 46302daace34..e77d3ee336a4 100644 > --- a/Documentation/gpu/nova/index.rst > +++ b/Documentation/gpu/nova/index.rst > @@ -33,3 +33,4 @@ vGPU manager VFIO driver and the nova-drm driver. > core/fwsec > core/falcon > core/msgq > + core/praminthanks, -- John Hubbard
Bagas Sanjaya
2025-Oct-22 02:09 UTC
[PATCH 4/7] docs: gpu: nova-core: Document the PRAMIN aperture mechanism
On Mon, Oct 20, 2025 at 02:55:36PM -0400, Joel Fernandes wrote:> +The window position is controlled via the PBUS BAR0_WINDOW register:: > + > + NV_PBUS_BAR0_WINDOW Register > + +-----+-----+--------------------------------------+ > + |31-26|25-24| 23-0 | > + | |TARG | BASE_ADDR | > + | | ET | (bits 39:16 of VRAM address) | > + +-----+-----+--------------------------------------+Shouldn't the TARGET field cell above be fitted (extended)?> + > + TARGET field values: > + - 0x0: VID_MEM (Video Memory / VRAM) > + - 0x1: SYS_MEM_COHERENT (Coherent system memory) > + - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory) > + > +64KB Alignment Requirement > +--------------------------- > + > +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced > +by the BASE_ADDR field representing bits [39:16] of the target address:: > + > + VRAM Address Calculation: > + actual_vram_addr = (BASE_ADDR << 16) + pramin_offset > + Where: > + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] > + - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF] > + Example Window Positioning:Move the supplementary explanation out of respective literal code blocks, like:: ---- >8 ---- diff --git a/Documentation/gpu/nova/core/pramin.rst b/Documentation/gpu/nova/core/pramin.rst index 19615e504db9d5..47abe539e32a81 100644 --- a/Documentation/gpu/nova/core/pramin.rst +++ b/Documentation/gpu/nova/core/pramin.rst @@ -70,23 +70,28 @@ The window position is controlled via the PBUS BAR0_WINDOW register:: | | ET | (bits 39:16 of VRAM address) | +-----+-----+--------------------------------------+ - TARGET field values: - - 0x0: VID_MEM (Video Memory / VRAM) - - 0x1: SYS_MEM_COHERENT (Coherent system memory) - - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory) +where TARGET field values are: + + - 0x0: VID_MEM (Video Memory / VRAM) + - 0x1: SYS_MEM_COHERENT (Coherent system memory) + - 0x2: SYS_MEM_NONCOHERENT (Non-coherent system memory) 64KB Alignment Requirement --------------------------- The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced -by the BASE_ADDR field representing bits [39:16] of the target address:: +by the BASE_ADDR field representing bits [39:16] of the target address. The +VRAM address calculation is determined by:: - VRAM Address Calculation: actual_vram_addr = (BASE_ADDR << 16) + pramin_offset - Where: - - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] - - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF] - Example Window Positioning: + +where: + + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] + - pramin_offset: 20-bit offset within PRAMIN window [0x00000-0xFFFFF] + +Example window positioning:: + +---------------------------------------------------------+ | VRAM Space | | | Thanks. -- An old man doll... just what I always wanted! - Clara -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <https://lists.freedesktop.org/archives/nouveau/attachments/20251022/e66447d9/attachment-0001.sig>