Joel Fernandes
2025-Apr-23 22:54 UTC
[PATCH 1/6] nova-core: doc: Add code comments related to devinit
Add several code comments to reduce acronym soup and explain how devinit magic
and bootflow works before driver loads. These are essential for debug and
development of the nova driver.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/devinit.rs | 36 ++++++++++++++++++++++++++++----
drivers/gpu/nova-core/regs.rs | 17 +++++++++++++--
2 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/devinit.rs b/drivers/gpu/nova-core/devinit.rs
index ee5685aff845..890bdc4d9522 100644
--- a/drivers/gpu/nova-core/devinit.rs
+++ b/drivers/gpu/nova-core/devinit.rs
@@ -1,6 +1,21 @@
// SPDX-License-Identifier: GPL-2.0
//! Methods for device initialization.
+//!
+//! A clarification about devinit terminology:
+//! devinit is a sequence of register read/writes after reset that performs
tasks
+//! such as:
+//! 1. Programming VRAM memory controller timings.
+//! 2. Power sequencing.
+//! 3. Clock and PLL configuration.
+//! 4. Thermal management.
+//! 5. Performs VRAM memory scrubbing (ECC initialization) - on some GPUs it
scrubs
+//! only part of memory and then kicks of 'async scrubbing'.
+//!
+//! devinit itself is a 'script' which is interpreted by the PMU
microcontroller of
+//! the GPU by an interpreter program.
+//!
+//! Note that the devinit sequence also needs to run during suspend/resume at
runtime.
use kernel::bindings;
use kernel::devres::Devres;
@@ -9,15 +24,28 @@
use crate::driver::Bar0;
use crate::regs;
-/// Wait for devinit FW completion.
+/// Wait for gfw (GPU firmware) boot completion signal (GFW_BOOT).
///
-/// Upon reset, the GPU runs some firmware code to setup its core parameters.
Most of the GPU is
-/// considered unusable until this step is completed, so it must be waited on
very early during
-/// driver initialization.
+/// Upon reset, several microcontrollers (such as PMU, SEC2, GSP etc) on the
GPU run some GPU
+/// firmware (gfw) code to setup its core parameters. Most of the GPU is
considered unusable until
+/// this step is completed, so it must be waited on very early during driver
initialization.
+///
+/// The GPU firmware (gfw) code includes several components that execute before
the driver loads.
+/// These components are located in the VBIOS ROM and are executed in a
sequence on these different
+/// microcontrollers. The devinit sequence itself runs on the PMU, and the
FWSEC runs on the GSP.
+///
+/// This function specifically waits for a signal indicating core
initialization is complete before
+/// which not much can be done. This signal is setup by the FWSEC running on
the GSP in high secure
+/// mode.
pub(crate) fn wait_gfw_boot_completion(bar: &Devres<Bar0>) ->
Result<()> {
let mut timeout = 2000;
loop {
+ // Before accessing the completion status in
PGC6_AON_SECURE_SCRATCH_GROUP_05, we must first
+ // check PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK. This is
because the register is
+ // accessible only after secure firmware (FWSEC) lowers the privilege
level to allow CPU
+ // (LS/low-secure) access. We can only safely read the status register
from CPU (LS/low-secure)
+ // once the mask indicates the privilege level has been appropriately
lowered.
let gfw_booted = with_bar!(
bar,
|b| regs::Pgc6AonSecureScratchGroup05PrivLevelMask::read(b)
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index eae5b7c13155..f4a6a382e83f 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -59,13 +59,26 @@
31:4 hi_val => as u32
);
-/* GC6 */
-
+/*
+ * GC6:
+ *
+ * GC6 is a GPU low-power state where VRAM is in self-refresh and the GPU
+ * is powered down (except for power rails needed to keep self-refresh
working).
+ *
+ * These scratch registers are "always-on" even in a low-power state
and have a
+ * designated group number.
+ */
+
+/// Privilege level mask register
(PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK)
+/// which dictates whether the host CPU has privilege to access to the
+/// PGC6_AON_SECURE_SCRATCH_GROUP_05 register.
register!(Pgc6AonSecureScratchGroup05PrivLevelMask at 0x00118128;
0:0 read_protection_level0_enabled => as_bit bool
);
/* TODO: This is an array of registers. */
+/// PGC6_AON_SECURE_SCRATCH_GROUP_05 scratch register used to check for
+/// GFW boot completion status.
register!(Pgc6AonSecureScratchGroup05 at 0x00118234;
31:0 value => as u32
);
--
2.43.0
Hello, Please find in this series, several clarifications, diagrams and code comments for various things in the nova-core driver. These are essential to develop an understanding how nova-core's boot initialization works and aid in development. These patches are on top of Alex's last posting for GSP WPR2 [1] [1] https://lore.kernel.org/all/20250420-nova-frts-v1-0-ecd1cca23963 at nvidia.com/ Joel Fernandes (6): nova-core: doc: Add code comments related to devinit nova-core: doc: Clarify sysmembar operations nova-core: docs: Document vbios layout nova-core: docs: Document fwsec operation and layout gpu: nova-core: Clarify fields in FalconAppifHdrV1 nova-core: docs: Document devinit process Documentation/gpu/nova/core/devinit.rst | 54 ++++++++ Documentation/gpu/nova/core/fwsec.rst | 173 ++++++++++++++++++++++++ Documentation/gpu/nova/core/vbios.rst | 155 +++++++++++++++++++++ Documentation/gpu/nova/index.rst | 2 + drivers/gpu/nova-core/devinit.rs | 36 ++++- drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++- drivers/gpu/nova-core/gpu.rs | 11 +- drivers/gpu/nova-core/regs.rs | 17 ++- 8 files changed, 452 insertions(+), 13 deletions(-) create mode 100644 Documentation/gpu/nova/core/devinit.rst create mode 100644 Documentation/gpu/nova/core/fwsec.rst create mode 100644 Documentation/gpu/nova/core/vbios.rst -- 2.43.0
Joel Fernandes
2025-Apr-23 22:54 UTC
[PATCH 2/6] nova-core: doc: Clarify sysmembar operations
sysmembar is a critical operation that the GSP falcon needs to perform
in the reset sequence. Add some code comments to clarify.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/gpu.rs | 11 ++++++++++-
drivers/gpu/nova-core/regs.rs | 2 ++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 4d03a0b11b64..61031bccb7d3 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -158,6 +158,9 @@ pub(crate) struct Gpu {
/// MMIO mapping of PCI BAR 0
bar: Devres<Bar0>,
fw: Firmware,
+ /// A system memory page for sysmembar (A GPU-initiated hardware
memory-barrier
+ /// operation that flushes all pending GPU-side memory writes that were
done
+ /// through PCIE, to system memory).
sysmem_flush: DmaObject,
timer: Timer,
bios: Vbios,
@@ -204,7 +207,13 @@ pub(crate) fn new(
devinit::wait_gfw_boot_completion(&bar)
.inspect_err(|_| pr_err!("GFW boot did not complete"))?;
- // System memory page required for sysmembar to properly flush into
system memory.
+ // System memory page required for sysmembar which is a GPU-initiated
hardware
+ // memory-barrier operation that flushes all pending GPU-side memory
writes that
+ // were done through PCIE, to system memory. It is required for Falcon
to be reset
+ // as the reset operation involves a reset handshake. When the falcon
acks the
+ // reset, it writes its acknowledgement into system memory, but for
this write to
+ // be visible to the host, the falcon needs to do sysmembar to flush
+ // its writes and prevent the driver from timing out.
let sysmem_flush = {
let page = DmaObject::new(
pdev.as_ref(),
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index f4a6a382e83f..22906ab1a43a 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -37,6 +37,8 @@
/* PFB */
+/// These two registers together hold the physical system memory address
+/// that is used by the GPU for perform sysmembar operation (see gpu.rs).
register!(PfbNisoFlushSysmemAddr at 0x00100c10;
31:0 adr_39_08 => as u32
);
--
2.43.0
Add detailed explanation and block diagrams of the layout of the vBIOS on Nvidia GPUs. This is important to understand how nova-core boots an Nvidia GPU. Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> --- Documentation/gpu/nova/core/vbios.rst | 154 ++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 Documentation/gpu/nova/core/vbios.rst diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst new file mode 100644 index 000000000000..17411f21b410 --- /dev/null +++ b/Documentation/gpu/nova/core/vbios.rst @@ -0,0 +1,154 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) +=========+VBIOS +=========+ +This document describes the layout of the VBIOS image which is a series of concatenated +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to +boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP. + +The format of the images in the ROM follow the "BIOS Specification" part of the +PCI specification, with Nvidia-specific extensions. + +As an example, the following are the different image types that can be found in the +VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver. + +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's naming likely + comes from the "IBM PC/AT" architecture. + +- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP + driver that is used to display UEFI graphics output. + +- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware) + +- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) contains + various microcodes that do a range of different functions - all in high secure mode + where they are allowed to perform highly privileged register accesses that the CPU + cannot do. The 2 most important parts (also known as Application Interfaces) of this + are: + 1. The devinit engine - this is loaded ontop the PMU before the driver loads + and interprets devinit commands which perform critical hardware initialization. + This will be described in a separate document. + 2. The DMEM mapper - this is loaded onto the GSP and one of the commands it + performs is carving out the WPR2 area (Write protected region) and placing + important data called 'FRTS' into it which contains things like voltage/frequency + curves etc. + +It is not clear why FwSec has 2 different images, but they both are of type 0xE0 +and can be identified as such. + +VBIOS ROM Layout +---------------- + +The VBIOS layout is roughly a series of concatenated images as follows: +(For more explanations of acronyms, see the detailed descriptions in vbios.rs). + +?????????????????????????????????????????????????????????????????????????? +? VBIOS (Starting at ROM_OFFSET: 0x300000) ? +?????????????????????????????????????????????????????????????????????????? +? ????????????????????????????????????????????????? ? +? ? PciAt Image (Type 0x00) ? ? +? ????????????????????????????????????????????????? ? +? ? ????????????????????? ? ? +? ? ? ROM Header ? ? ? +? ? ? (Signature 0xAA55)? ? ? +? ? ????????????????????? ? ? +? ? ? rom header's pci_data_struct_offset ? ? +? ? ? points to the PCIR structure ? ? +? ? V ? ? +? ? ????????????????????? ? ? +? ? ? PCIR Structure ? ? ? +? ? ? (Signature "PCIR")? ? ? +? ? ? last_image: 0x80 ? ? ? +? ? ? image_len: size ? ? ? +? ? ? in 512-byte units ? ? ? +? ? ????????????????????? ? ? +? ? ? ? ? +? ? ? NPDE immediately follows PCIR ? ? +? ? V ? ? +? ? ????????????????????? ? ? +? ? ? NPDE Structure ? ? ? +? ? ? (Signature "NPDE")? ? ? +? ? ? last_image: 0x00 ? ? ? +? ? ????????????????????? ? ? +? ? ? ? +? ? ????????????????????? ? ? +? ? ? BIT Header ? (Signature scanning ? ? +? ? ? (Signature "BIT") ? provides the location ? ? +? ? ????????????????????? of the BIT table) ? ? +? ? ? header is ? ? +? ? | followed by a table of tokens ? ? +? ? V one of which is for falcon data. ? ? +? ? ????????????????????? ? ? +? ? ? BIT Tokens ? ? ? +| | | ______________ | | | +? ? ? ? Falcon Data ? ? ? ? +? ? ? ? Token (0x70)?---+------------>------------???+ ? +? ? ? ??????????????? ? falcon_data_ptr() ? ? ? +? ? ????????????????????? ? V ? +? ????????????????????????????????????????????????? ? ? +? (no gap between images) ? ? +? ????????????????????????????????????????????????? ? ? +? ? EFI Image (Type 0x03) ? ? ? +? ????????????????????????????????????????????????? ? ? +| | Contains the UEFI GOP driver (Graphics Output)| | | +? ? ????????????????????? ? ? ? +? ? ? ROM Header ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? PCIR Structure ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? NPDE Structure ? ? ? ? +? ? ????????????????????? ? ? ? +? ? ? Image data ? ? ? ? +? ? ????????????????????? ? ? ? +? ????????????????????????????????????????????????? ? ? +? (no gap between images) ? ? +? ????????????????????????????????????????????????? ? ? +? ? First FwSec Image (Type 0xE0) ? ? ? +? ????????????????????????????????????????????????? ? ? +? ? ????????????????????? ? ? ? +? ? ? ROM Header ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? PCIR Structure ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? NPDE Structure ? ? ? ? +? ? ????????????????????? ? ? ? +? ? ? Image data ? ? ? ? +? ? ????????????????????? ? ? ? +? ????????????????????????????????????????????????? ? ? +? (no gap between images) ? ? +? ????????????????????????????????????????????????? ? ? +? ? Second FwSec Image (Type 0xE0) ? ? ? +? ????????????????????????????????????????????????? ? ? +? ? ????????????????????? ? ? ? +? ? ? ROM Header ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? PCIR Structure ? ? ? ? +? ? +???????????????????+ ? ? ? +? ? ? NPDE Structure ? ? ? ? +? ? ????????????????????? ? ? ? +? ? ? ? ? +? ? ????????????????????? ? ? ? +? ? ? PMU Lookup Table ? <- falcon_data_offset ?<?? ? +? ? ? ??????????????? ? pmu_lookup_table ? ? +? ? ? ? Entry 0x85 ? ? ? ? +? ? ? ? FWSEC_PROD ? ? ? ? +? ? ? ??????????????? ? ? ? +? ? ????????????????????? ? ? +? ? ? ? ? +? ? ? points to ? ? +? ? V ? ? +? ? ????????????????????? ? ? +? ? ? FalconUCodeDescV3 ? <- falcon_ucode_offset ? ? +? ? ? (FWSEC Firmware) ? fwsec_header() ? ? +? ? ????????????????????? ? ? +? ? ? immediately followed by... ? ? +? ? V ? ? +? ? ?????????????????????????????? ? ? +? ? ? Signatures + FWSEC Ucode ? ? ? +? ? ? fwsec_sigs(), fwsec_ucode()? ? ? +? ? ?????????????????????????????? ? ? +? ?????????????????????????????????????????????????______________________? + -- 2.43.0
Joel Fernandes
2025-Apr-23 22:54 UTC
[PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/nova-core/firmware/fwsec.rs
b/drivers/gpu/nova-core/firmware/fwsec.rs
index 664319d1d31c..79c21db9d89d 100644
--- a/drivers/gpu/nova-core/firmware/fwsec.rs
+++ b/drivers/gpu/nova-core/firmware/fwsec.rs
@@ -29,11 +29,14 @@
#[repr(C)]
#[derive(Debug)]
+
+/// The header of the Application Interface table, used
+/// to locate the DMEMMAPPER section in the DMEM (see fwsec.rst).
struct FalconAppifHdrV1 {
- ver: u8,
- hdr: u8,
- len: u8,
- cnt: u8,
+ version: u8,
+ header_size: u8,
+ entry_size: u8,
+ entry_count: u8,
}
// SAFETY: any byte sequence is valid for this struct.
unsafe impl FromBytes for FalconAppifHdrV1 {}
@@ -169,14 +172,14 @@ fn patch_command(fw: &mut DmaObject, v3_desc:
&FalconUCodeDescV3, cmd: FwsecComm
let hdr_offset = (v3_desc.imem_load_size + v3_desc.interface_offset) as
usize;
let hdr: &FalconAppifHdrV1 = unsafe { transmute(fw, hdr_offset) }?;
- if hdr.ver != 1 {
+ if hdr.version != 1 {
return Err(EINVAL);
}
// Find the DMEM mapper section in the firmware.
- for i in 0..hdr.cnt as usize {
+ for i in 0..hdr.entry_count as usize {
let app: &FalconAppifV1 - unsafe { transmute(fw,
hdr_offset + hdr.hdr as usize + i * hdr.len as usize) }?;
+ unsafe { transmute(fw, hdr_offset + hdr.header_size as usize + i *
hdr.entry_size as usize) }?;
if app.id != NVFW_FALCON_APPIF_ID_DMEMMAPPER {
continue;
--
2.43.0
Joel Fernandes
2025-Apr-23 22:54 UTC
[PATCH 6/6] nova-core: docs: Document devinit process
devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/devinit.rst | 54 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 2 +
2 files changed, 56 insertions(+)
create mode 100644 Documentation/gpu/nova/core/devinit.rst
diff --git a/Documentation/gpu/nova/core/devinit.rst
b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 000000000000..e504ed3efe15
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,54 @@
+.. SPDX-License-Identifier: GPL-2.0
+=================================+Device Initialization (devinit)
+=================================+
+Overview
+--------
+Device initialization (devinit) is a crucial sequence of register read/write
operations
+that occur after a GPU reset. The devinit sequence is essential for properly
configuring
+the GPU hardware before it can be used.
+
+The devinit is an interpreter program that typically runs on the PMU (Power
Management
+Unit) microcontroller of the GPU. This interpreter executes a
"script" of initialization
+commands. The devinit interpreter itself is part of the FWSEC (Firmware
Security)
+component provided by the GPU firmware in the VBIOS ROM (see fwsec.rst and
vbios.rst).
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+5. VRAM memory scrubbing (ECC initialization)
+ - On some GPUs, it scrubs only part of memory and then initiates 'async
scrubbing'
+
+Firmware Initialization Flow
+---------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.)
run GPU
+firmware (gfw) code to set up core parameters. Most of the GPU is considered
unusable
+until this initialization process completes.
+
+The GPU firmware components are:
+1. Located in the VBIOS ROM
+2. Executed in sequence on different microcontrollers:
+ - The devinit sequence runs on the PMU
+ - The FWSEC runs on the GSP (Graphics System Processor) in high secure mode
+
+Before the driver can proceed with further initialization, it must wait for a
signal
+indicating that core initialization is complete (known as GFW_BOOT). This
signal is
+set up by the FWSEC running on the GSP in high secure mode.
+
+Runtime Considerations
+---------------------
+It's important to note that the devinit sequence also needs to run during
suspend/resume
+operations at runtime, not just during initial boot.
+
+Security and Access Control
+--------------------------
+The initialization process involves careful privilege management. For example,
before
+accessing certain completion status registers, the driver must check privilege
level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers
the
+privilege level to allow CPU (LS/low-secure) access.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index c01dcc5657e2..301435c5cf67 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,5 +27,7 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/vbios
core/fwsec
+ core/devinit
core/todo
--
2.43.0
Joel Fernandes
2025-Apr-23 22:54 UTC
[PATCH 4/6] nova-core: docs: Document fwsec operation and layout
Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 173 ++++++++++++++++++++++++++
Documentation/gpu/nova/core/vbios.rst | 3 +-
Documentation/gpu/nova/index.rst | 1 +
3 files changed, 176 insertions(+), 1 deletion(-)
create mode 100644 Documentation/gpu/nova/core/fwsec.rst
diff --git a/Documentation/gpu/nova/core/fwsec.rst
b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..03d1f0d67414
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,173 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+========================+FWSEC (Firmware Security)
+========================+
+The role of FWSEC to provide firmware verification and perform secure
operations
+such as carving out the WPR2 region (Write protected region). It is critical to
+the boot sequence of the GPU.
+
+The FWSEC image is located in the VBIOS ROM. For how it is extracted, see
vbios.rst
+and the vbios.rs source code.
+
+The Falcon data in the FWSEC image is a combination of headers, data sections
+(DMEM) and instruction code sections (IMEM).
+
+FWSEC itself is a central location for all microcodes in the VBIOS that run on
+various microcontrollers (PMU, SEC, GSP etc) some of which (such as devinit)
are
+executed after GPU power-on before the nova-core driver is even loaded.
+
+For the purposes of nova-core driver, the FWSEC contains microcode called
+DMEMMAPPER which executes the "FWSEC-FRTS" command (among other
commands it is
+capable of executing) that is executed on the GSP to carve out the WPR2 region
+(Write protected region).
+
+FWSEC Memory Layout
+-------------------
+
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs):
+
+?????????????????????????????????????????????????????????????????
+? FWSEC image (type 0xE0) ?
+? ?
+? ??????????????????????????????????? ?
+? ? PMU Falcon Ucode Table ? ?
+? ? (PmuLookupTable) ? ?
+? ? ??????????????????????????? ? ?
+? ? ? Table Header ? ? ?
+? ? ? - version: 0x01 ? ? ?
+? ? ? - header_size: 6 ? ? ?
+? ? ? - entry_size: 6 ? ? ?
+? ? ? - entry_count: N ? ? ?
+? ? ? - desc_version:3(unused)? ? ?
+? ? ??????????????????????????? ? ?
+? ? ... ? ?
+? ? ??????????????????????????? ? ?
+? ? ? Entry for FWSEC (0x85) ? ? ?
+? ? ? (PmuLookupTableEntry) ? ? ?
+? ? ? - app_id: 0x85 (FWSEC) ? ????????? ?
+? ? ? - target_id: 0x01 (PMU) ? ? ? ?
+? ? ? - data: offset ???????????????????????? ?
+? ? ??????????????????????????? ? ? ? ?
+? ??????????????????????????????????? ? ? ?
+? ? ? ?
+? ? ? ?
+? ??????????????????????????????????? ? ? ?
+? ? FWSEC Ucode Component ?<???? ? ?
+? ? (aka Falcon data) ? ? ?
+? ? ??????????????????????????? ? ? ?
+? ? ? FalconUCodeDescV3 ?<????????????? ?
+? ? ? - hdr ? ? ?
+? ? ? - stored_size ? ? ?
+? ? ? - pkc_data_offset ? ? ?
+? ? ? - interface_offset ???????????????????????????? ?
+? ? ? - imem_phys_base ? ? ? ?
+? ? ? - imem_load_size ? ? ? ?
+? ? ? - imem_virt_base ? ? ? ?
+? ? ? - dmem_phys_base ? ? ? ?
+? ? ? - dmem_load_size ? ? ? ?
+? ? ? - engine_id_mask ? ? ? ?
+? ? ? - ucode_id ? ? ? ?
+? ? ? - signature_count ? ? look up sig ? ?
+? ? ? - signature_versions ???????????????using hal ? ?
+? ? ??????????????????????????? ? ? ? ?
+? ? (no gap) ? ? ? ?
+? ? ??????????????????????????? ? ? ? ?
+? ? ? Signatures Section ?<?????????? ? ?
+? ? ? (384 bytes per sig) ? ? ? ?
+? ? ? - RSA-3K Signature 1 ? ? ? ?
+? ? ? - RSA-3K Signature 2 ? ? ? ?
+? ? ? ... ? ? ? ?
+? ? ??????????????????????????? ? ? ?
+? ? ? ? ?
+? ? ??????????????????????????? ? ? ?
+? ? ? IMEM Section (Code) ? ? ? ?
+? ? ? ? ? ? ?
+? ? ? Contains devinit, fwsec ? ? ? ?
+? ? ? instruction code etc. ? ? ? ?
+? ? ??????????????????????????? ? ? ?
+? ? ? ? ?
+? ? ??????????????????????????? ? ? ?
+? ? ? DMEM Section (Data) ? ? ? ?
+? ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ?
+? ? ? ? Application ? ?<????????????????????? ?
+? ? ? ? Interface Table ? ? ? ?
+? ? ? ? (FalconAppifHdrV1) ? ? ? ?
+? ? ? ? Header: ? ? ? ?
+? ? ? ? - version: 0x01 ? ? ? ?
+? ? ? ? - header_size: 4 ? ? ? ?
+? ? ? ? - entry_size: 8 ? ? ? ?
+? ? ? ? - entry_count: N ? ? ? ?
+? ? ? ? ? ? ? ?
+? ? ? ? Entries: ? ? ? ?
+? ? ? ? ??????????????????? ? ? ? ?
+? ? ? ? ? DEVINIT (ID 1) ? ? ? ? ?
+? ? ? ? ? - id: 0x01 ? ? ? ? ?
+? ? ? ? ? - dmemOffset X ??????????? ?
+? ? ? ? ??????????????????? ? ? ? ?
+? ? ? ? ??????????????????? ? ? ? ?
+? ? ? ? ? DMEMMAPPER(ID 4)? ? ? ? ?
+? ? ? ? ? - id: 0x04 ? ? ? ? ?
+| | | | | (NVFW_FALCON_ | | | | |
+| | | | | APPIF_ID_DMEMMAPPER) | |
+? ? ? ? ? - dmemOffset Y ????????????????? ?
+? ? ? ? ??????????????????? ? ? ? ? ?
+? ? ? ? ??????????????????? ? ? ? ? ?
+? ? ? ? ? MULTI_FALCON (5)? ? ? ? ? ?
+? ? ? ? ? - id: 0x05 ? ? ? ? ? (See note [1]) ?
+? ? ? ? ? - dmemOffset Z ?????????????????????????????????? ?
+? ? ? ? ??????????????????? ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ? ?
+? ? ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ? ?
+? ? ? ? DEVINIT Engine ?<?????? ? ? ?
+? ? ? ? Interface ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ? ?
+? ? ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ? ?
+? ? ? ? DMEM Mapper (ID 4) ?<?????+?????? ? ?
+? ? ? ? (FalconAppifDmemmapperV3) | | |
+? ? ? ? - signature: "DMAP" ? ? ? | |
+? ? ? ? - version: 0x0003 ? ? ? | |
+? ? ? ? - Size: 64 bytes ? ? ? | |
+? ? ? ? - cmd_in_buffer_off ? ??????????????????? | |
+? ? ? ? - cmd_in_buffer_size? ? ? ? | |
+? ? ? ? - cmd_out_buffer_off? ????????????????????????? | |
+? ? ? ? - cmd_out_buffer_sz ? ? ? ? ? | |
+? ? ? ? - init_cmd ? ? ? ? ? | |
+? ? ? ? - features ? ? ? ? ? | |
+? ? ? ? - cmd_mask0/1 ? ? ? ? ? V |
+? ? ? ? - multiTgtTbl-------???????+????????????+?????+????? ?
+? ? ? ??????????????????????? ? ? ? ? ? ?
+? ? ? ? ? ? ? ? ?
+? ? ? ??????????????????????? ? ? ? ? ? ?
+? ? ? ? MULTI_FALCON ?<?????+????????????+?????+????? ?
+? ? ? ? - version ? ? ? ? ? |
+? ? ? ? - targetId (PMU/SEC)? ? ? ? ? |
+? ? ? ? - loadType ? ? ? ? ? |
+? ? ? ? - initStack ? ? ? ? ? |
+? ? ? ??????????????????????? ? ? ? ? |
+? ? ? ? ? ? ? |
+? ? ? ??????????????????????? ? ? ? ? |
+? ? ? ? Command Input Buffer?<??????????????????? ? |
+? ? ? ? - Command data ? ? ? ? |
+? ? ? ? - Arguments ? ? ? ? |
+? ? ? ??????????????????????? ? ? ? |
+? ? ? ? ? ? |
+? ? ? ??????????????????????? ? ? ? |
+? ? ? ? Command Output ?<????????????????????????? |
+? ? ? ? Buffer ? ? ? |
+? ? ? ? - Results ? ? ? |
+? ? ? ? - Status ? ? ? |
+? ? ? ??????????????????????? ? ? |
+? ? ??????????????????????????? ? |
+? ??????????????????????????????????? |
+? ?
+?????????????????????????????????????????????????????????????????
+
+Note:
+[1] MULTI_FALCON section in DMEM can either be looked up by
+ MULTI_FALCON App Interface, or from multi_tgt_table entry
+ in DMEM_MAPPER's App Interface.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/core/vbios.rst
b/Documentation/gpu/nova/core/vbios.rst
index 17411f21b410..da759b56fb2d 100644
--- a/Documentation/gpu/nova/core/vbios.rst
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -44,7 +44,8 @@ VBIOS ROM Layout
The VBIOS layout is roughly a series of concatenated images as follows:
(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
-
+Note: this diagram is using an GA-102 Ampere GPU as an example and could vary
+ for future GPUs.
??????????????????????????????????????????????????????????????????????????
? VBIOS (Starting at ROM_OFFSET: 0x300000) ?
??????????????????????????????????????????????????????????????????????????
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af35..c01dcc5657e2 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,4 +27,5 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/fwsec
core/todo
--
2.43.0
Alexandre Courbot
2025-Apr-24 01:18 UTC
[PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1
Since this just renames fields, would you be ok if I squashed this one into the relevant patch of my series, alongside a [joelagnelf at nvidia.com: give better names to FalconAppifHdrV1's fields] ? On Thu Apr 24, 2025 at 7:54 AM JST, Joel Fernandes wrote:> Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> > --- > drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++++++++++------- > 1 file changed, 10 insertions(+), 7 deletions(-) > > diff --git a/drivers/gpu/nova-core/firmware/fwsec.rs b/drivers/gpu/nova-core/firmware/fwsec.rs > index 664319d1d31c..79c21db9d89d 100644 > --- a/drivers/gpu/nova-core/firmware/fwsec.rs > +++ b/drivers/gpu/nova-core/firmware/fwsec.rs > @@ -29,11 +29,14 @@ > > #[repr(C)] > #[derive(Debug)] > + > +/// The header of the Application Interface table, used > +/// to locate the DMEMMAPPER section in the DMEM (see fwsec.rst). > struct FalconAppifHdrV1 { > - ver: u8, > - hdr: u8, > - len: u8, > - cnt: u8, > + version: u8, > + header_size: u8, > + entry_size: u8, > + entry_count: u8, > } > // SAFETY: any byte sequence is valid for this struct. > unsafe impl FromBytes for FalconAppifHdrV1 {} > @@ -169,14 +172,14 @@ fn patch_command(fw: &mut DmaObject, v3_desc: &FalconUCodeDescV3, cmd: FwsecComm > let hdr_offset = (v3_desc.imem_load_size + v3_desc.interface_offset) as usize; > let hdr: &FalconAppifHdrV1 = unsafe { transmute(fw, hdr_offset) }?; > > - if hdr.ver != 1 { > + if hdr.version != 1 { > return Err(EINVAL); > } > > // Find the DMEM mapper section in the firmware. > - for i in 0..hdr.cnt as usize { > + for i in 0..hdr.entry_count as usize { > let app: &FalconAppifV1 > - unsafe { transmute(fw, hdr_offset + hdr.hdr as usize + i * hdr.len as usize) }?; > + unsafe { transmute(fw, hdr_offset + hdr.header_size as usize + i * hdr.entry_size as usize) }?; > > if app.id != NVFW_FALCON_APPIF_ID_DMEMMAPPER { > continue;
Alexandre Courbot
2025-Apr-24 01:25 UTC
[PATCH 0/6] Additional documentation for nova-core
Thanks a lot for doing this, this was severely missing from the WPR2 patchset. Due to the strong focus on documentation, and in order to ease merging, I think it makes sense to keep this separate from the WPR2 patchset and merge it on top of it. Danilo, would that work for you? On Thu Apr 24, 2025 at 7:53 AM JST, Joel Fernandes wrote:> Hello, > Please find in this series, several clarifications, diagrams and code comments > for various things in the nova-core driver. These are essential to develop an > understanding how nova-core's boot initialization works and aid in development. > > These patches are on top of Alex's last posting for GSP WPR2 [1] > > [1] https://lore.kernel.org/all/20250420-nova-frts-v1-0-ecd1cca23963 at nvidia.com/ > > Joel Fernandes (6): > nova-core: doc: Add code comments related to devinit > nova-core: doc: Clarify sysmembar operations > nova-core: docs: Document vbios layout > nova-core: docs: Document fwsec operation and layout > gpu: nova-core: Clarify fields in FalconAppifHdrV1 > nova-core: docs: Document devinit process > > Documentation/gpu/nova/core/devinit.rst | 54 ++++++++ > Documentation/gpu/nova/core/fwsec.rst | 173 ++++++++++++++++++++++++ > Documentation/gpu/nova/core/vbios.rst | 155 +++++++++++++++++++++ > Documentation/gpu/nova/index.rst | 2 + > drivers/gpu/nova-core/devinit.rs | 36 ++++- > drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++- > drivers/gpu/nova-core/gpu.rs | 11 +- > drivers/gpu/nova-core/regs.rs | 17 ++- > 8 files changed, 452 insertions(+), 13 deletions(-) > create mode 100644 Documentation/gpu/nova/core/devinit.rst > create mode 100644 Documentation/gpu/nova/core/fwsec.rst > create mode 100644 Documentation/gpu/nova/core/vbios.rst