Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 1/7] nova-core: doc: Add code comments related to devinit
Add several code comments to reduce acronym soup and explain how devinit magic
and bootflow works before driver loads. These are essential for debug and
development of the nova driver.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/devinit.rs | 34 ++++++++++++++++++++++++++++----
drivers/gpu/nova-core/regs.rs | 16 +++++++++++++--
2 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/devinit.rs b/drivers/gpu/nova-core/devinit.rs
index 31a313a0652c..21fc9b96d498 100644
--- a/drivers/gpu/nova-core/devinit.rs
+++ b/drivers/gpu/nova-core/devinit.rs
@@ -1,6 +1,19 @@
// SPDX-License-Identifier: GPL-2.0
//! Methods for device initialization.
+//!
+//! A clarification about devinit terminology:
+//! devinit is a sequence of register read/writes after reset that performs
tasks
+//! such as:
+//! 1. Programming VRAM memory controller timings.
+//! 2. Power sequencing.
+//! 3. Clock and PLL configuration.
+//! 4. Thermal management.
+//!
+//! devinit itself is a 'script' which is interpreted by the PMU
microcontroller of
+//! the GPU by an interpreter program.
+//!
+//! Note that the devinit sequence also needs to run during suspend/resume at
runtime.
use kernel::bindings;
use kernel::devres::Devres;
@@ -9,15 +22,28 @@
use crate::driver::Bar0;
use crate::regs;
-/// Wait for devinit FW completion.
+/// Wait for gfw (GPU firmware) boot completion signal (GFW_BOOT).
///
-/// Upon reset, the GPU runs some firmware code to setup its core parameters.
Most of the GPU is
-/// considered unusable until this step is completed, so it must be waited on
very early during
-/// driver initialization.
+/// Upon reset, several microcontrollers (such as PMU, SEC2, GSP etc) on the
GPU run some GPU
+/// firmware (gfw) code to setup its core parameters. Most of the GPU is
considered unusable until
+/// this step is completed, so it must be waited on very early during driver
initialization.
+///
+/// The GPU firmware (gfw) code includes several components that execute before
the driver loads.
+/// These components are located in the VBIOS ROM and are executed in a
sequence on these different
+/// microcontrollers. The devinit sequence itself runs on the PMU, and the
FWSEC runs on the GSP.
+///
+/// This function specifically waits for a signal indicating core
initialization is complete before
+/// which not much can be done. This signal is setup by the FWSEC running on
the GSP in Heavy-secured
+/// mode.
pub(crate) fn wait_gfw_boot_completion(bar: &Devres<Bar0>) ->
Result<()> {
let mut timeout = 2000;
loop {
+ // Before accessing the completion status in
[`crate::regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05`],
+ // we must first check
[`crate::regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK`].
+ // This is because the register is accessible only after secure
firmware (FWSEC) lowers the
+ // privilege level to allow CPU (LS/Light-secured) access. We can only
safely read the status
+ // register from CPU (LS/Light-secured) once mask indicates privilege
level has been lowered.
let gfw_booted with_bar!(
bar,
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 3acec36f2d57..a2b506883654 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -74,8 +74,20 @@ pub(crate) fn chipset(self) -> Result<Chipset,
Error> {
31:4 hi_val as u32;
});
-/* PGC6 */
-
+/*
+ * PGC6 register space.
+ *
+ * GC6 is a GPU low-power state where VRAM is in self-refresh and the GPU
+ * is powered down (except for power rails needed to keep self-refresh working
+ * and important registers and hardware blocks).
+ *
+ * These scratch registers are "always-on" even in a low-power state
and have a
+ * designated group number.
+ */
+
+// Privilege level mask register.
+// It dictates whether the host CPU has privilege to access the
+// PGC6_AON_SECURE_SCRATCH_GROUP_05 register (which it needs to read GFW_BOOT).
register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128 {
0:0 read_protection_level0 as bool;
});
--
2.43.0
Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 180 ++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 181 insertions(+)
create mode 100644 Documentation/gpu/nova/core/fwsec.rst
diff --git a/Documentation/gpu/nova/core/fwsec.rst
b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..bed941ac3f2b
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,180 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+========================+FWSEC (Firmware Security)
+========================+This document briefly/conceptually describes the FWSEC
(Firmware Security) image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel
code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC to provide secure boot, it is running in Heavy-secure mode.
It does
+firmware verification after GPU reset and load various ucode images on to the
other
+microcontrollers on the GPU such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition
of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only
needs to
+to do FRTS, since SB is already already after reset by the time the kernel
driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which
contains
+data data required for power management. Once setup, only HS mode ucode can
+access it (see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that
contains
+various ucode images (also known as applications) -- one of them being FWSEC.
For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a
combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All
these
+ucode images are stored in the same ROM partition and the PMU table is used to
look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the purposes of nova-core driver, the FWSEC contains an 'application
interface'
+called DMEMMAPPER which is used to the "FWSEC-FRTS" command (among
other commands it
+is capable of executing). For Ampere, FWSEC is running on the GSP in
Heavy-secure
+mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs and is subject to
change
+completely, it is just provided as an example):
+
+Here is a block diagram of the FWSEC memory layout::
+ ?????????????????????????????????????????????????????????????????
+ ? FWSEC ROM image (type 0xE0) ?
+ ? ?
+ ? ??????????????????????????????????? ?
+ ? ? PMU Falcon Ucode Table ? ?
+ ? ? (PmuLookupTable) ? ?
+ ? ? ??????????????????????????? ? ?
+ ? ? ? Table Header ? ? ?
+ ? ? ? - version: 0x01 ? ? ?
+ ? ? ? - header_size: 6 ? ? ?
+ ? ? ? - entry_size: 6 ? ? ?
+ ? ? ? - entry_count: N ? ? ?
+ ? ? ? - desc_version:3(unused)? ? ?
+ ? ? ??????????????????????????? ? ?
+ ? ? ... ? ?
+ ? ? ??????????????????????????? ? ?
+ ? ? ? Entry for FWSEC (0x85) ? ? ?
+ ? ? ? (PmuLookupTableEntry) ? ? ?
+ ? ? ? - app_id: 0x85 (FWSEC) ? ????????? ?
+ ? ? ? - target_id: 0x01 (PMU) ? ? ? ?
+ ? ? ? - data: offset ???????????????????????? look up FWSEC ?
+ ? ? ??????????????????????????? ? ? ? application. ?
+ ? ??????????????????????????????????? ? ? ?
+ ? ? ? ?
+ ? ? ? ?
+ ? ??????????????????????????????????? ? ? ?
+ ? ? FWSEC Ucode Component ?<???? ? ?
+ ? ? (aka Falcon data) ? ? ?
+ ? ? ??????????????????????????? ? ? ?
+ ? ? ? FalconUCodeDescV3 ?<????????????? ?
+ ? ? ? - hdr ? ? ?
+ ? ? ? - stored_size ? ? ?
+ ? ? ? - pkc_data_offset ? ? ?
+ ? ? ? - interface_offset ???????????????????????????? ?
+ ? ? ? - imem_phys_base ? ? ? ?
+ ? ? ? - imem_load_size ? ? ? ?
+ ? ? ? - imem_virt_base ? ? ? ?
+ ? ? ? - dmem_phys_base ? ? ? ?
+ ? ? ? - dmem_load_size ? ? ? ?
+ ? ? ? - engine_id_mask ? ? ? ?
+ ? ? ? - ucode_id ? ? ? ?
+ ? ? ? - signature_count ? ? look up sig ? ?
+ ? ? ? - signature_versions --------------+ ? ?
+ ? ? ??????????????????????????? ? ? ? ?
+ ? ? (no gap) ? ? ? ?
+ ? ? ??????????????????????????? ? ? ? ?
+ ? ? ? Signatures Section ?<?????????? ? ?
+ ? ? ? (384 bytes per sig) ? ? ? ?
+ ? ? ? - RSA-3K Signature 1 ? ? ? ?
+ ? ? ? - RSA-3K Signature 2 ? ? ? ?
+ ? ? ? ... ? ? ? ?
+ ? ? ??????????????????????????? ? ? ?
+ ? ? ? ? ?
+ ? ? ??????????????????????????? ? ? ?
+ ? ? ? IMEM Section (Code) ? ? ? ?
+ ? ? ? ? ? ? ?
+ ? ? ? Contains instruction ? ? ? ?
+ ? ? ? code etc. ? ? ? ?
+ ? ? ??????????????????????????? ? ? ?
+ ? ? ? ? ?
+ ? ? ??????????????????????????? ? ? ?
+ ? ? ? DMEM Section (Data) ? ? ? ?
+ ? ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? Application ? ?<????????????????????? ?
+ ? ? ? ? Interface Table ? ? ? ?
+ ? ? ? ? (FalconAppifHdrV1) ? ? ? ?
+ ? ? ? ? Header: ? ? ? ?
+ ? ? ? ? - version: 0x01 ? ? ? ?
+ ? ? ? ? - header_size: 4 ? ? ? ?
+ ? ? ? ? - entry_size: 8 ? ? ? ?
+ ? ? ? ? - entry_count: N ? ? ? ?
+ ? ? ? ? ? ? ? ?
+ ? ? ? ? Entries: ? ? ? ?
+ ? ? ? ? ??????????????????? ? ? ? ?
+ ? ? ? ? ? DEVINIT (ID 1) ? ? ? ? ?
+ ? ? ? ? ? - id: 0x01 ? ? ? ? ?
+ ? ? ? ? ? - dmemOffset X ??????????? ?
+ ? ? ? ? ??????????????????? ? ? ? ?
+ ? ? ? ? ??????????????????? ? ? ? ?
+ ? ? ? ? ? DMEMMAPPER(ID 4)? ? ? ? ?
+ ? ? ? ? ? - id: 0x04 ? ? ? ? Used only for DevInit ?
+ ? ? ? ? ? (NVFW_FALCON_ ? ? ? ? application (not FWSEC) ?
+ ? ? ? ? ? APPIF_ID_DMEMMAPPER) ? ?
+ ? ? ? ? ? - dmemOffset Y ????????????????? ?
+ ? ? ? ? ??????????????????? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? DEVINIT Engine ?<?????? ? Used by FWSEC ?
+ ? ? ? ? Interface ? ? ? ? app. ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? DMEM Mapper (ID 4) ?<?????+?????? ?
+ ? ? ? ? (FalconAppifDmemmapperV3) ? ?
+ ? ? ? ? - signature: "DMAP" ? ? ? ?
+ ? ? ? ? - version: 0x0003 ? ? ? ?
+ ? ? ? ? - Size: 64 bytes ? ? ? ?
+ ? ? ? ? - cmd_in_buffer_off ? ??????????????????? ?
+ ? ? ? ? - cmd_in_buffer_size? ? ? ? ?
+ ? ? ? ? - cmd_out_buffer_off? ????????????????????????? ?
+ ? ? ? ? - cmd_out_buffer_sz ? ? ? ? ? ?
+ ? ? ? ? - init_cmd ? ? ? ? ? ?
+ ? ? ? ? - features ? ? ? ? ? ?
+ ? ? ? ? - cmd_mask0/1 ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ? ?
+ ? ? ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ? ?
+ ? ? ? ? Command Input Buffer?<??????????????????? ? ?
+ ? ? ? ? - Command data ? ? ? ? ?
+ ? ? ? ? - Arguments ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ? ?
+ ? ? ? ? Command Output ?<????????????????????????? ?
+ ? ? ? ? Buffer ? ? ? ?
+ ? ? ? ? - Results ? ? ? ?
+ ? ? ? ? - Status ? ? ? ?
+ ? ? ? ??????????????????????? ? ? ?
+ ? ? ??????????????????????????? ? ?
+ ? ??????????????????????????????????? ?
+ ? ?
+ ?????????????????????????????????????????????????????????????????
+
+.. note::
+ The FWSEC image also plays a role in memory scrubbing (ECC initialization)
and VPR
+ (Video Protected Region) initialization as well. Before the nova-core driver
is even
+ loaded, the FWSEC image is running on the GSP in heavy-secure mode. After
the devinit
+ sequence completes, it does VRAM memory scrubbing (ECC initialization). On
consumer
+ GPUs, it scrubs only part of memory and then initiates 'async
scrubbing'. Before this
+ async scrubbing completes, the unscrubbed VRAM cannot be used for allocation
(thus DRM
+ memory allocators need to wait for this scrubbing to complete).
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 91cc802ed94f..22e5712ac6b0 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
+ core/fwsec
core/todo
--
2.43.0
Hello, Please find attached initial documentation for nova-core mainly about the vbios and boot process. It helps build an understanding of the boot code. All the information is derived from publicly available code and sources. This series is based on Alex's v2 patches + nova-next: https://lore.kernel.org/all/20250501-nova-frts-v2-0-b4a137175337 at nvidia.com/ Previous posting: https://lore.kernel.org/all/20250423225405.139613-1-joelagnelf at nvidia.com/ Joel Fernandes (7): nova-core: doc: Add code comments related to devinit nova-core: doc: Clarify sysmembar operations nova-core: docs: Document vbios layout nova-core: docs: Document fwsec operation and layout docs: nova-core: Document devinit process docs: nova-core: Document basics of the Falcon gpu: nova-core: Clarify falcon code Documentation/gpu/nova/core/devinit.rst | 59 ++++++++ Documentation/gpu/nova/core/falcon.rst | 156 ++++++++++++++++++++ Documentation/gpu/nova/core/fwsec.rst | 180 ++++++++++++++++++++++++ Documentation/gpu/nova/core/vbios.rst | 177 +++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 4 + drivers/gpu/nova-core/devinit.rs | 36 ++++- drivers/gpu/nova-core/falcon.rs | 25 +++- drivers/gpu/nova-core/gpu.rs | 12 +- drivers/gpu/nova-core/regs.rs | 18 ++- 9 files changed, 652 insertions(+), 15 deletions(-) create mode 100644 Documentation/gpu/nova/core/devinit.rst create mode 100644 Documentation/gpu/nova/core/falcon.rst create mode 100644 Documentation/gpu/nova/core/fwsec.rst create mode 100644 Documentation/gpu/nova/core/vbios.rst -- 2.43.0
Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 3/7] nova-core: docs: Document vbios layout
Add detailed explanation and block diagrams of the layout of the vBIOS
on Nvidia GPUs. This is important to understand how nova-core boots an
Nvidia GPU.
[ Applied Timur Tabi's feedback on providing link to BIT documentation. ]
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/vbios.rst | 177 ++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 178 insertions(+)
create mode 100644 Documentation/gpu/nova/core/vbios.rst
diff --git a/Documentation/gpu/nova/core/vbios.rst
b/Documentation/gpu/nova/core/vbios.rst
new file mode 100644
index 000000000000..04ced35648cb
--- /dev/null
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -0,0 +1,177 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+=========+VBIOS
+=========+This document describes the layout of the VBIOS image which is a
series of concatenated
+images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is
read
+by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the
GPU to
+boot strap various microcontrollers (PMU, SEC, GSP) with critical
initialization before
+the driver loads, as well as by the nova-core driver in the kernel to boot the
GSP.
+
+The format of the images in the ROM follow the "BIOS Specification"
part of the
+PCI specification, with Nvidia-specific extensions. The ROM images of type
FwSec
+are the ones that contain Falcon ucode and what we are mainly looking for.
+
+As an example, the following are the different image types that can be found in
the
+VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
+
+- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's
naming likely
+ comes from the "IBM PC/AT" architecture.
+
+- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
+ driver that is used to display UEFI graphics output.
+
+- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
+
+- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
+ contains various different microcodes (also known as an applications) that do
+ a range of different functions. The FWSEC ucode is run in heavy-secure mode
and
+ typically runs directly on the GSP (it could be running on a different
designated
+ processor in future generations but as of Ampere, it is the GSP). This
firmware
+ then loads other firmware ucodes onto the PMU and SEC2 microcontrollers for
gfw
+ initialization after GPU reset and before the driver loads (see devinit.rst).
+ The DEVINIT ucode is itself another ucode that is stored in this ROM
partition.
+
+The Falcon ucodes once located have "Application Interfaces" in the
data memory (DMEM)
+of the ucode. For FWSEC, the application interface we use for FWSEC is the
"DMEM mapper"
+interface which is configured to run the "FRTS" command. This command
performs carving
+out of the WPR2 area (Write protected region) in the VRAM and placing important
data
+called 'FRTS' into it which contains power-management data. The WPR2
region is only
+accessible to heavy-secure ucode.
+
+.. note::
+ It is not clear why FwSec has 2 different partitions in the ROM, but they
both
+ are of type 0xE0 and can be identified as such. This could be subject to
change
+ in future generations.
+
+VBIOS ROM Layout
+----------------
+The VBIOS layout is roughly a series of concatenated images as follows:
+(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
+
+.. note::
+ This diagram is created based on an GA-102 Ampere GPU as an example and
could
+ vary for future or other GPUs.
+
+Here is a block diagram of the VBIOS layout::
+
+ ??????????????????????????????????????????????????????????????????????????
+ ? VBIOS (Starting at ROM_OFFSET: 0x300000) ?
+ ??????????????????????????????????????????????????????????????????????????
+ ? ????????????????????????????????????????????????? ?
+ ? ? PciAt Image (Type 0x00) ? ?
+ ? ????????????????????????????????????????????????? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? ROM Header ? ? ?
+ ? ? ? (Signature 0xAA55)? ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? rom header's pci_data_struct_offset ? ?
+ ? ? ? points to the PCIR structure ? ?
+ ? ? V ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? PCIR Structure ? ? ?
+ ? ? ? (Signature "PCIR")? ?
?
+ ? ? ? last_image: 0x80 ? ? ?
+ ? ? ? image_len: size ? ? ?
+ ? ? ? in 512-byte units ? ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? ? ?
+ ? ? ? NPDE immediately follows PCIR ? ?
+ ? ? V ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? NPDE Structure ? ? ?
+ ? ? ? (Signature "NPDE")? ?
?
+ ? ? ? last_image: 0x00 ? ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? BIT Header ? (Signature scanning ? ?
+ ? ? ? (Signature "BIT") ? provides the location ?
?
+ ? ? ????????????????????? of the BIT table) ? ?
+ ? ? ? header is ? ?
+ ? ? | followed by a table of tokens ? ?
+ ? ? V one of which is for falcon data. ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? BIT Tokens ? ? ?
+ ? | | ______________ | | |
+ ? ? ? ? Falcon Data ? ? ? ?
+ ? ? ? ? Token (0x70)?---+------------>------------???+ ?
+ ? ? ? ??????????????? ? falcon_data_ptr() ? ? ?
+ ? ? ????????????????????? ? V ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? (no gap between images) ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? ? EFI Image (Type 0x03) ? ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? | Contains the UEFI GOP driver (Graphics Output)| | |
+ ? ? ????????????????????? ? ? ?
+ ? ? ? ROM Header ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? PCIR Structure ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? NPDE Structure ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? Image data ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? (no gap between images) ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? ? First FwSec Image (Type 0xE0) ? ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? ROM Header ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? PCIR Structure ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? NPDE Structure ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? Image data ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? (no gap between images) ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? ? Second FwSec Image (Type 0xE0) ? ? ?
+ ? ????????????????????????????????????????????????? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? ROM Header ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? PCIR Structure ? ? ? ?
+ ? ? +???????????????????+ ? ? ?
+ ? ? ? NPDE Structure ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? ? ?
+ ? ? ????????????????????? ? ? ?
+ ? ? ? PMU Lookup Table ? <- falcon_data_offset ?<??
?
+ ? ? ? ??????????????? ? pmu_lookup_table ? ?
+ ? ? ? ? Entry 0x85 ? ? ? ?
+ ? ? ? ? FWSEC_PROD ? ? ? ?
+ ? ? ? ??????????????? ? ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? ? ?
+ ? ? ? points to ? ?
+ ? ? V ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? FalconUCodeDescV3 ? <- falcon_ucode_offset ? ?
+ ? ? ? (FWSEC Firmware) ? fwsec_header() ? ?
+ ? ? ????????????????????? ? ?
+ ? ? ? immediately followed by... ? ?
+ ? ? V ? ?
+ ? ? ?????????????????????????????? ? ?
+ ? ? ? Signatures + FWSEC Ucode ? ? ?
+ ? ? ? fwsec_sigs(), fwsec_ucode()? ? ?
+ ? ? ?????????????????????????????? ? ?
+ ? ?????????????????????????????????????????????????______________________?
+
+Falcon data Lookup
+------------------
+A key part of the VBIOS extraction code (vbios.rs) is to find the location of
the
+Falcon data in the VBIOS which contains the PMU lookup table. This lookup table
is
+used to find the required Falcon ucode based on an application ID.
+
+The location of the PMU lookup table is found by scanning the BIT (`BIOS
Information Table`_)
+tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which
indicates the
+offset of the same from the start of the VBIOS image. Unfortunately, the offset
+does not account for the EFI image that sits between the PciAt image and the
FwSec
+images, the vbios.rs code compensates for that with appropriate arithmetic.
+
+.. _`BIOS Information Table`:
https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af35..91cc802ed94f 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,4 +27,5 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/vbios
core/todo
--
2.43.0
Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 5/7] docs: nova-core: Document devinit process
devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/devinit.rst | 59 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
drivers/gpu/nova-core/devinit.rs | 6 +--
3 files changed, 63 insertions(+), 3 deletions(-)
create mode 100644 Documentation/gpu/nova/core/devinit.rst
diff --git a/Documentation/gpu/nova/core/devinit.rst
b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 000000000000..85e7033dd3a7
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,59 @@
+.. SPDX-License-Identifier: GPL-2.0
+=================================+Device Initialization (devinit)
+=================================+Note that devinit itself is a complicated
process, subject to change, and this document
+only roughly provides an overview of the process using the Ampere GPU family as
an example.
+The idea is to provide a conceptual overview of the process to help understand
the kernel
+code that deals with it.
+
+Device initialization (devinit) is a crucial sequence of register read/write
operations
+that occur after a GPU reset. The devinit sequence is essential for properly
configuring
+the GPU hardware before it can be used.
+
+The devinit engine is an interpreter program that typically runs on the PMU
(Power Management
+Unit) microcontroller of the GPU. This interpreter executes a
"script" of initialization
+commands. The devinit engine itself is part of the VBIOS ROM in the same ROM
image as the
+FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs
before the
+nova-core driver is even loaded. On an Ampere GPU, the devinit ucode (which is
separate
+from the FWSEC ucode), is launched by the FWSEC (FWSEC runs on GSP in
heavy-secure mode
+and devinit runs on PMU in light-secure mode).
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+
+Low-level Firmware Initialization Flow
+--------------------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.)
run GPU
+firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU
is
+considered unusable until this initialization process completes.
+
+These low-level GPU firmware components are typically:
+1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and
fwsec.rst).
+2. Executed in sequence on different microcontrollers:
+ - The devinit engine typically but not necessarily runs on the PMU.
+ - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System
Processor) in
+ heavy-secure mode.
+
+Before the driver can proceed with further initialization, it must wait for a
signal
+indicating that core initialization is complete (known as GFW_BOOT). This
signal is
+asserted by the FWSEC running on the GSP in heavy-secure mode.
+
+Runtime Considerations
+---------------------
+It's important to note that the devinit sequence also needs to run during
suspend/resume
+operations at runtime, not just during initial boot, as it is critical to power
management.
+
+Security and Access Control
+--------------------------
+The initialization process involves careful privilege management. For example,
before
+accessing certain completion status registers, the driver must check privilege
level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers
the
+privilege level to allow CPU (LS/low-secure) access. For example, to receive
the
+GFW_BOOT signal.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 22e5712ac6b0..301435c5cf67 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -29,4 +29,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
core/fwsec
+ core/devinit
core/todo
diff --git a/drivers/gpu/nova-core/devinit.rs b/drivers/gpu/nova-core/devinit.rs
index 21fc9b96d498..bd6a5c57c444 100644
--- a/drivers/gpu/nova-core/devinit.rs
+++ b/drivers/gpu/nova-core/devinit.rs
@@ -10,8 +10,8 @@
//! 3. Clock and PLL configuration.
//! 4. Thermal management.
//!
-//! devinit itself is a 'script' which is interpreted by the PMU
microcontroller of
-//! the GPU by an interpreter program.
+//! devinit itself is a 'script' which is interpreted by a
microcontroller of
+//! the GPU by an interpreter program (typically running on the PMU).
//!
//! Note that the devinit sequence also needs to run during suspend/resume at
runtime.
@@ -30,7 +30,7 @@
///
/// The GPU firmware (gfw) code includes several components that execute before
the driver loads.
/// These components are located in the VBIOS ROM and are executed in a
sequence on these different
-/// microcontrollers. The devinit sequence itself runs on the PMU, and the
FWSEC runs on the GSP.
+/// microcontrollers. The devinit sequence typically runs on the PMU, and the
FWSEC runs on the GSP.
///
/// This function specifically waits for a signal indicating core
initialization is complete before
/// which not much can be done. This signal is setup by the FWSEC running on
the GSP in Heavy-secured
--
2.43.0
Add documentation strings, comments and AES mode for completeness
to the Falcon signatures.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/falcon.rs | 25 +++++++++++++++++++------
1 file changed, 19 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon.rs b/drivers/gpu/nova-core/falcon.rs
index e9ee0c83dfc5..003db40d3303 100644
--- a/drivers/gpu/nova-core/falcon.rs
+++ b/drivers/gpu/nova-core/falcon.rs
@@ -89,13 +89,19 @@ fn try_from(value: u8) -> Result<Self> {
/// register.
#[repr(u8)]
#[derive(Debug, Default, Copy, Clone)]
+/// Security mode of the Falcon microprocessor.
+/// See falcon.rst for more details.
pub(crate) enum FalconSecurityModel {
/// Non-Secure: runs unsigned code without privileges.
#[default]
None = 0,
- /// Low-secure: runs unsigned code with some privileges. Can only be
entered from `Heavy` mode.
+ /// Light-Secured (LS): runs signed code with some privileges
+ /// Its signature can only be verified and entered from `Heavy` mode.
+ /// Also known as Privilege Level 2 or PL2.
Light = 2,
- /// High-Secure: runs signed code with full privileges.
+ /// Heavy-Secured: runs signed code with full privileges.
+ /// Its signature can only be verified by the Falcon Boot ROM (BROM).
+ /// Also known as Privilege Level 3 or PL3.
Heavy = 3,
}
@@ -117,10 +123,13 @@ fn try_from(value: u8) -> core::result::Result<Self,
Self::Error> {
}
/// Signing algorithm for a given firmware, used in the
[`crate::regs::NV_PFALCON2_FALCON_MOD_SEL`]
-/// register.
+/// register. It is passed to the Falcon Boot ROM (BROM) as a parameter.
#[repr(u8)]
#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
pub(crate) enum FalconModSelAlgo {
+ /// AES.
+ #[expect(dead_code)]
+ Aes = 0,
/// RSA3K.
#[default]
Rsa3k = 1,
@@ -184,15 +193,19 @@ pub(crate) enum FalconMem {
Dmem,
}
-/// Target/source of a DMA transfer to/from falcon memory.
+/// FBIF (Framebuffer Interface) aperture type. Used to determine
+/// the memory type of the external memory access for a DMA memory
+/// transfer (by the Falcon's FramebufferDMA (FBDMA) engine located
+/// inside the falcon). See falcon.rst for more details.
#[derive(Debug, Clone, Default)]
pub(crate) enum FalconFbifTarget {
/// VRAM.
#[default]
+ /// Local Framebuffer (GPU's VRAM memory)
LocalFb = 0,
- /// Coherent system memory.
+ /// Coherent system memory (System DRAM).
CoherentSysmem = 1,
- /// Non-coherent system memory.
+ /// Non-coherent system memory (System DRAM).
NoncoherentSysmem = 2,
}
--
2.43.0
Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 6/7] docs: nova-core: Document basics of the Falcon
Instances of the Falcon microcontroller appear in modern Nvidia GPUs and
are crucial to the GPU boot process. Document some concepts which will
make nova-core boot code easier to digest. All the information is
derived from public sources such as public documents, OpenRM and Nouveau
code.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
Documentation/gpu/nova/core/falcon.rst | 156 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 157 insertions(+)
create mode 100644 Documentation/gpu/nova/core/falcon.rst
diff --git a/Documentation/gpu/nova/core/falcon.rst
b/Documentation/gpu/nova/core/falcon.rst
new file mode 100644
index 000000000000..f2b89cc2a159
--- /dev/null
+++ b/Documentation/gpu/nova/core/falcon.rst
@@ -0,0 +1,156 @@
+=============================+Falcon (FAst Logic Controller)
+=============================+The following sections describe the Falcon core
and the ucode running on it.
+The descriptions are based on the Ampere GPU or earlier designs; however, they
+should mostly apply to future designs as well, but everything is subject to
+change. The overview provided here is mainly tailored towards understanding the
+interactions of nova-core driver with the Falcon.
+
+NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
+handle secure firmware tasks, initialization, and power management. Modern
+NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
+processor) and SEC2 (the security engine)) and also may integrate a RISC-V
core.
+This core is capable of running both RISC-V and Falcon code.
+
+The code running on the Falcons is also called Ucode and will be referred to as
+such in the following sections.
+
+Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
+small DMA engine (via the FBIF - "Frame Buffer Interface") to load
code from
+system memory. The nova-core driver must reset and configure the Falcon, load
+its firmware via DMA, and start its CPU.
+
+Falcon security levels
+=====================+Falcons can run in Non-secure (NS), Light Secure (LS), or
Heavy Secure (HS)
+modes.
+
+Heavy Secured (HS) also known as Privilege Level 3 (PL3)
+--------------------------------------------------------
+HS ucode is the most trusted code and has access to pretty much everything on
+the chip. The HS binary includes a signature in it which is verified at boot.
+This signature verification is done by the hardware itself, thus establishing a
+root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
+GSP in HS mode. FRTS, which involves setting up and loading content into the
WPR
+(Write Protect Region), has to be done by the HS ucode and cannot be done by
the
+host CPU or LS ucode.
+
+Light Secured (LS or PL2) and Non Secured (NS or PL0)
+-----------------------------------------------------
+These modes are less secure than HS. Like HS, the LS or NS ucode binary also
+typically includes a signature in it. To load firmware in LS or NS mode onto a
+Falcon, another Falcon needs to be running in HS mode, which also establishes
the
+root of trust. For example, in the case of an Ampere GPU, the CPU runs the
"Booter"
+ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
+run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
+example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
+devinit engine onto the PMU in LS mode.
+
+Root of trust establishment
+---------------------------
+To establish a root of trust, the code running of a Falcon has to be something
+that that cannot be erased and is hardwired into a read-only-memory (ROM). This
+follows industry norms for verification of firmware. This code is called the
+Boot ROM (BROM). The nova-core driver on the CPU communicates with Falcon's
Boot
+ROM through various Falcon registers prefixed with "BROM" (see
regs.rs).
+
+After nova-core driver reads the necessary ucode from VBIOS, it programs the
+BROM and DMA registers to trigger the Falcon to load the HS ucode from the
system
+memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is
verified
+by the Falcon's Boot ROM.
+
+Once the verified HS code is running on a Falcon, it can verify and load other
+LS/NS ucode binaries onto other Falcons and start them. The process of
signature
+verification is the same as HS; just in this case, the hardware (BROM)
doesn't
+compute the signature, but the HS ucode does.
+
+Thus the root of trust is:
+ Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS
ucode.
+
+Example on Ampere GPU, the boot verification flow is:
+ Hardware (Boot ROM running on the SEC2) ->
+ HS ucode (Booter running on the SEC2) ->
+ LS ucode (GSP-RM running on the GSP)
+
+.. note::
+ While the CPU can load HS ucode onto a Falcon microcontroller and have it
+ verified by the hardware and run, the CPU itself typically does not load
+ LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
+ HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on
the
+ SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
+ the "SEC2-RTOS" ucode at runtime. This presents a problem where
there is
+ no one to load the SEC2-RTOS ucode onto the SEC2 (i.e., the CPU is
incapable
+ of loading LS code, and GSP-RM has to run LS mode). To overcome this,
+ the GSP is temporarily made to run HS ucode (which is itself loaded by
+ the CPU via the nova-core driver using a "GSP-provided
sequencer")
+ which then loads the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP
+ then resumes running its own GSP-RM LS ucode.
+
+Falcon memory subsystem and DMA engine
+=====================================+Falcons have separate instruction and
data memories (IMEM/DMEM)
+and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
+DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
+(Framebuffer Interface), to external memory.
+
+DMA transfers are possible from the Falcon's memory to both the system
memory
+and the framebuffer memory (VRAM).
+
+To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
+is accessed (also known as aperture type). In the nova-core driver, this is
+determined by the `FalconFbifTarget` enum.
+
+The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
+controls access by the FBDMA to the external memory.
+
+Conceptual diagram (not exact) of the Falcon and its memory subsystem is as
follows:
+
+ External Memory (Framebuffer / System DRAM)
+ ? ?
+ ? ?
+ ? ?
+???????????????????????????????????????????????????????
+? ? ?
+? ????????????????? ? ?
+? ? FBIF ????????? ? FALCON
+? ? (FrameBuffer ? Memory Interface ? PROCESSOR
+? ? InterFace) ? ?
+? ? Apertures ? ?
+? ? Configures ? ?
+? ? mem access ? ?
+? ????????????????? ?
+? ? ?
+? ? FBDMA uses configured FBIF apertures ?
+? ? to access External Memory
+? ?
+? ????????????????? ??????????????????
+? ? FBDMA ? cfg ? RISC ?
+? ? (FrameBuffer ?<????>? CORE ??????>. Direct Core
Access
+? ? DMA Engine) ? ? ? ?
+? ? - Master dev. ? ? (can run both ? ?
+? ????????????????? ? Falcon and ? ?
+? ? cfg-->? RISC-V code) ? ?
+? ? / ? ? ?
+? ? | ?????????????????? ? ??????????????
+? ? ? ? ? BROM ?
+? ? ? <???>? (Boot ROM) ?
+? ? / ? ??????????????
+? ? ? ?
+? ????????????????? ?
+? ? IO-PMP ? Controls access by FBDMA ?
+? ? (IO Physical ? and other IO Masters ?
+? ? Memory Protect) ?
+? ????????????????? ?
+? ? ?
+? ? Protected Access Path for FBDMA ?
+? ? ?
+? ????????????????????????????????????????? ?
+? ? Memory ? ?
+? ? ????????????????? ??????????????? ? ?
+? ? ? IMEM ? ? DMEM ? ?<??????
+? ? ? (Instruction ? ? (Data ? ?
+? ? ? Memory) ? ? Memory) ? ?
+? ? ????????????????? ??????????????? ?
+? ?????????????????????????????????????????
+?????????????????????????????????????????????????????
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 301435c5cf67..75a98ab63055 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,6 +28,7 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
+ core/falcon
core/fwsec
core/devinit
core/todo
--
2.43.0
Joel Fernandes
2025-May-03 04:08 UTC
[PATCH v2 2/7] nova-core: doc: Clarify sysmembar operations
sysmembar is a critical operation that the GSP falcon needs to perform
in the reset sequence. Add some code comments to clarify.
Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
drivers/gpu/nova-core/gpu.rs | 12 +++++++++---
drivers/gpu/nova-core/regs.rs | 2 ++
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index a46768d18ac3..9bd73755c466 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -169,8 +169,8 @@ pub(crate) struct Gpu {
/// MMIO mapping of PCI BAR 0
bar: Devres<Bar0>,
fw: Firmware,
- // System memory page required for flushing all pending GPU-side memory
writes done through
- // PCIE into system memory.
+ /// System memory page required for flushing all pending GPU-side memory
writes done through
+ /// PCIE into system memory, via sysmembar (A GPU-initiated HW
memory-barrier operation).
sysmem_flush: DmaObject,
}
@@ -215,7 +215,13 @@ pub(crate) fn new(
devinit::wait_gfw_boot_completion(&bar)
.inspect_err(|_| dev_err!(pdev.as_ref(), "GFW boot did not
complete"))?;
- // System memory page required for sysmembar to properly flush into
system memory.
+ // System memory page required for sysmembar which is a GPU-initiated
hardware
+ // memory-barrier operation that flushes all pending GPU-side memory
writes that
+ // were done through PCIE, to system memory. It is required for Falcon
to be reset
+ // as the reset operation involves a reset handshake. When the falcon
acks the
+ // reset, it writes its acknowledgement into system memory, but for
this write to
+ // be visible to the host, the falcon needs to do sysmembar to flush
+ // its writes and prevent the driver from timing out.
let sysmem_flush = {
let page = DmaObject::new(pdev.as_ref(),
kernel::bindings::PAGE_SIZE)?;
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index a2b506883654..dcac7ab4619f 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -52,6 +52,8 @@ pub(crate) fn chipset(self) -> Result<Chipset, Error>
{
/* PFB */
+// These two registers together hold the physical system memory address
+// that is used by the GPU for perform sysmembar operation (see
[`crate::gpu`]).
register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR @ 0x00100c10 {
31:0 adr_39_08 as u32;
});
--
2.43.0
On Sat, May 03, 2025 at 12:07:55AM -0400, Joel Fernandes wrote:> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) > +=========> +VBIOS > +=========Separate SPDX line and title heading.> +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's naming likely > + comes from the "IBM PC/AT" architecture.Combine these two lines on the same bullet list item.> +Here is a block diagram of the VBIOS layout:: > + > + ?????????????????????????????????????????????????????????????????????????? > + ? VBIOS (Starting at ROM_OFFSET: 0x300000) ? > + ?????????????????????????????????????????????????????????????????????????? > + ? ????????????????????????????????????????????????? ? > + ? ? PciAt Image (Type 0x00) ? ? > + ? ????????????????????????????????????????????????? ? > + ? ? ????????????????????? ? ? > + ? ? ? ROM Header ? ? ? > + ? ? ? (Signature 0xAA55)? ? ? > + ? ? ????????????????????? ? ? > + ? ? ? rom header's pci_data_struct_offset ? ? > + ? ? ? points to the PCIR structure ? ? > + ? ? V ? ? > + ? ? ????????????????????? ? ? > + ? ? ? PCIR Structure ? ? ? > + ? ? ? (Signature "PCIR")? ? ? > + ? ? ? last_image: 0x80 ? ? ? > + ? ? ? image_len: size ? ? ? > + ? ? ? in 512-byte units ? ? ? > + ? ? ????????????????????? ? ? > + ? ? ? ? ? > + ? ? ? NPDE immediately follows PCIR ? ? > + ? ? V ? ? > + ? ? ????????????????????? ? ? > + ? ? ? NPDE Structure ? ? ? > + ? ? ? (Signature "NPDE")? ? ? > + ? ? ? last_image: 0x00 ? ? ? > + ? ? ????????????????????? ? ? > + ? ? ? ? > + ? ? ????????????????????? ? ? > + ? ? ? BIT Header ? (Signature scanning ? ? > + ? ? ? (Signature "BIT") ? provides the location ? ? > + ? ? ????????????????????? of the BIT table) ? ? > + ? ? ? header is ? ? > + ? ? | followed by a table of tokens ? ? > + ? ? V one of which is for falcon data. ? ? > + ? ? ????????????????????? ? ? > + ? ? ? BIT Tokens ? ? ? > + ? | | ______________ | | | > + ? ? ? ? Falcon Data ? ? ? ? > + ? ? ? ? Token (0x70)?---+------------>------------???+ ? > + ? ? ? ??????????????? ? falcon_data_ptr() ? ? ? > + ? ? ????????????????????? ? V ? > + ? ????????????????????????????????????????????????? ? ? > + ? (no gap between images) ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? ? EFI Image (Type 0x03) ? ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? | Contains the UEFI GOP driver (Graphics Output)| | | > + ? ? ????????????????????? ? ? ? > + ? ? ? ROM Header ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? PCIR Structure ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? NPDE Structure ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? Image data ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? (no gap between images) ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? ? First FwSec Image (Type 0xE0) ? ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? ROM Header ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? PCIR Structure ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? NPDE Structure ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? Image data ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? (no gap between images) ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? ? Second FwSec Image (Type 0xE0) ? ? ? > + ? ????????????????????????????????????????????????? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? ROM Header ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? PCIR Structure ? ? ? ? > + ? ? +???????????????????+ ? ? ? > + ? ? ? NPDE Structure ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ????????????????????? ? ? ? > + ? ? ? PMU Lookup Table ? <- falcon_data_offset ?<?? ? > + ? ? ? ??????????????? ? pmu_lookup_table ? ? > + ? ? ? ? Entry 0x85 ? ? ? ? > + ? ? ? ? FWSEC_PROD ? ? ? ? > + ? ? ? ??????????????? ? ? ? > + ? ? ????????????????????? ? ? > + ? ? ? ? ? > + ? ? ? points to ? ? > + ? ? V ? ? > + ? ? ????????????????????? ? ? > + ? ? ? FalconUCodeDescV3 ? <- falcon_ucode_offset ? ? > + ? ? ? (FWSEC Firmware) ? fwsec_header() ? ? > + ? ? ????????????????????? ? ? > + ? ? ? immediately followed by... ? ? > + ? ? V ? ? > + ? ? ?????????????????????????????? ? ? > + ? ? ? Signatures + FWSEC Ucode ? ? ? > + ? ? ? fwsec_sigs(), fwsec_ucode()? ? ? > + ? ? ?????????????????????????????? ? ? > + ? ?????????????????????????????????????????????????______________________?Diagram borders look messy in htmldocs output (due to Unicode characters ? and ?), so I use ASCII dash and vertical bar instead: ---- >8 ---- diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst index dd6ac891e5f1d0..c68ef0e7b70124 100644 --- a/Documentation/gpu/nova/core/vbios.rst +++ b/Documentation/gpu/nova/core/vbios.rst @@ -56,112 +56,113 @@ The VBIOS layout is roughly a series of concatenated images as follows: Here is a block diagram of the VBIOS layout:: - ?????????????????????????????????????????????????????????????????????????? - ? VBIOS (Starting at ROM_OFFSET: 0x300000) ? - ?????????????????????????????????????????????????????????????????????????? - ? ????????????????????????????????????????????????? ? - ? ? PciAt Image (Type 0x00) ? ? - ? ????????????????????????????????????????????????? ? - ? ? ????????????????????? ? ? - ? ? ? ROM Header ? ? ? - ? ? ? (Signature 0xAA55)? ? ? - ? ? ????????????????????? ? ? - ? ? ? rom header's pci_data_struct_offset ? ? - ? ? ? points to the PCIR structure ? ? - ? ? V ? ? - ? ? ????????????????????? ? ? - ? ? ? PCIR Structure ? ? ? - ? ? ? (Signature "PCIR")? ? ? - ? ? ? last_image: 0x80 ? ? ? - ? ? ? image_len: size ? ? ? - ? ? ? in 512-byte units ? ? ? - ? ? ????????????????????? ? ? - ? ? ? ? ? - ? ? ? NPDE immediately follows PCIR ? ? - ? ? V ? ? - ? ? ????????????????????? ? ? - ? ? ? NPDE Structure ? ? ? - ? ? ? (Signature "NPDE")? ? ? - ? ? ? last_image: 0x00 ? ? ? - ? ? ????????????????????? ? ? - ? ? ? ? - ? ? ????????????????????? ? ? - ? ? ? BIT Header ? (Signature scanning ? ? - ? ? ? (Signature "BIT") ? provides the location ? ? - ? ? ????????????????????? of the BIT table) ? ? - ? ? ? header is ? ? - ? ? | followed by a table of tokens ? ? - ? ? V one of which is for falcon data. ? ? - ? ? ????????????????????? ? ? - ? ? ? BIT Tokens ? ? ? - ? | | ______________ | | | - ? ? ? ? Falcon Data ? ? ? ? - ? ? ? ? Token (0x70)?---+------------>------------???+ ? - ? ? ? ??????????????? ? falcon_data_ptr() ? ? ? - ? ? ????????????????????? ? V ? - ? ????????????????????????????????????????????????? ? ? - ? (no gap between images) ? ? - ? ????????????????????????????????????????????????? ? ? - ? ? EFI Image (Type 0x03) ? ? ? - ? ????????????????????????????????????????????????? ? ? - ? | Contains the UEFI GOP driver (Graphics Output)| | | - ? ? ????????????????????? ? ? ? - ? ? ? ROM Header ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? PCIR Structure ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? NPDE Structure ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? Image data ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ????????????????????????????????????????????????? ? ? - ? (no gap between images) ? ? - ? ????????????????????????????????????????????????? ? ? - ? ? First FwSec Image (Type 0xE0) ? ? ? - ? ????????????????????????????????????????????????? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? ROM Header ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? PCIR Structure ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? NPDE Structure ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? Image data ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ????????????????????????????????????????????????? ? ? - ? (no gap between images) ? ? - ? ????????????????????????????????????????????????? ? ? - ? ? Second FwSec Image (Type 0xE0) ? ? ? - ? ????????????????????????????????????????????????? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? ROM Header ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? PCIR Structure ? ? ? ? - ? ? +???????????????????+ ? ? ? - ? ? ? NPDE Structure ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? ? ? - ? ? ????????????????????? ? ? ? - ? ? ? PMU Lookup Table ? <- falcon_data_offset ?<?? ? - ? ? ? ??????????????? ? pmu_lookup_table ? ? - ? ? ? ? Entry 0x85 ? ? ? ? - ? ? ? ? FWSEC_PROD ? ? ? ? - ? ? ? ??????????????? ? ? ? - ? ? ????????????????????? ? ? - ? ? ? ? ? - ? ? ? points to ? ? - ? ? V ? ? - ? ? ????????????????????? ? ? - ? ? ? FalconUCodeDescV3 ? <- falcon_ucode_offset ? ? - ? ? ? (FWSEC Firmware) ? fwsec_header() ? ? - ? ? ????????????????????? ? ? - ? ? ? immediately followed by... ? ? - ? ? V ? ? - ? ? ?????????????????????????????? ? ? - ? ? ? Signatures + FWSEC Ucode ? ? ? - ? ? ? fwsec_sigs(), fwsec_ucode()? ? ? - ? ? ?????????????????????????????? ? ? - ? ?????????????????????????????????????????????????______________________? + ?------------------------------------------------------------------------? + | VBIOS (Starting at ROM_OFFSET: 0x300000) | + ?------------------------------------------------------------------------? + | ?-----------------------------------------------? | + | | PciAt Image (Type 0x00) | | + | ?-----------------------------------------------? | + | | ?-------------------? | | + | | | ROM Header | | | + | | | (Signature 0xAA55)| | | + | | ?-------------------? | | + | | | rom header's pci_data_struct_offset | | + | | | points to the PCIR structure | | + | | V | | + | | ?-------------------? | | + | | | PCIR Structure | | | + | | | (Signature "PCIR")| | | + | | | last_image: 0x80 | | | + | | | image_len: size | | | + | | | in 512-byte units | | | + | | ?-------------------? | | + | | | | | + | | | NPDE immediately follows PCIR | | + | | V | | + | | ?-------------------? | | + | | | NPDE Structure | | | + | | | (Signature "NPDE")| | | + | | | last_image: 0x00 | | | + | | ?-------------------? | | + | | | | + | | ?-------------------? | | + | | | BIT Header | (Signature scanning | | + | | | (Signature "BIT") | provides the location | | + | | ?-------------------? of the BIT table) | | + | | | header is | | + | | | followed by a table of tokens | | + | | V one of which is for falcon data. | | + | | ?-------------------? | | + | | | BIT Tokens | | | + | | | ?-------------? | | | + | | | | Falcon Data | | | | + | | | | Token (0x70)|---+------------>------------?--+ | + | | | ?-------------? | falcon_data_ptr() | | | + | | ?-------------------? | V | + | ?-----------------------------------------------? | | + | (no gap between images) | | + | ?-----------------------------------------------? | | + | | EFI Image (Type 0x03) | | | + | ?-----------------------------------------------? | | + | | Contains the UEFI GOP driver (Graphics Output)| | | + | | ?-------------------? | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | ?-------------------? | | | + | | | Image data | | | | + | | ?-------------------? | | | + | ?-----------------------------------------------? | | + | (no gap between images) | | + | ?-----------------------------------------------? | | + | | First FwSec Image (Type 0xE0) | | | + | ?-----------------------------------------------? | | + | | ?-------------------? | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | ?-------------------? | | | + | | | Image data | | | | + | | ?-------------------? | | | + | ?-----------------------------------------------? | | + | (no gap between images) | | + | ?-----------------------------------------------? | | + | | Second FwSec Image (Type 0xE0) | | | + | ?-----------------------------------------------? | | + | | ?-------------------? | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | ?-------------------? | | | + | | | | | + | | ?-------------------? | | | + | | | PMU Lookup Table | <- falcon_data_offset |<-? | + | | | ?-------------? | pmu_lookup_table | | + | | | | Entry 0x85 | | | | + | | | | FWSEC_PROD | | | | + | | | ?-------------? | | | + | | ?-------------------? | | + | | | | | + | | | points to | | + | | V | | + | | ?-------------------? | | + | | | FalconUCodeDescV3 | <- falcon_ucode_offset | | + | | | (FWSEC Firmware) | fwsec_header() | | + | | ?-------------------? | | + | | | immediately followed by... | | + | | V | | + | | ?----------------------------? | | + | | | Signatures + FWSEC Ucode | | | + | | | fwsec_sigs(), fwsec_ucode()| | | + | | ?----------------------------? | | + | ?-----------------------------------------------? | + ?------------------------------------------------------------------------? Falcon data Lookup ------------------ Thanks. -- An old man doll... just what I always wanted! - Clara -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <https://lists.freedesktop.org/archives/nouveau/attachments/20250505/56a7297f/attachment-0001.sig>
Bagas Sanjaya
2025-May-05 03:52 UTC
[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
On Sat, May 03, 2025 at 12:07:56AM -0400, Joel Fernandes wrote:> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) > +========================> +FWSEC (Firmware Security) > +========================Separate SPDX line from title heading.> +Here is a block diagram of the FWSEC memory layout::Add blank line before actual diagram below.> + ????????????????????????????????????????????????????????????????? > + ? FWSEC ROM image (type 0xE0) ? > + ? ? > + ? ??????????????????????????????????? ? > + ? ? PMU Falcon Ucode Table ? ? > + ? ? (PmuLookupTable) ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Table Header ? ? ? > + ? ? ? - version: 0x01 ? ? ? > + ? ? ? - header_size: 6 ? ? ? > + ? ? ? - entry_size: 6 ? ? ? > + ? ? ? - entry_count: N ? ? ? > + ? ? ? - desc_version:3(unused)? ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ... ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Entry for FWSEC (0x85) ? ? ? > + ? ? ? (PmuLookupTableEntry) ? ? ? > + ? ? ? - app_id: 0x85 (FWSEC) ? ????????? ? > + ? ? ? - target_id: 0x01 (PMU) ? ? ? ? > + ? ? ? - data: offset ???????????????????????? look up FWSEC ? > + ? ? ??????????????????????????? ? ? ? application. ? > + ? ??????????????????????????????????? ? ? ? > + ? ? ? ? > + ? ? ? ? > + ? ??????????????????????????????????? ? ? ? > + ? ? FWSEC Ucode Component ?<???? ? ? > + ? ? (aka Falcon data) ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? FalconUCodeDescV3 ?<????????????? ? > + ? ? ? - hdr ? ? ? > + ? ? ? - stored_size ? ? ? > + ? ? ? - pkc_data_offset ? ? ? > + ? ? ? - interface_offset ???????????????????????????? ? > + ? ? ? - imem_phys_base ? ? ? ? > + ? ? ? - imem_load_size ? ? ? ? > + ? ? ? - imem_virt_base ? ? ? ? > + ? ? ? - dmem_phys_base ? ? ? ? > + ? ? ? - dmem_load_size ? ? ? ? > + ? ? ? - engine_id_mask ? ? ? ? > + ? ? ? - ucode_id ? ? ? ? > + ? ? ? - signature_count ? ? look up sig ? ? > + ? ? ? - signature_versions --------------+ ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? (no gap) ? ? ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? ? Signatures Section ?<?????????? ? ? > + ? ? ? (384 bytes per sig) ? ? ? ? > + ? ? ? - RSA-3K Signature 1 ? ? ? ? > + ? ? ? - RSA-3K Signature 2 ? ? ? ? > + ? ? ? ... ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? IMEM Section (Code) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? Contains instruction ? ? ? ? > + ? ? ? code etc. ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? DMEM Section (Data) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Application ? ?<????????????????????? ? > + ? ? ? ? Interface Table ? ? ? ? > + ? ? ? ? (FalconAppifHdrV1) ? ? ? ? > + ? ? ? ? Header: ? ? ? ? > + ? ? ? ? - version: 0x01 ? ? ? ? > + ? ? ? ? - header_size: 4 ? ? ? ? > + ? ? ? ? - entry_size: 8 ? ? ? ? > + ? ? ? ? - entry_count: N ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ? Entries: ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DEVINIT (ID 1) ? ? ? ? ? > + ? ? ? ? ? - id: 0x01 ? ? ? ? ? > + ? ? ? ? ? - dmemOffset X ??????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DMEMMAPPER(ID 4)? ? ? ? ? > + ? ? ? ? ? - id: 0x04 ? ? ? ? Used only for DevInit ? > + ? ? ? ? ? (NVFW_FALCON_ ? ? ? ? application (not FWSEC) ? > + ? ? ? ? ? APPIF_ID_DMEMMAPPER) ? ? > + ? ? ? ? ? - dmemOffset Y ????????????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DEVINIT Engine ?<?????? ? Used by FWSEC ? > + ? ? ? ? Interface ? ? ? ? app. ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DMEM Mapper (ID 4) ?<?????+?????? ? > + ? ? ? ? (FalconAppifDmemmapperV3) ? ? > + ? ? ? ? - signature: "DMAP" ? ? ? ? > + ? ? ? ? - version: 0x0003 ? ? ? ? > + ? ? ? ? - Size: 64 bytes ? ? ? ? > + ? ? ? ? - cmd_in_buffer_off ? ??????????????????? ? > + ? ? ? ? - cmd_in_buffer_size? ? ? ? ? > + ? ? ? ? - cmd_out_buffer_off? ????????????????????????? ? > + ? ? ? ? - cmd_out_buffer_sz ? ? ? ? ? ? > + ? ? ? ? - init_cmd ? ? ? ? ? ? > + ? ? ? ? - features ? ? ? ? ? ? > + ? ? ? ? - cmd_mask0/1 ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? Command Input Buffer?<??????????????????? ? ? > + ? ? ? ? - Command data ? ? ? ? ? > + ? ? ? ? - Arguments ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Command Output ?<????????????????????????? ? > + ? ? ? ? Buffer ? ? ? ? > + ? ? ? ? - Results ? ? ? ? > + ? ? ? ? - Status ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? > + ? ? ??????????????????????????? ? ? > + ? ??????????????????????????????????? ? > + ? ? > + ?????????????????????????????????????????????????????????????????The diagram can look messy on certain fonts in htmldocs output. Please use ASCII dash (-) and vertical bar (|) instead of Unicode box decoration variant (? and ?). Thanks. -- An old man doll... just what I always wanted! - Clara -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <https://lists.freedesktop.org/archives/nouveau/attachments/20250505/40d96535/attachment-0001.sig>
Bagas Sanjaya
2025-May-05 04:04 UTC
[PATCH v2 5/7] docs: nova-core: Document devinit process
On Sat, May 03, 2025 at 12:07:57AM -0400, Joel Fernandes wrote:> +.. SPDX-License-Identifier: GPL-2.0 > +=================================> +Device Initialization (devinit) > +=================================Separate SPDX line from title heading.> +These low-level GPU firmware components are typically: > +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst). > +2. Executed in sequence on different microcontrollers: > + - The devinit engine typically but not necessarily runs on the PMU. > + - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in > + heavy-secure mode.Please separate numbered list from preceding sentence, and the bullet sublist from parent numbered list by a line.> +Runtime Considerations > +--------------------- > <snipped>... > +Security and Access Control > +--------------------------Match section underline length with the text. Thanks. -- An old man doll... just what I always wanted! - Clara -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <https://lists.freedesktop.org/archives/nouveau/attachments/20250505/8406bae5/attachment.sig>
Bagas Sanjaya
2025-May-05 04:14 UTC
[PATCH v2 6/7] docs: nova-core: Document basics of the Falcon
On Sat, May 03, 2025 at 12:07:58AM -0400, Joel Fernandes wrote:> +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows: > + > + External Memory (Framebuffer / System DRAM) > + ? ? > + ? ? > + ? ? > +??????????????????????????????????????????????????????? > +? ? ? > +? ????????????????? ? ? > +? ? FBIF ????????? ? FALCON > +? ? (FrameBuffer ? Memory Interface ? PROCESSOR > +? ? InterFace) ? ? > +? ? Apertures ? ? > +? ? Configures ? ? > +? ? mem access ? ? > +? ????????????????? ? > +? ? ? > +? ? FBDMA uses configured FBIF apertures ? > +? ? to access External Memory > +? ? > +? ????????????????? ?????????????????? > +? ? FBDMA ? cfg ? RISC ? > +? ? (FrameBuffer ?<????>? CORE ??????>. Direct Core Access > +? ? DMA Engine) ? ? ? ? > +? ? - Master dev. ? ? (can run both ? ? > +? ????????????????? ? Falcon and ? ? > +? ? cfg-->? RISC-V code) ? ? > +? ? / ? ? ? > +? ? | ?????????????????? ? ?????????????? > +? ? ? ? ? BROM ? > +? ? ? <???>? (Boot ROM) ? > +? ? / ? ?????????????? > +? ? ? ? > +? ????????????????? ? > +? ? IO-PMP ? Controls access by FBDMA ? > +? ? (IO Physical ? and other IO Masters ? > +? ? Memory Protect) ? > +? ????????????????? ? > +? ? ? > +? ? Protected Access Path for FBDMA ? > +? ? ? > +? ????????????????????????????????????????? ? > +? ? Memory ? ? > +? ? ????????????????? ??????????????? ? ? > +? ? ? IMEM ? ? DMEM ? ?<?????? > +? ? ? (Instruction ? ? (Data ? ? > +? ? ? Memory) ? ? Memory) ? ? > +? ? ????????????????? ??????????????? ? > +? ????????????????????????????????????????? > +?????????????????????????????????????????????????????Wrap the diagram in literal code block. Also, note that it can look messy in htmldocs output with certain fonts due to box drawing characters. Consider using ASCII constructs (dashes and vertical bars) instead. Thanks. -- An old man doll... just what I always wanted! - Clara -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <https://lists.freedesktop.org/archives/nouveau/attachments/20250505/6014118a/attachment.sig>
On Sat, 3 May 2025 00:07:59 -0400 Joel Fernandes <joelagnelf at nvidia.com> wrote:> Add documentation strings, comments and AES mode for completeness > to the Falcon signatures. > > Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> > --- > drivers/gpu/nova-core/falcon.rs | 25 +++++++++++++++++++------ > 1 file changed, 19 insertions(+), 6 deletions(-) > > diff --git a/drivers/gpu/nova-core/falcon.rs > b/drivers/gpu/nova-core/falcon.rs index e9ee0c83dfc5..003db40d3303 > 100644 --- a/drivers/gpu/nova-core/falcon.rs > +++ b/drivers/gpu/nova-core/falcon.rs > @@ -89,13 +89,19 @@ fn try_from(value: u8) -> Result<Self> { > /// register. > #[repr(u8)] > #[derive(Debug, Default, Copy, Clone)] > +/// Security mode of the Falcon microprocessor. > +/// See falcon.rst for more details. > pub(crate) enum FalconSecurityModel { > /// Non-Secure: runs unsigned code without privileges. > #[default] > None = 0, > - /// Low-secure: runs unsigned code with some privileges. Can > only be entered from `Heavy` mode. > + /// Light-Secured (LS): runs signed code with some privileges > + /// Its signature can only be verified and entered from `Heavy` > mode. > + /// Also known as Privilege Level 2 or PL2. > Light = 2, > - /// High-Secure: runs signed code with full privileges. > + /// Heavy-Secured: runs signed code with full privileges. > + /// Its signature can only be verified by the Falcon Boot ROM > (BROM). > + /// Also known as Privilege Level 3 or PL3. > Heavy = 3, > } > > @@ -117,10 +123,13 @@ fn try_from(value: u8) -> > core::result::Result<Self, Self::Error> { } > > /// Signing algorithm for a given firmware, used in the > [`crate::regs::NV_PFALCON2_FALCON_MOD_SEL`] -/// register. > +/// register. It is passed to the Falcon Boot ROM (BROM) as a > parameter. #[repr(u8)] > #[derive(Debug, Default, Copy, Clone, PartialEq, Eq)] > pub(crate) enum FalconModSelAlgo { > + /// AES. > + #[expect(dead_code)] > + Aes = 0, > /// RSA3K. > #[default] > Rsa3k = 1, > @@ -184,15 +193,19 @@ pub(crate) enum FalconMem { > Dmem, > } > > -/// Target/source of a DMA transfer to/from falcon memory. > +/// FBIF (Framebuffer Interface) aperture type. Used to determine > +/// the memory type of the external memory access for a DMA memory > +/// transfer (by the Falcon's FramebufferDMA (FBDMA) engine locatedShould be Framebuffer DMA?^ So that it will be aligned with PATCH 6.> +/// inside the falcon). See falcon.rst for more details. > #[derive(Debug, Clone, Default)] > pub(crate) enum FalconFbifTarget { > /// VRAM. > #[default] > + /// Local Framebuffer (GPU's VRAM memory) > LocalFb = 0, > - /// Coherent system memory. > + /// Coherent system memory (System DRAM). > CoherentSysmem = 1, > - /// Non-coherent system memory. > + /// Non-coherent system memory (System DRAM). > NoncoherentSysmem = 2, > } >
Zhi Wang
2025-May-06 16:26 UTC
[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
On Sat, 3 May 2025 00:07:56 -0400 Joel Fernandes <joelagnelf at nvidia.com> wrote:> Add explanation of fwsec with diagrams. This helps clarify how the > nova-core falcon boot works. > > Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> > --- > Documentation/gpu/nova/core/fwsec.rst | 180 > ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | > 1 + 2 files changed, 181 insertions(+) > create mode 100644 Documentation/gpu/nova/core/fwsec.rst > > diff --git a/Documentation/gpu/nova/core/fwsec.rst > b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644 > index 000000000000..bed941ac3f2b > --- /dev/null > +++ b/Documentation/gpu/nova/core/fwsec.rst > @@ -0,0 +1,180 @@ > +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) > +========================> +FWSEC (Firmware Security) > +========================> +This document briefly/conceptually describes the FWSEC (Firmware > Security) image +and its role in the GPU boot sequence. As such, this > information is subject to +change in the future and is only current > as of the Ampere GPU family. However, +hopefully the concepts > described will be useful for understanding the kernel code +that > deals with it. All the information is derived from publicly available > +sources such as public drivers and documentation. + > +The role of FWSEC to provide secure boot, it is running in > Heavy-secure mode. It does +firmware verification after GPU reset and > load various ucode images on to the other +microcontrollers on the > GPU such as the PMU and GSP. + > +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC > partition of +ROM (see vbios.rst for more details). It contains > different commands like FRTS +(Firmware Runtime Services) and SB > (Secure Booting other microcontrollers after +reset and loading them > with other non-FWSEC ucode). The kernel driver only needs to +to do > FRTS, since SB is already already after reset by the time the kernel > driver +is loaded. + > +The FRTS command carves out the WPR2 region (Write protected region) > which contains +data data required for power management. Once setup,repetitive word "data" ^> only HS mode ucode can +access it (see falcon.rst for privilege > levels). + > +The FWSEC image is located in the VBIOS ROM in the partition of the > ROM that contains +various ucode images (also known as applications) > -- one of them being FWSEC. For how +it is extracted, see vbios.rst > and the vbios.rs source code. + > +The Falcon data for each ucode images (including the FWSEC image) is > a combination +of headers, data sections (DMEM) and instruction code > sections (IMEM). All these +ucode images are stored in the same ROM > partition and the PMU table is used to look +up the application to > load it based on its application ID (see vbios.rs). + > +For the purposes of nova-core driver, the FWSEC contains an > 'application interface' +called DMEMMAPPER which is used to the > "FWSEC-FRTS" command (among other commands it +is capable of > executing). For Ampere, FWSEC is running on the GSP in Heavy-secure > +mode and runs FRTS. + > +FWSEC Memory Layout > +------------------- > +The memory layout of the FWSEC image is as follows (this is using an > GA-102 +Ampere GPU as an example and could vary for future GPUs and > is subject to change +completely, it is just provided as an example): > + > +Here is a block diagram of the FWSEC memory layout:: > + ????????????????????????????????????????????????????????????????? > + ? FWSEC ROM image (type 0xE0) ? > + ? ? > + ? ??????????????????????????????????? ? > + ? ? PMU Falcon Ucode Table ? ? > + ? ? (PmuLookupTable) ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Table Header ? ? ? > + ? ? ? - version: 0x01 ? ? ? > + ? ? ? - header_size: 6 ? ? ? > + ? ? ? - entry_size: 6 ? ? ? > + ? ? ? - entry_count: N ? ? ? > + ? ? ? - desc_version:3(unused)? ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ... ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Entry for FWSEC (0x85) ? ? ? > + ? ? ? (PmuLookupTableEntry) ? ? ? > + ? ? ? - app_id: 0x85 (FWSEC) ? ????????? ? > + ? ? ? - target_id: 0x01 (PMU) ? ? ? ? > + ? ? ? - data: offset ???????????????????????? look up FWSEC ? > + ? ? ??????????????????????????? ? ? ? application. ? > + ? ??????????????????????????????????? ? ? ? > + ? ? ? ? > + ? ? ? ? > + ? ??????????????????????????????????? ? ? ? > + ? ? FWSEC Ucode Component ?<???? ? ? > + ? ? (aka Falcon data) ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? FalconUCodeDescV3 ?<????????????? ? > + ? ? ? - hdr ? ? ? > + ? ? ? - stored_size ? ? ? > + ? ? ? - pkc_data_offset ? ? ? > + ? ? ? - interface_offset ???????????????????????????? ? > + ? ? ? - imem_phys_base ? ? ? ? > + ? ? ? - imem_load_size ? ? ? ? > + ? ? ? - imem_virt_base ? ? ? ? > + ? ? ? - dmem_phys_base ? ? ? ? > + ? ? ? - dmem_load_size ? ? ? ? > + ? ? ? - engine_id_mask ? ? ? ? > + ? ? ? - ucode_id ? ? ? ? > + ? ? ? - signature_count ? ? look up sig ? ? > + ? ? ? - signature_versions --------------+ ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? (no gap) ? ? ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? ? Signatures Section ?<?????????? ? ? > + ? ? ? (384 bytes per sig) ? ? ? ? > + ? ? ? - RSA-3K Signature 1 ? ? ? ? > + ? ? ? - RSA-3K Signature 2 ? ? ? ? > + ? ? ? ... ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? IMEM Section (Code) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? Contains instruction ? ? ? ? > + ? ? ? code etc. ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? DMEM Section (Data) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Application ? ?<????????????????????? ? > + ? ? ? ? Interface Table ? ? ? ? > + ? ? ? ? (FalconAppifHdrV1) ? ? ? ? > + ? ? ? ? Header: ? ? ? ? > + ? ? ? ? - version: 0x01 ? ? ? ? > + ? ? ? ? - header_size: 4 ? ? ? ? > + ? ? ? ? - entry_size: 8 ? ? ? ? > + ? ? ? ? - entry_count: N ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ? Entries: ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DEVINIT (ID 1) ? ? ? ? ? > + ? ? ? ? ? - id: 0x01 ? ? ? ? ? > + ? ? ? ? ? - dmemOffset X ??????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DMEMMAPPER(ID 4)? ? ? ? ? > + ? ? ? ? ? - id: 0x04 ? ? ? ? Used only for DevInit ? > + ? ? ? ? ? (NVFW_FALCON_ ? ? ? ? application (not FWSEC) ? > + ? ? ? ? ? APPIF_ID_DMEMMAPPER) ? ? > + ? ? ? ? ? - dmemOffset Y ????????????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DEVINIT Engine ?<?????? ? Used by FWSEC ? > + ? ? ? ? Interface ? ? ? ? app. ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DMEM Mapper (ID 4) ?<?????+?????? ? > + ? ? ? ? (FalconAppifDmemmapperV3) ? ? > + ? ? ? ? - signature: "DMAP" ? ? ? ? > + ? ? ? ? - version: 0x0003 ? ? ? ? > + ? ? ? ? - Size: 64 bytes ? ? ? ? > + ? ? ? ? - cmd_in_buffer_off ? ??????????????????? ? > + ? ? ? ? - cmd_in_buffer_size? ? ? ? ? > + ? ? ? ? - cmd_out_buffer_off? ????????????????????????? ? > + ? ? ? ? - cmd_out_buffer_sz ? ? ? ? ? ? > + ? ? ? ? - init_cmd ? ? ? ? ? ? > + ? ? ? ? - features ? ? ? ? ? ? > + ? ? ? ? - cmd_mask0/1 ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? Command Input Buffer?<??????????????????? ? ? > + ? ? ? ? - Command data ? ? ? ? ? > + ? ? ? ? - Arguments ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Command Output ?<????????????????????????? ? > + ? ? ? ? Buffer ? ? ? ? > + ? ? ? ? - Results ? ? ? ? > + ? ? ? ? - Status ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? > + ? ? ??????????????????????????? ? ? > + ? ??????????????????????????????????? ? > + ? ? > + ????????????????????????????????????????????????????????????????? > + > +.. note:: > + The FWSEC image also plays a role in memory scrubbing (ECC > initialization) and VPR > + (Video Protected Region) initialization as well. Before the > nova-core driver is even > + loaded, the FWSEC image is running on the GSP in heavy-secure > mode. After the devinit > + sequence completes, it does VRAM memory scrubbing (ECC > initialization). On consumer > + GPUs, it scrubs only part of memory and then initiates 'async > scrubbing'. Before this > + async scrubbing completes, the unscrubbed VRAM cannot be used for > allocation (thus DRM > + memory allocators need to wait for this scrubbing to complete). > \ No newline at end of file > diff --git a/Documentation/gpu/nova/index.rst > b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0 > 100644 --- a/Documentation/gpu/nova/index.rst > +++ b/Documentation/gpu/nova/index.rst > @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver. > > core/guidelines > core/vbios > + core/fwsec > core/todo
On 5/3/25 6:07 AM, Joel Fernandes wrote:> Hello, > > Please find attached initial documentation for nova-core mainly about the vbios > and boot process. It helps build an understanding of the boot code. All the > information is derived from publicly available code and sources.Has there been a follow-up on the series? I think it'd be great to apply, now that we landed the FWSEC-FRTS series. - Danilo