This series adds some documentation that was relevant to the FWSEC-FRTS [1] series, but wasn't sent alongside it as it was worked on in parallel. It notably introduces a lot of HTMLdocs that adds key explanations to understand and work on the GSP boot process. [1] https://lore.kernel.org/all/DB0I8WAH970B.25D3S59AYF85P at nvidia.com/ Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- Changes in v4: - Convert `/*` comments into `//`. - Fix missing slashes in doccomments and minor typos. - Move some less-important paragraphs to notes in the RST files. - Link to v3: https://lore.kernel.org/r/20250702-nova-docs-v3-0-f362260813e2 at nvidia.com Changes in v3: - Rebase on top of latest nova-next. - Reorganize patches a bit and use standard prefixes. - Convert diagrams to only use ASCII characters (thanks Bagas!). - Move sysmembar documentation to the new `SysmemFlush` type. - Reword things a bit here and there. - Reorganize order of `index.rst` for a more natural flow. - Link to v2: https://lore.kernel.org/rust-for-linux/20250503040802.1411285-1-joelagnelf at nvidia.com/ --- Alexandre Courbot (1): gpu: nova-core: convert `/*` comments to `//` Joel Fernandes (7): gpu: nova-core: Add code comments related to devinit gpu: nova-core: Clarify sysmembar operations gpu: nova-core: Clarify falcon code Documentation: gpu: nova-core: Document vbios layout Documentation: gpu: nova-core: Document devinit process Documentation: gpu: nova-core: Document fwsec operation and layout Documentation: gpu: nova-core: Document basics of the Falcon Documentation/gpu/nova/core/devinit.rst | 61 +++++++++++ Documentation/gpu/nova/core/falcon.rst | 158 ++++++++++++++++++++++++++++ Documentation/gpu/nova/core/fwsec.rst | 181 ++++++++++++++++++++++++++++++++ Documentation/gpu/nova/core/vbios.rst | 181 ++++++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 4 + drivers/gpu/nova-core/falcon.rs | 29 +++-- drivers/gpu/nova-core/fb.rs | 10 ++ drivers/gpu/nova-core/gfw.rs | 39 ++++++- drivers/gpu/nova-core/gpu.rs | 3 +- drivers/gpu/nova-core/regs.rs | 35 ++++-- 10 files changed, 677 insertions(+), 24 deletions(-) --- base-commit: 4092e1b41202ff39aad75a40a03ac1d318443670 change-id: 20250702-nova-docs-b9900d0505b5 Best regards, -- Alexandre Courbot <acourbot at nvidia.com>
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 1/8] gpu: nova-core: Add code comments related to devinit
From: Joel Fernandes <joelagnelf at nvidia.com> Add several code comments to reduce acronym soup and explain how devinit magic and bootflow works before driver loads. These are essential for debug and development of the nova driver. [acourbot at nvidia.com: reformat and reword a couple of sentences] Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- drivers/gpu/nova-core/gfw.rs | 39 +++++++++++++++++++++++++++++++++++---- drivers/gpu/nova-core/regs.rs | 14 ++++++++++++-- 2 files changed, 47 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/nova-core/gfw.rs b/drivers/gpu/nova-core/gfw.rs index d5b68e02d405750b18d634d772f15f413453e80d..8ac1ed18719926493369c2aae9a59b2b55fa2b12 100644 --- a/drivers/gpu/nova-core/gfw.rs +++ b/drivers/gpu/nova-core/gfw.rs @@ -1,10 +1,22 @@ // SPDX-License-Identifier: GPL-2.0 -//! GPU Firmware (GFW) support. +//! GPU Firmware (`GFW`) support, a.k.a `devinit`. //! //! Upon reset, the GPU runs some firmware code from the BIOS to setup its core parameters. Most of //! the GPU is considered unusable until this step is completed, so we must wait on it before //! performing driver initialization. +//! +//! A clarification about devinit terminology: devinit is a sequence of register read/writes after +//! reset that performs tasks such as: +//! 1. Programming VRAM memory controller timings. +//! 2. Power sequencing. +//! 3. Clock and PLL configuration. +//! 4. Thermal management. +//! +//! devinit itself is a 'script' which is interpreted by an interpreter program typically running +//! on the PMU microcontroller. +//! +//! Note that the devinit sequence also needs to run during suspend/resume. use kernel::bindings; use kernel::prelude::*; @@ -14,13 +26,32 @@ use crate::regs; use crate::util; -/// Wait until `GFW` (GPU Firmware) completes, or a 4 seconds timeout elapses. +/// Wait for the `GFW` (GPU firmware) boot completion signal (`GFW_BOOT`), or a 4 seconds timeout. +/// +/// Upon GPU reset, several microcontrollers (such as PMU, SEC2, GSP etc) run some firmware code to +/// setup its core parameters. Most of the GPU is considered unusable until this step is completed, +/// so it must be waited on very early during driver initialization. +/// +/// The `GFW` code includes several components that need to execute before the driver loads. These +/// components are located in the VBIOS ROM and executed in a sequence on these different +/// microcontrollers. The devinit sequence typically runs on the PMU, and the FWSEC runs on the +/// GSP. +/// +/// This function waits for a signal indicating that core initialization is complete. Before this +/// signal is received, little can be done with the GPU. This signal is set by the FWSEC running on +/// the GSP in Heavy-secured mode. pub(crate) fn wait_gfw_boot_completion(bar: &Bar0) -> Result { + // Before accessing the completion status in `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05`, we must + // first check `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK`. This is because + // `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05` becomes accessible only after the secure firmware + // (FWSEC) lowers the privilege level to allow CPU (LS/Light-secured) access. We can only + // safely read the status register from CPU (LS/Light-secured) once the mask indicates + // that the privilege level has been lowered. + // // TIMEOUT: arbitrarily large value. GFW starts running immediately after the GPU is put out of // reset, and should complete in less time than that. util::wait_on(Delta::from_secs(4), || { - // Check that FWSEC has lowered its protection level before reading the GFW_BOOT - // status. + // Check that FWSEC has lowered its protection level before reading the GFW_BOOT status. let gfw_booted = regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK::read(bar) .read_protection_level0() && regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT::read(bar).completed(); diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs index e8b8aabce3f36abe6a7fba3e11f677e36baa3897..3bb38197a890bb32d54b9aa4df4d9ebd740dccca 100644 --- a/drivers/gpu/nova-core/regs.rs +++ b/drivers/gpu/nova-core/regs.rs @@ -104,9 +104,19 @@ pub(crate) fn higher_bound(self) -> u64 { } } -/* PGC6 */ +// PGC6 register space. +// +// `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except +// for power rails needed to keep self-refresh working and important registers and hardware +// blocks). +// +// These scratch registers remain powered on even in a low-power state and have a designated group +// number. -register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128 { +// Privilege level mask register. It dictates whether the host CPU has privilege to access the +// `PGC6_AON_SECURE_SCRATCH_GROUP_05` register (which it needs to read GFW_BOOT). +register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128, + "Privilege level mask register" { 0:0 read_protection_level0 as bool, "Set after FWSEC lowers its protection level"; }); -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 2/8] gpu: nova-core: Clarify sysmembar operations
From: Joel Fernandes <joelagnelf at nvidia.com> sysmembar is a critical operation that the GSP falcon needs to perform in the reset sequence. Add some code comments to clarify. [acourbot at nvdidia.com: move relevant documentation to SysmemFlush type] Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- drivers/gpu/nova-core/fb.rs | 10 ++++++++++ drivers/gpu/nova-core/gpu.rs | 3 +-- drivers/gpu/nova-core/regs.rs | 3 +++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/nova-core/fb.rs b/drivers/gpu/nova-core/fb.rs index 172b4a12ba2afc05860cc004fd1f0154402f467a..4a702525fff4f394b75fcf54145ba78e34a1a539 100644 --- a/drivers/gpu/nova-core/fb.rs +++ b/drivers/gpu/nova-core/fb.rs @@ -17,6 +17,16 @@ /// Type holding the sysmem flush memory page, a page of memory to be written into the /// `NV_PFB_NISO_FLUSH_SYSMEM_ADDR*` registers and used to maintain memory coherency. /// +/// A system memory page is required for `sysmembar`, which is a GPU-initiated hardware +/// memory-barrier operation that flushes all pending GPU-side memory writes that were done through +/// PCIE to system memory. It is required for falcons to be reset as the reset operation involves a +/// reset handshake. When the falcon acknowledges a reset, it writes into system memory. To ensure +/// this write is visible to the host and prevent driver timeouts, the falcon must perform a +/// sysmembar operation to flush its writes. +/// +/// Because of this, the sysmem flush memory page must be registered as early as possible during +/// driver initialization, and before any falcon is reset. +/// /// Users are responsible for manually calling [`Self::unregister`] before dropping this object, /// otherwise the GPU might still use it even after it has been freed. pub(crate) struct SysmemFlush { diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs index 8e32af16b669ca773e63e184d34c3e0427bc9b76..72d40b0124f0c1a2a381484172c289af523511df 100644 --- a/drivers/gpu/nova-core/gpu.rs +++ b/drivers/gpu/nova-core/gpu.rs @@ -170,7 +170,7 @@ pub(crate) struct Gpu { bar: Devres<Bar0>, fw: Firmware, /// System memory page required for flushing all pending GPU-side memory writes done through - /// PCIE into system memory. + /// PCIE into system memory, via sysmembar (A GPU-initiated HW memory-barrier operation). sysmem_flush: SysmemFlush, } @@ -283,7 +283,6 @@ pub(crate) fn new( gfw::wait_gfw_boot_completion(bar) .inspect_err(|_| dev_err!(pdev.as_ref(), "GFW boot did not complete"))?; - // System memory page required for sysmembar to properly flush into system memory. let sysmem_flush = SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?; let gsp_falcon = Falcon::<Gsp>::new( diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs index 3bb38197a890bb32d54b9aa4df4d9ebd740dccca..b934ffe8e81390b36f5a39af39a9b9f337aa66bf 100644 --- a/drivers/gpu/nova-core/regs.rs +++ b/drivers/gpu/nova-core/regs.rs @@ -51,6 +51,9 @@ pub(crate) fn chipset(self) -> Result<Chipset> { /* PFB */ +// These two registers together hold the physical system memory address that is used by the GPU for +// perform sysmembar operation (see `fb::SysmemFlush`). + register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR @ 0x00100c10 { 31:0 adr_39_08 as u32; }); -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 3/8] gpu: nova-core: Clarify falcon code
From: Joel Fernandes <joelagnelf at nvidia.com> Add documentation strings, comments and AES mode for completeness to the Falcon signatures. Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- drivers/gpu/nova-core/falcon.rs | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/nova-core/falcon.rs b/drivers/gpu/nova-core/falcon.rs index c2c6f9eb380ab390befe5af1b3c5df260ccd7595..50437c67c14a89b6974a121d4408efbcdcb3fdd0 100644 --- a/drivers/gpu/nova-core/falcon.rs +++ b/drivers/gpu/nova-core/falcon.rs @@ -105,14 +105,23 @@ fn try_from(value: u8) -> Result<Self> { /// register. #[repr(u8)] #[derive(Debug, Default, Copy, Clone)] +/// Security mode of the Falcon microprocessor. +/// +/// See `falcon.rst` for more details. pub(crate) enum FalconSecurityModel { /// Non-Secure: runs unsigned code without privileges. #[default] None = 0, - /// Low-Secure: runs code with some privileges. Can only be entered from `Heavy` mode, which - /// will typically validate the LS code through some signature. + /// Light-Secured (LS): Runs signed code with some privileges. + /// Entry into this mode is only possible from 'Heavy-secure' mode, which verifies the code's + /// signature. + /// + /// Also known as Low-Secure, Privilege Level 2 or PL2. Light = 2, - /// High-Secure: runs signed code with full privileges. Signature is validated by boot ROM. + /// Heavy-Secured (HS): Runs signed code with full privileges. + /// The code's signature is verified by the Falcon Boot ROM (BROM). + /// + /// Also known as High-Secure, Privilege Level 3 or PL3. Heavy = 3, } impl_from_enum_to_u32!(FalconSecurityModel); @@ -136,10 +145,13 @@ fn try_from(value: u8) -> Result<Self> { } /// Signing algorithm for a given firmware, used in the [`crate::regs::NV_PFALCON2_FALCON_MOD_SEL`] -/// register. +/// register. It is passed to the Falcon Boot ROM (BROM) as a parameter. #[repr(u8)] #[derive(Debug, Default, Copy, Clone, PartialEq, Eq)] pub(crate) enum FalconModSelAlgo { + /// AES. + #[expect(dead_code)] + Aes = 0, /// RSA3K. #[default] Rsa3k = 1, @@ -209,15 +221,18 @@ pub(crate) enum FalconMem { Dmem, } -/// Target/source of a DMA transfer to/from falcon memory. +/// Defines the Framebuffer Interface (FBIF) aperture type. +/// This determines the memory type for external memory access during a DMA transfer, which is +/// performed by the Falcon's Framebuffer DMA (FBDMA) engine. See falcon.rst for more details. #[derive(Debug, Clone, Default)] pub(crate) enum FalconFbifTarget { /// VRAM. #[default] + /// Local Framebuffer (GPU's VRAM memory). LocalFb = 0, - /// Coherent system memory. + /// Coherent system memory (System DRAM). CoherentSysmem = 1, - /// Non-coherent system memory. + /// Non-coherent system memory (System DRAM). NoncoherentSysmem = 2, } impl_from_enum_to_u32!(FalconFbifTarget); -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 4/8] gpu: nova-core: convert `/*` comments to `//`
The second form is preferred, and there was no reason to use the first. Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- drivers/gpu/nova-core/regs.rs | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs index b934ffe8e81390b36f5a39af39a9b9f337aa66bf..6f774d4f79a5eb253df90e335d05cf6bc1fa5857 100644 --- a/drivers/gpu/nova-core/regs.rs +++ b/drivers/gpu/nova-core/regs.rs @@ -14,7 +14,7 @@ use crate::gpu::{Architecture, Chipset}; use kernel::prelude::*; -/* PMC */ +// PMC register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" { 3:0 minor_revision as u8, "Minor revision of the chip"; @@ -42,14 +42,14 @@ pub(crate) fn chipset(self) -> Result<Chipset> { } } -/* PBUS */ +// PBUS // TODO[REGA]: this is an array of registers. register!(NV_PBUS_SW_SCRATCH_0E at 0x00001438 { 31:16 frts_err_code as u16; }); -/* PFB */ +// PFB // These two registers together hold the physical system memory address that is used by the GPU for // perform sysmembar operation (see `fb::SysmemFlush`). @@ -160,7 +160,7 @@ pub(crate) fn usable_fb_size(self) -> u64 { } } -/* PDISP */ +// PDISP register!(NV_PDISP_VGA_WORKSPACE_BASE @ 0x00625f04 { 3:3 status_valid as bool, "Set if the `addr` field is valid"; @@ -178,7 +178,7 @@ pub(crate) fn vga_workspace_addr(self) -> Option<u64> { } } -/* FUSE */ +// FUSE register!(NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION @ 0x00824100 { 15:0 data as u16; @@ -192,7 +192,7 @@ pub(crate) fn vga_workspace_addr(self) -> Option<u64> { 15:0 data as u16; }); -/* PFALCON */ +// PFALCON register!(NV_PFALCON_FALCON_IRQSCLR @ +0x00000004 { 4:4 halt as bool; @@ -312,7 +312,7 @@ pub(crate) fn mem_scrubbing_done(self) -> bool { 31:0 value as u32; }); -/* PRISCV */ +// PRISCV register!(NV_PRISCV_RISCV_BCR_CTRL @ +0x00001668 { 0:0 valid as bool; @@ -324,7 +324,7 @@ pub(crate) fn mem_scrubbing_done(self) -> bool { // only be used in HAL modules. pub(crate) mod gm107 { - /* FUSE */ + // FUSE register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00021c04 { 0:0 display_disabled as bool; @@ -332,7 +332,7 @@ pub(crate) mod gm107 { } pub(crate) mod ga100 { - /* FUSE */ + // FUSE register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00820c04 { 0:0 display_disabled as bool; -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 5/8] Documentation: gpu: nova-core: Document vbios layout
From: Joel Fernandes <joelagnelf at nvidia.com> Add detailed explanation and block diagrams of the layout of the vBIOS on Nvidia GPUs. This is important to understand how nova-core boots an Nvidia GPU. [ Applied Timur Tabi's feedback on providing link to BIT documentation. ] Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- Documentation/gpu/nova/core/vbios.rst | 181 ++++++++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 182 insertions(+) diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst new file mode 100644 index 0000000000000000000000000000000000000000..efd40087480c4fd6cc8ef10ddfaa9f0297417e5b --- /dev/null +++ b/Documentation/gpu/nova/core/vbios.rst @@ -0,0 +1,181 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +=========+VBIOS +=========+This document describes the layout of the VBIOS image which is a series of concatenated +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to +bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP. + +The format of the images in the ROM follow the "BIOS Specification" part of the +PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec +are the ones that contain Falcon ucode and what we are mainly looking for. + +As an example, the following are the different image types that can be found in the +VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver. + +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name + likely comes from the "IBM PC/AT" architecture. + +- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP + driver that is used to display UEFI graphics output. + +- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware) + +- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) + contains various microcodes (also known as an applications) that do a range + of different functions. The FWSEC ucode is run in heavy-secure mode and + typically runs directly on the GSP (it could be running on a different + designated processor in future generations but as of Ampere, it is the GSP). + This firmware then loads other firmware ucodes onto the PMU and SEC2 + microcontrollers for gfw initialization after GPU reset and before the driver + loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is + stored in this ROM partition. + +Once located, the Falcon ucodes have "Application Interfaces" in their data +memory (DMEM). For FWSEC, the application interface we use for FWSEC is the +"DMEM mapper" interface which is configured to run the "FRTS" command. This +command carves out the WPR2 (Write-Protected Region) in VRAM. It then places +important power-management data, called 'FRTS', into this region. The WPR2 +region is only accessible to heavy-secure ucode. + +.. note:: + It is not clear why FwSec has 2 different partitions in the ROM, but they both + are of type 0xE0 and can be identified as such. This could be subject to change + in future generations. + +VBIOS ROM Layout +---------------- +The VBIOS layout is roughly a series of concatenated images laid out as follows:: + + +----------------------------------------------------------------------------+ + | VBIOS (Starting at ROM_OFFSET: 0x300000) | + +----------------------------------------------------------------------------+ + | +-----------------------------------------------+ | + | | PciAt Image (Type 0x00) | | + | +-----------------------------------------------+ | + | | +-------------------+ | | + | | | ROM Header | | | + | | | (Signature 0xAA55)| | | + | | +-------------------+ | | + | | | rom header's pci_data_struct_offset | | + | | | points to the PCIR structure | | + | | V | | + | | +-------------------+ | | + | | | PCIR Structure | | | + | | | (Signature "PCIR")| | | + | | | last_image: 0x80 | | | + | | | image_len: size | | | + | | | in 512-byte units | | | + | | +-------------------+ | | + | | | | | + | | | NPDE immediately follows PCIR | | + | | V | | + | | +-------------------+ | | + | | | NPDE Structure | | | + | | | (Signature "NPDE")| | | + | | | last_image: 0x00 | | | + | | +-------------------+ | | + | | | | + | | +-------------------+ | | + | | | BIT Header | (Signature scanning | | + | | | (Signature "BIT") | provides the location | | + | | +-------------------+ of the BIT table) | | + | | | header is | | + | | | followed by a table of tokens | | + | | V one of which is for falcon data. | | + | | +-------------------+ | | + | | | BIT Tokens | | | + | | | ______________ | | | + | | | | Falcon Data | | | | + | | | | Token (0x70)|---+------------>------------+--+ | + | | | +-------------+ | falcon_data_ptr() | | | + | | +-------------------+ | V | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | EFI Image (Type 0x03) | | | + | +-----------------------------------------------+ | | + | | Contains the UEFI GOP driver (Graphics Output)| | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | Image data | | | | + | | +-------------------+ | | | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | First FwSec Image (Type 0xE0) | | | + | +-----------------------------------------------+ | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | Image data | | | | + | | +-------------------+ | | | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | Second FwSec Image (Type 0xE0) | | | + | +-----------------------------------------------+ | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | | | + | | +-------------------+ | | | + | | | PMU Lookup Table | <- falcon_data_offset <----+ | + | | | +-------------+ | pmu_lookup_table | | + | | | | Entry 0x85 | | | | + | | | | FWSEC_PROD | | | | + | | | +-------------+ | | | + | | +-------------------+ | | + | | | | | + | | | points to | | + | | V | | + | | +-------------------+ | | + | | | FalconUCodeDescV3 | <- falcon_ucode_offset | | + | | | (FWSEC Firmware) | fwsec_header() | | + | | +-------------------+ | | + | | | immediately followed by... | | + | | V | | + | | +----------------------------+ | | + | | | Signatures + FWSEC Ucode | | | + | | | fwsec_sigs(), fwsec_ucode()| | | + | | +----------------------------+ | | + | +-----------------------------------------------+ | + | | + +----------------------------------------------------------------------------+ + +.. note:: + This diagram is created based on an GA-102 Ampere GPU as an example and could + vary for future or other GPUs. + +.. note:: + For more explanations of acronyms, see the detailed descriptions in `vbios.rs`. + +Falcon data Lookup +------------------ +A key part of the VBIOS extraction code (vbios.rs) is to find the location of the +Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is +used to find the required Falcon ucode based on an application ID. + +The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_) +tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the +offset of the same from the start of the VBIOS image. Unfortunately, the offset +does not account for the EFI image located between the PciAt and FwSec images. +The `vbios.rs` code compensates for this with appropriate arithmetic. + +.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst index 2701b3f4af358b373d4ef47975d5a80213f3e276..f38041fcd595524b204eabf6ca3aad51038682cf 100644 --- a/Documentation/gpu/nova/index.rst +++ b/Documentation/gpu/nova/index.rst @@ -28,3 +28,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/guidelines core/todo + core/vbios -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 6/8] Documentation: gpu: nova-core: Document devinit process
From: Joel Fernandes <joelagnelf at nvidia.com> devinit is mentioned in the code. This patch explains it so it is clear what it does. devinit is not only essential at boot-time, but also at runtime due to suspend-resume and things like re-clocking. Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Reviewed-by: Bagas Sanjaya <bagasdotme at gmail.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- Documentation/gpu/nova/core/devinit.rst | 61 +++++++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 62 insertions(+) diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst new file mode 100644 index 0000000000000000000000000000000000000000..70c819a96a00a0a27846e7e96525470d07721a10 --- /dev/null +++ b/Documentation/gpu/nova/core/devinit.rst @@ -0,0 +1,61 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================+Device Initialization (devinit) +=================================+The devinit process is complex and subject to change. This document provides a high-level +overview using the Ampere GPU family as an example. The goal is to provide a conceptual +overview of the process to aid in understanding the corresponding kernel code. + +Device initialization (devinit) is a crucial sequence of register read/write operations +that occur after a GPU reset. The devinit sequence is essential for properly configuring +the GPU hardware before it can be used. + +The devinit engine is an interpreter program that typically runs on the PMU (Power Management +Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization +commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the +FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the +nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the +FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while +devinit runs on the PMU in 'light-secure' mode. + +Key Functions of devinit +------------------------ +devinit performs several critical tasks: + +1. Programming VRAM memory controller timings +2. Power sequencing +3. Clock and PLL (Phase-Locked Loop) configuration +4. Thermal management + +Low-level Firmware Initialization Flow +-------------------------------------- +Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU +firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is +considered unusable until this initialization process completes. + +These low-level GPU firmware components are typically: + +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst). +2. Executed in sequence on different microcontrollers: + + - The devinit engine typically but not necessarily runs on the PMU. + - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in + heavy-secure mode. + +Before the driver can proceed with further initialization, it must wait for a signal +indicating that core initialization is complete (known as GFW_BOOT). This signal is +asserted by the FWSEC running on the GSP in heavy-secure mode. + +Runtime Considerations +---------------------- +It's important to note that the devinit sequence also needs to run during suspend/resume +operations at runtime, not just during initial boot, as it is critical to power management. + +Security and Access Control +--------------------------- +The initialization process involves careful privilege management. For example, before +accessing certain completion status registers, the driver must check privilege level +masks. Some registers are only accessible after secure firmware (FWSEC) lowers the +privilege level to allow CPU (LS/low-secure) access. This is the case, for example, +when receiving the GFW_BOOT signal. \ No newline at end of file diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst index f38041fcd595524b204eabf6ca3aad51038682cf..e4e017d926767284b5cee844d8dba32be6bc064c 100644 --- a/Documentation/gpu/nova/index.rst +++ b/Documentation/gpu/nova/index.rst @@ -29,3 +29,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/guidelines core/todo core/vbios + core/devinit -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 7/8] Documentation: gpu: nova-core: Document fwsec operation and layout
From: Joel Fernandes <joelagnelf at nvidia.com> Add explanation of fwsec with diagrams. This helps clarify how the nova-core falcon boot works. Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- Documentation/gpu/nova/core/fwsec.rst | 181 ++++++++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 182 insertions(+) diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644 index 0000000000000000000000000000000000000000..c440edbe420c39d7f405316a13472f8a13fe1d8d --- /dev/null +++ b/Documentation/gpu/nova/core/fwsec.rst @@ -0,0 +1,181 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========================+FWSEC (Firmware Security) +========================+This document briefly/conceptually describes the FWSEC (Firmware Security) image +and its role in the GPU boot sequence. As such, this information is subject to +change in the future and is only current as of the Ampere GPU family. However, +hopefully the concepts described will be useful for understanding the kernel code +that deals with it. All the information is derived from publicly available +sources such as public drivers and documentation. + +The role of FWSEC is to provide a secure boot process. It runs in +'Heavy-secure' mode, and performs firmware verification after a GPU reset +before loading various ucode images onto other microcontrollers on the GPU, +such as the PMU and GSP. + +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of +ROM (see vbios.rst for more details). It contains different commands like FRTS +(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after +reset and loading them with other non-FWSEC ucode). The kernel driver only needs +to perform FRTS, since Secure Boot (SB) has already completed by the time the driver +is loaded. + +The FRTS command carves out the WPR2 region (Write protected region) which contains +data required for power management. Once setup, only HS mode ucode can access it +(see falcon.rst for privilege levels). + +The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains +various ucode images (also known as applications) -- one of them being FWSEC. For how +it is extracted, see vbios.rst and the vbios.rs source code. + +The Falcon data for each ucode images (including the FWSEC image) is a combination +of headers, data sections (DMEM) and instruction code sections (IMEM). All these +ucode images are stored in the same ROM partition and the PMU table is used to look +up the application to load it based on its application ID (see vbios.rs). + +For the nova-core driver, the FWSEC contains an 'application interface' called +DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others. +For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS. + +FWSEC Memory Layout +------------------- +The memory layout of the FWSEC image is as follows:: + + +---------------------------------------------------------------+ + | FWSEC ROM image (type 0xE0) | + | | + | +---------------------------------+ | + | | PMU Falcon Ucode Table | | + | | (PmuLookupTable) | | + | | +-------------------------+ | | + | | | Table Header | | | + | | | - version: 0x01 | | | + | | | - header_size: 6 | | | + | | | - entry_size: 6 | | | + | | | - entry_count: N | | | + | | | - desc_version:3(unused)| | | + | | +-------------------------+ | | + | | ... | | + | | +-------------------------+ | | + | | | Entry for FWSEC (0x85) | | | + | | | (PmuLookupTableEntry) | | | + | | | - app_id: 0x85 (FWSEC) |----|----+ | + | | | - target_id: 0x01 (PMU) | | | | + | | | - data: offset ---------|----|----|---+ look up FWSEC | + | | +-------------------------+ | | | | + | +---------------------------------+ | | | + | | | | + | | | | + | +---------------------------------+ | | | + | | FWSEC Ucode Component |<---+ | | + | | (aka Falcon data) | | | + | | +-------------------------+ | | | + | | | FalconUCodeDescV3 |<---|--------+ | + | | | - hdr | | | + | | | - stored_size | | | + | | | - pkc_data_offset | | | + | | | - interface_offset -----|----|----------------+ | + | | | - imem_phys_base | | | | + | | | - imem_load_size | | | | + | | | - imem_virt_base | | | | + | | | - dmem_phys_base | | | | + | | | - dmem_load_size | | | | + | | | - engine_id_mask | | | | + | | | - ucode_id | | | | + | | | - signature_count | | look up sig | | + | | | - signature_versions --------------+ | | + | | +-------------------------+ | | | | + | | (no gap) | | | | + | | +-------------------------+ | | | | + | | | Signatures Section |<---|-----+ | | + | | | (384 bytes per sig) | | | | + | | | - RSA-3K Signature 1 | | | | + | | | - RSA-3K Signature 2 | | | | + | | | ... | | | | + | | +-------------------------+ | | | + | | | | | + | | +-------------------------+ | | | + | | | IMEM Section (Code) | | | | + | | | | | | | + | | | Contains instruction | | | | + | | | code etc. | | | | + | | +-------------------------+ | | | + | | | | | + | | +-------------------------+ | | | + | | | DMEM Section (Data) | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | Application | |<---|----------------+ | + | | | | Interface Table | | | | + | | | | (FalconAppifHdrV1) | | | | + | | | | Header: | | | | + | | | | - version: 0x01 | | | | + | | | | - header_size: 4 | | | | + | | | | - entry_size: 8 | | | | + | | | | - entry_count: N | | | | + | | | | | | | | + | | | | Entries: | | | | + | | | | +-----------------+ | | | | + | | | | | DEVINIT (ID 1) | | | | | + | | | | | - id: 0x01 | | | | | + | | | | | - dmemOffset X -|-|-|----+ | + | | | | +-----------------+ | | | | + | | | | +-----------------+ | | | | + | | | | | DMEMMAPPER(ID 4)| | | | | + | | | | | - id: 0x04 | | | | Used only for DevInit | + | | | | | (NVFW_FALCON_ | | | | application (not FWSEC) | + | | | | | APPIF_ID_DMEMMAPPER) | | + | | | | | - dmemOffset Y -|-|-|----|-----+ | + | | | | +-----------------+ | | | | | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | DEVINIT Engine |<|----+ | Used by FWSEC | + | | | | Interface | | | | app. | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | DMEM Mapper (ID 4) |<|----+-----+ | + | | | | (FalconAppifDmemmapperV3) | | + | | | | - signature: "DMAP" | | | | + | | | | - version: 0x0003 | | | | + | | | | - Size: 64 bytes | | | | + | | | | - cmd_in_buffer_off | |----|------------+ | + | | | | - cmd_in_buffer_size| | | | | + | | | | - cmd_out_buffer_off| |----|------------|-----+ | + | | | | - cmd_out_buffer_sz | | | | | | + | | | | - init_cmd | | | | | | + | | | | - features | | | | | | + | | | | - cmd_mask0/1 | | | | | | + | | | +---------------------+ | | | | | + | | | | | | | | + | | | +---------------------+ | | | | | + | | | | Command Input Buffer|<|----|------------+ | | + | | | | - Command data | | | | | + | | | | - Arguments | | | | | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | Command Output |<|----|------------------+ | + | | | | Buffer | | | | + | | | | - Results | | | | + | | | | - Status | | | | + | | | +---------------------+ | | | + | | +-------------------------+ | | + | +---------------------------------+ | + | | + +---------------------------------------------------------------+ + +.. note:: + This is using an GA-102 Ampere GPU as an example and could vary for future GPUs. + +.. note:: + The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR + (Video Protected Region) initialization as well. Before the nova-core driver is even + loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit + sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer + GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this + async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM + memory allocators need to wait for this scrubbing to complete). diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst index e4e017d926767284b5cee844d8dba32be6bc064c..e3650f53ff53113f31f63f67cf26116b5c070693 100644 --- a/Documentation/gpu/nova/index.rst +++ b/Documentation/gpu/nova/index.rst @@ -30,3 +30,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/todo core/vbios core/devinit + core/fwsec -- 2.50.0
Alexandre Courbot
2025-Jul-08 06:49 UTC
[PATCH v4 8/8] Documentation: gpu: nova-core: Document basics of the Falcon
From: Joel Fernandes <joelagnelf at nvidia.com> Instances of the Falcon microcontroller appear in modern Nvidia GPUs and are crucial to the GPU boot process. Document some concepts which will make nova-core boot code easier to digest. All the information is derived from public sources such as public documents, OpenRM and Nouveau code. Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> Reviewed-by: Bagas Sanjaya <bagasdotme at gmail.com> Signed-off-by: Alexandre Courbot <acourbot at nvidia.com> --- Documentation/gpu/nova/core/falcon.rst | 158 +++++++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 159 insertions(+) diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst new file mode 100644 index 0000000000000000000000000000000000000000..33137082eb6c14cecda2fbe6fdb79e63ee9ca2e6 --- /dev/null +++ b/Documentation/gpu/nova/core/falcon.rst @@ -0,0 +1,158 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============================+Falcon (FAst Logic Controller) +=============================+The following sections describe the Falcon core and the ucode running on it. +The descriptions are based on the Ampere GPU or earlier designs; however, they +should mostly apply to future designs as well, but everything is subject to +change. The overview provided here is mainly tailored towards understanding the +interactions of nova-core driver with the Falcon. + +NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which +handle secure firmware tasks, initialization, and power management. Modern +NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system +processor) and SEC2 (the security engine)) and also may integrate a RISC-V core. +This core is capable of running both RISC-V and Falcon code. + +The code running on the Falcon cores is also called 'ucode', and will be +referred to as such in the following sections. + +Falcons have separate instruction and data memories (IMEM/DMEM) and provide a +small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from +system memory. The nova-core driver must reset and configure the Falcon, load +its firmware via DMA, and start its CPU. + +Falcon security levels +=====================+Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS) +modes. + +Heavy Secured (HS) also known as Privilege Level 3 (PL3) +-------------------------------------------------------- +HS ucode is the most trusted code and has access to pretty much everything on +the chip. The HS binary includes a signature in it which is verified at boot. +This signature verification is done by the hardware itself, thus establishing a +root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the +GSP in HS mode. FRTS, which involves setting up and loading content into the WPR +(Write Protect Region), has to be done by the HS ucode and cannot be done by the +host CPU or LS ucode. + +Light Secured (LS or PL2) and Non Secured (NS or PL0) +----------------------------------------------------- +These modes are less secure than HS. Like HS, the LS or NS ucode binary also +typically includes a signature in it. To load firmware in LS or NS mode onto a +Falcon, another Falcon needs to be running in HS mode, which also establishes the +root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter" +ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the +run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an +example, after reset on an Ampere, FWSEC runs on the GSP which then loads the +devinit engine onto the PMU in LS mode. + +Root of trust establishment +--------------------------- +To establish a root of trust, the code running on a Falcon must be immutable and +hardwired into a read-only memory (ROM). This follows industry norms for +verification of firmware. This code is called the Boot ROM (BROM). The nova-core +driver on the CPU communicates with Falcon's Boot ROM through various Falcon +registers prefixed with "BROM" (see regs.rs). + +After nova-core driver reads the necessary ucode from VBIOS, it programs the +BROM and DMA registers to trigger the Falcon to load the HS ucode from the system +memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified +by the Falcon's Boot ROM. + +Once the verified HS code is running on a Falcon, it can verify and load other +LS/NS ucode binaries onto other Falcons and start them. The process of signature +verification is the same as HS; just in this case, the hardware (BROM) doesn't +compute the signature, but the HS ucode does. + +The root of trust is therefore established as follows: + Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode. + +On an Ampere GPU, for example, the boot verification flow is: + Hardware (Boot ROM running on the SEC2) -> + HS ucode (Booter running on the SEC2) -> + LS ucode (GSP-RM running on the GSP) + +.. note:: + While the CPU can load HS ucode onto a Falcon microcontroller and have it + verified by the hardware and run, the CPU itself typically does not load + LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the + HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the + SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run + the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no + component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load + LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is + temporarily made to run HS ucode (which is itself loaded by the CPU via + the nova-core driver using a "GSP-provided sequencer") which then loads + the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes + running its own GSP-RM LS ucode. + +Falcon memory subsystem and DMA engine +=====================================+Falcons have separate instruction and data memories (IMEM/DMEM) +and contains a small DMA engine called FBDMA (Framebuffer DMA) which does +DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF +(Framebuffer Interface), to external memory. + +DMA transfers are possible from the Falcon's memory to both the system memory +and the framebuffer memory (VRAM). + +To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory +is accessed (also known as aperture type). In the nova-core driver, this is +determined by the `FalconFbifTarget` enum. + +The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon +controls access by the FBDMA to the external memory. + +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:: + + External Memory (Framebuffer / System DRAM) + ^ | + | | + | v + +-----------------------------------------------------+ + | | | + | +---------------+ | | + | | FBIF |-------+ | FALCON + | | (FrameBuffer | Memory Interface | PROCESSOR + | | InterFace) | | + | | Apertures | | + | | Configures | | + | | mem access | | + | +-------^-------+ | + | | | + | | FBDMA uses configured FBIF apertures | + | | to access External Memory + | | + | +-------v--------+ +---------------+ + | | FBDMA | cfg | RISC | + | | (FrameBuffer |<---->| CORE |----->. Direct Core Access + | | DMA Engine) | | | | + | | - Master dev. | | (can run both | | + | +-------^--------+ | Falcon and | | + | | cfg--->| RISC-V code) | | + | | / | | | + | | | +---------------+ | +------------+ + | | | | | BROM | + | | | <--->| (Boot ROM) | + | | / | +------------+ + | | v | + | +---------------+ | + | | IO-PMP | Controls access by FBDMA | + | | (IO Physical | and other IO Masters | + | | Memory Protect) | + | +-------^-------+ | + | | | + | | Protected Access Path for FBDMA | + | v | + | +---------------------------------------+ | + | | Memory | | + | | +---------------+ +------------+ | | + | | | IMEM | | DMEM | |<-----+ + | | | (Instruction | | (Data | | + | | | Memory) | | Memory) | | + | | +---------------+ +------------+ | + | +---------------------------------------+ + +-----------------------------------------------------+ diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst index e3650f53ff53113f31f63f67cf26116b5c070693..e39cb3163581ea4ff5b441b82e9efa4282c946f8 100644 --- a/Documentation/gpu/nova/index.rst +++ b/Documentation/gpu/nova/index.rst @@ -31,3 +31,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/vbios core/devinit core/fwsec + core/falcon -- 2.50.0
On Tue, Jul 08, 2025 at 03:49:40PM +0900, Alexandre Courbot wrote:> This series adds some documentation that was relevant to the FWSEC-FRTS > [1] series, but wasn't sent alongside it as it was worked on in > parallel.Joel, thanks a lot for this series. It's great to have this documentation -- very much appreciated. Applied to nova-next, thanks!