Zhi Wang
2025-May-06 16:26 UTC
[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
On Sat, 3 May 2025 00:07:56 -0400 Joel Fernandes <joelagnelf at nvidia.com> wrote:> Add explanation of fwsec with diagrams. This helps clarify how the > nova-core falcon boot works. > > Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com> > --- > Documentation/gpu/nova/core/fwsec.rst | 180 > ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | > 1 + 2 files changed, 181 insertions(+) > create mode 100644 Documentation/gpu/nova/core/fwsec.rst > > diff --git a/Documentation/gpu/nova/core/fwsec.rst > b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644 > index 000000000000..bed941ac3f2b > --- /dev/null > +++ b/Documentation/gpu/nova/core/fwsec.rst > @@ -0,0 +1,180 @@ > +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) > +========================> +FWSEC (Firmware Security) > +========================> +This document briefly/conceptually describes the FWSEC (Firmware > Security) image +and its role in the GPU boot sequence. As such, this > information is subject to +change in the future and is only current > as of the Ampere GPU family. However, +hopefully the concepts > described will be useful for understanding the kernel code +that > deals with it. All the information is derived from publicly available > +sources such as public drivers and documentation. + > +The role of FWSEC to provide secure boot, it is running in > Heavy-secure mode. It does +firmware verification after GPU reset and > load various ucode images on to the other +microcontrollers on the > GPU such as the PMU and GSP. + > +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC > partition of +ROM (see vbios.rst for more details). It contains > different commands like FRTS +(Firmware Runtime Services) and SB > (Secure Booting other microcontrollers after +reset and loading them > with other non-FWSEC ucode). The kernel driver only needs to +to do > FRTS, since SB is already already after reset by the time the kernel > driver +is loaded. + > +The FRTS command carves out the WPR2 region (Write protected region) > which contains +data data required for power management. Once setup,repetitive word "data" ^> only HS mode ucode can +access it (see falcon.rst for privilege > levels). + > +The FWSEC image is located in the VBIOS ROM in the partition of the > ROM that contains +various ucode images (also known as applications) > -- one of them being FWSEC. For how +it is extracted, see vbios.rst > and the vbios.rs source code. + > +The Falcon data for each ucode images (including the FWSEC image) is > a combination +of headers, data sections (DMEM) and instruction code > sections (IMEM). All these +ucode images are stored in the same ROM > partition and the PMU table is used to look +up the application to > load it based on its application ID (see vbios.rs). + > +For the purposes of nova-core driver, the FWSEC contains an > 'application interface' +called DMEMMAPPER which is used to the > "FWSEC-FRTS" command (among other commands it +is capable of > executing). For Ampere, FWSEC is running on the GSP in Heavy-secure > +mode and runs FRTS. + > +FWSEC Memory Layout > +------------------- > +The memory layout of the FWSEC image is as follows (this is using an > GA-102 +Ampere GPU as an example and could vary for future GPUs and > is subject to change +completely, it is just provided as an example): > + > +Here is a block diagram of the FWSEC memory layout:: > + ????????????????????????????????????????????????????????????????? > + ? FWSEC ROM image (type 0xE0) ? > + ? ? > + ? ??????????????????????????????????? ? > + ? ? PMU Falcon Ucode Table ? ? > + ? ? (PmuLookupTable) ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Table Header ? ? ? > + ? ? ? - version: 0x01 ? ? ? > + ? ? ? - header_size: 6 ? ? ? > + ? ? ? - entry_size: 6 ? ? ? > + ? ? ? - entry_count: N ? ? ? > + ? ? ? - desc_version:3(unused)? ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ... ? ? > + ? ? ??????????????????????????? ? ? > + ? ? ? Entry for FWSEC (0x85) ? ? ? > + ? ? ? (PmuLookupTableEntry) ? ? ? > + ? ? ? - app_id: 0x85 (FWSEC) ? ????????? ? > + ? ? ? - target_id: 0x01 (PMU) ? ? ? ? > + ? ? ? - data: offset ???????????????????????? look up FWSEC ? > + ? ? ??????????????????????????? ? ? ? application. ? > + ? ??????????????????????????????????? ? ? ? > + ? ? ? ? > + ? ? ? ? > + ? ??????????????????????????????????? ? ? ? > + ? ? FWSEC Ucode Component ?<???? ? ? > + ? ? (aka Falcon data) ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? FalconUCodeDescV3 ?<????????????? ? > + ? ? ? - hdr ? ? ? > + ? ? ? - stored_size ? ? ? > + ? ? ? - pkc_data_offset ? ? ? > + ? ? ? - interface_offset ???????????????????????????? ? > + ? ? ? - imem_phys_base ? ? ? ? > + ? ? ? - imem_load_size ? ? ? ? > + ? ? ? - imem_virt_base ? ? ? ? > + ? ? ? - dmem_phys_base ? ? ? ? > + ? ? ? - dmem_load_size ? ? ? ? > + ? ? ? - engine_id_mask ? ? ? ? > + ? ? ? - ucode_id ? ? ? ? > + ? ? ? - signature_count ? ? look up sig ? ? > + ? ? ? - signature_versions --------------+ ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? (no gap) ? ? ? ? > + ? ? ??????????????????????????? ? ? ? ? > + ? ? ? Signatures Section ?<?????????? ? ? > + ? ? ? (384 bytes per sig) ? ? ? ? > + ? ? ? - RSA-3K Signature 1 ? ? ? ? > + ? ? ? - RSA-3K Signature 2 ? ? ? ? > + ? ? ? ... ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? IMEM Section (Code) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? Contains instruction ? ? ? ? > + ? ? ? code etc. ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? ? ? > + ? ? ??????????????????????????? ? ? ? > + ? ? ? DMEM Section (Data) ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Application ? ?<????????????????????? ? > + ? ? ? ? Interface Table ? ? ? ? > + ? ? ? ? (FalconAppifHdrV1) ? ? ? ? > + ? ? ? ? Header: ? ? ? ? > + ? ? ? ? - version: 0x01 ? ? ? ? > + ? ? ? ? - header_size: 4 ? ? ? ? > + ? ? ? ? - entry_size: 8 ? ? ? ? > + ? ? ? ? - entry_count: N ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ? Entries: ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DEVINIT (ID 1) ? ? ? ? ? > + ? ? ? ? ? - id: 0x01 ? ? ? ? ? > + ? ? ? ? ? - dmemOffset X ??????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ??????????????????? ? ? ? ? > + ? ? ? ? ? DMEMMAPPER(ID 4)? ? ? ? ? > + ? ? ? ? ? - id: 0x04 ? ? ? ? Used only for DevInit ? > + ? ? ? ? ? (NVFW_FALCON_ ? ? ? ? application (not FWSEC) ? > + ? ? ? ? ? APPIF_ID_DMEMMAPPER) ? ? > + ? ? ? ? ? - dmemOffset Y ????????????????? ? > + ? ? ? ? ??????????????????? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DEVINIT Engine ?<?????? ? Used by FWSEC ? > + ? ? ? ? Interface ? ? ? ? app. ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? DMEM Mapper (ID 4) ?<?????+?????? ? > + ? ? ? ? (FalconAppifDmemmapperV3) ? ? > + ? ? ? ? - signature: "DMAP" ? ? ? ? > + ? ? ? ? - version: 0x0003 ? ? ? ? > + ? ? ? ? - Size: 64 bytes ? ? ? ? > + ? ? ? ? - cmd_in_buffer_off ? ??????????????????? ? > + ? ? ? ? - cmd_in_buffer_size? ? ? ? ? > + ? ? ? ? - cmd_out_buffer_off? ????????????????????????? ? > + ? ? ? ? - cmd_out_buffer_sz ? ? ? ? ? ? > + ? ? ? ? - init_cmd ? ? ? ? ? ? > + ? ? ? ? - features ? ? ? ? ? ? > + ? ? ? ? - cmd_mask0/1 ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? ? > + ? ? ? ? Command Input Buffer?<??????????????????? ? ? > + ? ? ? ? - Command data ? ? ? ? ? > + ? ? ? ? - Arguments ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? ? > + ? ? ? ? Command Output ?<????????????????????????? ? > + ? ? ? ? Buffer ? ? ? ? > + ? ? ? ? - Results ? ? ? ? > + ? ? ? ? - Status ? ? ? ? > + ? ? ? ??????????????????????? ? ? ? > + ? ? ??????????????????????????? ? ? > + ? ??????????????????????????????????? ? > + ? ? > + ????????????????????????????????????????????????????????????????? > + > +.. note:: > + The FWSEC image also plays a role in memory scrubbing (ECC > initialization) and VPR > + (Video Protected Region) initialization as well. Before the > nova-core driver is even > + loaded, the FWSEC image is running on the GSP in heavy-secure > mode. After the devinit > + sequence completes, it does VRAM memory scrubbing (ECC > initialization). On consumer > + GPUs, it scrubs only part of memory and then initiates 'async > scrubbing'. Before this > + async scrubbing completes, the unscrubbed VRAM cannot be used for > allocation (thus DRM > + memory allocators need to wait for this scrubbing to complete). > \ No newline at end of file > diff --git a/Documentation/gpu/nova/index.rst > b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0 > 100644 --- a/Documentation/gpu/nova/index.rst > +++ b/Documentation/gpu/nova/index.rst > @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver. > > core/guidelines > core/vbios > + core/fwsec > core/todo