[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout

Zhi Wang
2025-05-06T16:26:41Z

On Sat,  3 May 2025 00:07:56 -0400
Joel Fernandes <joelagnelf at nvidia.com> wrote:

> Add explanation of fwsec with diagrams. This helps clarify how the
> nova-core falcon boot works.
>
> Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
> ---
>  Documentation/gpu/nova/core/fwsec.rst | 180
> ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst      |
> 1 + 2 files changed, 181 insertions(+)
>  create mode 100644 Documentation/gpu/nova/core/fwsec.rst
>
> diff --git a/Documentation/gpu/nova/core/fwsec.rst
> b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644
> index 000000000000..bed941ac3f2b
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/fwsec.rst
> @@ -0,0 +1,180 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +========================> +FWSEC (Firmware Security)
> +========================> +This document briefly/conceptually describes
the FWSEC (Firmware
> Security) image +and its role in the GPU boot sequence. As such, this
> information is subject to +change in the future and is only current
> as of the Ampere GPU family. However, +hopefully the concepts
> described will be useful for understanding the kernel code +that
> deals with it. All the information is derived from publicly available
> +sources such as public drivers and documentation. +
> +The role of FWSEC to provide secure boot, it is running in
> Heavy-secure mode. It does +firmware verification after GPU reset and
> load various ucode images on to the other +microcontrollers on the
> GPU such as the PMU and GSP. +
> +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC
> partition of +ROM (see vbios.rst for more details). It contains
> different commands like FRTS +(Firmware Runtime Services) and SB
> (Secure Booting other microcontrollers after +reset and loading them
> with other non-FWSEC ucode). The kernel driver only needs to +to do
> FRTS, since SB is already already after reset by the time the kernel
> driver +is loaded. +
> +The FRTS command carves out the WPR2 region (Write protected region)
> which contains +data data required for power management. Once setup,
repetitive word "data" ^

> only HS mode ucode can +access it (see falcon.rst for privilege
> levels). +
> +The FWSEC image is located in the VBIOS ROM in the partition of the
> ROM that contains +various ucode images (also known as applications)
> -- one of them being FWSEC. For how +it is extracted, see vbios.rst
> and the vbios.rs source code. +
> +The Falcon data for each ucode images (including the FWSEC image) is
> a combination +of headers, data sections (DMEM) and instruction code
> sections (IMEM). All these +ucode images are stored in the same ROM
> partition and the PMU table is used to look +up the application to
> load it based on its application ID (see vbios.rs). +
> +For the purposes of nova-core driver, the FWSEC contains an
> 'application interface' +called DMEMMAPPER which is used to the
> "FWSEC-FRTS" command (among other commands it +is capable of
> executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
> +mode and runs FRTS. +
> +FWSEC Memory Layout
> +-------------------
> +The memory layout of the FWSEC image is as follows (this is using an
> GA-102 +Ampere GPU as an example and could vary for future GPUs and
> is subject to change +completely, it is just provided as an example):
> +
> +Here is a block diagram of the FWSEC memory layout::
> + ?????????????????????????????????????????????????????????????????
> + ?                         FWSEC ROM image (type 0xE0)           ?
> + ?                                                               ?
> + ?  ???????????????????????????????????                          ?
> + ?  ?     PMU Falcon Ucode Table      ?                          ?
> + ?  ?     (PmuLookupTable)            ?                          ?
> + ?  ?  ???????????????????????????    ?                          ?
> + ?  ?  ? Table Header            ?    ?                          ?
> + ?  ?  ? - version: 0x01         ?    ?                          ?
> + ?  ?  ? - header_size: 6        ?    ?                          ?
> + ?  ?  ? - entry_size: 6         ?    ?                          ?
> + ?  ?  ? - entry_count: N        ?    ?                          ?
> + ?  ?  ? - desc_version:3(unused)?    ?                          ?
> + ?  ?  ???????????????????????????    ?                          ?
> + ?  ?         ...                     ?                          ?
> + ?  ?  ???????????????????????????    ?                          ?
> + ?  ?  ? Entry for FWSEC (0x85)  ?    ?                          ?
> + ?  ?  ? (PmuLookupTableEntry)   ?    ?                          ?
> + ?  ?  ? - app_id: 0x85 (FWSEC)  ? ?????????                     ?
> + ?  ?  ? - target_id: 0x01 (PMU) ?    ?    ?                     ?
> + ?  ?  ? - data: offset ???????????????????????? look up FWSEC   ?
> + ?  ?  ???????????????????????????    ?    ?   ? application.    ?
> + ?  ???????????????????????????????????    ?   ?                 ?
> + ?                                         ?   ?                 ?
> + ?                                         ?   ?                 ?
> + ?  ???????????????????????????????????    ?   ?                 ?
> + ?  ?     FWSEC Ucode Component       ?<????   ?                 ?
> + ?  ?     (aka Falcon data)           ?        ?                 ?
> + ?  ?  ???????????????????????????    ?        ?                 ?
> + ?  ?  ? FalconUCodeDescV3       ?<?????????????                 ?
> + ?  ?  ? - hdr                   ?    ?                          ?
> + ?  ?  ? - stored_size           ?    ?                          ?
> + ?  ?  ? - pkc_data_offset       ?    ?                          ?
> + ?  ?  ? - interface_offset ????????????????????????????         ?
> + ?  ?  ? - imem_phys_base        ?    ?                ?         ?
> + ?  ?  ? - imem_load_size        ?    ?                ?         ?
> + ?  ?  ? - imem_virt_base        ?    ?                ?         ?
> + ?  ?  ? - dmem_phys_base        ?    ?                ?         ?
> + ?  ?  ? - dmem_load_size        ?    ?                ?         ?
> + ?  ?  ? - engine_id_mask        ?    ?                ?         ?
> + ?  ?  ? - ucode_id              ?    ?                ?         ?
> + ?  ?  ? - signature_count       ?    ?    look up sig ?         ?
> + ?  ?  ? - signature_versions --------------+          ?         ?
> + ?  ?  ???????????????????????????    ?     ?          ?         ?
> + ?  ?         (no gap)                ?     ?          ?         ?
> + ?  ?  ???????????????????????????    ?     ?          ?         ?
> + ?  ?  ? Signatures Section      ?<??????????          ?         ?
> + ?  ?  ? (384 bytes per sig)     ?    ?                ?         ?
> + ?  ?  ? - RSA-3K Signature 1    ?    ?                ?         ?
> + ?  ?  ? - RSA-3K Signature 2    ?    ?                ?         ?
> + ?  ?  ?   ...                   ?    ?                ?         ?
> + ?  ?  ???????????????????????????    ?                ?         ?
> + ?  ?                                 ?                ?         ?
> + ?  ?  ???????????????????????????    ?                ?         ?
> + ?  ?  ? IMEM Section (Code)     ?    ?                ?         ?
> + ?  ?  ?                         ?    ?                ?         ?
> + ?  ?  ? Contains instruction    ?    ?                ?         ?
> + ?  ?  ? code etc.               ?    ?                ?         ?
> + ?  ?  ???????????????????????????    ?                ?         ?
> + ?  ?                                 ?                ?         ?
> + ?  ?  ???????????????????????????    ?                ?         ?
> + ?  ?  ? DMEM Section (Data)     ?    ?                ?         ?
> + ?  ?  ?                         ?    ?                ?         ?
> + ?  ?  ? ??????????????????????? ?    ?                ?         ?
> + ?  ?  ? ? Application         ? ?<?????????????????????         ?
> + ?  ?  ? ? Interface Table     ? ?    ?                          ?
> + ?  ?  ? ? (FalconAppifHdrV1)  ? ?    ?                          ?
> + ?  ?  ? ? Header:             ? ?    ?                          ?
> + ?  ?  ? ? - version: 0x01     ? ?    ?                          ?
> + ?  ?  ? ? - header_size: 4    ? ?    ?                          ?
> + ?  ?  ? ? - entry_size: 8     ? ?    ?                          ?
> + ?  ?  ? ? - entry_count: N    ? ?    ?                          ?
> + ?  ?  ? ?                     ? ?    ?                          ?
> + ?  ?  ? ? Entries:            ? ?    ?                          ?
> + ?  ?  ? ? ??????????????????? ? ?    ?                          ?
> + ?  ?  ? ? ? DEVINIT (ID 1)  ? ? ?    ?                          ?
> + ?  ?  ? ? ? - id: 0x01      ? ? ?    ?                          ?
> + ?  ?  ? ? ? - dmemOffset X ???????????                          ?
> + ?  ?  ? ? ??????????????????? ? ?    ?                          ?
> + ?  ?  ? ? ??????????????????? ? ?    ?                          ?
> + ?  ?  ? ? ? DMEMMAPPER(ID 4)? ? ?    ?                          ?
> + ?  ?  ? ? ? - id: 0x04      ? ? ?    ? Used only for DevInit    ?
> + ?  ?  ? ? ?  (NVFW_FALCON_  ? ? ?    ? application (not FWSEC)  ?
> + ?  ?  ? ? ?   APPIF_ID_DMEMMAPPER)   ?                          ?
> + ?  ?  ? ? ? - dmemOffset Y ?????????????????                    ?
> + ?  ?  ? ? ??????????????????? ? ?    ?     ?                    ?
> + ?  ?  ? ??????????????????????? ?    ?     ?                    ?
> + ?  ?  ?                         ?    ?     ?                    ?
> + ?  ?  ? ??????????????????????? ?    ?     ?                    ?
> + ?  ?  ? ? DEVINIT Engine      ?<??????     ? Used by FWSEC      ?
> + ?  ?  ? ? Interface           ? ?    ?     ?         app.       ?
> + ?  ?  ? ??????????????????????? ?    ?     ?                    ?
> + ?  ?  ?                         ?    ?     ?                    ?
> + ?  ?  ? ??????????????????????? ?    ?     ?                    ?
> + ?  ?  ? ? DMEM Mapper (ID 4)  ?<?????+??????                    ?
> + ?  ?  ? ? (FalconAppifDmemmapperV3)  ?                          ?
> + ?  ?  ? ? - signature: "DMAP" ? ?    ?
?
> + ?  ?  ? ? - version: 0x0003   ? ?    ?                          ?
> + ?  ?  ? ? - Size: 64 bytes    ? ?    ?                          ?
> + ?  ?  ? ? - cmd_in_buffer_off ? ???????????????????             ?
> + ?  ?  ? ? - cmd_in_buffer_size? ?    ?            ?             ?
> + ?  ?  ? ? - cmd_out_buffer_off? ?????????????????????????       ?
> + ?  ?  ? ? - cmd_out_buffer_sz ? ?    ?            ?     ?       ?
> + ?  ?  ? ? - init_cmd          ? ?    ?            ?     ?       ?
> + ?  ?  ? ? - features          ? ?    ?            ?     ?       ?
> + ?  ?  ? ? - cmd_mask0/1       ? ?    ?            ?     ?       ?
> + ?  ?  ? ??????????????????????? ?    ?            ?     ?       ?
> + ?  ?  ?                         ?    ?            ?     ?       ?
> + ?  ?  ? ??????????????????????? ?    ?            ?     ?       ?
> + ?  ?  ? ? Command Input Buffer?<???????????????????     ?       ?
> + ?  ?  ? ? - Command data      ? ?    ?                  ?       ?
> + ?  ?  ? ? - Arguments         ? ?    ?                  ?       ?
> + ?  ?  ? ??????????????????????? ?    ?                  ?       ?
> + ?  ?  ?                         ?    ?                  ?       ?
> + ?  ?  ? ??????????????????????? ?    ?                  ?       ?
> + ?  ?  ? ? Command Output      ?<?????????????????????????       ?
> + ?  ?  ? ? Buffer              ? ?    ?                          ?
> + ?  ?  ? ? - Results           ? ?    ?                          ?
> + ?  ?  ? ? - Status            ? ?    ?                          ?
> + ?  ?  ? ??????????????????????? ?    ?                          ?
> + ?  ?  ???????????????????????????    ?                          ?
> + ?  ???????????????????????????????????                          ?
> + ?                                                               ?
> + ?????????????????????????????????????????????????????????????????
> +
> +.. note::
> +   The FWSEC image also plays a role in memory scrubbing (ECC
> initialization) and VPR
> +   (Video Protected Region) initialization as well. Before the
> nova-core driver is even
> +   loaded, the FWSEC image is running on the GSP in heavy-secure
> mode. After the devinit
> +   sequence completes, it does VRAM memory scrubbing (ECC
> initialization). On consumer
> +   GPUs, it scrubs only part of memory and then initiates 'async
> scrubbing'. Before this
> +   async scrubbing completes, the unscrubbed VRAM cannot be used for
> allocation (thus DRM
> +   memory allocators need to wait for this scrubbing to complete).
> \ No newline at end of file
> diff --git a/Documentation/gpu/nova/index.rst
> b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0
> 100644 --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
>
>     core/guidelines
>     core/vbios
> +   core/fwsec
>     core/todo