1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
=========================
FWSEC (Firmware Security)
=========================
This document briefly/conceptually describes the FWSEC (Firmware Security) image
and its role in the GPU boot sequence. As such, this information is subject to
change in the future and is only current as of the Ampere GPU family. However,
hopefully the concepts described will be useful for understanding the kernel code
that deals with it. All the information is derived from publicly available
sources such as public drivers and documentation.
The role of FWSEC is to provide a secure boot process. It runs in
'Heavy-secure' mode, and performs firmware verification after a GPU reset
before loading various ucode images onto other microcontrollers on the GPU,
such as the PMU and GSP.
FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
ROM (see vbios.rst for more details). It contains different commands like FRTS
(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
reset and loading them with other non-FWSEC ucode). The kernel driver only needs
to perform FRTS, since Secure Boot (SB) has already completed by the time the driver
is loaded.
The FRTS command carves out the WPR2 region (Write protected region) which contains
data required for power management. Once setup, only HS mode ucode can access it
(see falcon.rst for privilege levels).
The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
various ucode images (also known as applications) -- one of them being FWSEC. For how
it is extracted, see vbios.rst and the vbios.rs source code.
The Falcon data for each ucode images (including the FWSEC image) is a combination
of headers, data sections (DMEM) and instruction code sections (IMEM). All these
ucode images are stored in the same ROM partition and the PMU table is used to look
up the application to load it based on its application ID (see vbios.rs).
For the nova-core driver, the FWSEC contains an 'application interface' called
DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others.
For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS.
FWSEC Memory Layout
-------------------
The memory layout of the FWSEC image is as follows::
+---------------------------------------------------------------+
| FWSEC ROM image (type 0xE0) |
| |
| +---------------------------------+ |
| | PMU Falcon Ucode Table | |
| | (PmuLookupTable) | |
| | +-------------------------+ | |
| | | Table Header | | |
| | | - version: 0x01 | | |
| | | - header_size: 6 | | |
| | | - entry_size: 6 | | |
| | | - entry_count: N | | |
| | | - desc_version:3(unused)| | |
| | +-------------------------+ | |
| | ... | |
| | +-------------------------+ | |
| | | Entry for FWSEC (0x85) | | |
| | | (PmuLookupTableEntry) | | |
| | | - app_id: 0x85 (FWSEC) |----|----+ |
| | | - target_id: 0x01 (PMU) | | | |
| | | - data: offset ---------|----|----|---+ look up FWSEC |
| | +-------------------------+ | | | |
| +---------------------------------+ | | |
| | | |
| | | |
| +---------------------------------+ | | |
| | FWSEC Ucode Component |<---+ | |
| | (aka Falcon data) | | |
| | +-------------------------+ | | |
| | | FalconUCodeDescV3 |<---|--------+ |
| | | - hdr | | |
| | | - stored_size | | |
| | | - pkc_data_offset | | |
| | | - interface_offset -----|----|----------------+ |
| | | - imem_phys_base | | | |
| | | - imem_load_size | | | |
| | | - imem_virt_base | | | |
| | | - dmem_phys_base | | | |
| | | - dmem_load_size | | | |
| | | - engine_id_mask | | | |
| | | - ucode_id | | | |
| | | - signature_count | | look up sig | |
| | | - signature_versions --------------+ | |
| | +-------------------------+ | | | |
| | (no gap) | | | |
| | +-------------------------+ | | | |
| | | Signatures Section |<---|-----+ | |
| | | (384 bytes per sig) | | | |
| | | - RSA-3K Signature 1 | | | |
| | | - RSA-3K Signature 2 | | | |
| | | ... | | | |
| | +-------------------------+ | | |
| | | | |
| | +-------------------------+ | | |
| | | IMEM Section (Code) | | | |
| | | | | | |
| | | Contains instruction | | | |
| | | code etc. | | | |
| | +-------------------------+ | | |
| | | | |
| | +-------------------------+ | | |
| | | DMEM Section (Data) | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | Application | |<---|----------------+ |
| | | | Interface Table | | | |
| | | | (FalconAppifHdrV1) | | | |
| | | | Header: | | | |
| | | | - version: 0x01 | | | |
| | | | - header_size: 4 | | | |
| | | | - entry_size: 8 | | | |
| | | | - entry_count: N | | | |
| | | | | | | |
| | | | Entries: | | | |
| | | | +-----------------+ | | | |
| | | | | DEVINIT (ID 1) | | | | |
| | | | | - id: 0x01 | | | | |
| | | | | - dmemOffset X -|-|-|----+ |
| | | | +-----------------+ | | | |
| | | | +-----------------+ | | | |
| | | | | DMEMMAPPER(ID 4)| | | | |
| | | | | - id: 0x04 | | | | Used only for DevInit |
| | | | | (NVFW_FALCON_ | | | | application (not FWSEC) |
| | | | | APPIF_ID_DMEMMAPPER) | |
| | | | | - dmemOffset Y -|-|-|----|-----+ |
| | | | +-----------------+ | | | | |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | DEVINIT Engine |<|----+ | Used by FWSEC |
| | | | Interface | | | | app. |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | DMEM Mapper (ID 4) |<|----+-----+ |
| | | | (FalconAppifDmemmapperV3) | |
| | | | - signature: "DMAP" | | | |
| | | | - version: 0x0003 | | | |
| | | | - Size: 64 bytes | | | |
| | | | - cmd_in_buffer_off | |----|------------+ |
| | | | - cmd_in_buffer_size| | | | |
| | | | - cmd_out_buffer_off| |----|------------|-----+ |
| | | | - cmd_out_buffer_sz | | | | | |
| | | | - init_cmd | | | | | |
| | | | - features | | | | | |
| | | | - cmd_mask0/1 | | | | | |
| | | +---------------------+ | | | | |
| | | | | | | |
| | | +---------------------+ | | | | |
| | | | Command Input Buffer|<|----|------------+ | |
| | | | - Command data | | | | |
| | | | - Arguments | | | | |
| | | +---------------------+ | | | |
| | | | | | |
| | | +---------------------+ | | | |
| | | | Command Output |<|----|------------------+ |
| | | | Buffer | | | |
| | | | - Results | | | |
| | | | - Status | | | |
| | | +---------------------+ | | |
| | +-------------------------+ | |
| +---------------------------------+ |
| |
+---------------------------------------------------------------+
.. note::
This is using an GA-102 Ampere GPU as an example and could vary for future GPUs.
.. note::
The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
(Video Protected Region) initialization as well. Before the nova-core driver is even
loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
memory allocators need to wait for this scrubbing to complete).
|
