diff options
Diffstat (limited to 'Documentation/admin-guide')
| -rw-r--r-- | Documentation/admin-guide/RAS/address-translation.rst | 24 | ||||
| -rw-r--r-- | Documentation/admin-guide/RAS/error-decoding.rst | 21 | ||||
| -rw-r--r-- | Documentation/admin-guide/RAS/index.rst | 7 | ||||
| -rw-r--r-- | Documentation/admin-guide/RAS/main.rst (renamed from Documentation/admin-guide/ras.rst) | 10 | ||||
| -rw-r--r-- | Documentation/admin-guide/README.rst | 69 | ||||
| -rw-r--r-- | Documentation/admin-guide/cgroup-v1/cpusets.rst | 2 | ||||
| -rw-r--r-- | Documentation/admin-guide/cgroup-v1/hugetlb.rst | 20 | ||||
| -rw-r--r-- | Documentation/admin-guide/hw-vuln/index.rst | 1 | ||||
| -rw-r--r-- | Documentation/admin-guide/hw-vuln/reg-file-data-sampling.rst | 104 | ||||
| -rw-r--r-- | Documentation/admin-guide/hw-vuln/spectre.rst | 8 | ||||
| -rw-r--r-- | Documentation/admin-guide/index.rst | 4 | ||||
| -rw-r--r-- | Documentation/admin-guide/kdump/kdump.rst | 7 | ||||
| -rw-r--r-- | Documentation/admin-guide/kernel-parameters.rst | 1 | ||||
| -rw-r--r-- | Documentation/admin-guide/kernel-parameters.txt | 554 | ||||
| -rw-r--r-- | Documentation/admin-guide/tainted-kernels.rst | 4 | ||||
| -rw-r--r-- | Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst | 1952 |
16 files changed, 2485 insertions, 303 deletions
diff --git a/Documentation/admin-guide/RAS/address-translation.rst b/Documentation/admin-guide/RAS/address-translation.rst new file mode 100644 index 000000000000..f0ca17b43cd3 --- /dev/null +++ b/Documentation/admin-guide/RAS/address-translation.rst @@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Address translation +=================== + +x86 AMD +------- + +Zen-based AMD systems include a Data Fabric that manages the layout of +physical memory. Devices attached to the Fabric, like memory controllers, +I/O, etc., may not have a complete view of the system physical memory map. +These devices may provide a "normalized", i.e. device physical, address +when reporting memory errors. Normalized addresses must be translated to +a system physical address for the kernel to action on the memory. + +AMD Address Translation Library (CONFIG_AMD_ATL) provides translation for +this case. + +Glossary of acronyms used in address translation for Zen-based systems + +* CCM = Cache Coherent Moderator +* COD = Cluster-on-Die +* COH_ST = Coherent Station +* DF = Data Fabric diff --git a/Documentation/admin-guide/RAS/error-decoding.rst b/Documentation/admin-guide/RAS/error-decoding.rst new file mode 100644 index 000000000000..26a72f3fe5de --- /dev/null +++ b/Documentation/admin-guide/RAS/error-decoding.rst @@ -0,0 +1,21 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Error decoding +============== + +x86 +--- + +Error decoding on AMD systems should be done using the rasdaemon tool: +https://github.com/mchehab/rasdaemon/ + +While the daemon is running, it would automatically log and decode +errors. If not, one can still decode such errors by supplying the +hardware information from the error:: + + $ rasdaemon -p --status <STATUS> --ipid <IPID> --smca + +Also, the user can pass particular family and model to decode the error +string:: + + $ rasdaemon -p --status <STATUS> --ipid <IPID> --smca --family <CPU Family> --model <CPU Model> --bank <BANK_NUM> diff --git a/Documentation/admin-guide/RAS/index.rst b/Documentation/admin-guide/RAS/index.rst new file mode 100644 index 000000000000..f4087040a7c0 --- /dev/null +++ b/Documentation/admin-guide/RAS/index.rst @@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. toctree:: + :maxdepth: 2 + + main + error-decoding + address-translation diff --git a/Documentation/admin-guide/ras.rst b/Documentation/admin-guide/RAS/main.rst index 8e03751d126d..7ac1d4ccc509 100644 --- a/Documentation/admin-guide/ras.rst +++ b/Documentation/admin-guide/RAS/main.rst @@ -1,8 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 .. include:: <isonum.txt> -============================================ -Reliability, Availability and Serviceability -============================================ +================================================== +Reliability, Availability and Serviceability (RAS) +================================================== + +This documents different aspects of the RAS functionality present in the +kernel. RAS concepts ************ diff --git a/Documentation/admin-guide/README.rst b/Documentation/admin-guide/README.rst index 9a969c0157f1..f2bebff6a733 100644 --- a/Documentation/admin-guide/README.rst +++ b/Documentation/admin-guide/README.rst @@ -262,9 +262,11 @@ Compiling the kernel - Make sure you have at least gcc 5.1 available. For more information, refer to :ref:`Documentation/process/changes.rst <changes>`. - - Do a ``make`` to create a compressed kernel image. It is also - possible to do ``make install`` if you have lilo installed to suit the - kernel makefiles, but you may want to check your particular lilo setup first. + - Do a ``make`` to create a compressed kernel image. It is also possible to do + ``make install`` if you have lilo installed or if your distribution has an + install script recognised by the kernel's installer. Most popular + distributions will have a recognized install script. You may want to + check your distribution's setup first. To do the actual install, you have to be root, but none of the normal build should require that. Don't take the name of root in vain. @@ -301,32 +303,51 @@ Compiling the kernel image (e.g. .../linux/arch/x86/boot/bzImage after compilation) to the place where your regular bootable kernel is found. - - Booting a kernel directly from a floppy without the assistance of a - bootloader such as LILO, is no longer supported. - - If you boot Linux from the hard drive, chances are you use LILO, which - uses the kernel image as specified in the file /etc/lilo.conf. The - kernel image file is usually /vmlinuz, /boot/vmlinuz, /bzImage or - /boot/bzImage. To use the new kernel, save a copy of the old image - and copy the new image over the old one. Then, you MUST RERUN LILO - to update the loading map! If you don't, you won't be able to boot - the new kernel image. - - Reinstalling LILO is usually a matter of running /sbin/lilo. - You may wish to edit /etc/lilo.conf to specify an entry for your - old kernel image (say, /vmlinux.old) in case the new one does not - work. See the LILO docs for more information. - - After reinstalling LILO, you should be all set. Shutdown the system, + - Booting a kernel directly from a storage device without the assistance + of a bootloader such as LILO or GRUB, is no longer supported in BIOS + (non-EFI systems). On UEFI/EFI systems, however, you can use EFISTUB + which allows the motherboard to boot directly to the kernel. + On modern workstations and desktops, it's generally recommended to use a + bootloader as difficulties can arise with multiple kernels and secure boot. + For more details on EFISTUB, + see "Documentation/admin-guide/efi-stub.rst". + + - It's important to note that as of 2016 LILO (LInux LOader) is no longer in + active development, though as it was extremely popular, it often comes up + in documentation. Popular alternatives include GRUB2, rEFInd, Syslinux, + systemd-boot, or EFISTUB. For various reasons, it's not recommended to use + software that's no longer in active development. + + - Chances are your distribution includes an install script and running + ``make install`` will be all that's needed. Should that not be the case + you'll have to identify your bootloader and reference its documentation or + configure your EFI. + +Legacy LILO Instructions +------------------------ + + + - If you use LILO the kernel images are specified in the file /etc/lilo.conf. + The kernel image file is usually /vmlinuz, /boot/vmlinuz, /bzImage or + /boot/bzImage. To use the new kernel, save a copy of the old image and copy + the new image over the old one. Then, you MUST RERUN LILO to update the + loading map! If you don't, you won't be able to boot the new kernel image. + + - Reinstalling LILO is usually a matter of running /sbin/lilo. You may wish + to edit /etc/lilo.conf to specify an entry for your old kernel image + (say, /vmlinux.old) in case the new one does not work. See the LILO docs + for more information. + + - After reinstalling LILO, you should be all set. Shutdown the system, reboot, and enjoy! - If you ever need to change the default root device, video mode, - etc. in the kernel image, use your bootloader's boot options - where appropriate. No need to recompile the kernel to change - these parameters. + - If you ever need to change the default root device, video mode, etc. in the + kernel image, use your bootloader's boot options where appropriate. No need + to recompile the kernel to change these parameters. - Reboot with the new kernel and enjoy. + If something goes wrong ----------------------- diff --git a/Documentation/admin-guide/cgroup-v1/cpusets.rst b/Documentation/admin-guide/cgroup-v1/cpusets.rst index ae646d621a8a..7d3415eea05d 100644 --- a/Documentation/admin-guide/cgroup-v1/cpusets.rst +++ b/Documentation/admin-guide/cgroup-v1/cpusets.rst @@ -179,7 +179,7 @@ files describing that cpuset: - cpuset.mem_hardwall flag: is memory allocation hardwalled - cpuset.memory_pressure: measure of how much paging pressure in cpuset - cpuset.memory_spread_page flag: if set, spread page cache evenly on allowed nodes - - cpuset.memory_spread_slab flag: if set, spread slab cache evenly on allowed nodes + - cpuset.memory_spread_slab flag: OBSOLETE. Doesn't have any function. - cpuset.sched_load_balance flag: if set, load balance within CPUs on that cpuset - cpuset.sched_relax_domain_level: the searching range when migrating tasks diff --git a/Documentation/admin-guide/cgroup-v1/hugetlb.rst b/Documentation/admin-guide/cgroup-v1/hugetlb.rst index 0fa724d82abb..493a8e386700 100644 --- a/Documentation/admin-guide/cgroup-v1/hugetlb.rst +++ b/Documentation/admin-guide/cgroup-v1/hugetlb.rst @@ -65,10 +65,12 @@ files include:: 1. Page fault accounting -hugetlb.<hugepagesize>.limit_in_bytes -hugetlb.<hugepagesize>.max_usage_in_bytes -hugetlb.<hugepagesize>.usage_in_bytes -hugetlb.<hugepagesize>.failcnt +:: + + hugetlb.<hugepagesize>.limit_in_bytes + hugetlb.<hugepagesize>.max_usage_in_bytes + hugetlb.<hugepagesize>.usage_in_bytes + hugetlb.<hugepagesize>.failcnt The HugeTLB controller allows users to limit the HugeTLB usage (page fault) per control group and enforces the limit during page fault. Since HugeTLB @@ -82,10 +84,12 @@ getting SIGBUS. 2. Reservation accounting -hugetlb.<hugepagesize>.rsvd.limit_in_bytes -hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes -hugetlb.<hugepagesize>.rsvd.usage_in_bytes -hugetlb.<hugepagesize>.rsvd.failcnt +:: + + hugetlb.<hugepagesize>.rsvd.limit_in_bytes + hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes + hugetlb.<hugepagesize>.rsvd.usage_in_bytes + hugetlb.<hugepagesize>.rsvd.failcnt The HugeTLB controller allows to limit the HugeTLB reservations per control group and enforces the controller limit at reservation time and at the fault of diff --git a/Documentation/admin-guide/hw-vuln/index.rst b/Documentation/admin-guide/hw-vuln/index.rst index de99caabf65a..ff0b440ef2dc 100644 --- a/Documentation/admin-guide/hw-vuln/index.rst +++ b/Documentation/admin-guide/hw-vuln/index.rst @@ -21,3 +21,4 @@ are configurable at compile, boot or run time. cross-thread-rsb srso gather_data_sampling + reg-file-data-sampling diff --git a/Documentation/admin-guide/hw-vuln/reg-file-data-sampling.rst b/Documentation/admin-guide/hw-vuln/reg-file-data-sampling.rst new file mode 100644 index 000000000000..0585d02b9a6c --- /dev/null +++ b/Documentation/admin-guide/hw-vuln/reg-file-data-sampling.rst @@ -0,0 +1,104 @@ +================================== +Register File Data Sampling (RFDS) +================================== + +Register File Data Sampling (RFDS) is a microarchitectural vulnerability that +only affects Intel Atom parts(also branded as E-cores). RFDS may allow +a malicious actor to infer data values previously used in floating point +registers, vector registers, or integer registers. RFDS does not provide the +ability to choose which data is inferred. CVE-2023-28746 is assigned to RFDS. + +Affected Processors +=================== +Below is the list of affected Intel processors [#f1]_: + + =================== ============ + Common name Family_Model + =================== ============ + ATOM_GOLDMONT 06_5CH + ATOM_GOLDMONT_D 06_5FH + ATOM_GOLDMONT_PLUS 06_7AH + ATOM_TREMONT_D 06_86H + ATOM_TREMONT 06_96H + ALDERLAKE 06_97H + ALDERLAKE_L 06_9AH + ATOM_TREMONT_L 06_9CH + RAPTORLAKE 06_B7H + RAPTORLAKE_P 06_BAH + ATOM_GRACEMONT 06_BEH + RAPTORLAKE_S 06_BFH + =================== ============ + +As an exception to this table, Intel Xeon E family parts ALDERLAKE(06_97H) and +RAPTORLAKE(06_B7H) codenamed Catlow are not affected. They are reported as +vulnerable in Linux because they share the same family/model with an affected +part. Unlike their affected counterparts, they do not enumerate RFDS_CLEAR or +CPUID.HYBRID. This information could be used to distinguish between the +affected and unaffected parts, but it is deemed not worth adding complexity as +the reporting is fixed automatically when these parts enumerate RFDS_NO. + +Mitigation +========== +Intel released a microcode update that enables software to clear sensitive +information using the VERW instruction. Like MDS, RFDS deploys the same +mitigation strategy to force the CPU to clear the affected buffers before an +attacker can extract the secrets. This is achieved by using the otherwise +unused and obsolete VERW instruction in combination with a microcode update. +The microcode clears the affected CPU buffers when the VERW instruction is +executed. + +Mitigation points +----------------- +VERW is executed by the kernel before returning to user space, and by KVM +before VMentry. None of the affected cores support SMT, so VERW is not required +at C-state transitions. + +New bits in IA32_ARCH_CAPABILITIES +---------------------------------- +Newer processors and microcode update on existing affected processors added new +bits to IA32_ARCH_CAPABILITIES MSR. These bits can be used to enumerate +vulnerability and mitigation capability: + +- Bit 27 - RFDS_NO - When set, processor is not affected by RFDS. +- Bit 28 - RFDS_CLEAR - When set, processor is affected by RFDS, and has the + microcode that clears the affected buffers on VERW execution. + +Mitigation control on the kernel command line +--------------------------------------------- +The kernel command line allows to control RFDS mitigation at boot time with the +parameter "reg_file_data_sampling=". The valid arguments are: + + ========== ================================================================= + on If the CPU is vulnerable, enable mitigation; CPU buffer clearing + on exit to userspace and before entering a VM. + off Disables mitigation. + ========== ================================================================= + +Mitigation default is selected by CONFIG_MITIGATION_RFDS. + +Mitigation status information +----------------------------- +The Linux kernel provides a sysfs interface to enumerate the current +vulnerability status of the system: whether the system is vulnerable, and +which mitigations are active. The relevant sysfs file is: + + /sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling + +The possible values in this file are: + + .. list-table:: + + * - 'Not affected' + - The processor is not vulnerable + * - 'Vulnerable' + - The processor is vulnerable, but no mitigation enabled + * - 'Vulnerable: No microcode' + - The processor is vulnerable but microcode is not updated. + * - 'Mitigation: Clear Register File' + - The processor is vulnerable and the CPU buffer clearing mitigation is + enabled. + +References +---------- +.. [#f1] Affected Processors + https://www.intel.com/content/www/us/en/developer/topic-technology/software-security-guidance/processors-affected-consolidated-product-cpu-model.html diff --git a/Documentation/admin-guide/hw-vuln/spectre.rst b/Documentation/admin-guide/hw-vuln/spectre.rst index 32a8893e5617..cce768afec6b 100644 --- a/Documentation/admin-guide/hw-vuln/spectre.rst +++ b/Documentation/admin-guide/hw-vuln/spectre.rst @@ -473,8 +473,8 @@ Spectre variant 2 -mindirect-branch=thunk-extern -mindirect-branch-register options. If the kernel is compiled with a Clang compiler, the compiler needs to support -mretpoline-external-thunk option. The kernel config - CONFIG_RETPOLINE needs to be turned on, and the CPU needs to run with - the latest updated microcode. + CONFIG_MITIGATION_RETPOLINE needs to be turned on, and the CPU needs + to run with the latest updated microcode. On Intel Skylake-era systems the mitigation covers most, but not all, cases. See :ref:`[3] <spec_ref3>` for more details. @@ -609,8 +609,8 @@ kernel command line. Selecting 'on' will, and 'auto' may, choose a mitigation method at run time according to the CPU, the available microcode, the setting of the - CONFIG_RETPOLINE configuration option, and the - compiler with which the kernel was built. + CONFIG_MITIGATION_RETPOLINE configuration option, + and the compiler with which the kernel was built. Selecting 'on' will also enable the mitigation against user space to user space task attacks. diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst index fb40a1f6f79e..32ea52f1d150 100644 --- a/Documentation/admin-guide/index.rst +++ b/Documentation/admin-guide/index.rst @@ -1,3 +1,4 @@ +================================================= The Linux kernel user's and administrator's guide ================================================= @@ -37,6 +38,7 @@ problems and bugs in particular. reporting-issues reporting-regressions quickly-build-trimmed-linux + verify-bugs-and-bisect-regressions bug-hunting bug-bisect tainted-kernels @@ -122,7 +124,7 @@ configure specific aspects of kernel behavior to your liking. pmf pnp rapidio - ras + RAS/index rtc serial-console svga diff --git a/Documentation/admin-guide/kdump/kdump.rst b/Documentation/admin-guide/kdump/kdump.rst index 5762e7477a0c..0302a93b1d40 100644 --- a/Documentation/admin-guide/kdump/kdump.rst +++ b/Documentation/admin-guide/kdump/kdump.rst @@ -191,9 +191,7 @@ Dump-capture kernel config options (Arch Dependent, i386 and x86_64) CPU is enough for kdump kernel to dump vmcore on most of systems. However, you can also specify nr_cpus=X to enable multiple processors - in kdump kernel. In this case, "disable_cpu_apicid=" is needed to - tell kdump kernel which cpu is 1st kernel's BSP. Please refer to - admin-guide/kernel-parameters.txt for more details. + in kdump kernel. With CONFIG_SMP=n, the above things are not related. @@ -454,8 +452,7 @@ Notes on loading the dump-capture kernel: to use multi-thread programs with it, such as parallel dump feature of makedumpfile. Otherwise, the multi-thread program may have a great performance degradation. To enable multi-cpu support, you should bring up an - SMP dump-capture kernel and specify maxcpus/nr_cpus, disable_cpu_apicid=[X] - options while loading it. + SMP dump-capture kernel and specify maxcpus/nr_cpus options while loading it. * For s390x there are two kdump modes: If a ELF header is specified with the elfcorehdr= kernel parameter, it is used by the kdump kernel as it diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst index 4410384596a9..e8bdf5e86a9b 100644 --- a/Documentation/admin-guide/kernel-parameters.rst +++ b/Documentation/admin-guide/kernel-parameters.rst @@ -108,6 +108,7 @@ is applicable:: CMA Contiguous Memory Area support is enabled. DRM Direct Rendering Management support is enabled. DYNAMIC_DEBUG Build in debug messages and enable them at runtime + EARLY Parameter processed too early to be embedded in initrd. EDD BIOS Enhanced Disk Drive Services (EDD) is enabled EFI EFI Partitioning (GPT) is enabled EVM Extended Verification Module diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 31b3a25680d0..b3b5ce13de2b 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -9,7 +9,7 @@ accept_memory=eager can be used to accept all memory at once during boot. - acpi= [HW,ACPI,X86,ARM64,RISCV64] + acpi= [HW,ACPI,X86,ARM64,RISCV64,EARLY] Advanced Configuration and Power Interface Format: { force | on | off | strict | noirq | rsdt | copy_dsdt } @@ -26,7 +26,7 @@ See also Documentation/power/runtime_pm.rst, pci=noacpi - acpi_apic_instance= [ACPI, IOAPIC] + acpi_apic_instance= [ACPI,IOAPIC,EARLY] Format: <int> 2: use 2nd APIC table, if available 1,0: use 1st APIC table @@ -41,7 +41,7 @@ If set to native, use the device's native backlight mode. If set to none, disable the ACPI backlight interface. - acpi_force_32bit_fadt_addr + acpi_force_32bit_fadt_addr [ACPI,EARLY] force FADT to use 32 bit addresses rather than the 64 bit X_* addresses. Some firmware have broken 64 bit addresses for force ACPI ignore these and use @@ -97,7 +97,7 @@ no: ACPI OperationRegions are not marked as reserved, no further checks are performed. - acpi_force_table_verification [HW,ACPI] + acpi_force_table_verification [HW,ACPI,EARLY] Enable table checksum verification during early stage. By default, this is disabled due to x86 early mapping size limitation. @@ -137,7 +137,7 @@ acpi_no_memhotplug [ACPI] Disable memory hotplug. Useful for kdump kernels. - acpi_no_static_ssdt [HW,ACPI] + acpi_no_static_ssdt [HW,ACPI,EARLY] Disable installation of static SSDTs at early boot time By default, SSDTs contained in the RSDT/XSDT will be installed automatically and they will appear under @@ -151,7 +151,7 @@ Ignore the ACPI-based watchdog interface (WDAT) and let a native driver control the watchdog device instead. - acpi_rsdp= [ACPI,EFI,KEXEC] + acpi_rsdp= [ACPI,EFI,KEXEC,EARLY] Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump. @@ -228,10 +228,10 @@ to assume that this machine's pmtimer latches its value and always returns good values. - acpi_sci= [HW,ACPI] ACPI System Control Interrupt trigger mode + acpi_sci= [HW,ACPI,EARLY] ACPI System Control Interrupt trigger mode Format: { level | edge | high | low } - acpi_skip_timer_override [HW,ACPI] + acpi_skip_timer_override [HW,ACPI,EARLY] Recognize and ignore IRQ0/pin2 Interrupt Override. For broken nForce2 BIOS resulting in XT-PIC timer. @@ -266,11 +266,11 @@ behave incorrectly in some ways with respect to system suspend and resume to be ignored (use wisely). - acpi_use_timer_override [HW,ACPI] + acpi_use_timer_override [HW,ACPI,EARLY] Use timer override. For some broken Nvidia NF5 boards that require a timer override, but don't have HPET - add_efi_memmap [EFI; X86] Include EFI memory map in + add_efi_memmap [EFI,X86,EARLY] Include EFI memory map in kernel's map of available physical RAM. agp= [AGP] @@ -307,7 +307,7 @@ do not want to use tracing_snapshot_alloc() as it needs to be done where GFP_KERNEL allocations are allowed. - allow_mismatched_32bit_el0 [ARM64] + allow_mismatched_32bit_el0 [ARM64,EARLY] Allow execve() of 32-bit applications and setting of the PER_LINUX32 personality on systems where only a strict subset of the CPUs support 32-bit EL0. When this @@ -351,7 +351,7 @@ This mode requires kvm-amd.avic=1. (Default when IOMMU HW support is present.) - amd_pstate= [X86] + amd_pstate= [X86,EARLY] disable Do not enable amd_pstate as the default scaling driver for the supported processors @@ -391,7 +391,7 @@ not play well with APC CPU idle - disable it if you have APC and your system crashes randomly. - apic= [APIC,X86] Advanced Programmable Interrupt Controller + apic= [APIC,X86,EARLY] Advanced Programmable Interrupt Controller Change the output verbosity while booting Format: { quiet (default) | verbose | debug } Change the amount of debugging information output @@ -401,7 +401,7 @@ Format: apic=driver_name Examples: apic=bigsmp - apic_extnmi= [APIC,X86] External NMI delivery setting + apic_extnmi= [APIC,X86,EARLY] External NMI delivery setting Format: { bsp (default) | all | none } bsp: External NMI is delivered only to CPU 0 all: External NMIs are broadcast to all CPUs as a @@ -508,21 +508,22 @@ bert_disable [ACPI] Disable BERT OS support on buggy BIOSes. - bgrt_disable [ACPI][X86] + bgrt_disable [ACPI,X86,EARLY] Disable BGRT to avoid flickering OEM logo. blkdevparts= Manual partition parsing of block device(s) for embedded devices based on command line input. See Documentation/block/cmdline-partition.rst - boot_delay= Milliseconds to delay each printk during boot. + boot_delay= [KNL,EARLY] + Milliseconds to delay each printk during boot. Only works if CONFIG_BOOT_PRINTK_DELAY is enabled, and you may also have to specify "lpj=". Boot_delay values larger than 10 seconds (10000) are assumed erroneous and ignored. Format: integer - bootconfig [KNL] + bootconfig [KNL,EARLY] Extended command line options can be added to an initrd and this will cause the kernel to look for it. @@ -557,7 +558,7 @@ trust validation. format: { id:<keyid> | builtin } - cca= [MIPS] Override the kernel pages' cache coherency + cca= [MIPS,EARLY] Override the kernel pages' cache coherency algorithm. Accepted values range from 0 to 7 inclusive. See arch/mips/include/asm/pgtable-bits.h for platform specific values (SB1, Loongson3 and @@ -672,19 +673,13 @@ [X86-64] hpet,tsc clocksource.arm_arch_timer.evtstrm= - [ARM,ARM64] + [ARM,ARM64,EARLY] Format: <bool> Enable/disable the eventstream feature of the ARM architected timer so that code using WFE-based polling loops can be debugged more effectively on production systems. - clocksource.max_cswd_read_retries= [KNL] - Number of clocksource_watchdog() retries due to - external delays before the clock will be marked - unstable. Defaults to two retries, that is, - three attempts to read the clock under test. - clocksource.verify_n_cpus= [KNL] Limit the number of CPUs checked for clocksources marked with CLOCK_SOURCE_VERIFY_PERCPU that @@ -702,7 +697,7 @@ 10 seconds when built into the kernel. cma=nn[MG]@[start[MG][-end[MG]]] - [KNL,CMA] + [KNL,CMA,EARLY] Sets the size of kernel global memory area for contiguous memory allocations and optionally the placement constraint by the physical address range of @@ -711,7 +706,7 @@ kernel/dma/contiguous.c cma_pernuma=nn[MG] - [KNL,CMA] + [KNL,CMA,EARLY] Sets the size of kernel per-numa memory area for contiguous memory allocations. A value of 0 disables per-numa CMA altogether. And If this option is not @@ -722,7 +717,7 @@ they will fallback to the global default memory area. numa_cma=<node>:nn[MG][,<node>:nn[MG]] - [KNL,CMA] + [KNL,CMA,EARLY] Sets the size of kernel numa memory area for contiguous memory allocations. It will reserve CMA area for the specified node. @@ -739,7 +734,7 @@ a hypervisor. Default: yes - coherent_pool=nn[KMG] [ARM,KNL] + coherent_pool=nn[KMG] [ARM,KNL,EARLY] Sets the size of memory pool for coherent, atomic dma allocations, by default set to 256K. @@ -757,7 +752,7 @@ condev= [HW,S390] console device conmode= - con3215_drop= [S390] 3215 console drop mode. + con3215_drop= [S390,EARLY] 3215 console drop mode. Format: y|n|Y|N|1|0 When set to true, drop data on the 3215 console when the console buffer is full. In this case the @@ -863,7 +858,7 @@ kernel before the cpufreq driver probes. cpu_init_udelay=N - [X86] Delay for N microsec between assert and de-assert + [X86,EARLY] Delay for N microsec between assert and de-assert of APIC INIT to start processors. This delay occurs on every CPU online, such as boot, and resume from suspend. Default: 10000 @@ -883,7 +878,7 @@ kernel more unstable. crashkernel=size[KMG][@offset[KMG]] - [KNL] Using kexec, Linux can switch to a 'crash kernel' + [KNL,EARLY] Using kexec, Linux can switch to a 'crash kernel' upon panic. This parameter reserves the physical memory region [offset, offset + size] for that kernel image. If '@offset' is omitted, then a suitable offset @@ -954,10 +949,10 @@ Format: <port#>,<type> See also Documentation/input/devices/joystick-parport.rst - debug [KNL] Enable kernel debugging (events log level). + debug [KNL,EARLY] Enable kernel debugging (events log level). debug_boot_weak_hash - [KNL] Enable printing [hashed] pointers early in the + [KNL,EARLY] Enable printing [hashed] pointers early in the boot sequence. If enabled, we use a weak hash instead of siphash to hash pointers. Use this option if you are seeing instances of '(___ptrval___)') and need to see a @@ -974,10 +969,10 @@ will print _a_lot_ more information - normally only useful to lockdep developers. - debug_objects [KNL] Enable object debugging + debug_objects [KNL,EARLY] Enable object debugging debug_guardpage_minorder= - [KNL] When CONFIG_DEBUG_PAGEALLOC is set, this + [KNL,EARLY] When CONFIG_DEBUG_PAGEALLOC is set, this parameter allows control of the order of pages that will be intentionally kept free (and hence protected) by the buddy allocator. Bigger value increase the probability @@ -996,7 +991,7 @@ help tracking down these problems. debug_pagealloc= - [KNL] When CONFIG_DEBUG_PAGEALLOC is set, this parameter + [KNL,EARLY] When CONFIG_DEBUG_PAGEALLOC is set, this parameter enables the feature at boot time. By default, it is disabled and the system will work mostly the same as a kernel built without CONFIG_DEBUG_PAGEALLOC. @@ -1004,8 +999,8 @@ useful to also enable the page_owner functionality. on: enable the feature - debugfs= [KNL] This parameter enables what is exposed to userspace - and debugfs internal clients. + debugfs= [KNL,EARLY] This parameter enables what is exposed to + userspace and debugfs internal clients. Format: { on, no-mount, off } on: All functions are enabled. no-mount: @@ -1084,7 +1079,7 @@ dhash_entries= [KNL] Set number of hash buckets for dentry cache. - disable_1tb_segments [PPC] + disable_1tb_segments [PPC,EARLY] Disables the use of 1TB hash page table segments. This causes the kernel to fall back to 256MB segments which can be useful when debugging issues that require an SLB @@ -1093,41 +1088,32 @@ disable= [IPV6] See Documentation/networking/ipv6.rst. - disable_radix [PPC] + disable_radix [PPC,EARLY] Disable RADIX MMU mode on POWER9 disable_tlbie [PPC] Disable TLBIE instruction. Currently does not work with KVM, with HASH MMU, or with coherent accelerators. - disable_cpu_apicid= [X86,APIC,SMP] - Format: <int> - The number of initial APIC ID for the - corresponding CPU to be disabled at boot, - mostly used for the kdump 2nd kernel to - disable BSP to wake up multiple CPUs without - causing system reset or hang due to sending - INIT from AP to BSP. - - disable_ddw [PPC/PSERIES] + disable_ddw [PPC/PSERIES,EARLY] Disable Dynamic DMA Window support. Use this to workaround buggy firmware. disable_ipv6= [IPV6] See Documentation/networking/ipv6.rst. - disable_mtrr_cleanup [X86] + disable_mtrr_cleanup [X86,EARLY] The kernel tries to adjust MTRR layout from continuous to discrete, to make X server driver able to add WB entry later. This parameter disables that. - disable_mtrr_trim [X86, Intel and AMD only] + disable_mtrr_trim [X86, Intel and AMD only,EARLY] By default the kernel will trim any uncacheable memory out of your available memory pool based on MTRR settings. This parameter disables that behavior, possibly causing your machine to run very slowly. - disable_timer_pin_1 [X86] + disable_timer_pin_1 [X86,EARLY] Disable PIN 1 of APIC timer Can be useful to work around chipset bugs. @@ -1150,6 +1136,26 @@ The filter can be disabled or changed to another driver later using sysfs. + reg_file_data_sampling= + [X86] Controls mitigation for Register File Data + Sampling (RFDS) vulnerability. RFDS is a CPU + vulnerability which may allow userspace to infer + kernel data values previously stored in floating point + registers, vector registers, or integer registers. + RFDS only affects Intel Atom processors. + + on: Turns ON the mitigation. + off: Turns OFF the mitigation. + + This parameter overrides the compile time default set + by CONFIG_MITIGATION_RFDS. Mitigation cannot be + disabled when other VERW based mitigations (like MDS) + are enabled. In order to disable RFDS mitigation all + VERW based mitigations need to be disabled. + + For details see: + Documentation/admin-guide/hw-vuln/reg-file-data-sampling.rst + driver_async_probe= [KNL] List of driver names to be probed asynchronously. * matches with all driver names. If * is specified, the @@ -1177,7 +1183,7 @@ dscc4.setup= [NET] - dt_cpu_ftrs= [PPC] + dt_cpu_ftrs= [PPC,EARLY] Format: {"off" | "known"} Control how the dt_cpu_ftrs device-tree binding is used for CPU feature discovery and setup (if it @@ -1197,12 +1203,12 @@ Documentation/admin-guide/dynamic-debug-howto.rst for details. - early_ioremap_debug [KNL] + early_ioremap_debug [KNL,EARLY] Enable debug messages in early_ioremap support. This is useful for tracking down temporary early mappings which are not unmapped. - earlycon= [KNL] Output early console device and options. + earlycon= [KNL,EARLY] Output early console device and options. When used with no options, the early console is determined by stdout-path property in device tree's @@ -1338,7 +1344,7 @@ address must be provided, and the serial port must already be setup and configured. - earlyprintk= [X86,SH,ARM,M68k,S390] + earlyprintk= [X86,SH,ARM,M68k,S390,UM,EARLY] earlyprintk=vga earlyprintk=sclp earlyprintk=xen @@ -1396,7 +1402,7 @@ edd= [EDD] Format: {"off" | "on" | "skip[mbr]"} - efi= [EFI] + efi= [EFI,EARLY] Format: { "debug", "disable_early_pci_dma", "nochunk", "noruntime", "nosoftreserve", "novamap", "no_disable_early_pci_dma" } @@ -1417,13 +1423,13 @@ no_disable_early_pci_dma: Leave the busmaster bit set on all PCI bridges while in the EFI boot stub - efi_no_storage_paranoia [EFI; X86] + efi_no_storage_paranoia [EFI,X86,EARLY] Using this parameter you can use more than 50% of your efi variable storage. Use this parameter only if you are really sure that your UEFI does sane gc and fulfills the spec otherwise your board may brick. - efi_fake_mem= nn[KMG]@ss[KMG]:aa[,nn[KMG]@ss[KMG]:aa,..] [EFI; X86] + efi_fake_mem= nn[KMG]@ss[KMG]:aa[,nn[KMG]@ss[KMG]:aa,..] [EFI,X86,EARLY] Add arbitrary attribute to specific memory range by updating original EFI memory map. Region of memory which aa attribute is added to is @@ -1454,7 +1460,7 @@ eisa_irq_edge= [PARISC,HW] See header of drivers/parisc/eisa.c. - ekgdboc= [X86,KGDB] Allow early kernel console debugging + ekgdboc= [X86,KGDB,EARLY] Allow early kernel console debugging Format: ekgdboc=kbd This is designed to be used in conjunction with @@ -1469,13 +1475,13 @@ See comment before function elanfreq_setup() in arch/x86/kernel/cpu/cpufreq/elanfreq.c. - elfcorehdr=[size[KMG]@]offset[KMG] [PPC,SH,X86,S390] + elfcorehdr=[size[KMG]@]offset[KMG] [PPC,SH,X86,S390,EARLY] Specifies physical address of start of kernel core image elf header and optionally the size. Generally kexec loader will pass this option to capture kernel. See Documentation/admin-guide/kdump/kdump.rst for details. - enable_mtrr_cleanup [X86] + enable_mtrr_cleanup [X86,EARLY] The kernel tries to adjust MTRR layout from continuous to discrete, to make X server driver able to add WB entry later. This parameter enables that. @@ -1508,7 +1514,7 @@ Permit 'security.evm' to be updated regardless of current integrity status. - early_page_ext [KNL] Enforces page_ext initialization to earlier + early_page_ext [KNL,EARLY] Enforces page_ext initialization to earlier stages so cover more early boot allocations. Please note that as side effect some optimizations might be disabled to achieve that (e.g. parallelized @@ -1539,6 +1545,12 @@ Warning: use of this parameter will taint the kernel and may cause unknown problems. + fred= [X86-64] + Enable/disable Flexible Return and Event Delivery. + Format: { on | off } + on: enable FRED when it's present. + off: disable FRED, the default setting. + ftrace=[tracer] [FTRACE] will set and start the specified tracer as early as possible in order to facilitate early @@ -1600,7 +1612,7 @@ can be changed at run time by the max_graph_depth file in the tracefs tracing directory. default: 0 (no limit) - fw_devlink= [KNL] Create device links between consumer and supplier + fw_devlink= [KNL,EARLY] Create device links between consumer and supplier devices by scanning the firmware to infer the consumer/supplier relationships. This feature is especially useful when drivers are loaded as modules as @@ -1619,12 +1631,12 @@ rpm -- Like "on", but also use to order runtime PM. fw_devlink.strict=<bool> - [KNL] Treat all inferred dependencies as mandatory + [KNL,EARLY] Treat all inferred dependencies as mandatory dependencies. This only applies for fw_devlink=on|rpm. Format: <bool> fw_devlink.sync_state = - [KNL] When all devices that could probe have finished + [KNL,EARLY] When all devices that could probe have finished probing, this parameter controls what to do with devices that haven't yet received their sync_state() calls. @@ -1645,12 +1657,12 @@ gamma= [HW,DRM] - gart_fix_e820= [X86-64] disable the fix e820 for K8 GART + gart_fix_e820= [X86-64,EARLY] disable the fix e820 for K8 GART Format: off | on default: on gather_data_sampling= - [X86,INTEL] Control the Gather Data Sampling (GDS) + [X86,INTEL,EARLY] Control the Gather Data Sampling (GDS) mitigation. Gather Data Sampling is a hardware vulnerability which @@ -1748,7 +1760,7 @@ (that will set all pages holding image data during restoration read-only). - highmem=nn[KMG] [KNL,BOOT] forces the highmem zone to have an exact + highmem=nn[KMG] [KNL,BOOT,EARLY] forces the highmem zone to have an exact size of <nn>. This works even on boxes that have no highmem otherwise. This also works to reduce highmem size on bigger boxes. @@ -1759,7 +1771,7 @@ hlt [BUGS=ARM,SH] - hostname= [KNL] Set the hostname (aka UTS nodename). + hostname= [KNL,EARLY] Set the hostname (aka UTS nodename). Format: <string> This allows setting the system's hostname during early startup. This sets the name returned by gethostname. @@ -1804,7 +1816,7 @@ Documentation/admin-guide/mm/hugetlbpage.rst. Format: size[KMG] - hugetlb_cma= [HW,CMA] The size of a CMA area used for allocation + hugetlb_cma= [HW,CMA,EARLY] The size of a CMA area used for allocation of gigantic hugepages. Or using node format, the size of a CMA area per node can be specified. Format: nn[KMGTPE] or (node format) @@ -1850,9 +1862,10 @@ If specified, z/VM IUCV HVC accepts connections from listed z/VM user IDs only. - hv_nopvspin [X86,HYPER_V] Disables the paravirt spinlock optimizations - which allow the hypervisor to 'idle' the - guest on lock contention. + hv_nopvspin [X86,HYPER_V,EARLY] + Disables the paravirt spinlock optimizations + which allow the hypervisor to 'idle' the guest + on lock contention. i2c_bus= [HW] Override the default board specific I2C bus speed or register an additional I2C bus that is not @@ -1917,7 +1930,7 @@ Format: <io>[,<membase>[,<icn_id>[,<icn_id2>]]] - idle= [X86] + idle= [X86,EARLY] Format: idle=poll, idle=halt, idle=nomwait Poll forces a polling idle loop that can slightly improve the performance of waking up a idle CPU, but @@ -1973,7 +1986,7 @@ mode generally follows that for the NaN encoding, except where unsupported by hardware. - ignore_loglevel [KNL] + ignore_loglevel [KNL,EARLY] Ignore loglevel setting - this will print /all/ kernel messages to the console. Useful for debugging. We also add it as printk module parameter, so users @@ -2091,21 +2104,21 @@ unpacking being completed before device_ and late_ initcalls. - initrd= [BOOT] Specify the location of the initial ramdisk + initrd= [BOOT,EARLY] Specify the location of the initial ramdisk - initrdmem= [KNL] Specify a physical address and size from which to + initrdmem= [KNL,EARLY] Specify a physical address and size from which to load the initrd. If an initrd is compiled in or specified in the bootparams, it takes priority over this setting. Format: ss[KMG],nn[KMG] Default is 0, 0 - init_on_alloc= [MM] Fill newly allocated pages and heap objects with + init_on_alloc= [MM,EARLY] Fill newly allocated pages and heap objects with zeroes. Format: 0 | 1 Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. - init_on_free= [MM] Fill freed pages and heap objects with zeroes. + init_on_free= [MM,EARLY] Fill freed pages and heap objects with zeroes. Format: 0 | 1 Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. @@ -2161,7 +2174,7 @@ 0 disables intel_idle and fall back on acpi_idle. 1 to 9 specify maximum depth of C-state. - intel_pstate= [X86] + intel_pstate= [X86,EARLY] disable Do not enable intel_pstate as the default scaling driver for the supported processors @@ -2205,7 +2218,7 @@ Allow per-logical-CPU P-State performance control limits using cpufreq sysfs interface - intremap= [X86-64, Intel-IOMMU] + intremap= [X86-64,Intel-IOMMU,EARLY] on enable Interrupt Remapping (default) off disable Interrupt Remapping nosid disable Source ID checking @@ -2217,7 +2230,7 @@ strict regions from userspace. relaxed - iommu= [X86] + iommu= [X86,EARLY] off force noforce @@ -2232,7 +2245,7 @@ nobypass [PPC/POWERNV] Disable IOMMU bypass, using IOMMU for PCI devices. - iommu.forcedac= [ARM64, X86] Control IOVA allocation for PCI devices. + iommu.forcedac= [ARM64,X86,EARLY] Control IOVA allocation for PCI devices. Format: { "0" | "1" } 0 - Try to allocate a 32-bit DMA address first, before falling back to the full range if needed. @@ -2240,7 +2253,7 @@ forcing Dual Address Cycle for PCI cards supporting greater than 32-bit addressing. - iommu.strict= [ARM64, X86, S390] Configure TLB invalidation behaviour + iommu.strict= [ARM64,X86,S390,EARLY] Configure TLB invalidation behaviour Format: { "0" | "1" } 0 - Lazy mode. Request that DMA unmap operations use deferred @@ -2256,7 +2269,7 @@ legacy driver-specific options takes precedence. iommu.passthrough= - [ARM64, X86] Configure DMA to bypass the IOMMU by default. + [ARM64,X86,EARLY] Configure DMA to bypass the IOMMU by default. Format: { "0" | "1" } 0 - Use IOMMU translation for DMA. 1 - Bypass the IOMMU for DMA. @@ -2266,7 +2279,7 @@ See comment before marvel_specify_io7 in arch/alpha/kernel/core_marvel.c. - io_delay= [X86] I/O delay method + io_delay= [X86,EARLY] I/O delay method 0x80 Standard port 0x80 based delay 0xed @@ -2279,28 +2292,28 @@ ip= [IP_PNP] See Documentation/admin-guide/nfs/nfsroot.rst. - ipcmni_extend [KNL] Extend the maximum number of unique System V + ipcmni_extend [KNL,EARLY] Extend the maximum number of unique System V IPC identifiers from 32,768 to 16,777,216. irqaffinity= [SMP] Set the default irq affinity mask The argument is a cpu list, as described above. irqchip.gicv2_force_probe= - [ARM, ARM64] + [ARM,ARM64,EARLY] Format: <bool> Force the kernel to look for the second 4kB page of a GICv2 controller even if the memory range exposed by the device tree is too small. irqchip.gicv3_nolpi= - [ARM, ARM64] + [ARM,ARM64,EARLY] Force the kernel to ignore the availability of LPIs (and by consequence ITSs). Intended for system that use the kernel as a bootloader, and thus want to let secondary kernels in charge of setting up LPIs. - irqchip.gicv3_pseudo_nmi= [ARM64] + irqchip.gicv3_pseudo_nmi= [ARM64,EARLY] Enables support for pseudo-NMIs in the kernel. This requires the kernel to be built with CONFIG_ARM64_PSEUDO_NMI. @@ -2445,7 +2458,7 @@ parameter KASAN will print report only for the first invalid access. - keep_bootcon [KNL] + keep_bootcon [KNL,EARLY] Do not unregister boot console at start. This is only useful for debugging when something happens in the window between unregistering the boot console and initializing @@ -2453,7 +2466,7 @@ keepinitrd [HW,ARM] See retain_initrd. - kernelcore= [KNL,X86,IA-64,PPC] + kernelcore= [KNL,X86,IA-64,PPC,EARLY] Format: nn[KMGTPE] | nn% | "mirror" This parameter specifies the amount of memory usable by the kernel for non-movable allocations. The requested @@ -2478,7 +2491,7 @@ for Movable pages. "nn[KMGTPE]", "nn%", and "mirror" are exclusive, so you cannot specify multiple forms. - kgdbdbgp= [KGDB,HW] kgdb over EHCI usb debug port. + kgdbdbgp= [KGDB,HW,EARLY] kgdb over EHCI usb debug port. Format: <Controller#>[,poll interval] The controller # is the number of the ehci usb debug port as it is probed via PCI. The poll interval is @@ -2499,7 +2512,7 @@ kms, kbd format: kms,kbd kms, kbd and serial format: kms,kbd,<ser_dev>[,baud] - kgdboc_earlycon= [KGDB,HW] + kgdboc_earlycon= [KGDB,HW,EARLY] If the boot console provides the ability to read characters and can work in polling mode, you can use this parameter to tell kgdb to use it as a backend @@ -2514,14 +2527,14 @@ blank and the first boot console that implements read() will be picked. - kgdbwait [KGDB] Stop kernel execution and enter the + kgdbwait [KGDB,EARLY] Stop kernel execution and enter the kernel debugger at the earliest opportunity. kmac= [MIPS] Korina ethernet MAC address. Configure the RouterBoard 532 series on-chip Ethernet adapter MAC address. - kmemleak= [KNL] Boot-time kmemleak enable/disable + kmemleak= [KNL,EARLY] Boot-time kmemleak enable/disable Valid arguments: on, off Default: on Built with CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y, @@ -2540,8 +2553,8 @@ See also Documentation/trace/kprobetrace.rst "Kernel Boot Parameter" section. - kpti= [ARM64] Control page table isolation of user - and kernel address spaces. + kpti= [ARM64,EARLY] Control page table isolation of + user and kernel address spaces. Default: enabled on cores which need mitigation. 0: force disabled 1: force enabled @@ -2618,7 +2631,8 @@ for NPT. kvm-arm.mode= - [KVM,ARM] Select one of KVM/arm64's modes of operation. + [KVM,ARM,EARLY] Select one of KVM/arm64's modes of + operation. none: Forcefully disable KVM. @@ -2638,22 +2652,22 @@ used with extreme caution. kvm-arm.vgic_v3_group0_trap= - [KVM,ARM] Trap guest accesses to GICv3 group-0 + [KVM,ARM,EARLY] Trap guest accesses to GICv3 group-0 system registers kvm-arm.vgic_v3_group1_trap= - [KVM,ARM] Trap guest accesses to GICv3 group-1 + [KVM,ARM,EARLY] Trap guest accesses to GICv3 group-1 system registers kvm-arm.vgic_v3_common_trap= - [KVM,ARM] Trap guest accesses to GICv3 common + [KVM,ARM,EARLY] Trap guest accesses to GICv3 common system registers kvm-arm.vgic_v4_enable= - [KVM,ARM] Allow use of GICv4 for direct injection of - LPIs. + [KVM,ARM,EARLY] Allow use of GICv4 for direct + injection of LPIs. - kvm_cma_resv_ratio=n [PPC] + kvm_cma_resv_ratio=n [PPC,EARLY] Reserves given percentage from system memory area for contiguous memory allocation for KVM hash pagetable allocation. @@ -2706,7 +2720,7 @@ (enabled). Disable by KVM if hardware lacks support for it. - l1d_flush= [X86,INTEL] + l1d_flush= [X86,INTEL,EARLY] Control mitigation for L1D based snooping vulnerability. Certain CPUs are vulnerable to an exploit against CPU @@ -2723,7 +2737,7 @@ on - enable the interface for the mitigation - l1tf= [X86] Control mitigation of the L1TF vulnerability on + l1tf= [X86,EARLY] Control mitigation of the L1TF vulnerability on affected CPUs The kernel PTE inversion protection is unconditionally @@ -2792,7 +2806,7 @@ l3cr= [PPC] - lapic [X86-32,APIC] Enable the local APIC even if BIOS + lapic [X86-32,APIC,EARLY] Enable the local APIC even if BIOS disabled it. lapic= [X86,APIC] Do not use TSC deadline @@ -2800,7 +2814,7 @@ back to the programmable timer unit in the LAPIC. Format: notscdeadline - lapic_timer_c2_ok [X86,APIC] trust the local apic timer + lapic_timer_c2_ok [X86,APIC,EARLY] trust the local apic timer in C2 power state. libata.dma= [LIBATA] DMA control @@ -2924,7 +2938,7 @@ lockd.nlm_udpport=M [NFS] Assign UDP port. Format: <integer> - lockdown= [SECURITY] + lockdown= [SECURITY,EARLY] { integrity | confidentiality } Enable the kernel lockdown feature. If set to integrity, kernel features that allow userland to @@ -3031,7 +3045,8 @@ logibm.irq= [HW,MOUSE] Logitech Bus Mouse Driver Format: <irq> - loglevel= All Kernel Messages with a loglevel smaller than the + loglevel= [KNL,EARLY] + All Kernel Messages with a loglevel smaller than the console loglevel will be printed to the console. It can also be changed with klogd or other programs. The loglevels are defined as follows: @@ -3045,13 +3060,15 @@ 6 (KERN_INFO) informational 7 (KERN_DEBUG) debug-level messages - log_buf_len=n[KMG] Sets the size of the printk ring buffer, - in bytes. n must be a power of two and greater - than the minimal size. The minimal size is defined - by LOG_BUF_SHIFT kernel config parameter. There is - also CONFIG_LOG_CPU_MAX_BUF_SHIFT config parameter - that allows to increase the default size depending on - the number of CPUs. See init/Kconfig for more details. + log_buf_len=n[KMG] [KNL,EARLY] + Sets the size of the printk ring buffer, in bytes. + n must be a power of two and greater than the + minimal size. The minimal size is defined by + LOG_BUF_SHIFT kernel config parameter. There + is also CONFIG_LOG_CPU_MAX_BUF_SHIFT config + parameter that allows to increase the default size + depending on the number of CPUs. See init/Kconfig + for more details. logo.nologo [FB] Disables display of the built-in Linux logo. This may be used to provide more screen space for @@ -3109,7 +3126,7 @@ max_addr=nn[KMG] [KNL,BOOT,IA-64] All physical memory greater than or equal to this physical address is ignored. - maxcpus= [SMP] Maximum number of processors that an SMP kernel + maxcpus= [SMP,EARLY] Maximum number of processors that an SMP kernel will bring up during bootup. maxcpus=n : n >= 0 limits the kernel to bring up 'n' processors. Surely after bootup you can bring up the other plugged cpu by executing @@ -3136,7 +3153,7 @@ Format: <first>,<last> Specifies range of consoles to be captured by the MDA. - mds= [X86,INTEL] + mds= [X86,INTEL,EARLY] Control mitigation for the Micro-architectural Data Sampling (MDS) vulnerability. @@ -3168,11 +3185,12 @@ For details see: Documentation/admin-guide/hw-vuln/mds.rst - mem=nn[KMG] [HEXAGON] Set the memory size. + mem=nn[KMG] [HEXAGON,EARLY] Set the memory size. Must be specified, otherwise memory size will be 0. - mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory - Amount of memory to be used in cases as follows: + mem=nn[KMG] [KNL,BOOT,EARLY] Force usage of a specific amount + of memory Amount of memory to be used in cases + as follows: 1 for test; 2 when the kernel is not able to see the whole system memory; @@ -3196,8 +3214,8 @@ if system memory of hypervisor is not sufficient. mem=nn[KMG]@ss[KMG] - [ARM,MIPS] - override the memory layout reported by - firmware. + [ARM,MIPS,EARLY] - override the memory layout + reported by firmware. Define a memory region of size nn[KMG] starting at ss[KMG]. Multiple different regions can be specified with @@ -3206,7 +3224,7 @@ mem=nopentium [BUGS=X86-32] Disable usage of 4MB pages for kernel memory. - memblock=debug [KNL] Enable memblock debug messages. + memblock=debug [KNL,EARLY] Enable memblock debug messages. memchunk=nn[KMG] [KNL,SH] Allow user to override the default size for @@ -3220,14 +3238,14 @@ option. See Documentation/admin-guide/mm/memory-hotplug.rst. - memmap=exactmap [KNL,X86] Enable setting of an exact + memmap=exactmap [KNL,X86,EARLY] Enable setting of an exact E820 memory map, as specified by the user. Such memmap=exactmap lines can be constructed based on BIOS output or other requirements. See the memmap=nn@ss option description. memmap=nn[KMG]@ss[KMG] - [KNL, X86, MIPS, XTENSA] Force usage of a specific region of memory. + [KNL, X86,MIPS,XTENSA,EARLY] Force usage of a specific region of memory. Region of memory to be used is from ss to ss+nn. If @ss[KMG] is omitted, it is equivalent to mem=nn[KMG], which limits max address to nn[KMG]. @@ -3237,11 +3255,11 @@ memmap=100M@2G,100M#3G,1G!1024G memmap=nn[KMG]#ss[KMG] - [KNL,ACPI] Mark specific memory as ACPI data. + [KNL,ACPI,EARLY] Mark specific memory as ACPI data. Region of memory to be marked is from ss to ss+nn. memmap=nn[KMG]$ss[KMG] - [KNL,ACPI] Mark specific memory as reserved. + [KNL,ACPI,EARLY] Mark specific memory as reserved. Region of memory to be reserved is from ss to ss+nn. Example: Exclude memory from 0x18690000-0x1869ffff memmap=64K$0x18690000 @@ -3251,14 +3269,14 @@ like Grub2, otherwise '$' and the following number will be eaten. - memmap=nn[KMG]!ss[KMG] + memmap=nn[KMG]!ss[KMG,EARLY] [KNL,X86] Mark specific memory as protected. Region of memory to be used, from ss to ss+nn. The memory region may be marked as e820 type 12 (0xc) and is NVDIMM or ADR memory. memmap=<size>%<offset>-<oldtype>+<newtype> - [KNL,ACPI] Convert memory within the specified region + [KNL,ACPI,EARLY] Convert memory within the specified region from <oldtype> to <newtype>. If "-<oldtype>" is left out, the whole region will be marked as <newtype>, even if previously unavailable. If "+<newtype>" is left @@ -3266,7 +3284,7 @@ specified as e820 types, e.g., 1 = RAM, 2 = reserved, 3 = ACPI, 12 = PRAM. - memory_corruption_check=0/1 [X86] + memory_corruption_check=0/1 [X86,EARLY] Some BIOSes seem to corrupt the first 64k of memory when doing things like suspend/resume. Setting this option will scan the memory @@ -3278,13 +3296,13 @@ affects the same memory, you can use memmap= to prevent the kernel from using that memory. - memory_corruption_check_size=size [X86] + memory_corruption_check_size=size [X86,EARLY] By default it checks for corruption in the low 64k, making this memory unavailable for normal use. Use this parameter to scan for corruption in more or less memory. - memory_corruption_check_period=seconds [X86] + memory_corruption_check_period=seconds [X86,EARLY] By default it checks for corruption every 60 seconds. Use this parameter to check at some other rate. 0 disables periodic checking. @@ -3308,7 +3326,7 @@ Note that even when enabled, there are a few cases where the feature is not effective. - memtest= [KNL,X86,ARM,M68K,PPC,RISCV] Enable memtest + memtest= [KNL,X86,ARM,M68K,PPC,RISCV,EARLY] Enable memtest Format: <integer> default : 0 <disable> Specifies the number of memtest passes to be @@ -3320,9 +3338,7 @@ mem_encrypt= [X86-64] AMD Secure Memory Encryption (SME) control Valid arguments: on, off - Default (depends on kernel configuration option): - on (CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y) - off (CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=n) + Default: off mem_encrypt=on: Activate SME mem_encrypt=off: Do not activate SME @@ -3376,7 +3392,7 @@ https://repo.or.cz/w/linux-2.6/mini2440.git mitigations= - [X86,PPC,S390,ARM64] Control optional mitigations for + [X86,PPC,S390,ARM64,EARLY] Control optional mitigations for CPU vulnerabilities. This is a set of curated, arch-independent options, each of which is an aggregation of existing arch-specific options. @@ -3398,7 +3414,9 @@ nospectre_bhb [ARM64] nospectre_v1 [X86,PPC] nospectre_v2 [X86,PPC,S390,ARM64] + reg_file_data_sampling=off [X86] retbleed=off [X86] + spec_rstack_overflow=off [X86] spec_store_bypass_disable=off [X86,PPC] spectre_v2_user=off [X86] srbds=off [X86,INTEL] @@ -3429,7 +3447,7 @@ retbleed=auto,nosmt [X86] mminit_loglevel= - [KNL] When CONFIG_DEBUG_MEMORY_INIT is set, this + [KNL,EARLY] When CONFIG_DEBUG_MEMORY_INIT is set, this parameter allows control of the logging verbosity for the additional memory initialisation checks. A value of 0 disables mminit logging and a level of 4 will @@ -3437,7 +3455,7 @@ so loglevel=8 may also need to be specified. mmio_stale_data= - [X86,INTEL] Control mitigation for the Processor + [X86,INTEL,EARLY] Control mitigation for the Processor MMIO Stale Data vulnerabilities. Processor MMIO Stale Data is a class of @@ -3512,7 +3530,7 @@ mousedev.yres= [MOUSE] Vertical screen resolution, used for devices reporting absolute coordinates, such as tablets - movablecore= [KNL,X86,IA-64,PPC] + movablecore= [KNL,X86,IA-64,PPC,EARLY] Format: nn[KMGTPE] | nn% This parameter is the complement to kernelcore=, it specifies the amount of memory used for migratable @@ -3523,7 +3541,7 @@ that the amount of memory usable for all allocations is not too small. - movable_node [KNL] Boot-time switch to make hotplugable memory + movable_node [KNL,EARLY] Boot-time switch to make hotplugable memory NUMA nodes to be movable. This means that the memory of such nodes will be usable only for movable allocations which rules out almost all kernel @@ -3547,21 +3565,21 @@ [HW] Make the MicroTouch USB driver use raw coordinates ('y', default) or cooked coordinates ('n') - mtrr=debug [X86] + mtrr=debug [X86,EARLY] Enable printing debug information related to MTRR registers at boot time. - mtrr_chunk_size=nn[KMG] [X86] + mtrr_chunk_size=nn[KMG,X86,EARLY] used for mtrr cleanup. It is largest continuous chunk that could hold holes aka. UC entries. - mtrr_gran_size=nn[KMG] [X86] + mtrr_gran_size=nn[KMG,X86,EARLY] Used for mtrr cleanup. It is granularity of mtrr block. Default is 1. Large value could prevent small alignment from using up MTRRs. - mtrr_spare_reg_nr=n [X86] + mtrr_spare_reg_nr=n [X86,EARLY] Format: <integer> Range: 0,7 : spare reg number Default : 1 @@ -3747,10 +3765,10 @@ emulation library even if a 387 maths coprocessor is present. - no4lvl [RISCV] Disable 4-level and 5-level paging modes. Forces - kernel to use 3-level paging instead. + no4lvl [RISCV,EARLY] Disable 4-level and 5-level paging modes. + Forces kernel to use 3-level paging instead. - no5lvl [X86-64,RISCV] Disable 5-level paging mode. Forces + no5lvl [X86-64,RISCV,EARLY] Disable 5-level paging mode. Forces kernel to use 4-level paging instead. noaliencache [MM, NUMA, SLAB] Disables the allocation of alien @@ -3759,15 +3777,15 @@ noalign [KNL,ARM] - noaltinstr [S390] Disables alternative instructions patching - (CPU alternatives feature). + noaltinstr [S390,EARLY] Disables alternative instructions + patching (CPU alternatives feature). - noapic [SMP,APIC] Tells the kernel to not make use of any + noapic [SMP,APIC,EARLY] Tells the kernel to not make use of any IOAPICs that may be present in the system. noautogroup Disable scheduler automatic task group creation. - nocache [ARM] + nocache [ARM,EARLY] no_console_suspend [HW] Never suspend the console @@ -3785,13 +3803,13 @@ turn on/off it dynamically. no_debug_objects - [KNL] Disable object debugging + [KNL,EARLY] Disable object debugging nodsp [SH] Disable hardware DSP at boot time. - noefi Disable EFI runtime services support. + noefi [EFI,EARLY] Disable EFI runtime services support. - no_entry_flush [PPC] Don't flush the L1-D cache when entering the kernel. + no_entry_flush [PPC,EARLY] Don't flush the L1-D cache when entering the kernel. noexec [IA-64] @@ -3822,6 +3840,7 @@ real-time systems. no_hash_pointers + [KNL,EARLY] Force pointers printed to the console or buffers to be unhashed. By default, when a pointer is printed via %p format string, that pointer is "hashed", i.e. obscured @@ -3846,9 +3865,9 @@ the impact of the sleep instructions. This is also useful when using JTAG debugger. - nohugeiomap [KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings. + nohugeiomap [KNL,X86,PPC,ARM64,EARLY] Disable kernel huge I/O mappings. - nohugevmalloc [KNL,X86,PPC,ARM64] Disable kernel huge vmalloc mappings. + nohugevmalloc [KNL,X86,PPC,ARM64,EARLY] Disable kernel huge vmalloc mappings. nohz= [KNL] Boottime enable/disable dynamic ticks Valid arguments: on, off @@ -3870,13 +3889,13 @@ noinitrd [RAM] Tells the kernel not to load any configured initial RAM disk. - nointremap [X86-64, Intel-IOMMU] Do not enable interrupt + nointremap [X86-64,Intel-IOMMU,EARLY] Do not enable interrupt remapping. [Deprecated - use intremap=off] nointroute [IA-64] - noinvpcid [X86] Disable the INVPCID cpu feature. + noinvpcid [X86,EARLY] Disable the INVPCID cpu feature. noiotrap [SH] Disables trapped I/O port accesses. @@ -3887,19 +3906,19 @@ nojitter [IA-64] Disables jitter checking for ITC timers. - nokaslr [KNL] + nokaslr [KNL,EARLY] When CONFIG_RANDOMIZE_BASE is set, this disables kernel and module base offset ASLR (Address Space Layout Randomization). - no-kvmapf [X86,KVM] Disable paravirtualized asynchronous page + no-kvmapf [X86,KVM,EARLY] Disable paravirtualized asynchronous page fault handling. - no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver + no-kvmclock [X86,KVM,EARLY] Disable paravirtualized KVM clock driver - nolapic [X86-32,APIC] Do not enable or use the local APIC. + nolapic [X86-32,APIC,EARLY] Do not enable or use the local APIC. - nolapic_timer [X86-32,APIC] Do not use the local APIC timer. + nolapic_timer [X86-32,APIC,EARLY] Do not use the local APIC timer. nomca [IA-64] Disable machine check abort handling @@ -3924,23 +3943,23 @@ shutdown the other cpus. Instead use the REBOOT_VECTOR irq. - nopat [X86] Disable PAT (page attribute table extension of + nopat [X86,EARLY] Disable PAT (page attribute table extension of pagetables) support. - nopcid [X86-64] Disable the PCID cpu feature. + nopcid [X86-64,EARLY] Disable the PCID cpu feature. nopku [X86] Disable Memory Protection Keys CPU feature found in some Intel CPUs. - nopti [X86-64] + nopti [X86-64,EARLY] Equivalent to pti=off - nopv= [X86,XEN,KVM,HYPER_V,VMWARE] + nopv= [X86,XEN,KVM,HYPER_V,VMWARE,EARLY] Disables the PV optimizations forcing the guest to run as generic guest with no PV drivers. Currently support XEN HVM, KVM, HYPER_V and VMWARE guest. - nopvspin [X86,XEN,KVM] + nopvspin [X86,XEN,KVM,EARLY] Disables the qspinlock slow path using PV optimizations which allow the hypervisor to 'idle' the guest on lock contention. @@ -3960,20 +3979,20 @@ This is required for the Braillex ib80-piezo Braille reader made by F.H. Papenmeier (Germany). - nosgx [X86-64,SGX] Disables Intel SGX kernel support. + nosgx [X86-64,SGX,EARLY] Disables Intel SGX kernel support. - nosmap [PPC] + nosmap [PPC,EARLY] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. - nosmep [PPC64s] + nosmep [PPC64s,EARLY] Disable SMEP (Supervisor Mode Execution Prevention) even if it is supported by processor. - nosmp [SMP] Tells an SMP kernel to act as a UP kernel, + nosmp [SMP,EARLY] Tells an SMP kernel to act as a UP kernel, and disable the IO APIC. legacy for "maxcpus=0". - nosmt [KNL,MIPS,PPC,S390] Disable symmetric multithreading (SMT). + nosmt [KNL,MIPS,PPC,S390,EARLY] Disable symmetric multithreading (SMT). Equivalent to smt=1. [KNL,X86,PPC] Disable symmetric multithreading (SMT). @@ -3983,22 +4002,23 @@ nosoftlockup [KNL] Disable the soft-lockup detector. nospec_store_bypass_disable - [HW] Disable all mitigations for the Speculative Store Bypass vulnerability + [HW,EARLY] Disable all mitigations for the Speculative + Store Bypass vulnerability - nospectre_bhb [ARM64] Disable all mitigations for Spectre-BHB (branch + nospectre_bhb [ARM64,EARLY] Disable all mitigations for Spectre-BHB (branch history injection) vulnerability. System may allow data leaks with this option. - nospectre_v1 [X86,PPC] Disable mitigations for Spectre Variant 1 + nospectre_v1 [X86,PPC,EARLY] Disable mitigations for Spectre Variant 1 (bounds check bypass). With this option data leaks are possible in the system. - nospectre_v2 [X86,PPC_E500,ARM64] Disable all mitigations for - the Spectre variant 2 (indirect branch prediction) - vulnerability. System may allow data leaks with this - option. + nospectre_v2 [X86,PPC_E500,ARM64,EARLY] Disable all mitigations + for the Spectre variant 2 (indirect branch + prediction) vulnerability. System may allow data + leaks with this option. - no-steal-acc [X86,PV_OPS,ARM64,PPC/PSERIES,RISCV] Disable + no-steal-acc [X86,PV_OPS,ARM64,PPC/PSERIES,RISCV,EARLY] Disable paravirtualized steal time accounting. steal time is computed, but won't influence scheduler behaviour @@ -4008,7 +4028,7 @@ broken timer IRQ sources. no_uaccess_flush - [PPC] Don't flush the L1-D cache after accessing user data. + [PPC,EARLY] Don't flush the L1-D cache after accessing user data. novmcoredd [KNL,KDUMP] Disable device dump. Device dump allows drivers to @@ -4022,15 +4042,15 @@ is set. no-vmw-sched-clock - [X86,PV_OPS] Disable paravirtualized VMware scheduler - clock and use the default one. + [X86,PV_OPS,EARLY] Disable paravirtualized VMware + scheduler clock and use the default one. nowatchdog [KNL] Disable both lockup detectors, i.e. soft-lockup and NMI watchdog (hard-lockup). - nowb [ARM] + nowb [ARM,EARLY] - nox2apic [X86-64,APIC] Do not enable x2APIC mode. + nox2apic [X86-64,APIC,EARLY] Do not enable x2APIC mode. NOTE: this parameter will be ignored on systems with the LEGACY_XAPIC_DISABLED bit set in the @@ -4068,7 +4088,7 @@ purges which is reported from either PAL_VM_SUMMARY or SAL PALO. - nr_cpus= [SMP] Maximum number of processors that an SMP kernel + nr_cpus= [SMP,EARLY] Maximum number of processors that an SMP kernel could support. nr_cpus=n : n >= 1 limits the kernel to support 'n' processors. It could be larger than the number of already plugged CPU during bootup, later in @@ -4079,8 +4099,9 @@ nr_uarts= [SERIAL] maximum number of UARTs to be registered. - numa=off [KNL, ARM64, PPC, RISCV, SPARC, X86] Disable NUMA, Only - set up a single NUMA node spanning all memory. + numa=off [KNL, ARM64, PPC, RISCV, SPARC, X86, EARLY] + Disable NUMA, Only set up a single NUMA node + spanning all memory. numa_balancing= [KNL,ARM64,PPC,RISCV,S390,X86] Enable or disable automatic NUMA balancing. @@ -4091,7 +4112,7 @@ This can be set from sysctl after boot. See Documentation/admin-guide/sysctl/vm.rst for details. - ohci1394_dma=early [HW] enable debugging via the ohci1394 driver. + ohci1394_dma=early [HW,EARLY] enable debugging via the ohci1394 driver. See Documentation/core-api/debugging-via-ohci1394.rst for more info. @@ -4117,7 +4138,8 @@ Once locked, the boundary cannot be changed. 1 indicates lock status, 0 indicates unlock status. - oops=panic Always panic on oopses. Default is to just kill the + oops=panic [KNL,EARLY] + Always panic on oopses. Default is to just kill the process, but there is a small probability of deadlocking the machine. This will also cause panics on machine check exceptions. @@ -4133,13 +4155,13 @@ can be read from sysfs at: /sys/module/page_alloc/parameters/shuffle. - page_owner= [KNL] Boot-time page_owner enabling option. + page_owner= [KNL,EARLY] Boot-time page_owner enabling option. Storage of the information about who allocated each page is disabled in default. With this switch, we can turn it on. on: enable the feature - page_poison= [KNL] Boot-time parameter changing the state of + page_poison= [KNL,EARLY] Boot-time parameter changing the state of poisoning on the buddy allocator, available with CONFIG_PAGE_POISONING=y. off: turn off poisoning (default) @@ -4157,7 +4179,8 @@ timeout < 0: reboot immediately Format: <timeout> - panic_on_taint= Bitmask for conditionally calling panic() in add_taint() + panic_on_taint= [KNL,EARLY] + Bitmask for conditionally calling panic() in add_taint() Format: <hex>[,nousertaint] Hexadecimal bitmask representing the set of TAINT flags that will cause the kernel to panic when add_taint() is @@ -4313,7 +4336,7 @@ pcbit= [HW,ISDN] - pci=option[,option...] [PCI] various PCI subsystem options. + pci=option[,option...] [PCI,EARLY] various PCI subsystem options. Some options herein operate on a specific device or a set of devices (<pci_dev>). These are @@ -4582,7 +4605,8 @@ Format: { 0 | 1 } See arch/parisc/kernel/pdc_chassis.c - percpu_alloc= Select which percpu first chunk allocator to use. + percpu_alloc= [MM,EARLY] + Select which percpu first chunk allocator to use. Currently supported values are "embed" and "page". Archs may support subset or none of the selections. See comments in mm/percpu.c for details on each @@ -4644,6 +4668,11 @@ may be specified. Format: <port>,<port>.... + possible_cpus= [SMP,S390,X86] + Format: <unsigned int> + Set the number of possible CPUs, overriding the + regular discovery mechanisms (such as ACPI/FW, etc). + powersave=off [PPC] This option disables power saving features. It specifically disables cpuidle and sets the platform machine description specific power_save @@ -4651,12 +4680,12 @@ execution priority. ppc_strict_facility_enable - [PPC] This option catches any kernel floating point, + [PPC,ENABLE] This option catches any kernel floating point, Altivec, VSX and SPE outside of regions specifically allowed (eg kernel_enable_fpu()/kernel_disable_fpu()). There is some performance impact when enabling this. - ppc_tm= [PPC] + ppc_tm= [PPC,EARLY] Format: {"off"} Disable Hardware Transactional Memory @@ -4766,7 +4795,7 @@ [KNL] Number of legacy pty's. Overwrites compiled-in default number. - quiet [KNL] Disable most log messages + quiet [KNL,EARLY] Disable most log messages r128= [HW,DRM] @@ -4783,17 +4812,17 @@ ramdisk_start= [RAM] RAM disk image start address random.trust_cpu=off - [KNL] Disable trusting the use of the CPU's + [KNL,EARLY] Disable trusting the use of the CPU's random number generator (if available) to initialize the kernel's RNG. random.trust_bootloader=off - [KNL] Disable trusting the use of the a seed + [KNL,EARLY] Disable trusting the use of the a seed passed by the bootloader (if available) to initialize the kernel's RNG. randomize_kstack_offset= - [KNL] Enable or disable kernel stack offset + [KNL,EARLY] Enable or disable kernel stack offset randomization, which provides roughly 5 bits of entropy, frustrating memory corruption attacks that depend on stack address determinism or @@ -5034,6 +5063,11 @@ this kernel boot parameter, forcibly setting it to zero. + rcutree.enable_rcu_lazy= [KNL] + To save power, batch RCU callbacks and flush after + delay, memory pressure or callback list growing too + big. + rcuscale.gp_async= [KNL] Measure performance of asynchronous grace-period primitives such as call_rcu(). @@ -5484,7 +5518,7 @@ Run specified binary instead of /init from the ramdisk, used for early userspace startup. See initrd. - rdrand= [X86] + rdrand= [X86,EARLY] force - Override the decision by the kernel to hide the advertisement of RDRAND support (this affects certain AMD processors because of buggy BIOS @@ -5580,7 +5614,7 @@ them. If <base> is less than 0x10000, the region is assumed to be I/O ports; otherwise it is memory. - reservetop= [X86-32] + reservetop= [X86-32,EARLY] Format: nn[KMG] Reserves a hole at the top of the kernel virtual address space. @@ -5665,7 +5699,7 @@ [KNL] Disable ring 3 MONITOR/MWAIT feature on supported CPUs. - riscv_isa_fallback [RISCV] + riscv_isa_fallback [RISCV,EARLY] When CONFIG_RISCV_ISA_FALLBACK is not enabled, permit falling back to detecting extension support by parsing "riscv,isa" property on devicetree systems when the @@ -5674,13 +5708,14 @@ ro [KNL] Mount root device read-only on boot - rodata= [KNL] + rodata= [KNL,EARLY] on Mark read-only kernel memory as read-only (default). off Leave read-only kernel memory writable for debugging. full Mark read-only kernel memory and aliases as read-only [arm64] rockchip.usb_uart + [EARLY] Enable the uart passthrough on the designated usb port on Rockchip SoCs. When active, the signals of the debug-uart get routed to the D+ and D- pins of the usb @@ -5741,7 +5776,7 @@ sa1100ir [NET] See drivers/net/irda/sa1100_ir.c. - sched_verbose [KNL] Enables verbose scheduler debug messages. + sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics. Allowed values are enable and disable. This feature @@ -5856,7 +5891,7 @@ non-zero "wait" parameter. See weight_single and weight_many. - skew_tick= [KNL] Offset the periodic timer tick per cpu to mitigate + skew_tick= [KNL,EARLY] Offset the periodic timer tick per cpu to mitigate xtime_lock contention on larger systems, and/or RCU lock contention on all systems with CONFIG_MAXSMP set. Format: { "0" | "1" } @@ -5987,10 +6022,10 @@ 1: Fast pin select (default) 2: ATC IRMode - smt= [KNL,MIPS,S390] Set the maximum number of threads (logical - CPUs) to use per physical CPU on systems capable of - symmetric multithreading (SMT). Will be capped to the - actual hardware limit. + smt= [KNL,MIPS,S390,EARLY] Set the maximum number of threads + (logical CPUs) to use per physical CPU on systems + capable of symmetric multithreading (SMT). Will + be capped to the actual hardware limit. Format: <integer> Default: -1 (no limit) @@ -6012,7 +6047,7 @@ sonypi.*= [HW] Sony Programmable I/O Control Device driver See Documentation/admin-guide/laptops/sonypi.rst - spectre_v2= [X86] Control mitigation of Spectre variant 2 + spectre_v2= [X86,EARLY] Control mitigation of Spectre variant 2 (indirect branch speculation) vulnerability. The default operation protects the kernel from user space attacks. @@ -6027,8 +6062,8 @@ Selecting 'on' will, and 'auto' may, choose a mitigation method at run time according to the CPU, the available microcode, the setting of the - CONFIG_RETPOLINE configuration option, and the - compiler with which the kernel was built. + CONFIG_MITIGATION_RETPOLINE configuration option, + and the compiler with which the kernel was built. Selecting 'on' will also enable the mitigation against user space to user space task attacks. @@ -6092,7 +6127,7 @@ spectre_v2_user=auto. spec_rstack_overflow= - [X86] Control RAS overflow mitigation on AMD Zen CPUs + [X86,EARLY] Control RAS overflow mitigation on AMD Zen CPUs off - Disable mitigation microcode - Enable microcode mitigation only @@ -6103,7 +6138,7 @@ (cloud-specific mitigation) spec_store_bypass_disable= - [HW] Control Speculative Store Bypass (SSB) Disable mitigation + [HW,EARLY] Control Speculative Store Bypass (SSB) Disable mitigation (Speculative Store Bypass vulnerability) Certain CPUs are vulnerable to an exploit against a @@ -6199,7 +6234,7 @@ #DB exception for bus lock is triggered only when CPL > 0. - srbds= [X86,INTEL] + srbds= [X86,INTEL,EARLY] Control the Special Register Buffer Data Sampling (SRBDS) mitigation. @@ -6286,7 +6321,7 @@ srcutree.convert_to_big must have the 0x10 bit set for contention-based conversions to occur. - ssbd= [ARM64,HW] + ssbd= [ARM64,HW,EARLY] Speculative Store Bypass Disable control On CPUs that are vulnerable to the Speculative @@ -6310,7 +6345,7 @@ growing up) the main stack are reserved for no other mapping. Default value is 256 pages. - stack_depot_disable= [KNL] + stack_depot_disable= [KNL,EARLY] Setting this to true through kernel command line will disable the stack depot thereby saving the static memory consumed by the stack hash table. By default this is set @@ -6349,12 +6384,12 @@ be used to filter out binaries which have not yet been made aware of AT_MINSIGSTKSZ. - stress_hpt [PPC] + stress_hpt [PPC,EARLY] Limits the number of kernel HPT entries in the hash page table to increase the rate of hash page table faults on kernel addresses. - stress_slb [PPC] + stress_slb [PPC,EARLY] Limits the number of kernel SLB entries, and flushes them frequently to increase the rate of SLB faults on kernel addresses. @@ -6414,7 +6449,7 @@ This parameter controls use of the Protected Execution Facility on pSeries. - swiotlb= [ARM,IA-64,PPC,MIPS,X86] + swiotlb= [ARM,IA-64,PPC,MIPS,X86,EARLY] Format: { <int> [,<int>] | force | noforce } <int> -- Number of I/O TLB slabs <int> -- Second integer after comma. Number of swiotlb @@ -6424,7 +6459,7 @@ wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging) - switches= [HW,M68k] + switches= [HW,M68k,EARLY] sysctl.*= [KNL] Set a sysctl parameter, right before loading the init @@ -6483,11 +6518,11 @@ <deci-seconds>: poll all this frequency 0: no polling (default) - threadirqs [KNL] + threadirqs [KNL,EARLY] Force threading of all interrupt handlers except those marked explicitly IRQF_NO_THREAD. - topology= [S390] + topology= [S390,EARLY] Format: {off | on} Specify if the kernel should make use of the cpu topology information if the hardware supports this. @@ -6728,7 +6763,7 @@ can be overridden by a later tsc=nowatchdog. A console message will flag any such suppression or overriding. - tsc_early_khz= [X86] Skip early TSC calibration and use the given + tsc_early_khz= [X86,EARLY] Skip early TSC calibration and use the given value instead. Useful when the early TSC frequency discovery procedure is not reliable, such as on overclocked systems with CPUID.16h support and partial CPUID.15h support. @@ -6763,7 +6798,7 @@ See Documentation/admin-guide/hw-vuln/tsx_async_abort.rst for more details. - tsx_async_abort= [X86,INTEL] Control mitigation for the TSX Async + tsx_async_abort= [X86,INTEL,EARLY] Control mitigation for the TSX Async Abort (TAA) vulnerability. Similar to Micro-architectural Data Sampling (MDS) @@ -6829,7 +6864,7 @@ unknown_nmi_panic [X86] Cause panic on unknown NMI. - unwind_debug [X86-64] + unwind_debug [X86-64,EARLY] Enable unwinder debug output. This can be useful for debugging certain unwinder error conditions, including corrupt stacks and @@ -7019,7 +7054,7 @@ Example: user_debug=31 userpte= - [X86] Flags controlling user PTE allocations. + [X86,EARLY] Flags controlling user PTE allocations. nohigh = do not allocate PTE pages in HIGHMEM regardless of setting @@ -7048,7 +7083,7 @@ vector= [IA-64,SMP] vector=percpu: enable percpu vector domain - video= [FB] Frame buffer configuration + video= [FB,EARLY] Frame buffer configuration See Documentation/fb/modedb.rst. video.brightness_switch_enabled= [ACPI] @@ -7096,13 +7131,13 @@ P Enable page structure init time poisoning - Disable all of the above options - vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact - size of <nn>. This can be used to increase the - minimum size (128MB on x86). It can also be used to - decrease the size and leave more room for directly - mapped kernel RAM. + vmalloc=nn[KMG] [KNL,BOOT,EARLY] Forces the vmalloc area to have an + exact size of <nn>. This can be used to increase + the minimum size (128MB on x86). It can also be + used to decrease the size and leave more room + for directly mapped kernel RAM. - vmcp_cma=nn[MG] [KNL,S390] + vmcp_cma=nn[MG] [KNL,S390,EARLY] Sets the memory size reserved for contiguous memory allocations for the vmcp device driver. @@ -7115,7 +7150,7 @@ vmpoff= [KNL,S390] Perform z/VM CP command after power off. Format: <command> - vsyscall= [X86-64] + vsyscall= [X86-64,EARLY] Controls the behavior of vsyscalls (i.e. calls to fixed addresses of 0xffffffffff600x00 from legacy code). Most statically-linked binaries and older @@ -7225,6 +7260,15 @@ threshold repeatedly. They are likely good candidates for using WQ_UNBOUND workqueues instead. + workqueue.cpu_intensive_warning_thresh=<uint> + If CONFIG_WQ_CPU_INTENSIVE_REPORT is set, the kernel + will report the work functions which violate the + intensive_threshold_us repeatedly. In order to prevent + spurious warnings, start printing only after a work + function has violated this threshold number of times. + + The default is 4 times. 0 disables the warning. + workqueue.power_efficient Per-cpu workqueues are generally preferred because they show better performance thanks to cache @@ -7263,13 +7307,13 @@ When enabled, memory and cache locality will be impacted. - writecombine= [LOONGARCH] Control the MAT (Memory Access Type) of - ioremap_wc(). + writecombine= [LOONGARCH,EARLY] Control the MAT (Memory Access + Type) of ioremap_wc(). on - Enable writecombine, use WUC for ioremap_wc() off - Disable writecombine, use SUC for ioremap_wc() - x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of + x2apic_phys [X86-64,APIC,EARLY] Use x2apic physical mode instead of default x2apic cluster mode on platforms supporting x2apic. @@ -7280,7 +7324,7 @@ save/restore/migration must be enabled to handle larger domains. - xen_emul_unplug= [HW,X86,XEN] + xen_emul_unplug= [HW,X86,XEN,EARLY] Unplug Xen emulated devices Format: [unplug0,][unplug1] ide-disks -- unplug primary master IDE devices @@ -7292,17 +7336,17 @@ the unplug protocol never -- do not unplug even if version check succeeds - xen_legacy_crash [X86,XEN] + xen_legacy_crash [X86,XEN,EARLY] Crash from Xen panic notifier, without executing late panic() code such as dumping handler. - xen_msr_safe= [X86,XEN] + xen_msr_safe= [X86,XEN,EARLY] Format: <bool> Select whether to always use non-faulting (safe) MSR access functions when running as Xen PV guest. The default value is controlled by CONFIG_XEN_PV_MSR_SAFE. - xen_nopvspin [X86,XEN] + xen_nopvspin [X86,XEN,EARLY] Disables the qspinlock slowpath using Xen PV optimizations. This parameter is obsoleted by "nopvspin" parameter, which has equivalent effect for XEN platform. @@ -7314,7 +7358,7 @@ has equivalent effect for XEN platform. xen_no_vector_callback - [KNL,X86,XEN] Disable the vector callback for Xen + [KNL,X86,XEN,EARLY] Disable the vector callback for Xen event channel interrupts. xen_scrub_pages= [XEN] @@ -7323,7 +7367,7 @@ with /sys/devices/system/xen_memory/xen_memory0/scrub_pages. Default value controlled with CONFIG_XEN_SCRUB_PAGES_DEFAULT. - xen_timer_slop= [X86-64,XEN] + xen_timer_slop= [X86-64,XEN,EARLY] Set the timer slop (in nanoseconds) for the virtual Xen timers (default is 100000). This adjusts the minimum delta of virtualized Xen timers, where lower values @@ -7376,7 +7420,7 @@ host controller quirks. Meaning of each bit can be consulted in header drivers/usb/host/xhci.h. - xmon [PPC] + xmon [PPC,EARLY] Format: { early | on | rw | ro | off } Controls if xmon debugger is enabled. Default is off. Passing only "xmon" is equivalent to "xmon=early". diff --git a/Documentation/admin-guide/tainted-kernels.rst b/Documentation/admin-guide/tainted-kernels.rst index 92a8a07f5c43..f92551539e8a 100644 --- a/Documentation/admin-guide/tainted-kernels.rst +++ b/Documentation/admin-guide/tainted-kernels.rst @@ -34,7 +34,7 @@ name of the command ('Comm:') that triggered the event:: You'll find a 'Not tainted: ' there if the kernel was not tainted at the time of the event; if it was, then it will print 'Tainted: ' and characters -either letters or blanks. In above example it looks like this:: +either letters or blanks. In the example above it looks like this:: Tainted: P W O @@ -52,7 +52,7 @@ At runtime, you can query the tainted state by reading tainted; any other number indicates the reasons why it is. The easiest way to decode that number is the script ``tools/debugging/kernel-chktaint``, which your distribution might ship as part of a package called ``linux-tools`` or -``kernel-tools``; if it doesn't you can download the script from +``kernel-tools``; if it doesn't, you can download the script from `git.kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/debugging/kernel-chktaint>`_ and execute it with ``sh kernel-chktaint``, which would print something like this on the machine that had the statements in the logs that were quoted earlier:: diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst new file mode 100644 index 000000000000..58211840ac6f --- /dev/null +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -0,0 +1,1952 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) +.. [see the bottom of this file for redistribution information] + +========================================= +How to verify bugs and bisect regressions +========================================= + +This document describes how to check if some Linux kernel problem occurs in code +currently supported by developers -- to then explain how to locate the change +causing the issue, if it is a regression (e.g. did not happen with earlier +versions). + +The text aims at people running kernels from mainstream Linux distributions on +commodity hardware who want to report a kernel bug to the upstream Linux +developers. Despite this intent, the instructions work just as well for users +who are already familiar with building their own kernels: they help avoid +mistakes occasionally made even by experienced developers. + +.. + Note: if you see this note, you are reading the text's source file. You + might want to switch to a rendered version: it makes it a lot easier to + read and navigate this document -- especially when you want to look something + up in the reference section, then jump back to where you left off. +.. + Find the latest rendered version of this text here: + https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.rst.html + +The essence of the process (aka 'TL;DR') +======================================== + +*[If you are new to building or bisecting Linux, ignore this section and head +over to the* ":ref:`step-by-step guide<introguide_bissbs>`" *below. It utilizes +the same commands as this section while describing them in brief fashion. The +steps are nevertheless easy to follow and together with accompanying entries +in a reference section mention many alternatives, pitfalls, and additional +aspects, all of which might be essential in your present case.]* + +**In case you want to check if a bug is present in code currently supported by +developers**, execute just the *preparations* and *segment 1*; while doing so, +consider the newest Linux kernel you regularly use to be the 'working' kernel. +In the following example that's assumed to be 6.0.13, which is why the sources +of v6.0 will be used to prepare the .config file. + +**In case you face a regression**, follow the steps at least till the end of +*segment 2*. Then you can submit a preliminary report -- or continue with +*segment 3*, which describes how to perform a bisection needed for a +full-fledged regression report. In the following example 6.0.13 is assumed to be +the 'working' kernel and 6.1.5 to be the first 'broken', which is why v6.0 +will be considered the 'good' release and used to prepare the .config file. + +* **Preparations**: set up everything to build your own kernels:: + + # * Remove any software that depends on externally maintained kernel modules + # or builds any automatically during bootup. + # * Ensure Secure Boot permits booting self-compiled Linux kernels. + # * If you are not already running the 'working' kernel, reboot into it. + # * Install compilers and everything else needed for building Linux. + # * Ensure to have 15 Gigabyte free space in your home directory. + git clone -o mainline --no-checkout \ + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ + cd ~/linux/ + git remote add -t master stable \ + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + git checkout --detach v6.0 + # * Hint: if you used an existing clone, ensure no stale .config is around. + make olddefconfig + # * Ensure the former command picked the .config of the 'working' kernel. + # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up + # VPNs, mount network shares, and briefly try the feature that is broken. + yes '' | make localmodconfig + ./scripts/config --set-str CONFIG_LOCALVERSION '-local' + ./scripts/config -e CONFIG_LOCALVERSION_AUTO + # * Note, when short on storage space, check the guide for an alternative: + ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \ + -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS + # * Hint: at this point you might want to adjust the build configuration; + # you'll have to, if you are running Debian. + make olddefconfig + cp .config ~/kernel-config-working + +* **Segment 1**: build a kernel from the latest mainline codebase. + + This among others checks if the problem was fixed already and which developers + later need to be told about the problem; in case of a regression, this rules + out a .config change as root of the problem. + + a) Checking out latest mainline code:: + + cd ~/linux/ + git checkout --force --detach mainline/master + + b) Build, install, and boot a kernel:: + + cp ~/kernel-config-working .config + make olddefconfig + make -j $(nproc --all) + # * Make sure there is enough disk space to hold another kernel: + df -h /boot/ /lib/modules/ + # * Note: on Arch Linux, its derivatives and a few other distributions + # the following commands will do nothing at all or only part of the + # job. See the step-by-step guide for further details. + command -v installkernel && sudo make modules_install install + # * Check how much space your self-built kernel actually needs, which + # enables you to make better estimates later: + du -ch /boot/*$(make -s kernelrelease)* | tail -n 1 + du -sh /lib/modules/$(make -s kernelrelease)/ + # * Hint: the output of the following command will help you pick the + # right kernel from the boot menu: + make -s kernelrelease | tee -a ~/kernels-built + reboot + # * Once booted, ensure you are running the kernel you just built by + # checking if the output of the next two commands matches: + tail -n 1 ~/kernels-built + uname -r + + c) Check if the problem occurs with this kernel as well. + +* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel. + + This among others verifies the trimmed .config file actually works well, as + bisecting with it otherwise would be a waste of time: + + a) Start by checking out the sources of the 'good' version:: + + cd ~/linux/ + git checkout --force --detach v6.0 + + b) Build, install, and boot a kernel as described earlier in *segment 1, + section b* -- just feel free to skip the 'du' commands, as you have a rough + estimate already. + + c) Ensure the feature that regressed with the 'broken' kernel actually works + with this one. + +* **Segment 3**: perform and validate the bisection. + + a) In case your 'broken' version is a stable/longterm release, add the Git + branch holding it:: + + git remote set-branches --add stable linux-6.1.y + git fetch stable + + b) Initialize the bisection:: + + cd ~/linux/ + git bisect start + git bisect good v6.0 + git bisect bad v6.1.5 + + c) Build, install, and boot a kernel as described earlier in *segment 1, + section b*. + + In case building or booting the kernel fails for unrelated reasons, run + ``git bisect skip``. In all other outcomes, check if the regressed feature + works with the newly built kernel. If it does, tell Git by executing + ``git bisect good``; if it does not, run ``git bisect bad`` instead. + + All three commands will make Git checkout another commit; then re-execute + this step (e.g. build, install, boot, and test a kernel to then tell Git + the outcome). Do so again and again until Git shows which commit broke + things. If you run short of disk space during this process, check the + "Supplementary tasks" section below. + + d) Once your finished the bisection, put a few things away:: + + cd ~/linux/ + git bisect log > ~/bisect-log + cp .config ~/bisection-config-culprit + git bisect reset + + e) Try to verify the bisection result:: + + git checkout --force --detach mainline/master + git revert --no-edit cafec0cacaca0 + + This is optional, as some commits are impossible to revert. But if the + second command worked flawlessly, build, install, and boot one more kernel + kernel, which should not show the regression. + +* **Supplementary tasks**: cleanup during and after the process. + + a) To avoid running out of disk space during a bisection, you might need to + remove some kernels you built earlier. You most likely want to keep those + you built during segment 1 and 2 around for a while, but you will most + likely no longer need kernels tested during the actual bisection + (Segment 3 c). You can list them in build order using:: + + ls -ltr /lib/modules/*-local* + + To then for example erase a kernel that identifies itself as + '6.0-rc1-local-gcafec0cacaca0', use this:: + + sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0 + sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0 + # * Note, on some distributions kernel-install is missing + # or does only part of the job. + + b) If you performed a bisection and successfully validated the result, feel + free to remove all kernels built during the actual bisection (Segment 3 c); + the kernels you built earlier and later you might want to keep around for + a week or two. + +.. _introguide_bissbs: + +Step-by-step guide on how to verify bugs and bisect regressions +=============================================================== + +This guide describes how to set up your own Linux kernels for investigating bugs +or regressions you intent to report. How far you want to follow the instructions +depends on your issue: + +Execute all steps till the end of *segment 1* to **verify if your kernel problem +is present in code supported by Linux kernel developers**. If it is, you are all +set to report the bug -- unless it did not happen with earlier kernel versions, +as then your want to at least continue with *segment 2* to **check if the issue +qualifies as regression** which receive priority treatment. Depending on the +outcome you then are ready to report a bug or submit a preliminary regression +report; instead of the latter your could also head straight on and follow +*segment 3* to **perform a bisection** for a full-fledged regression report +developers are obliged to act upon. + + :ref:`Preparations: set up everything to build your own kernels.<introprep_bissbs>` + + :ref:`Segment 1: try to reproduce the problem with the latest codebase.<introlatestcheck_bissbs>` + + :ref:`Segment 2: check if the kernels you build work fine.<introworkingcheck_bissbs>` + + :ref:`Segment 3: perform a bisection and validate the result.<introbisect_bissbs>` + + :ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>` + +The steps in each segment illustrate the important aspects of the process, while +a comprehensive reference section holds additional details. The latter sometimes +also outlines alternative approaches, pitfalls, as well as problems that might +occur at the particular step -- and how to get things rolling again. + +For further details on how to report Linux kernel issues or regressions check +out Documentation/admin-guide/reporting-issues.rst, which works in conjunction +with this document. It among others explains why you need to verify bugs with +the latest 'mainline' kernel, even if you face a problem with a kernel from a +'stable/longterm' series; for users facing a regression it also explains that +sending a preliminary report after finishing segment 2 might be wise, as the +regression and its culprit might be known already. For further details on +what actually qualifies as a regression check out +Documentation/admin-guide/reporting-regressions.rst. + +.. _introprep_bissbs: + +Preparations: set up everything to build your own kernels +--------------------------------------------------------- + +.. _backup_bissbs: + +* Create a fresh backup and put system repair and restore tools at hand, just + to be prepared for the unlikely case of something going sideways. + + [:ref:`details<backup_bisref>`] + +.. _vanilla_bissbs: + +* Remove all software that depends on externally developed kernel drivers or + builds them automatically. That includes but is not limited to DKMS, openZFS, + VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module). + + [:ref:`details<vanilla_bisref>`] + +.. _secureboot_bissbs: + +* On platforms with 'Secure Boot' or similar solutions, prepare everything to + ensure the system will permit your self-compiled kernel to boot. The + quickest and easiest way to achieve this on commodity x86 systems is to + disable such techniques in the BIOS setup utility; alternatively, remove + their restrictions through a process initiated by + ``mokutil --disable-validation``. + + [:ref:`details<secureboot_bisref>`] + +.. _rangecheck_bissbs: + +* Determine the kernel versions considered 'good' and 'bad' throughout this + guide. + + Do you follow this guide to verify if a bug is present in the code developers + care for? Then consider the mainline release your 'working' kernel (the newest + one you regularly use) is based on to be the 'good' version; if your 'working' + kernel for example is '6.0.11', then your 'good' kernel is 'v6.0'. + + In case you face a regression, it depends on the version range where the + regression was introduced: + + * Something which used to work in Linux 6.0 broke when switching to Linux + 6.1-rc1? Then henceforth regard 'v6.0' as the last known 'good' version + and 'v6.1-rc1' as the first 'bad' one. + + * Some function stopped working when updating from 6.0.11 to 6.1.4? Then for + the time being consider 'v6.0' as the last 'good' version and 'v6.1.4' as + the 'bad' one. Note, at this point it is merely assumed that 6.0 is fine; + this assumption will be checked in segment 2. + + * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? In + that case you want to bisect within a stable/longterm series: consider + 'v6.0.11' as the last known 'good' version and 'v6.0.13' as the first 'bad' + one. Note, in this case you still want to compile and test a mainline kernel + as explained in segment 1: the outcome will determine if you need to report + your issue to the regular developers or the stable team. + + *Note, do not confuse 'good' version with 'working' kernel; the latter term + throughout this guide will refer to the last kernel that has been working + fine.* + + [:ref:`details<rangecheck_bisref>`] + +.. _bootworking_bissbs: + +* Boot into the 'working' kernel and briefly use the apparently broken feature. + + [:ref:`details<bootworking_bisref>`] + +.. _diskspace_bissbs: + +* Ensure to have enough free space for building Linux. 15 Gigabyte in your home + directory should typically suffice. If you have less available, be sure to pay + attention to later steps about retrieving the Linux sources and handling of + debug symbols: both explain approaches reducing the amount of space, which + should allow you to master these tasks with about 4 Gigabytes free space. + + [:ref:`details<diskspace_bisref>`] + +.. _buildrequires_bissbs: + +* Install all software required to build a Linux kernel. Often you will need: + 'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl', + 'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The + reference section shows how to quickly install those on various popular Linux + distributions. + + [:ref:`details<buildrequires_bisref>`] + +.. _sources_bissbs: + +* Retrieve the mainline Linux sources; then change into the directory holding + them, as all further commands in this guide are meant to be executed from + there. + + *Note, the following describe how to retrieve the sources using a full + mainline clone, which downloads about 2,75 GByte as of early 2024. The* + :ref:`reference section describes two alternatives <sources_bisref>` *: + one downloads less than 500 MByte, the other works better with unreliable + internet connections.* + + Execute the following command to retrieve a fresh mainline codebase while + preparing things to add stable/longterm branches later:: + + git clone -o mainline --no-checkout \ + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ + cd ~/linux/ + git remote add -t master stable \ + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + + [:ref:`details<sources_bisref>`] + +.. _oldconfig_bissbs: + +* Start preparing a kernel build configuration (the '.config' file). + + Before doing so, ensure you are still running the 'working' kernel an earlier + step told you to boot; if you are unsure, check the current kernel release + identifier using ``uname -r``. + + Afterwards check out the source code for the version earlier established as + 'good' (in this example this is assumed to be 6.0) and create a .config file:: + + git checkout --detach v6.0 + make olddefconfig + + The second command will try to locate the build configuration file for the + running kernel and then adjust it for the needs of the kernel sources you + checked out. While doing so, it will print a few lines you need to check. + + Look out for a line starting with '# using defaults found in'. It should be + followed by a path to a file in '/boot/' that contains the release identifier + of your currently working kernel. If the line instead continues with something + like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find + the .config file for your running kernel -- in which case you have to put one + there manually, as explained in the reference section. + + In case you can not find such a line, look for one containing '# configuration + written to .config'. If that's the case you have a stale build configuration + lying around. Unless you intend to use it, delete it; afterwards run + 'make olddefconfig' again and check if it now picked up the right config file + as base. + + [:ref:`details<oldconfig_bisref>`] + +.. _localmodconfig_bissbs: + +* Disable any kernel modules apparently superfluous for your setup. This is + optional, but especially wise for bisections, as it speeds up the build + process enormously -- at least unless the .config file picked up in the + previous step was already tailored to your and your hardware needs, in which + case you should skip this step. + + To prepare the trimming, connect external hardware you occasionally use (USB + keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted + since you started that guide, ensure that you tried using the feature causing + trouble since you started the system. Only then trim your .config:: + + yes '' | make localmodconfig + + There is a catch to this, as the 'apparently' in initial sentence of this step + and the preparation instructions already hinted at: + + The 'localmodconfig' target easily disables kernel modules for features only + used occasionally -- like modules for external peripherals not yet connected + since booting, virtualization software not yet utilized, VPN tunnels, and a + few other things. That's because some tasks rely on kernel modules Linux only + loads when you execute tasks like the aforementioned ones for the first time. + + This drawback of localmodconfig is nothing you should lose sleep over, but + something to keep in mind: if something is misbehaving with the kernels built + during this guide, this is most likely the reason. You can reduce or nearly + eliminate the risk with tricks outlined in the reference section; but when + building a kernel just for quick testing purposes this is usually not worth + spending much effort on, as long as it boots and allows to properly test the + feature that causes trouble. + + [:ref:`details<localmodconfig_bisref>`] + +.. _tagging_bissbs: + +* Ensure all the kernels you will build are clearly identifiable using a special + tag and a unique version number:: + + ./scripts/config --set-str CONFIG_LOCALVERSION '-local' + ./scripts/config -e CONFIG_LOCALVERSION_AUTO + + [:ref:`details<tagging_bisref>`] + +.. _debugsymbols_bissbs: + +* Decide how to handle debug symbols. + + In the context of this document it is often wise to enable them, as there is a + decent chance you will need to decode a stack trace from a 'panic', 'Oops', + 'warning', or 'BUG':: + + ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \ + -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS + + But if you are extremely short on storage space, you might want to disable + debug symbols instead:: + + ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \ + -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE + + [:ref:`details<debugsymbols_bisref>`] + +.. _configmods_bissbs: + +* Check if you may want or need to adjust some other kernel configuration + options: + + * Are you running Debian? Then you want to avoid known problems by performing + additional adjustments explained in the reference section. + + [:ref:`details<configmods_distros_bisref>`]. + + * If you want to influence other aspects of the configuration, do so now using + your preferred tool. Note, to use make targets like 'menuconfig' or + 'nconfig', you will need to install the development files of ncurses; for + 'xconfig' you likewise need the Qt5 or Qt6 headers. + + [:ref:`details<configmods_individual_bisref>`]. + +.. _saveconfig_bissbs: + +* Reprocess the .config after the latest adjustments and store it in a safe + place:: + + make olddefconfig + cp .config ~/kernel-config-working + + [:ref:`details<saveconfig_bisref>`] + +.. _introlatestcheck_bissbs: + +Segment 1: try to reproduce the problem with the latest codebase +---------------------------------------------------------------- + +The following steps verify if the problem occurs with the code currently +supported by developers. In case you face a regression, it also checks that the +problem is not caused by some .config change, as reporting the issue then would +be a waste of time. [:ref:`details<introlatestcheck_bisref>`] + +.. _checkoutmaster_bissbs: + +* Check out the latest Linux codebase:: + + cd ~/linux/ + git checkout --force --detach mainline/master + + [:ref:`details<checkoutmaster_bisref>`] + +.. _build_bissbs: + +* Build the image and the modules of your first kernel using the config file you + prepared:: + + cp ~/kernel-config-working .config + make olddefconfig + make -j $(nproc --all) + + If you want your kernel packaged up as deb, rpm, or tar file, see the + reference section for alternatives, which obviously will require other + steps to install as well. + + [:ref:`details<build_bisref>`] + +.. _install_bissbs: + +* Install your newly built kernel. + + Before doing so, consider checking if there is still enough room for it:: + + df -h /boot/ /lib/modules/ + + 150 MByte in /boot/ and 200 in /lib/modules/ usually suffice. Those are rough + estimates assuming the worst case. How much your kernels actually require will + be determined later. + + Now install the kernel, which will be saved in parallel to the kernels from + your Linux distribution:: + + command -v installkernel && sudo make modules_install install + + On many commodity Linux distributions this will take care of everything + required to boot your kernel. You might want to ensure that's the case by + checking if your boot loader's configuration was updated; furthermore ensure + an initramfs (also known as initrd) exists, which on many distributions can be + achieved by running ``ls -l /boot/init*$(make -s kernelrelease)*``. Those + steps are recommended, as there are quite a few Linux distribution where above + command is insufficient: + + * On Arch Linux, its derivatives, many immutable Linux distributions, and a + few others the above command does nothing at, as they lack 'installkernel' + executable. + + * Some distributions install the kernel, but don't add an entry for your + kernel in your boot loader's configuration -- the kernel thus won't show up + in the boot menu. + + * Some distributions add a boot loader menu entry, but don't create an + initramfs on installation -- in that case your kernel most likely will be + unable to mount the root partition during bootup. + + If any of that applies to you, see the reference section for further guidance. + Once you figured out what to do, consider writing down the necessary + installation steps: if you will build more kernels as described in + segment 2 and 3, you will have to execute these commands every time that + ``command -v installkernel [...]`` comes up again. + + [:ref:`details<install_bisref>`] + +.. _storagespace_bissbs: + +* In case you plan to follow this guide further, check how much storage space + the kernel, its modules, and other related files like the initramfs consume:: + + du -ch /boot/*$(make -s kernelrelease)* | tail -n 1 + du -sh /lib/modules/$(make -s kernelrelease)/ + + Write down or remember those two values for later: they enable you to prevent + running out of disk space accidentally during a bisection. + + [:ref:`details<storagespace_bisref>`] + +.. _kernelrelease_bissbs: + +* Show and store the kernelrelease identifier of the kernel you just built:: + + make -s kernelrelease | tee -a ~/kernels-built + + Remember the identifier momentarily, as it will help you pick the right kernel + from the boot menu upon restarting. + +.. _recheckbroken_bissbs: + +* Reboot into the kernel you just built and check if the feature that is + expected to be broken really is. + + Start by making sure the kernel you booted is the one you just built. When + unsure, check if the output of these commands show the exact same release + identifier:: + + tail -n 1 ~/kernels-built + uname -r + + Now verify if the feature that causes trouble works with your newly built + kernel. If things work while investigating a regression, check the reference + section for further details. + + [:ref:`details<recheckbroken_bisref>`] + +.. _recheckstablebroken_bissbs: + +* Are you facing a problem within a stable/longterm release, but failed to + reproduce it with the mainline kernel you just built? Then check if the latest + codebase for the particular series might already fix the problem. To do so, + add the stable series Git branch for your 'good' kernel (again, this here is + assumed to be 6.0) and check out the latest version:: + + cd ~/linux/ + git remote set-branches --add stable linux-6.0.y + git fetch stable + git checkout --force --detach linux-6.0.y + + Now use the checked out code to build and install another kernel using the + commands the earlier steps already described in more detail:: + + cp ~/kernel-config-working .config + make olddefconfig + make -j $(nproc --all) + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + command -v installkernel && sudo make modules_install install + make -s kernelrelease | tee -a ~/kernels-built + reboot + + Now verify if you booted the kernel you intended to start, to then check if + everything works fine with this kernel:: + + tail -n 1 ~/kernels-built + uname -r + + [:ref:`details<recheckstablebroken_bisref>`] + +Do you follow this guide to verify if a problem is present in the code +currently supported by Linux kernel developers? Then you are done at this +point. If you later want to remove the kernel you just built, check out +:ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`. + +In case you face a regression, move on and execute at least the next segment +as well. + +.. _introworkingcheck_bissbs: + +Segment 2: check if the kernels you build work fine +--------------------------------------------------- + +In case of a regression, you now want to ensure the trimmed configuration file +you created earlier works as expected; a bisection with the .config file +otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] + +.. _recheckworking_bissbs: + +* Build your own variant of the 'working' kernel and check if the feature that + regressed works as expected with it. + + Start by checking out the sources for the version earlier established as + 'good' (once again assumed to be 6.0 here):: + + cd ~/linux/ + git checkout --detach v6.0 + + Now use the checked out code to configure, build, and install another kernel + using the commands the previous subsection explained in more detail:: + + cp ~/kernel-config-working .config + make olddefconfig + make -j $(nproc --all) + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + command -v installkernel && sudo make modules_install install + make -s kernelrelease | tee -a ~/kernels-built + reboot + + When the system booted, you may want to verify once again that the + kernel you started is the one you just built: + + tail -n 1 ~/kernels-built + uname -r + + Now check if this kernel works as expected; if not, consult the reference + section for further instructions. + + [:ref:`details<recheckworking_bisref>`] + +.. _introbisect_bissbs: + +Segment 3: perform the bisection and validate the result +-------------------------------------------------------- + +With all the preparations and precaution builds taken care of, you are now ready +to begin the bisection. This will make you build quite a few kernels -- usually +about 15 in case you encountered a regression when updating to a newer series +(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build +configuration created earlier this works a lot faster than many people assume: +overall on average it will often just take about 10 to 15 minutes to compile +each kernel on commodity x86 machines. + +* In case your 'bad' version is a stable/longterm release (say v6.1.5), add its + stable branch, unless you already did so earlier:: + + cd ~/linux/ + git remote set-branches --add stable linux-6.1.y + git fetch stable + +.. _bisectstart_bissbs: + +* Start the bisection and tell Git about the versions earlier established as + 'good' (6.0 in the following example command) and 'bad' (6.1.5):: + + cd ~/linux/ + git bisect start + git bisect good v6.0 + git bisect bad v6.1.5 + + [:ref:`details<bisectstart_bisref>`] + +.. _bisectbuild_bissbs: + +* Now use the code Git checked out to build, install, and boot a kernel using + the commands introduced earlier:: + + cp ~/kernel-config-working .config + make olddefconfig + make -j $(nproc --all) + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + command -v installkernel && sudo make modules_install install + make -s kernelrelease | tee -a ~/kernels-built + reboot + + If compilation fails for some reason, run ``git bisect skip`` and restart + executing the stack of commands from the beginning. + + In case you skipped the "test latest codebase" step in the guide, check its + description as for why the 'df [...]' and 'make -s kernelrelease [...]' + commands are here. + + Important note: the latter command from this point on will print release + identifiers that might look odd or wrong to you -- which they are not, as it's + totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0' + if you bisect between versions 6.1 and 6.2 for example. + + [:ref:`details<bisectbuild_bisref>`] + +.. _bisecttest_bissbs: + +* Now check if the feature that regressed works in the kernel you just built. + + You again might want to start by making sure the kernel you booted is the one + you just built:: + + cd ~/linux/ + tail -n 1 ~/kernels-built + uname -r + + Now verify if the feature that regressed works at this kernel bisection point. + If it does, run this:: + + git bisect good + + If it does not, run this:: + + git bisect bad + + Be sure about what you tell Git, as getting this wrong just once will send the + rest of the bisection totally off course. + + While the bisection is ongoing, Git will use the information you provided to + find and check out another bisection point for you to test. While doing so, it + will print something like 'Bisecting: 675 revisions left to test after this + (roughly 10 steps)' to indicate how many further changes it expects to be + tested. Now build and install another kernel using the instructions from the + previous step; afterwards follow the instructions in this step again. + + Repeat this again and again until you finish the bisection -- that's the case + when Git after tagging a change as 'good' or 'bad' prints something like + 'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right + afterwards it will show some details about the culprit including the patch + description of the change. The latter might fill your terminal screen, so you + might need to scroll up to see the message mentioning the culprit; + alternatively, run ``git bisect log > ~/bisection-log``. + + [:ref:`details<bisecttest_bisref>`] + +.. _bisectlog_bissbs: + +* Store Git's bisection log and the current .config file in a safe place before + telling Git to reset the sources to the state before the bisection:: + + cd ~/linux/ + git bisect log > ~/bisection-log + cp .config ~/bisection-config-culprit + git bisect reset + + [:ref:`details<bisectlog_bisref>`] + +.. _revert_bissbs: + +* Try reverting the culprit on top of latest mainline to see if this fixes your + regression. + + This is optional, as it might be impossible or hard to realize. The former is + the case, if the bisection determined a merge commit as the culprit; the + latter happens if other changes depend on the culprit. But if the revert + succeeds, it is worth building another kernel, as it validates the result of + a bisection, which can easily deroute; it furthermore will let kernel + developers know, if they can resolve the regression with a quick revert. + + Begin by checking out the latest codebase depending on the range you bisected: + + * Did you face a regression within a stable/longterm series (say between + 6.0.11 and 6.0.13) that does not happen in mainline? Then check out the + latest codebase for the affected series like this:: + + git fetch stable + git checkout --force --detach linux-6.0.y + + * In all other cases check out latest mainline:: + + git fetch mainline + git checkout --force --detach mainline/master + + If you bisected a regression within a stable/longterm series that also + happens in mainline, there is one more thing to do: look up the mainline + commit-id. To do so, use a command like ``git show abcdcafecabcd`` to + view the patch description of the culprit. There will be a line near + the top which looks like 'commit cafec0cacaca0 upstream.' or + 'Upstream commit cafec0cacaca0'; use that commit-id in the next command + and not the one the bisection blamed. + + Now try reverting the culprit by specifying its commit id:: + + git revert --no-edit cafec0cacaca0 + + If that fails, give up trying and move on to the next step. But if it works, + build a kernel again using the familiar command sequence:: + + cp ~/kernel-config-working .config + make olddefconfig && + make -j $(nproc --all) && + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + command -v installkernel && sudo make modules_install install + Make -s kernelrelease | tee -a ~/kernels-built + reboot + + Now check one last time if the feature that made you perform a bisection work + with that kernel. + + [:ref:`details<revert_bisref>`] + +.. _introclosure_bissbs: + +Supplementary tasks: cleanup during and after the bisection +----------------------------------------------------------- + +During and after following this guide you might want or need to remove some of +the kernels you installed: the boot menu otherwise will become confusing or +space might run out. + +.. _makeroom_bissbs: + +* To remove one of the kernels you installed, look up its 'kernelrelease' + identifier. This guide stores them in '~/kernels-built', but the following + command will print them as well:: + + ls -ltr /lib/modules/*-local* + + You in most situations want to remove the oldest kernels built during the + actual bisection (e.g. segment 3 of this guide). The two ones you created + beforehand (e.g. to test the latest codebase and the version considered + 'good') might become handy to verify something later -- thus better keep them + around, unless you are really short on storage space. + + To remove the modules of a kernel with the kernelrelease identifier + '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its + modules:: + + sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0 + + Afterwards try the following command:: + + sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0 + + On quite a few distributions this will delete all other kernel files installed + while also removing the kernel's entry from the boot menu. But on some + distributions kernel-install does not exist or leaves boot-loader entries or + kernel image and related files behind; in that case remove them as described + in the reference section. + + [:ref:`details<makeroom_bisref>`] + +.. _finishingtouch_bissbs: + +* Once you have finished the bisection, do not immediately remove anything you + set up, as you might need a few things again. What is safe to remove depends + on the outcome of the bisection: + + * Could you initially reproduce the regression with the latest codebase and + after the bisection were able to fix the problem by reverting the culprit on + top of the latest codebase? Then you want to keep those two kernels around + for a while, but safely remove all others with a '-local' in the release + identifier. + + * Did the bisection end on a merge-commit or seems questionable for other + reasons? Then you want to keep as many kernels as possible around for a few + days: it's pretty likely that you will be asked to recheck something. + + * In other cases it likely is a good idea to keep the following kernels around + for some time: the one built from the latest codebase, the one created from + the version considered 'good', and the last three or four you compiled + during the actual bisection process. + + [:ref:`details<finishingtouch_bisref>`] + +.. _submit_improvements: + +This concludes the step-by-step guide. + +Did you run into trouble following any of the above steps not cleared up by the +reference section below? Did you spot errors? Or do you have ideas how to +improve the guide? Then please take a moment and let the maintainer of this +document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while +CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is +vital to improve this document further, which is in everybody's interest, as it +will enable more people to master the task described here -- and hopefully also +improve similar guides inspired by this one. + + +Reference section for the step-by-step guide +============================================ + +This section holds additional information for almost all the items in the above +step-by-step guide. + +.. _backup_bisref: + +Prepare for emergencies +----------------------- + + *Create a fresh backup and put system repair and restore tools at hand.* + [:ref:`... <backup_bissbs>`] + +Remember, you are dealing with computers, which sometimes do unexpected things +-- especially if you fiddle with crucial parts like the kernel of an operating +system. That's what you are about to do in this process. Hence, better prepare +for something going sideways, even if that should not happen. + +[:ref:`back to step-by-step guide <backup_bissbs>`] + +.. _vanilla_bisref: + +Remove anything related to externally maintained kernel modules +--------------------------------------------------------------- + + *Remove all software that depends on externally developed kernel drivers or + builds them automatically.* [:ref:`...<vanilla_bissbs>`] + +Externally developed kernel modules can easily cause trouble during a bisection. + +But there is a more important reason why this guide contains this step: most +kernel developers will not care about reports about regressions occurring with +kernels that utilize such modules. That's because such kernels are not +considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst +explains in more detail. + +[:ref:`back to step-by-step guide <vanilla_bissbs>`] + +.. _secureboot_bisref: + +Deal with techniques like Secure Boot +------------------------------------- + + *On platforms with 'Secure Boot' or similar techniques, prepare everything to + ensure the system will permit your self-compiled kernel to boot later.* + [:ref:`... <secureboot_bissbs>`] + +Many modern systems allow only certain operating systems to start; that's why +they reject booting self-compiled kernels by default. + +You ideally deal with this by making your platform trust your self-built kernels +with the help of a certificate. How to do that is not described +here, as it requires various steps that would take the text too far away from +its purpose; 'Documentation/admin-guide/module-signing.rst' and various web +sides already explain everything needed in more detail. + +Temporarily disabling solutions like Secure Boot is another way to make your own +Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup +utility; the required steps vary a lot between machines and therefore cannot be +described here. + +On mainstream x86 Linux distributions there is a third and universal option: +disable all Secure Boot restrictions for your Linux environment. You can +initiate this process by running ``mokutil --disable-validation``; this will +tell you to create a one-time password, which is safe to write down. Now +restart; right after your BIOS performed all self-tests the bootloader Shim will +show a blue box with a message 'Press any key to perform MOK management'. Hit +some key before the countdown exposes, which will open a menu. Choose 'Change +Secure Boot state'. Shim's 'MokManager' will now ask you to enter three +randomly chosen characters from the one-time password specified earlier. Once +you provided them, confirm you really want to disable the validation. +Afterwards, permit MokManager to reboot the machine. + +[:ref:`back to step-by-step guide <secureboot_bissbs>`] + +.. _bootworking_bisref: + +Boot the last kernel that was working +------------------------------------- + + *Boot into the last working kernel and briefly recheck if the feature that + regressed really works.* [:ref:`...<bootworking_bissbs>`] + +This will make later steps that cover creating and trimming the configuration do +the right thing. + +[:ref:`back to step-by-step guide <bootworking_bissbs>`] + +.. _diskspace_bisref: + +Space requirements +------------------ + + *Ensure to have enough free space for building Linux.* + [:ref:`... <diskspace_bissbs>`] + +The numbers mentioned are rough estimates with a big extra charge to be on the +safe side, so often you will need less. + +If you have space constraints, be sure to hay attention to the :ref:`step about +debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference +section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk +space by quite a few gigabytes. + +[:ref:`back to step-by-step guide <diskspace_bissbs>`] + +.. _rangecheck_bisref: + +Bisection range +--------------- + + *Determine the kernel versions considered 'good' and 'bad' throughout this + guide.* [:ref:`...<rangecheck_bissbs>`] + +Establishing the range of commits to be checked is mostly straightforward, +except when a regression occurred when switching from a release of one stable +series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that case +Git will need some hand holding, as there is no straight line of descent. + +That's because with the release of 6.0 mainline carried on to 6.1 while the +stable series 6.0.y branched to the side. It's therefore theoretically possible +that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed by a +commit that went into one of the 6.0.y releases, but never hit mainline or the +6.1.y series. Thankfully that normally should not happen due to the way the +stable/longterm maintainers maintain the code. It's thus pretty safe to assume +6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel +will be built and tested in the segment '2' of this guide; Git would force you +to do this as well, if you tried bisecting between 6.0.11 and 6.1.13. + +[:ref:`back to step-by-step guide <rangecheck_bissbs>`] + +.. _buildrequires_bisref: + +Install build requirements +-------------------------- + + *Install all software required to build a Linux kernel.* + [:ref:`...<buildrequires_bissbs>`] + +The kernel is pretty stand-alone, but besides tools like the compiler you will +sometimes need a few libraries to build one. How to install everything needed +depends on your Linux distribution and the configuration of the kernel you are +about to build. + +Here are a few examples what you typically need on some mainstream +distributions: + +* Arch Linux and derivatives:: + + sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \ + pahole perl zlib ncurses qt6-base + +* Debian, Ubuntu, and derivatives:: + + sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \ + libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \ + libncurses-dev qt6-base-dev g++ + +* Fedora and derivatives:: + + sudo dnf install binutils \ + /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \ + /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction} + +* openSUSE and derivatives:: + + sudo zypper install bc binutils bison dwarves flex gcc git \ + kernel-install-tools libelf-devel make modutils openssl openssl-devel \ + perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel + +These commands install a few packages that are often, but not always needed. You +for example might want to skip installing the development headers for ncurses, +which you will only need in case you later might want to adjust the kernel build +configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit +the headers of Qt6 is you do not plan to adjust the .config using 'xconfig'. + +You furthermore might need additional libraries and their development headers +for tasks not covered in this guide -- for example when building utilities from +the kernel's tools/ directory. + +[:ref:`back to step-by-step guide <buildrequires_bissbs>`] + +.. _sources_bisref: + +Download the sources using Git +------------------------------ + + *Retrieve the Linux mainline sources.* + [:ref:`...<sources_bissbs>`] + +The step-by-step guide outlines how to download the Linux sources using a full +Git clone of Linus' mainline repository. There is nothing more to say about +that -- but there are two alternatives ways to retrieve the sources that might +work better for you: + + * If you have an unreliable internet connection, consider + :ref:`using a 'Git bundle'<sources_bundle_bisref>`. + + * If downloading the complete repository would take too long or requires too + much storage space, consider :ref:`using a 'shallow + clone'<sources_shallow_bisref>`. + +.. _sources_bundle_bisref: + +Downloading Linux mainline sources using a bundle +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the following commands to retrieve the Linux mainline sources using a +bundle:: + + wget -c \ + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle + git clone --no-checkout clone.bundle ~/linux/ + cd ~/linux/ + git remote remove origin + git remote add mainline \ + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git + git fetch mainline + git remote add -t master stable \ + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + +In case the 'wget' command fails, just re-execute it, it will pick up where +it left off. + +[:ref:`back to step-by-step guide <sources_bissbs>`] +[:ref:`back to section intro <sources_bisref>`] + +.. _sources_shallow_bisref: + +Downloading Linux mainline sources using a shallow clone +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, execute the following command to retrieve the latest mainline codebase:: + + git clone -o mainline --no-checkout --depth 1 -b master \ + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ + cd ~/linux/ + git remote add -t master stable \ + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + +Now deepen your clone's history to the second predecessor of the mainline +release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19 would +be the first predecessor and 5.18 the second -- hence deepen the history up to +that version:: + + git fetch --shallow-exclude=v5.18 mainline + +Afterwards add the stable Git repository as remote and all required stable +branches as explained in the step-by-step guide. + +Note, shallow clones have a few peculiar characteristics: + + * For bisections the history needs to be deepened a few mainline versions + farther than it seems necessary, as explained above already. That's because + Git otherwise will be unable to revert or describe most of the commits within + a range (say v6.1..v6.2), as they are internally based on earlier kernels + releases (like v6.0-rc2 or 5.19-rc3). + + * This document in most places uses ``git fetch`` with ``--shallow-exclude=`` + to specify the earliest version you care about (or to be precise: its git + tag). You alternatively can use the parameter ``--shallow-since=`` to specify + an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to + define the depth of the history you want to download. When using them while + bisecting mainline, ensure to deepen the history to at least 7 months before + the release of the mainline release your 'good' kernel is based on. + + * Be warned, when deepening your clone you might encounter an error like + 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'. + In that case run ``git repack -d`` and try again. + +[:ref:`back to step-by-step guide <sources_bissbs>`] +[:ref:`back to section intro <sources_bisref>`] + +.. _oldconfig_bisref: + +Start defining the build configuration for your kernel +------------------------------------------------------ + + *Start preparing a kernel build configuration (the '.config' file).* + [:ref:`... <oldconfig_bissbs>`] + +*Note, this is the first of multiple steps in this guide that create or modify +build artifacts. The commands used in this guide store them right in the source +tree to keep things simple. In case you prefer storing the build artifacts +separately, create a directory like '~/linux-builddir/' and add the parameter +``O=~/linux-builddir/`` to all make calls used throughout this guide. You will +have to point other commands there as well -- among them the ``./scripts/config +[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to +locate the right build configuration.* + +Two things can easily go wrong when creating a .config file as advised: + + * The oldconfig target will use a .config file from your build directory, if + one is already present there (e.g. '~/linux/.config'). That's totally fine if + that's what you intend (see next step), but in all other cases you want to + delete it. This for example is important in case you followed this guide + further, but due to problems come back here to redo the configuration from + scratch. + + * Sometimes olddefconfig is unable to locate the .config file for your running + kernel and will use defaults, as briefly outlined in the guide. In that case + check if your distribution ships the configuration somewhere and manually put + it in the right place (e.g. '~/linux/.config') if it does. On distributions + where /proc/config.gz exists this can be achieved using this command:: + + zcat /proc/config.gz > .config + + Once you put it there, run ``make olddefconfig`` again to adjust it to the + needs of the kernel about to be built. + +Note, the olddefconfig target will set any undefined build options to their +default value. If you prefer to set such configuration options manually, use +``make oldconfig`` instead. Then for each undefined configuration option you +will be asked how to proceed; in case you are unsure what to answer, simply hit +'enter' to apply the default value. Note though that for bisections you normally +want to go with the defaults, as you otherwise might enable a new feature that +causes a problem looking like regressions (for example due to security +restrictions). + +Occasionally odd things happen when trying to use a config file prepared for one +kernel (say 6.1) on an older mainline release -- especially if it is much older +(say v5.15). That's one of the reasons why the previous step in the guide told +you to boot the kernel where everything works. If you manually add a .config +file you thus want to ensure it's from the working kernel and not from a one +that shows the regression. + +In case you want to build kernels for another machine, locate its kernel build +configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy +that file to the build machine and store it as ~/linux/.config; afterwards run +``make olddefconfig`` to adjust it. + +[:ref:`back to step-by-step guide <oldconfig_bissbs>`] + +.. _localmodconfig_bisref: + +Trim the build configuration for your kernel +-------------------------------------------- + + *Disable any kernel modules apparently superfluous for your setup.* + [:ref:`... <localmodconfig_bissbs>`] + +As explained briefly in the step-by-step guide already: with localmodconfig it +can easily happen that your self-built kernels will lack modules for tasks you +did not perform at least once before utilizing this make target. That happens +when a task requires kernel modules which are only autoloaded when you execute +it for the first time. So when you never performed that task since starting your +kernel the modules will not have been loaded -- and from localmodonfig's point +of view look superfluous, which thus disables them to reduce the amount of code +to be compiled. + +You can try to avoid this by performing typical tasks that often will autoload +additional kernel modules: start a VM, establish VPN connections, loop-mount a +CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external +devices (2FA keys, headsets, webcams, ...) as well as storage devices with file +systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it +is hard to think of everything that might be needed -- even kernel developers +often forget one thing or another at this point. + +Do not let that risk bother you, especially when compiling a kernel only for +testing purposes: everything typically crucial will be there. And if you forget +something important you can turn on a missing feature manually later and quickly +run the commands again to compile and install a kernel that has everything you +need. + +But if you plan to build and use self-built kernels regularly, you might want to +reduce the risk by recording which modules your system loads over the course of +a few weeks. You can automate this with `modprobed-db +<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to +point localmodconfig to the list of modules modprobed-db noticed being used:: + + yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig + +That parameter also allows you to build trimmed kernels for another machine in +case you copied a suitable .config over to use as base (see previous step). Just +run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to +your build's host home directory. Then run these commands instead of the one the +step-by-step guide mentions:: + + yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig + +[:ref:`back to step-by-step guide <localmodconfig_bissbs>`] + +.. _tagging_bisref: + +Tag the kernels about to be build +--------------------------------- + + *Ensure all the kernels you will build are clearly identifiable using a + special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`] + +This allows you to differentiate your distribution's kernels from those created +during this process, as the file or directories for the latter will contain +'-local' in the name; it also helps picking the right entry in the boot menu and +not lose track of you kernels, as their version numbers will look slightly +confusing during the bisection. + +[:ref:`back to step-by-step guide <tagging_bissbs>`] + +.. _debugsymbols_bisref: + +Decide to enable or disable debug symbols +----------------------------------------- + + *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`] + +Having debug symbols available can be important when your kernel throws a +'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be +able to find the exact place where the problem occurred in the code. But +collecting and embedding the needed debug information takes time and consumes +quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel +trimmed with localmodconfig consumed around 5 Gigabyte of space with debug +symbols, but less than 1 when they were disabled. The resulting kernel image and +modules are bigger as well, which increases storage requirements for /boot/ and +load times. + +In case you want a small kernel and are unlikely to decode a stack trace later, +you thus might want to disable debug symbols to avoid those downsides. If it +later turns out that you need them, just enable them as shown and rebuild the +kernel. + +You on the other hand definitely want to enable them for this process, if there +is a decent chance that you need to decode a stack trace later. The section +'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst +explains this process in more detail. + +[:ref:`back to step-by-step guide <debugsymbols_bissbs>`] + +.. _configmods_bisref: + +Adjust build configuration +-------------------------- + + *Check if you may want or need to adjust some other kernel configuration + options:* + +Depending on your needs you at this point might want or have to adjust some +kernel configuration options. + +.. _configmods_distros_bisref: + +Distro specific adjustments +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + *Are you running* [:ref:`... <configmods_bissbs>`] + +The following sections help you to avoid build problems that are known to occur +when following this guide on a few commodity distributions. + +**Debian:** + + * Remove a stale reference to a certificate file that would cause your build to + fail:: + + ./scripts/config --set-str SYSTEM_TRUSTED_KEYS '' + + Alternatively, download the needed certificate and make that configuration + option point to it, as `the Debian handbook explains in more detail + <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_ + -- or generate your own, as explained in + Documentation/admin-guide/module-signing.rst. + +[:ref:`back to step-by-step guide <configmods_bissbs>`] + +.. _configmods_individual_bisref: + +Individual adjustments +~~~~~~~~~~~~~~~~~~~~~~ + + *If you want to influence the other aspects of the configuration, do so + now.* [:ref:`... <configmods_bissbs>`] + +You at this point can use a command like ``make menuconfig`` to enable or +disable certain features using a text-based user interface; to use a graphical +configuration utility, call the make target ``xconfig`` or ``gconfig`` instead. +All of them require development libraries from toolkits they are based on +(ncurses, Qt5, Gtk2); an error message will tell you if something required is +missing. + +[:ref:`back to step-by-step guide <configmods_bissbs>`] + +.. _saveconfig_bisref: + +Put the .config file aside +-------------------------- + + *Reprocess the .config after the latest changes and store it in a safe place.* + [:ref:`... <saveconfig_bissbs>`] + +Put the .config you prepared aside, as you want to copy it back to the build +directory every time during this guide before you start building another +kernel. That's because going back and forth between different versions can alter +.config files in odd ways; those occasionally cause side effects that could +confuse testing or in some cases render the result of your bisection +meaningless. + +[:ref:`back to step-by-step guide <saveconfig_bissbs>`] + +.. _introlatestcheck_bisref: + +Try to reproduce the regression +----------------------------------------- + + *Verify the regression is not caused by some .config change and check if it + still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`] + +For some readers it might seem unnecessary to check the latest codebase at this +point, especially if you did that already with a kernel prepared by your +distributor or face a regression within a stable/longterm series. But it's +highly recommended for these reasons: + +* You will run into any problems caused by your setup before you actually begin + a bisection. That will make it a lot easier to differentiate between 'this + most likely is some problem in my setup' and 'this change needs to be skipped + during the bisection, as the kernel sources at that stage contain an unrelated + problem that causes building or booting to fail'. + +* These steps will rule out if your problem is caused by some change in the + build configuration between the 'working' and the 'broken' kernel. This for + example can happen when your distributor enabled an additional security + feature in the newer kernel which was disabled or not yet supported by the + older kernel. That security feature might get into the way of something you + do -- in which case your problem from the perspective of the Linux kernel + upstream developers is not a regression, as + Documentation/admin-guide/reporting-regressions.rst explains in more detail. + You thus would waste your time if you'd try to bisect this. + +* If the cause for your regression was already fixed in the latest mainline + codebase, you'd perform the bisection for nothing. This holds true for a + regression you encountered with a stable/longterm release as well, as they are + often caused by problems in mainline changes that were backported -- in which + case the problem will have to be fixed in mainline first. Maybe it already was + fixed there and the fix is already in the process of being backported. + +* For regressions within a stable/longterm series it's furthermore crucial to + know if the issue is specific to that series or also happens in the mainline + kernel, as the report needs to be sent to different people: + + * Regressions specific to a stable/longterm series are the stable team's + responsibility; mainline Linux developers might or might not care. + + * Regressions also happening in mainline are something the regular Linux + developers and maintainers have to handle; the stable team does not care + and does not need to be involved in the report, they just should be told + to backport the fix once it's ready. + + Your report might be ignored if you send it to the wrong party -- and even + when you get a reply there is a decent chance that developers tell you to + evaluate which of the two cases it is before they take a closer look. + +[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] + +.. _checkoutmaster_bisref: + +Checkout the latest Linux codebase +---------------------------------- + + *Checkout the latest Linux codebase.* + [:ref:`... <introlatestcheck_bissbs>`] + +In case you later want to recheck if an ever newer codebase might fix the +problem, remember to run that ``git fetch --shallow-exclude [...]`` command +again mentioned earlier to update your local Git repository. + +[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] + +.. _build_bisref: + +Build your kernel +----------------- + + *Build the image and the modules of your first kernel using the config file + you prepared.* [:ref:`... <build_bissbs>`] + +A lot can go wrong at this stage, but the instructions below will help you help +yourself. Another subsection explains how to directly package your kernel up as +deb, rpm or tar file. + +Dealing with build errors +~~~~~~~~~~~~~~~~~~~~~~~~~ + +When a build error occurs, it might be caused by some aspect of your machine's +setup that often can be fixed quickly; other times though the problem lies in +the code and can only be fixed by a developer. A close examination of the +failure messages coupled with some research on the internet will often tell you +which of the two it is. To perform such a investigation, restart the build +process like this:: + + make V=1 + +The ``V=1`` activates verbose output, which might be needed to see the actual +error. To make it easier to spot, this command also omits the ``-j $(nproc +--all)`` used earlier to utilize every CPU core in the system for the job -- but +this parallelism also results in some clutter when failures occur. + +After a few seconds the build process should run into the error again. Now try +to find the most crucial line describing the problem. Then search the internet +for the most important and non-generic section of that line (say 4 to 8 words); +avoid or remove anything that looks remotely system-specific, like your username +or local path names like ``/home/username/linux/``. First try your regular +internet search engine with that string, afterwards search Linux kernel mailing +lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_. + +This most of the time will find something that will explain what is wrong; quite +often one of the hits will provide a solution for your problem, too. If you +do not find anything that matches your problem, try again from a different angle +by modifying your search terms or using another line from the error messages. + +In the end, most trouble you are to run into has likely been encountered and +reported by others already. That includes issues where the cause is not your +system, but lies the code. If you run into one of those, you might thus find a +solution (e.g. a patch) or workaround for your problem, too. + +Package your kernel up +~~~~~~~~~~~~~~~~~~~~~~ + +The step-by-step guide uses the default make targets (e.g. 'bzImage' and +'modules' on x86) to build the image and the modules of your kernel, which later +steps of the guide then install. You instead can also directly build everything +and directly package it up by using one of the following targets: + + * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package + + * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package + + * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball + +This is just a selection of available make targets for this purpose, see +``make help`` for others. You can also use these targets after running +``make -j $(nproc --all)``, as they will pick up everything already built. + +If you employ the targets to generate deb or rpm packages, ignore the +step-by-step guide's instructions on installing and removing your kernel; +instead install and remove the packages using the package utility for the format +(e.g. dpkg and rpm) or a package management utility build on top of them (apt, +aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using +these two make targets are designed to work on various distributions utilizing +those formats, they thus will sometimes behave differently than your +distribution's kernel packages. + +[:ref:`back to step-by-step guide <build_bissbs>`] + +.. _install_bisref: + +Put the kernel in place +----------------------- + + *Install the kernel you just built.* [:ref:`... <install_bissbs>`] + +What you need to do after executing the command in the step-by-step guide +depends on the existence and the implementation of an ``installkernel`` +executable. Many commodity Linux distributions ship such a kernel installer in +'/sbin/' that does everything needed, hence there is nothing left for you +except rebooting. But some distributions contain an installkernel that does +only part of the job -- and a few lack it completely and leave all the work to +you. + +If ``installkernel`` is found, the kernel's build system will delegate the +actual installation of your kernel's image and related files to this executable. +On almost all Linux distributions it will store the image as '/boot/vmlinuz- +<kernelrelease identifier>' and put a 'System.map-<kernelrelease +identifier>' alongside it. Your kernel will thus be installed in parallel to any +existing ones, unless you already have one with exactly the same release name. + +Installkernel on many distributions will afterwards generate an 'initramfs' +(often also called 'initrd'), which commodity distributions rely on for booting; +hence be sure to keep the order of the two make targets used in the step-by-step +guide, as things will go sideways if you install your kernel's image before its +modules. Often installkernel will then add your kernel to the bootloader +configuration, too. You have to take care of one or both of these tasks +yourself, if your distributions installkernel doesn't handle them. + +A few distributions like Arch Linux and its derivatives totally lack an +installkernel executable. On those just install the modules using the kernel's +build system and then install the image and the System.map file manually:: + + sudo make modules_install + sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease) + sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease) + +If your distribution boots with the help of an initramfs, now generate one for +your kernel using the tools your distribution provides for this process. +Afterwards add your kernel to your bootloader configuration and reboot. + +[:ref:`back to step-by-step guide <install_bissbs>`] + +.. _storagespace_bisref: + +Storage requirements per kernel +------------------------------- + + *Check how much storage space the kernel, its modules, and other related files + like the initramfs consume.* [:ref:`... <storagespace_bissbs>`] + +The kernels built during a bisection consume quite a bit of space in /boot/ and +/lib/modules/, especially if you enabled debug symbols. That makes it easy to +fill up volumes during a bisection -- and due to that even kernels which used to +work earlier might fail to boot. To prevent that you will need to know how much +space each installed kernel typically requires. + +Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in +the guide will match all files needed to boot your kernel -- but neither the +path nor the naming scheme are mandatory. On some distributions you thus will +need to look in different places. + +[:ref:`back to step-by-step guide <storagespace_bissbs>`] + +.. _recheckbroken_bisref: + +Check the kernel built from the latest codebase +----------------------------------------------- + + *Reboot into the kernel you just built and check if the feature that regressed + is really broken there.* [:ref:`... <recheckbroken_bissbs>`] + +There are a couple of reasons why the regression you face might not show up with +your own kernel built from the latest codebase. These are the most frequent: + +* The cause for the regression was fixed meanwhile. + +* The regression with the broken kernel was caused by a change in the build + configuration the provider of your kernel carried out. + +* Your problem might be a race condition that does not show up with your kernel; + the trimmed build configuration, a different setting for debug symbols, the + compiler used, and various other things can cause this. + +* In case you encountered the regression with a stable/longterm kernel it might + be a problem that is specific to that series; the next step in this guide will + check this. + +[:ref:`back to step-by-step guide <recheckbroken_bissbs>`] + +.. _recheckstablebroken_bisref: + +Check the kernel built from the latest stable/longterm codebase +--------------------------------------------------------------- + + *Are you facing a regression within a stable/longterm release, but failed to + reproduce it with the kernel you just built using the latest mainline sources? + Then check if the latest codebase for the particular series might already fix + the problem.* [:ref:`... <recheckstablebroken_bissbs>`] + +If this kernel does not show the regression either, there most likely is no need +for a bisection. + +[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`] + +.. _introworkingcheck_bisref: + +Ensure the 'good' version is really working well +------------------------------------------------ + + *Check if the kernels you build work fine.* + [:ref:`... <introworkingcheck_bissbs>`] + +This section will reestablish a known working base. Skipping it might be +appealing, but is usually a bad idea, as it does something important: + +It will ensure the .config file you prepared earlier actually works as expected. +That is in your own interest, as trimming the configuration is not foolproof -- +and you might be building and testing ten or more kernels for nothing before +starting to suspect something might be wrong with the build configuration. + +That alone is reason enough to spend the time on this, but not the only reason. + +Many readers of this guide normally run kernels that are patched, use add-on +modules, or both. Those kernels thus are not considered 'vanilla' -- therefore +it's possible that the thing that regressed might never have worked in vanilla +builds of the 'good' version in the first place. + +There is a third reason for those that noticed a regression between +stable/longterm kernels of different series (e.g. v6.0.13..v6.1.5): it will +ensure the kernel version you assumed to be 'good' earlier in the process (e.g. +v6.0) actually is working. + +[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`] + +.. _recheckworking_bisref: + +Build your own version of the 'good' kernel +------------------------------------------- + + *Build your own variant of the working kernel and check if the feature that + regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`] + +In case the feature that broke with newer kernels does not work with your first +self-built kernel, find and resolve the cause before moving on. There are a +multitude of reasons why this might happen. Some ideas where to look: + +* Maybe localmodconfig did something odd and disabled the module required to + test the feature? Then you might want to recreate a .config file based on the + one from the last working kernel and skip trimming it down; manually disabling + some features in the .config might work as well to reduce the build time. + +* Maybe it's not a kernel regression and something that is caused by some fluke, + a broken initramfs (also known as initrd), new firmware files, or an updated + userland software? + +* Maybe it was a feature added to your distributor's kernel which vanilla Linux + at that point never supported? + +Note, if you found and fixed problems with the .config file, you want to use it +to build another kernel from the latest codebase, as your earlier tests with +mainline and the latest version from an affected stable/longterm series most +likely has been flawed. + +[:ref:`back to step-by-step guide <recheckworking_bissbs>`] + +.. _bisectstart_bisref: + +Start the bisection +------------------- + + *Start the bisection and tell Git about the versions earlier established as + 'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`] + +This will start the bisection process; the last of the commands will make Git +checkout a commit round about half-way between the 'good' and the 'bad' changes +for your to test. + +[:ref:`back to step-by-step guide <bisectstart_bissbs>`] + +.. _bisectbuild_bisref: + +Build a kernel from the bisection point +--------------------------------------- + + *Build, install, and boot a kernel from the code Git checked out using the + same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`] + +There are two things worth of note here: + +* Occasionally building the kernel will fail or it might not boot due some + problem in the code at the bisection point. In that case run this command:: + + git bisect skip + + Git will then check out another commit nearby which with a bit of luck should + work better. Afterwards restart executing this step. + +* Those slightly odd looking version identifiers can happen during bisections, + because the Linux kernel subsystems prepare their changes for a new mainline + release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus + base them on a somewhat earlier point like v6.1-rc1 or even v6.0 -- and then + get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This + leads to those slightly odd looking version identifiers coming up during + bisections. + +[:ref:`back to step-by-step guide <bisectbuild_bissbs>`] + +.. _bisecttest_bisref: + +Bisection checkpoint +-------------------- + + *Check if the feature that regressed works in the kernel you just built.* + [:ref:`... <bisecttest_bissbs>`] + +Ensure what you tell Git is accurate: getting it wrong just one time will bring +the rest of the bisection totally of course, hence all testing after that point +will be for nothing. + +[:ref:`back to step-by-step guide <bisecttest_bissbs>`] + +.. _bisectlog_bisref: + +Put the bisection log away +-------------------------- + + *Store Git's bisection log and the current .config file in a safe place.* + [:ref:`... <bisectlog_bissbs>`] + +As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will +render the end result of a bisection useless. In that case you'd normally have +to restart the bisection from scratch. The log can prevent that, as it might +allow someone to point out where a bisection likely went sideways -- and then +instead of testing ten or more kernels you might only have to build a few to +resolve things. + +The .config file is put aside, as there is a decent chance that developers might +ask for it after you reported the regression. + +[:ref:`back to step-by-step guide <bisectlog_bissbs>`] + +.. _revert_bisref: + +Try reverting the culprit +------------------------- + + *Try reverting the culprit on top of the latest codebase to see if this fixes + your regression.* [:ref:`... <revert_bissbs>`] + +This is an optional step, but whenever possible one you should try: there is a +decent chance that developers will ask you to perform this step when you bring +the bisection result up. So give it a try, you are in the flow already, building +one more kernel shouldn't be a big deal at this point. + +The step-by-step guide covers everything relevant already except one slightly +rare thing: did you bisected a regression that also happened with mainline using +a stable/longterm series, but Git failed to revert the commit in mainline? Then +try to revert the culprit in the affected stable/longterm series -- and if that +succeeds, test that kernel version instead. + +[:ref:`back to step-by-step guide <revert_bissbs>`] + + +Supplementary tasks: cleanup during and after the bisection +----------------------------------------------------------- + +.. _makeroom_bisref: + +Cleaning up during the bisection +-------------------------------- + + *To remove one of the kernels you installed, look up its 'kernelrelease' + identifier.* [:ref:`... <makeroom_bissbs>`] + +The kernels you install during this process are easy to remove later, as its +parts are only stored in two places and clearly identifiable. You thus do not +need to worry to mess up your machine when you install a kernel manually (and +thus bypass your distribution's packaging system): all parts of your kernels are +relatively easy to remove later. + +One of the two places is a directory in /lib/modules/, which holds the modules +for each installed kernel. This directory is named after the kernel's release +identifier; hence, to remove all modules for one of the kernels you built, +simply remove its modules directory in /lib/modules/. + +The other place is /boot/, where typically two up to five files will be placed +during installation of a kernel. All of them usually contain the release name in +their file name, but how many files and their exact name depends somewhat on +your distribution's installkernel executable and its initramfs generator. On +some distributions the ``kernel-install remove...`` command mentioned in the +step-by-step guide will delete all of these files for you while also removing +the menu entry for the kernel from your bootloader configuration. On others you +have to take care of these two tasks yourself. The following command should +interactively remove the three main files of a kernel with the release name +'6.0-rc1-local-gcafec0cacaca0':: + + rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0 + +Afterwards check for other files in /boot/ that have +'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well. +Now remove the boot entry for the kernel from your bootloader's configuration; +the steps to do that vary quite a bit between Linux distributions. + +Note, be careful with wildcards like '*' when deleting files or directories +for kernels manually: you might accidentally remove files of a 6.0.11 kernel +when all you want is to remove 6.0 or 6.0.1. + +[:ref:`back to step-by-step guide <makeroom_bissbs>`] + +Cleaning up after the bisection +------------------------------- + +.. _finishingtouch_bisref: + + *Once you have finished the bisection, do not immediately remove anything + you set up, as you might need a few things again.* + [:ref:`... <finishingtouch_bissbs>`] + +When you are really short of storage space removing the kernels as described in +the step-by-step guide might not free as much space as you would like. In that +case consider running ``rm -rf ~/linux/*`` as well now. This will remove the +build artifacts and the Linux sources, but will leave the Git repository +(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the +sources back. + +Removing the repository as well would likely be unwise at this point: there is a +decent chance developers will ask you to build another kernel to perform +additional tests. This is often required to debug an issue or check proposed +fixes. Before doing so you want to run the ``git fetch mainline`` command again +followed by ``git checkout mainline/master`` to bring your clone up to date and +checkout the latest codebase. Then apply the patch using ``git apply +<filename>`` or ``git am <filename>`` and build yet another kernel using the +familiar commands. + +Additional tests are also the reason why you want to keep the +~/kernel-config-working file around for a few weeks. + +[:ref:`back to step-by-step guide <finishingtouch_bissbs>`] + + +Additional reading material +=========================== + +Further sources +--------------- + +* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and + `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_ + in the Git documentation. +* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_ + from kernel developer Nathan Chancellor. +* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_. +* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_. + +.. + end-of-content +.. + This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If + you spot a typo or small mistake, feel free to let him know directly and + he'll fix it. You are free to do the same in a mostly informal way if you + want to contribute changes to the text -- but for copyright reasons please CC + linux-doc@vger.kernel.org and 'sign-off' your contribution as + Documentation/process/submitting-patches.rst explains in the section 'Sign + your work - the Developer's Certificate of Origin'. +.. + This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top + of the file. If you want to distribute this text under CC-BY-4.0 only, + please use 'The Linux kernel development community' for author attribution + and link this as source: + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst + +.. + Note: Only the content of this RST file as found in the Linux kernel sources + is available under CC-BY-4.0, as versions of this text that were processed + (for example by the kernel's build system) might contain content taken from + files which use a more restrictive license. |