diff options
Diffstat (limited to 'Documentation/admin-guide')
| -rw-r--r-- | Documentation/admin-guide/bug-bisect.rst | 78 | ||||
| -rw-r--r-- | Documentation/admin-guide/bug-hunting.rst | 426 | ||||
| -rw-r--r-- | Documentation/admin-guide/index.rst | 11 | ||||
| -rw-r--r-- | Documentation/admin-guide/oops-tracing.rst | 300 | ||||
| -rw-r--r-- | Documentation/admin-guide/tainted-kernels.rst | 59 |
5 files changed, 416 insertions, 458 deletions
diff --git a/Documentation/admin-guide/bug-bisect.rst b/Documentation/admin-guide/bug-bisect.rst new file mode 100644 index 000000000000..5682d742017c --- /dev/null +++ b/Documentation/admin-guide/bug-bisect.rst @@ -0,0 +1,78 @@ +Bisecting a bug ++++++++++++++++ + +Last updated: 28 October 2016 + +Introduction +============ + +Always try the latest kernel from kernel.org and build from source. If you are +not confident in doing that please report the bug to your distribution vendor +instead of to a kernel developer. + +Finding bugs is not always easy. Have a go though. If you can't find it don't +give up. Report as much as you have found to the relevant maintainer. See +MAINTAINERS for who that is for the subsystem you have worked on. + +Before you submit a bug report read +:ref:`Documentation/admin-guide/reporting-bugs.rst <reportingbugs>`. + +Devices not appearing +===================== + +Often this is caused by udev/systemd. Check that first before blaming it +on the kernel. + +Finding patch that caused a bug +=============================== + +Using the provided tools with ``git`` makes finding bugs easy provided the bug +is reproducible. + +Steps to do it: + +- build the Kernel from its git source +- start bisect with [#f1]_:: + + $ git bisect start + +- mark the broken changeset with:: + + $ git bisect bad [commit] + +- mark a changeset where the code is known to work with:: + + $ git bisect good [commit] + +- rebuild the Kernel and test +- interact with git bisect by using either:: + + $ git bisect good + + or:: + + $ git bisect bad + + depending if the bug happened on the changeset you're testing +- After some interactions, git bisect will give you the changeset that + likely caused the bug. + +- For example, if you know that the current version is bad, and version + 4.8 is good, you could do:: + + $ git bisect start + $ git bisect bad # Current version is bad + $ git bisect good v4.8 + + +.. [#f1] You can, optionally, provide both good and bad arguments at git + start:: + + git bisect start [BAD] [GOOD] + +For further references, please read: + +- The man page for ``git-bisect`` +- `Fighting regressions with git bisect <https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html>`_ +- `Fully automated bisecting with "git bisect run" <https://lwn.net/Articles/317154>`_ +- `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_ diff --git a/Documentation/admin-guide/bug-hunting.rst b/Documentation/admin-guide/bug-hunting.rst index d35dd9fd1af0..08c4b1308189 100644 --- a/Documentation/admin-guide/bug-hunting.rst +++ b/Documentation/admin-guide/bug-hunting.rst @@ -1,145 +1,190 @@ Bug hunting -+++++++++++ +=========== + +Kernel bug reports often come with a stack dump like the one below:: + + ------------[ cut here ]------------ + WARNING: CPU: 1 PID: 28102 at kernel/module.c:1108 module_put+0x57/0x70 + Modules linked in: dvb_usb_gp8psk(-) dvb_usb dvb_core nvidia_drm(PO) nvidia_modeset(PO) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd soundcore nvidia(PO) [last unloaded: rc_core] + CPU: 1 PID: 28102 Comm: rmmod Tainted: P WC O 4.8.4-build.1 #1 + Hardware name: MSI MS-7309/MS-7309, BIOS V1.12 02/23/2009 + 00000000 c12ba080 00000000 00000000 c103ed6a c1616014 00000001 00006dc6 + c1615862 00000454 c109e8a7 c109e8a7 00000009 ffffffff 00000000 f13f6a10 + f5f5a600 c103ee33 00000009 00000000 00000000 c109e8a7 f80ca4d0 c109f617 + Call Trace: + [<c12ba080>] ? dump_stack+0x44/0x64 + [<c103ed6a>] ? __warn+0xfa/0x120 + [<c109e8a7>] ? module_put+0x57/0x70 + [<c109e8a7>] ? module_put+0x57/0x70 + [<c103ee33>] ? warn_slowpath_null+0x23/0x30 + [<c109e8a7>] ? module_put+0x57/0x70 + [<f80ca4d0>] ? gp8psk_fe_set_frontend+0x460/0x460 [dvb_usb_gp8psk] + [<c109f617>] ? symbol_put_addr+0x27/0x50 + [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb] + [<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0 [dvb_usb] + [<c13d03bc>] ? usb_disable_endpoint+0x7c/0xb0 + [<f80bb48a>] ? dvb_usb_device_exit+0x2a/0x50 [dvb_usb] + [<c13d2882>] ? usb_unbind_interface+0x62/0x250 + [<c136b514>] ? __pm_runtime_idle+0x44/0x70 + [<c13620d8>] ? __device_release_driver+0x78/0x120 + [<c1362907>] ? driver_detach+0x87/0x90 + [<c1361c48>] ? bus_remove_driver+0x38/0x90 + [<c13d1c18>] ? usb_deregister+0x58/0xb0 + [<c109fbb0>] ? SyS_delete_module+0x130/0x1f0 + [<c1055654>] ? task_work_run+0x64/0x80 + [<c1000fa5>] ? exit_to_usermode_loop+0x85/0x90 + [<c10013f0>] ? do_fast_syscall_32+0x80/0x130 + [<c1549f43>] ? sysenter_past_esp+0x40/0x6a + ---[ end trace 6ebc60ef3981792f ]--- + +Such stack traces provide enough information to identify the line inside the +Kernel's source code where the bug happened. Depending on the severity of +the issue, it may also contain the word **Oops**, as on this one:: + + BUG: unable to handle kernel NULL pointer dereference at (null) + IP: [<c06969d4>] iret_exc+0x7d0/0xa59 + *pdpt = 000000002258a001 *pde = 0000000000000000 + Oops: 0002 [#1] PREEMPT SMP + ... + +Despite being an **Oops** or some other sort of stack trace, the offended +line is usually required to identify and handle the bug. Along this chapter, +we'll refer to "Oops" for all kinds of stack traces that need to be analized. -Last updated: 20 December 2005 - -Introduction -============ - -Always try the latest kernel from kernel.org and build from source. If you are -not confident in doing that please report the bug to your distribution vendor -instead of to a kernel developer. - -Finding bugs is not always easy. Have a go though. If you can't find it don't -give up. Report as much as you have found to the relevant maintainer. See -MAINTAINERS for who that is for the subsystem you have worked on. - -Before you submit a bug report read -:ref:`Documentation/admin-guide/reporting-bugs.rst <reportingbugs>`. +.. note:: -Devices not appearing -===================== + ``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original + format (from ``dmesg``, etc). Ignore any references in this or other docs to + "decoding the Oops" or "running it through ksymoops". + If you post an Oops from 2.6+ that has been run through ``ksymoops``, + people will just tell you to repost it. -Often this is caused by udev. Check that first before blaming it on the -kernel. +Where is the Oops message is located? +------------------------------------- -Finding patch that caused a bug -=============================== +Normally the Oops text is read from the kernel buffers by klogd and +handed to ``syslogd`` which writes it to a syslog file, typically +``/var/log/messages`` (depends on ``/etc/syslog.conf``). On systems with +systemd, it may also be stored by the ``journald`` daemon, and accessed +by running ``journalctl`` command. +Sometimes ``klogd`` dies, in which case you can run ``dmesg > file`` to +read the data from the kernel buffers and save it. Or you can +``cat /proc/kmsg > file``, however you have to break in to stop the transfer, +``kmsg`` is a "never ending file". +If the machine has crashed so badly that you cannot enter commands or +the disk is not available then you have three options: -Finding using ``git-bisect`` ----------------------------- +(1) Hand copy the text from the screen and type it in after the machine + has restarted. Messy but it is the only option if you have not + planned for a crash. Alternatively, you can take a picture of + the screen with a digital camera - not nice, but better than + nothing. If the messages scroll off the top of the console, you + may find that booting with a higher resolution (eg, ``vga=791``) + will allow you to read more of the text. (Caveat: This needs ``vesafb``, + so won't help for 'early' oopses) -Using the provided tools with ``git`` makes finding bugs easy provided the bug -is reproducible. +(2) Boot with a serial console (see + :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`), + run a null modem to a second machine and capture the output there + using your favourite communication program. Minicom works well. -Steps to do it: +(3) Use Kdump (see Documentation/kdump/kdump.txt), + extract the kernel ring buffer from old memory with using dmesg + gdbmacro in Documentation/kdump/gdbmacros.txt. -- start using git for the kernel source -- read the man page for ``git-bisect`` -- have fun +Finding the bug's location +-------------------------- -Finding it the old way ----------------------- +Reporting a bug works best if you point the location of the bug at the +Kernel source file. There are two methods for doing that. Usually, using +``gdb`` is easier, but the Kernel should be pre-compiled with debug info. -[Sat Mar 2 10:32:33 PST 1996 KERNEL_BUG-HOWTO lm@sgi.com (Larry McVoy)] +gdb +^^^ -This is how to track down a bug if you know nothing about kernel hacking. -It's a brute force approach but it works pretty well. +The GNU debug (``gdb``) is the best way to figure out the exact file and line +number of the OOPS from the ``vmlinux`` file. -You need: +The usage of gdb works best on a kernel compiled with ``CONFIG_DEBUG_INFO``. +This can be set by running:: - - A reproducible bug - it has to happen predictably (sorry) - - All the kernel tar files from a revision that worked to the - revision that doesn't + $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO -You will then do: +On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the +EIP value from the OOPS:: - - Rebuild a revision that you believe works, install, and verify that. - - Do a binary search over the kernels to figure out which one - introduced the bug. I.e., suppose 1.3.28 didn't have the bug, but - you know that 1.3.69 does. Pick a kernel in the middle and build - that, like 1.3.50. Build & test; if it works, pick the mid point - between .50 and .69, else the mid point between .28 and .50. - - You'll narrow it down to the kernel that introduced the bug. You - can probably do better than this but it gets tricky. + EIP: 0060:[<c021e50e>] Not tainted VLI - - Narrow it down to a subdirectory +And use GDB to translate that to human-readable form:: - - Copy kernel that works into "test". Let's say that 3.62 works, - but 3.63 doesn't. So you diff -r those two kernels and come - up with a list of directories that changed. For each of those - directories: + $ gdb vmlinux + (gdb) l *0xc021e50e - Copy the non-working directory next to the working directory - as "dir.63". - One directory at time, try moving the working directory to - "dir.62" and mv dir.63 dir"time, try:: +If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function +offset from the OOPS:: - mv dir dir.62 - mv dir.63 dir - find dir -name '*.[oa]' -print | xargs rm -f + EIP is at vt_ioctl+0xda8/0x1482 - And then rebuild and retest. Assuming that all related - changes were contained in the sub directory, this should - isolate the change to a directory. +And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled:: - Problems: changes in header files may have occurred; I've - found in my case that they were self explanatory - you may - or may not want to give up when that happens. + $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO + $ make vmlinux + $ gdb vmlinux + (gdb) l *vt_ioctl+0xda8 + 0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293). + 288 { + 289 struct vc_data *vc = NULL; + 290 int ret = 0; + 291 + 292 console_lock(); + 293 if (VT_BUSY(vc_num)) + 294 ret = -EBUSY; + 295 else if (vc_num) + 296 vc = vc_deallocate(vc_num); + 297 console_unlock(); + +or, if you want to be more verbose:: - - Narrow it down to a file + (gdb) p vt_ioctl + $1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl> + (gdb) l *0xae0+0xda8 - - You can apply the same technique to each file in the directory, - hoping that the changes in that file are self contained. +You could, instead, use the object file:: - - Narrow it down to a routine + $ make drivers/tty/ + $ gdb drivers/tty/vt/vt_ioctl.o + (gdb) l *vt_ioctl+0xda8 - - You can take the old file and the new file and manually create - a merged file that has:: +If you have a call trace, such as:: - #ifdef VER62 - routine() - { - ... - } - #else - routine() - { - ... - } - #endif + Call Trace: + [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5 + [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e + [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee + ... - And then walk through that file, one routine at a time and - prefix it with:: +this shows the problem likely in the :jbd: module. You can load that module +in gdb and list the relevant code:: - #define VER62 - /* both routines here */ - #undef VER62 + $ gdb fs/jbd/jbd.ko + (gdb) l *log_wait_commit+0xa3 - Then recompile, retest, move the ifdefs until you find the one - that makes the difference. +.. note:: -Finally, you take all the info that you have, kernel revisions, bug -description, the extent to which you have narrowed it down, and pass -that off to whomever you believe is the maintainer of that section. -A post to linux.dev.kernel isn't such a bad idea if you've done some -work to narrow it down. + You can also do the same for any function call at the stack trace, + like this one:: -If you get it down to a routine, you'll probably get a fix in 24 hours. + [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb] -My apologies to Linus and the other kernel hackers for describing this -brute force approach, it's hardly what a kernel hacker would do. However, -it does work and it lets non-hackers help fix bugs. And it is cool -because Linux snapshots will let you do this - something that you can't -do with vendor supplied releases. + The position where the above call happened can be seen with:: -Fixing the bug -============== + $ gdb drivers/media/usb/dvb-usb/dvb-usb.o + (gdb) l *dvb_usb_adapter_frontend_exit+0x3a -Nobody is going to tell you how to fix bugs. Seriously. You need to work it -out. But below are some hints on how to use the tools. +objdump +^^^^^^^ To debug a kernel, use objdump and look for the hex offset from the crash output to find the valid line of code/assembler. Without debug symbols, you @@ -147,7 +192,7 @@ will see the assembler code for the routine shown, but if your kernel has debug symbols the C code will also be available. (Debug symbols can be enabled in the kernel hacking menu of the menu configuration.) For example:: - objdump -r -S -l --disassemble net/dccp/ipv4.o + $ objdump -r -S -l --disassemble net/dccp/ipv4.o .. note:: @@ -157,7 +202,7 @@ in the kernel hacking menu of the menu configuration.) For example:: If you don't have access to the code you can also debug on some crash dumps e.g. crash dump output as shown by Dave Miller:: - EIP is at ip_queue_xmit+0x14/0x4c0 + EIP is at +0x14/0x4c0 ... Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00 00 00 55 57 56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08 @@ -185,65 +230,140 @@ e.g. crash dump output as shown by Dave Miller:: mov 0x8(%ebp), %ebx ! %ebx = skb->sk mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt -In addition, you can use GDB to figure out the exact file and line -number of the OOPS from the ``vmlinux`` file. If you have -``CONFIG_DEBUG_INFO`` enabled, you can simply copy the EIP value from the -OOPS:: - - EIP: 0060:[<c021e50e>] Not tainted VLI - -And use GDB to translate that to human-readable form:: - - gdb vmlinux - (gdb) l *0xc021e50e - -If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function -offset from the OOPS:: - - EIP is at vt_ioctl+0xda8/0x1482 - -And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled:: +Reporting the bug +----------------- - make vmlinux - gdb vmlinux - (gdb) p vt_ioctl - (gdb) l *(0x<address of vt_ioctl> + 0xda8) +Once you find where the bug happened, by inspecting its location, +you could either try to fix it yourself or report it upstream. -or, as one command:: +In order to report it upstream, you should identify the mailing list +used for the development of the affected code. This can be done by using +the ``get_maintainer.pl`` script. - (gdb) l *(vt_ioctl + 0xda8) +For example, if you find a bug at the gspca's conex.c file, you can get +their maintainers with:: -If you have a call trace, such as:: + $ ./scripts/get_maintainer.pl -f drivers/media/usb/gspca/sonixj.c + Hans Verkuil <hverkuil@xs4all.nl> (odd fixer:GSPCA USB WEBCAM DRIVER,commit_signer:1/1=100%) + Mauro Carvalho Chehab <mchehab@kernel.org> (maintainer:MEDIA INPUT INFRASTRUCTURE (V4L/DVB),commit_signer:1/1=100%) + Tejun Heo <tj@kernel.org> (commit_signer:1/1=100%) + Bhaktipriya Shridhar <bhaktipriya96@gmail.com> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:4/4=100%,removed_lines:9/9=100%) + linux-media@vger.kernel.org (open list:GSPCA USB WEBCAM DRIVER) + linux-kernel@vger.kernel.org (open list) - Call Trace: - [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5 - [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e - [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee - ... +Please notice that it will point to: -this shows the problem in the :jbd: module. You can load that module in gdb -and list the relevant code:: +- The last developers that touched on the source code. On the above example, + Tejun and Bhaktipriya (in this specific case, none really envolved on the + development of this file); +- The driver maintainer (Hans Verkuil); +- The subsystem maintainer (Mauro Carvalho Chehab) +- The driver and/or subsystem mailing list (linux-media@vger.kernel.org); +- the Linux Kernel mailing list (linux-kernel@vger.kernel.org). - gdb fs/jbd/jbd.ko - (gdb) p log_wait_commit - (gdb) l *(0x<address> + 0xa3) +Usually, the fastest way to have your bug fixed is to report it to mailing +list used for the development of the code (linux-media ML) copying the driver maintainer (Hans). -or:: +If you are totally stumped as to whom to send the report, and +``get_maintainer.pl`` didn't provide you anything useful, send it to +linux-kernel@vger.kernel.org. - (gdb) l *(log_wait_commit + 0xa3) +Thanks for your help in making Linux as stable as humanly possible. +Fixing the bug +-------------- -Another very useful option of the Kernel Hacking section in menuconfig is -Debug memory allocations. This will help you see whether data has been -initialised and not set before use etc. To see the values that get assigned -with this look at ``mm/slab.c`` and search for ``POISON_INUSE``. When using -this an Oops will often show the poisoned data instead of zero which is the -default. +If you know programming, you could help us by not only reporting the bug, +but also providing us with a solution. After all open source is about +sharing what you do and don't you want to be recognised for your genius? -Once you have worked out a fix please submit it upstream. After all open -source is about sharing what you do and don't you want to be recognised for -your genius? +If you decide to take this way, once you have worked out a fix please submit +it upstream. Please do read ref:`Documentation/process/submitting-patches.rst <submittingpatches>` though to help your code get accepted. + + +--------------------------------------------------------------------------- + +Notes on Oops tracing with ``klogd`` +------------------------------------ + +In order to help Linus and the other kernel developers there has been +substantial support incorporated into ``klogd`` for processing protection +faults. In order to have full support for address resolution at least +version 1.3-pl3 of the ``sysklogd`` package should be used. + +When a protection fault occurs the ``klogd`` daemon automatically +translates important addresses in the kernel log messages to their +symbolic equivalents. This translated kernel message is then +forwarded through whatever reporting mechanism ``klogd`` is using. The +protection fault message can be simply cut out of the message files +and forwarded to the kernel developers. + +Two types of address resolution are performed by ``klogd``. The first is +static translation and the second is dynamic translation. Static +translation uses the System.map file in much the same manner that +ksymoops does. In order to do static translation the ``klogd`` daemon +must be able to find a system map file at daemon initialization time. +See the klogd man page for information on how ``klogd`` searches for map +files. + +Dynamic address translation is important when kernel loadable modules +are being used. Since memory for kernel modules is allocated from the +kernel's dynamic memory pools there are no fixed locations for either +the start of the module or for functions and symbols in the module. + +The kernel supports system calls which allow a program to determine +which modules are loaded and their location in memory. Using these +system calls the klogd daemon builds a symbol table which can be used +to debug a protection fault which occurs in a loadable kernel module. + +At the very minimum klogd will provide the name of the module which +generated the protection fault. There may be additional symbolic +information available if the developer of the loadable module chose to +export symbol information from the module. + +Since the kernel module environment can be dynamic there must be a +mechanism for notifying the ``klogd`` daemon when a change in module +environment occurs. There are command line options available which +allow klogd to signal the currently executing daemon that symbol +information should be refreshed. See the ``klogd`` manual page for more +information. + +A patch is included with the sysklogd distribution which modifies the +``modules-2.0.0`` package to automatically signal klogd whenever a module +is loaded or unloaded. Applying this patch provides essentially +seamless support for debugging protection faults which occur with +kernel loadable modules. + +The following is an example of a protection fault in a loadable module +processed by ``klogd``:: + + Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc + Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000 + Aug 29 09:51:01 blizard kernel: *pde = 00000000 + Aug 29 09:51:01 blizard kernel: Oops: 0002 + Aug 29 09:51:01 blizard kernel: CPU: 0 + Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868] + Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212 + Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c + Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c + Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 + Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000) + Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 + Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 + Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 + Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] + Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 + +--------------------------------------------------------------------------- + +:: + + Dr. G.W. Wettstein Oncology Research Div. Computing Facility + Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com + 820 4th St. N. + Fargo, ND 58122 + Phone: 701-234-7556 diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst index 368845e9900a..2681cbd24cdd 100644 --- a/Documentation/admin-guide/index.rst +++ b/Documentation/admin-guide/index.rst @@ -8,7 +8,7 @@ document! With luck things will improve quickly over time. This initial section contains overall information, including the README file describing the kernel as a whole, documentation on kernel parameters, -etc. +etc. .. toctree:: :maxdepth: 1 @@ -22,11 +22,12 @@ problems and bugs in particular. .. toctree:: :maxdepth: 1 - + reporting-bugs security-bugs bug-hunting - oops-tracing + bug-bisect + tainted-kernels ramoops dynamic-debug-howto init @@ -37,7 +38,7 @@ ABI will be found here. .. toctree:: :maxdepth: 1 - + sysfs-rules The rest of this manual consists of various unordered guides on how to @@ -45,7 +46,7 @@ configure specific aspects of kernel behavior to your liking. .. toctree:: :maxdepth: 1 - + initrd serial-console braille-console diff --git a/Documentation/admin-guide/oops-tracing.rst b/Documentation/admin-guide/oops-tracing.rst deleted file mode 100644 index 13be8d7bcfe7..000000000000 --- a/Documentation/admin-guide/oops-tracing.rst +++ /dev/null @@ -1,300 +0,0 @@ -OOPS tracing -============ - -.. note:: - - ``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original - format (from ``dmesg``, etc). Ignore any references in this or other docs to - "decoding the Oops" or "running it through ksymoops". - If you post an Oops from 2.6+ that has been run through ``ksymoops``, - people will just tell you to repost it. - -Quick Summary -------------- - -Find the Oops and send it to the maintainer of the kernel area that seems to be -involved with the problem. Don't worry too much about getting the wrong person. -If you are unsure send it to the person responsible for the code relevant to -what you were doing. If it occurs repeatably try and describe how to recreate -it. That's worth even more than the oops. - -If you are totally stumped as to whom to send the report, send it to -linux-kernel@vger.kernel.org. Thanks for your help in making Linux as -stable as humanly possible. - -Where is the Oops? ----------------------- - -Normally the Oops text is read from the kernel buffers by klogd and -handed to ``syslogd`` which writes it to a syslog file, typically -``/var/log/messages`` (depends on ``/etc/syslog.conf``). Sometimes ``klogd`` -dies, in which case you can run ``dmesg > file`` to read the data from the -kernel buffers and save it. Or you can ``cat /proc/kmsg > file``, however you -have to break in to stop the transfer, ``kmsg`` is a "never ending file". -If the machine has crashed so badly that you cannot enter commands or -the disk is not available then you have three options : - -(1) Hand copy the text from the screen and type it in after the machine - has restarted. Messy but it is the only option if you have not - planned for a crash. Alternatively, you can take a picture of - the screen with a digital camera - not nice, but better than - nothing. If the messages scroll off the top of the console, you - may find that booting with a higher resolution (eg, ``vga=791``) - will allow you to read more of the text. (Caveat: This needs ``vesafb``, - so won't help for 'early' oopses) - -(2) Boot with a serial console (see - :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`), - run a null modem to a second machine and capture the output there - using your favourite communication program. Minicom works well. - -(3) Use Kdump (see Documentation/kdump/kdump.txt), - extract the kernel ring buffer from old memory with using dmesg - gdbmacro in Documentation/kdump/gdbmacros.txt. - - -Full Information ----------------- - -.. note:: - - the message from Linus below applies to 2.4 kernel. I have preserved it - for historical reasons, and because some of the information in it still - applies. Especially, please ignore any references to ksymoops. - - :: - - From: Linus Torvalds <torvalds@osdl.org> - - How to track down an Oops.. [originally a mail to linux-kernel] - - The main trick is having 5 years of experience with those pesky oops - messages ;-) - -Actually, there are things you can do that make this easier. I have two -separate approaches:: - - gdb /usr/src/linux/vmlinux - gdb> disassemble <offending_function> - -That's the easy way to find the problem, at least if the bug-report is -well made (like this one was - run through ``ksymoops`` to get the -information of which function and the offset in the function that it -happened in). - -Oh, it helps if the report happens on a kernel that is compiled with the -same compiler and similar setups. - -The other thing to do is disassemble the "Code:" part of the bug report: -ksymoops will do this too with the correct tools, but if you don't have -the tools you can just do a silly program:: - - char str[] = "\xXX\xXX\xXX..."; - main(){} - -and compile it with ``gcc -g`` and then do ``disassemble str`` (where the ``XX`` -stuff are the values reported by the Oops - you can just cut-and-paste -and do a replace of spaces to ``\x`` - that's what I do, as I'm too lazy -to write a program to automate this all). - -Alternatively, you can use the shell script in ``scripts/decodecode``. -Its usage is:: - - decodecode < oops.txt - -The hex bytes that follow "Code:" may (in some architectures) have a series -of bytes that precede the current instruction pointer as well as bytes at and -following the current instruction pointer. In some cases, one instruction -byte or word is surrounded by ``<>`` or ``()``, as in ``<86>`` or ``(f00d)``. -These ``<>`` or ``()`` markings indicate the current instruction pointer. - -Example from i386, split into multiple lines for readability:: - - Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1 - 64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54 - 7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0 - -Finally, if you want to see where the code comes from, you can do:: - - cd /usr/src/linux - make fs/buffer.s # or whatever file the bug happened in - -and then you get a better idea of what happens than with the gdb -disassembly. - -Now, the trick is just then to combine all the data you have: the C -sources (and general knowledge of what it **should** do), the assembly -listing and the code disassembly (and additionally the register dump you -also get from the "oops" message - that can be useful to see **what** the -corrupted pointers were, and when you have the assembler listing you can -also match the other registers to whatever C expressions they were used -for). - -Essentially, you just look at what doesn't match (in this case it was the -"Code" disassembly that didn't match with what the compiler generated). -Then you need to find out **why** they don't match. Often it's simple - you -see that the code uses a NULL pointer and then you look at the code and -wonder how the NULL pointer got there, and if it's a valid thing to do -you just check against it.. - -Now, if somebody gets the idea that this is time-consuming and requires -some small amount of concentration, you're right. Which is why I will -mostly just ignore any panic reports that don't have the symbol table -info etc looked up: it simply gets too hard to look it up (I have some -programs to search for specific patterns in the kernel code segment, and -sometimes I have been able to look up those kinds of panics too, but -that really requires pretty good knowledge of the kernel just to be able -to pick out the right sequences etc..) - -**Sometimes** it happens that I just see the disassembled code sequence -from the panic, and I know immediately where it's coming from. That's when -I get worried that I've been doing this for too long ;-) - - Linus - - ---------------------------------------------------------------------------- - -Notes on Oops tracing with ``klogd`` ------------------------------------- - -In order to help Linus and the other kernel developers there has been -substantial support incorporated into ``klogd`` for processing protection -faults. In order to have full support for address resolution at least -version 1.3-pl3 of the ``sysklogd`` package should be used. - -When a protection fault occurs the ``klogd`` daemon automatically -translates important addresses in the kernel log messages to their -symbolic equivalents. This translated kernel message is then -forwarded through whatever reporting mechanism ``klogd`` is using. The -protection fault message can be simply cut out of the message files -and forwarded to the kernel developers. - -Two types of address resolution are performed by ``klogd``. The first is -static translation and the second is dynamic translation. Static -translation uses the System.map file in much the same manner that -ksymoops does. In order to do static translation the ``klogd`` daemon -must be able to find a system map file at daemon initialization time. -See the klogd man page for information on how ``klogd`` searches for map -files. - -Dynamic address translation is important when kernel loadable modules -are being used. Since memory for kernel modules is allocated from the -kernel's dynamic memory pools there are no fixed locations for either -the start of the module or for functions and symbols in the module. - -The kernel supports system calls which allow a program to determine -which modules are loaded and their location in memory. Using these -system calls the klogd daemon builds a symbol table which can be used -to debug a protection fault which occurs in a loadable kernel module. - -At the very minimum klogd will provide the name of the module which -generated the protection fault. There may be additional symbolic -information available if the developer of the loadable module chose to -export symbol information from the module. - -Since the kernel module environment can be dynamic there must be a -mechanism for notifying the ``klogd`` daemon when a change in module -environment occurs. There are command line options available which -allow klogd to signal the currently executing daemon that symbol -information should be refreshed. See the ``klogd`` manual page for more -information. - -A patch is included with the sysklogd distribution which modifies the -``modules-2.0.0`` package to automatically signal klogd whenever a module -is loaded or unloaded. Applying this patch provides essentially -seamless support for debugging protection faults which occur with -kernel loadable modules. - -The following is an example of a protection fault in a loadable module -processed by ``klogd``:: - - Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc - Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000 - Aug 29 09:51:01 blizard kernel: *pde = 00000000 - Aug 29 09:51:01 blizard kernel: Oops: 0002 - Aug 29 09:51:01 blizard kernel: CPU: 0 - Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868] - Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212 - Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c - Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c - Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 - Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000) - Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 - Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 - Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 - Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] - Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 - ---------------------------------------------------------------------------- - -:: - - Dr. G.W. Wettstein Oncology Research Div. Computing Facility - Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com - 820 4th St. N. - Fargo, ND 58122 - Phone: 701-234-7556 - - ---------------------------------------------------------------------------- - -Tainted kernels ---------------- - -Some oops reports contain the string **'Tainted: '** after the program -counter. This indicates that the kernel has been tainted by some -mechanism. The string is followed by a series of position-sensitive -characters, each representing a particular tainted value. - - 1) 'G' if all modules loaded have a GPL or compatible license, 'P' if - any proprietary module has been loaded. Modules without a - MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by - insmod as GPL compatible are assumed to be proprietary. - - 2) ``F`` if any module was force loaded by ``insmod -f``, ``' '`` if all - modules were loaded normally. - - 3) ``S`` if the oops occurred on an SMP kernel running on hardware that - hasn't been certified as safe to run multiprocessor. - Currently this occurs only on various Athlons that are not - SMP capable. - - 4) ``R`` if a module was force unloaded by ``rmmod -f``, ``' '`` if all - modules were unloaded normally. - - 5) ``M`` if any processor has reported a Machine Check Exception, - ``' '`` if no Machine Check Exceptions have occurred. - - 6) ``B`` if a page-release function has found a bad page reference or - some unexpected page flags. - - 7) ``U`` if a user or user application specifically requested that the - Tainted flag be set, ``' '`` otherwise. - - 8) ``D`` if the kernel has died recently, i.e. there was an OOPS or BUG. - - 9) ``A`` if the ACPI table has been overridden. - - 10) ``W`` if a warning has previously been issued by the kernel. - (Though some warnings may set more specific taint flags.) - - 11) ``C`` if a staging driver has been loaded. - - 12) ``I`` if the kernel is working around a severe bug in the platform - firmware (BIOS or similar). - - 13) ``O`` if an externally-built ("out-of-tree") module has been loaded. - - 14) ``E`` if an unsigned module has been loaded in a kernel supporting - module signature. - - 15) ``L`` if a soft lockup has previously occurred on the system. - - 16) ``K`` if the kernel has been live patched. - -The primary reason for the **'Tainted: '** string is to tell kernel -debuggers if this is a clean kernel or if anything unusual has -occurred. Tainting is permanent: even if an offending module is -unloaded, the tainted value remains to indicate that the kernel is not -trustworthy. diff --git a/Documentation/admin-guide/tainted-kernels.rst b/Documentation/admin-guide/tainted-kernels.rst new file mode 100644 index 000000000000..1df03b5cb02f --- /dev/null +++ b/Documentation/admin-guide/tainted-kernels.rst @@ -0,0 +1,59 @@ +Tainted kernels +--------------- + +Some oops reports contain the string **'Tainted: '** after the program +counter. This indicates that the kernel has been tainted by some +mechanism. The string is followed by a series of position-sensitive +characters, each representing a particular tainted value. + + 1) 'G' if all modules loaded have a GPL or compatible license, 'P' if + any proprietary module has been loaded. Modules without a + MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by + insmod as GPL compatible are assumed to be proprietary. + + 2) ``F`` if any module was force loaded by ``insmod -f``, ``' '`` if all + modules were loaded normally. + + 3) ``S`` if the oops occurred on an SMP kernel running on hardware that + hasn't been certified as safe to run multiprocessor. + Currently this occurs only on various Athlons that are not + SMP capable. + + 4) ``R`` if a module was force unloaded by ``rmmod -f``, ``' '`` if all + modules were unloaded normally. + + 5) ``M`` if any processor has reported a Machine Check Exception, + ``' '`` if no Machine Check Exceptions have occurred. + + 6) ``B`` if a page-release function has found a bad page reference or + some unexpected page flags. + + 7) ``U`` if a user or user application specifically requested that the + Tainted flag be set, ``' '`` otherwise. + + 8) ``D`` if the kernel has died recently, i.e. there was an OOPS or BUG. + + 9) ``A`` if the ACPI table has been overridden. + + 10) ``W`` if a warning has previously been issued by the kernel. + (Though some warnings may set more specific taint flags.) + + 11) ``C`` if a staging driver has been loaded. + + 12) ``I`` if the kernel is working around a severe bug in the platform + firmware (BIOS or similar). + + 13) ``O`` if an externally-built ("out-of-tree") module has been loaded. + + 14) ``E`` if an unsigned module has been loaded in a kernel supporting + module signature. + + 15) ``L`` if a soft lockup has previously occurred on the system. + + 16) ``K`` if the kernel has been live patched. + +The primary reason for the **'Tainted: '** string is to tell kernel +debuggers if this is a clean kernel or if anything unusual has +occurred. Tainting is permanent: even if an offending module is +unloaded, the tainted value remains to indicate that the kernel is not +trustworthy. |