diff options
Diffstat (limited to 'Documentation/admin-guide/oops-tracing.rst')
| -rw-r--r-- | Documentation/admin-guide/oops-tracing.rst | 241 |
1 files changed, 0 insertions, 241 deletions
diff --git a/Documentation/admin-guide/oops-tracing.rst b/Documentation/admin-guide/oops-tracing.rst deleted file mode 100644 index 1f5e2b716631..000000000000 --- a/Documentation/admin-guide/oops-tracing.rst +++ /dev/null @@ -1,241 +0,0 @@ -OOPS tracing -============ - -.. note:: - - ``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original - format (from ``dmesg``, etc). Ignore any references in this or other docs to - "decoding the Oops" or "running it through ksymoops". - If you post an Oops from 2.6+ that has been run through ``ksymoops``, - people will just tell you to repost it. - -Quick Summary -------------- - -Find the Oops and send it to the maintainer of the kernel area that seems to be -involved with the problem. Don't worry too much about getting the wrong person. -If you are unsure send it to the person responsible for the code relevant to -what you were doing. If it occurs repeatably try and describe how to recreate -it. That's worth even more than the oops. - -If you are totally stumped as to whom to send the report, send it to -linux-kernel@vger.kernel.org. Thanks for your help in making Linux as -stable as humanly possible. - -Where is the Oops? ----------------------- - -Normally the Oops text is read from the kernel buffers by klogd and -handed to ``syslogd`` which writes it to a syslog file, typically -``/var/log/messages`` (depends on ``/etc/syslog.conf``). Sometimes ``klogd`` -dies, in which case you can run ``dmesg > file`` to read the data from the -kernel buffers and save it. Or you can ``cat /proc/kmsg > file``, however you -have to break in to stop the transfer, ``kmsg`` is a "never ending file". -If the machine has crashed so badly that you cannot enter commands or -the disk is not available then you have three options : - -(1) Hand copy the text from the screen and type it in after the machine - has restarted. Messy but it is the only option if you have not - planned for a crash. Alternatively, you can take a picture of - the screen with a digital camera - not nice, but better than - nothing. If the messages scroll off the top of the console, you - may find that booting with a higher resolution (eg, ``vga=791``) - will allow you to read more of the text. (Caveat: This needs ``vesafb``, - so won't help for 'early' oopses) - -(2) Boot with a serial console (see - :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`), - run a null modem to a second machine and capture the output there - using your favourite communication program. Minicom works well. - -(3) Use Kdump (see Documentation/kdump/kdump.txt), - extract the kernel ring buffer from old memory with using dmesg - gdbmacro in Documentation/kdump/gdbmacros.txt. - - -Full Information ----------------- - -.. note:: - - the message from Linus below applies to 2.4 kernel. I have preserved it - for historical reasons, and because some of the information in it still - applies. Especially, please ignore any references to ksymoops. - - :: - - From: Linus Torvalds <torvalds@osdl.org> - - How to track down an Oops.. [originally a mail to linux-kernel] - - The main trick is having 5 years of experience with those pesky oops - messages ;-) - -Actually, there are things you can do that make this easier. I have two -separate approaches:: - - gdb /usr/src/linux/vmlinux - gdb> disassemble <offending_function> - -That's the easy way to find the problem, at least if the bug-report is -well made (like this one was - run through ``ksymoops`` to get the -information of which function and the offset in the function that it -happened in). - -Oh, it helps if the report happens on a kernel that is compiled with the -same compiler and similar setups. - -The other thing to do is disassemble the "Code:" part of the bug report: -ksymoops will do this too with the correct tools, but if you don't have -the tools you can just do a silly program:: - - char str[] = "\xXX\xXX\xXX..."; - main(){} - -and compile it with ``gcc -g`` and then do ``disassemble str`` (where the ``XX`` -stuff are the values reported by the Oops - you can just cut-and-paste -and do a replace of spaces to ``\x`` - that's what I do, as I'm too lazy -to write a program to automate this all). - -Alternatively, you can use the shell script in ``scripts/decodecode``. -Its usage is:: - - decodecode < oops.txt - -The hex bytes that follow "Code:" may (in some architectures) have a series -of bytes that precede the current instruction pointer as well as bytes at and -following the current instruction pointer. In some cases, one instruction -byte or word is surrounded by ``<>`` or ``()``, as in ``<86>`` or ``(f00d)``. -These ``<>`` or ``()`` markings indicate the current instruction pointer. - -Example from i386, split into multiple lines for readability:: - - Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1 - 64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54 - 7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0 - -Finally, if you want to see where the code comes from, you can do:: - - cd /usr/src/linux - make fs/buffer.s # or whatever file the bug happened in - -and then you get a better idea of what happens than with the gdb -disassembly. - -Now, the trick is just then to combine all the data you have: the C -sources (and general knowledge of what it **should** do), the assembly -listing and the code disassembly (and additionally the register dump you -also get from the "oops" message - that can be useful to see **what** the -corrupted pointers were, and when you have the assembler listing you can -also match the other registers to whatever C expressions they were used -for). - -Essentially, you just look at what doesn't match (in this case it was the -"Code" disassembly that didn't match with what the compiler generated). -Then you need to find out **why** they don't match. Often it's simple - you -see that the code uses a NULL pointer and then you look at the code and -wonder how the NULL pointer got there, and if it's a valid thing to do -you just check against it.. - -Now, if somebody gets the idea that this is time-consuming and requires -some small amount of concentration, you're right. Which is why I will -mostly just ignore any panic reports that don't have the symbol table -info etc looked up: it simply gets too hard to look it up (I have some -programs to search for specific patterns in the kernel code segment, and -sometimes I have been able to look up those kinds of panics too, but -that really requires pretty good knowledge of the kernel just to be able -to pick out the right sequences etc..) - -**Sometimes** it happens that I just see the disassembled code sequence -from the panic, and I know immediately where it's coming from. That's when -I get worried that I've been doing this for too long ;-) - - Linus - - ---------------------------------------------------------------------------- - -Notes on Oops tracing with ``klogd`` ------------------------------------- - -In order to help Linus and the other kernel developers there has been -substantial support incorporated into ``klogd`` for processing protection -faults. In order to have full support for address resolution at least -version 1.3-pl3 of the ``sysklogd`` package should be used. - -When a protection fault occurs the ``klogd`` daemon automatically -translates important addresses in the kernel log messages to their -symbolic equivalents. This translated kernel message is then -forwarded through whatever reporting mechanism ``klogd`` is using. The -protection fault message can be simply cut out of the message files -and forwarded to the kernel developers. - -Two types of address resolution are performed by ``klogd``. The first is -static translation and the second is dynamic translation. Static -translation uses the System.map file in much the same manner that -ksymoops does. In order to do static translation the ``klogd`` daemon -must be able to find a system map file at daemon initialization time. -See the klogd man page for information on how ``klogd`` searches for map -files. - -Dynamic address translation is important when kernel loadable modules -are being used. Since memory for kernel modules is allocated from the -kernel's dynamic memory pools there are no fixed locations for either -the start of the module or for functions and symbols in the module. - -The kernel supports system calls which allow a program to determine -which modules are loaded and their location in memory. Using these -system calls the klogd daemon builds a symbol table which can be used -to debug a protection fault which occurs in a loadable kernel module. - -At the very minimum klogd will provide the name of the module which -generated the protection fault. There may be additional symbolic -information available if the developer of the loadable module chose to -export symbol information from the module. - -Since the kernel module environment can be dynamic there must be a -mechanism for notifying the ``klogd`` daemon when a change in module -environment occurs. There are command line options available which -allow klogd to signal the currently executing daemon that symbol -information should be refreshed. See the ``klogd`` manual page for more -information. - -A patch is included with the sysklogd distribution which modifies the -``modules-2.0.0`` package to automatically signal klogd whenever a module -is loaded or unloaded. Applying this patch provides essentially -seamless support for debugging protection faults which occur with -kernel loadable modules. - -The following is an example of a protection fault in a loadable module -processed by ``klogd``:: - - Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc - Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000 - Aug 29 09:51:01 blizard kernel: *pde = 00000000 - Aug 29 09:51:01 blizard kernel: Oops: 0002 - Aug 29 09:51:01 blizard kernel: CPU: 0 - Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868] - Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212 - Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c - Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c - Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 - Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000) - Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 - Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 - Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 - Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] - Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 - ---------------------------------------------------------------------------- - -:: - - Dr. G.W. Wettstein Oncology Research Div. Computing Facility - Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com - 820 4th St. N. - Fargo, ND 58122 - Phone: 701-234-7556 - - ---------------------------------------------------------------------------- - |