summaryrefslogtreecommitdiff
path: root/scripts/coccicheck
AgeCommit message (Collapse)Author
2017-11-02License cleanup: add SPDX GPL-2.0 license identifier to files with no licenseGreg Kroah-Hartman
Many source files in the tree are missing licensing information, which makes it harder for compliance tools to determine the correct license. By default all files without license information are under the default license of the kernel, which is GPL version 2. Update the files which contain no license information with the 'GPL-2.0' SPDX license identifier. The SPDX identifier is a legally binding shorthand, which can be used instead of the full boiler plate text. This patch is based on work done by Thomas Gleixner and Kate Stewart and Philippe Ombredanne. How this work was done: Patches were generated and checked against linux-4.14-rc6 for a subset of the use cases: - file had no licensing information it it. - file was a */uapi/* one with no licensing information in it, - file was a */uapi/* one with existing licensing information, Further patches will be generated in subsequent months to fix up cases where non-standard license headers were used, and references to license had to be inferred by heuristics based on keywords. The analysis to determine which SPDX License Identifier to be applied to a file was done in a spreadsheet of side by side results from of the output of two independent scanners (ScanCode & Windriver) producing SPDX tag:value files created by Philippe Ombredanne. Philippe prepared the base worksheet, and did an initial spot review of a few 1000 files. The 4.13 kernel was the starting point of the analysis with 60,537 files assessed. Kate Stewart did a file by file comparison of the scanner results in the spreadsheet to determine which SPDX license identifier(s) to be applied to the file. She confirmed any determination that was not immediately clear with lawyers working with the Linux Foundation. Criteria used to select files for SPDX license identifier tagging was: - Files considered eligible had to be source code files. - Make and config files were included as candidates if they contained >5 lines of source - File already had some variant of a license header in it (even if <5 lines). All documentation files were explicitly excluded. The following heuristics were used to determine which SPDX license identifiers to apply. - when both scanners couldn't find any license traces, file was considered to have no license information in it, and the top level COPYING file license applied. For non */uapi/* files that summary was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 11139 and resulted in the first patch in this series. If that file was a */uapi/* path one, it was "GPL-2.0 WITH Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 WITH Linux-syscall-note 930 and resulted in the second patch in this series. - if a file had some form of licensing information in it, and was one of the */uapi/* ones, it was denoted with the Linux-syscall-note if any GPL family license was found in the file or had no licensing in it (per prior point). Results summary: SPDX license identifier # files ---------------------------------------------------|------ GPL-2.0 WITH Linux-syscall-note 270 GPL-2.0+ WITH Linux-syscall-note 169 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17 LGPL-2.1+ WITH Linux-syscall-note 15 GPL-1.0+ WITH Linux-syscall-note 14 ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5 LGPL-2.0+ WITH Linux-syscall-note 4 LGPL-2.1 WITH Linux-syscall-note 3 ((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3 ((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1 and that resulted in the third patch in this series. - when the two scanners agreed on the detected license(s), that became the concluded license(s). - when there was disagreement between the two scanners (one detected a license but the other didn't, or they both detected different licenses) a manual inspection of the file occurred. - In most cases a manual inspection of the information in the file resulted in a clear resolution of the license that should apply (and which scanner probably needed to revisit its heuristics). - When it was not immediately clear, the license identifier was confirmed with lawyers working with the Linux Foundation. - If there was any question as to the appropriate license identifier, the file was flagged for further research and to be revisited later in time. In total, over 70 hours of logged manual review was done on the spreadsheet to determine the SPDX license identifiers to apply to the source files by Kate, Philippe, Thomas and, in some cases, confirmation by lawyers working with the Linux Foundation. Kate also obtained a third independent scan of the 4.13 code base from FOSSology, and compared selected files where the other two scanners disagreed against that SPDX file, to see if there was new insights. The Windriver scanner is based on an older version of FOSSology in part, so they are related. Thomas did random spot checks in about 500 files from the spreadsheets for the uapi headers and agreed with SPDX license identifier in the files he inspected. For the non-uapi files Thomas did random spot checks in about 15000 files. In initial set of patches against 4.14-rc6, 3 files were found to have copy/paste license identifier errors, and have been fixed to reflect the correct identifier. Additionally Philippe spent 10 hours this week doing a detailed manual inspection and review of the 12,461 patched files from the initial patch version early this week with: - a full scancode scan run, collecting the matched texts, detected license ids and scores - reviewing anything where there was a license detected (about 500+ files) to ensure that the applied SPDX license was correct - reviewing anything where there was no detection but the patch license was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied SPDX license was correct This produced a worksheet with 20 files needing minor correction. This worksheet was then exported into 3 different .csv files for the different types of files to be modified. These .csv files were then reviewed by Greg. Thomas wrote a script to parse the csv files and add the proper SPDX tag to the file, in the format that the file expected. This script was further refined by Greg based on the output to detect more types of files automatically and to distinguish between header and source .c files (which need different comment types.) Finally Greg ran the script using the .csv files to generate the patches. Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com> Reviewed-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2016-10-11scripts/coccicheck: Update reference for the corresponding documentationMarkus Elfring
Use the current name (in a comment at the beginning of this script) for the file which was converted to the documentation format "reStructuredText" in August 2016. Fixes: 4b9033a33494 ("docs: sphinxify coccinelle.txt and add it to dev-tools") Signed-off-by: Markus Elfring <elfring@users.sourceforge.net> Acked-by: Julia Lawall <julia.lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: refer to Documentation/coccinelle.txt and wikiLuis R. Rodriguez
Refer to the Documentation/coccinelle.txt and supplemental documentation on the wiki: https://bottest.wiki.kernel.org/coccicheck This page shall always refer to the linux-next iteration of scripts/coccicheck. v4: only refer to the wiki as supplemental documentation, and also update Documentation/coccinelle.txt. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: add support for requring a coccinelle versionLuis R. Rodriguez
Enable Coccinelle SmPL patches to require a specific version of Coccinelle. In the event that the version does not match we just inform the user, if the user asked to go through all SmPL patches we just inform them of the need for a new version of coccinelle for the SmPL patch and continue on with the rest. This uses the simple kernel scripts/ld-version.sh to create a weight on the version provided by spatch. The -dirty attribute is ignored if supplied, the benefit of scripts/ld-version.sh is it has a long history and well tested. While at it, document the // Options stuff as well. v4: Document // Options and // Requires as well on Documentation/coccinelle.txt. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: replace --very-quiet with --quiet when debuggingLuis R. Rodriguez
When debugging (using --profile or --show-trying) you want to avoid supressing output, use --quiet instead. While at it, extend documentation for SPFLAGS use. For instance one can use: $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci $ make coccicheck DEBUG_FILE="poo.err" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c Expand Documentation/coccinelle.txt as well. v4: expand Documentation/coccinelle.txt v3: rebased, resolve conflicts, expand Documentation/coccinelle.txt v2: use egrep instead of the *"=--option"* check, this doesn't work for disjunctions. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Julia Lawall <julia.lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: add support for DEBUG_FILELuis R. Rodriguez
Enable to capture stderr via a DEBUG_FILE variable passed to coccicheck. You can now do: $ rm -f cocci.err $ export COCCI=scripts/coccinelle/free/kfree.cocci $ make coccicheck MODE=report DEBUG_FILE=cocci.err ... $ cat cocci.err This will be come more useful once we add support to use more things which would go into stderr, such as profiling. That will be done separately in another commit. Expand Documentation/coccinelle.txt with details. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: enable parmap supportLuis R. Rodriguez
Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: make SPFLAGS more usefulLuis R. Rodriguez
SPFLAGS is set early, it means that any heuristics done on coccicheck cannot be overridden currently. Move SPFLAGS after OPTIONS and set this at the end. This lets you override any heuristics as coccinelle treats conflicts by only listening to the last option that makes sense. v3: this patch was added in the v3 series v4: Update Documentation/coccinelle.txt explaining how SPFLAGS works as well. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-07-22coccicheck: move spatch binary check upLuis R. Rodriguez
This has no functional changes. This is being done to enable us to later use spatch binary for some flag checking for certain features early on. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-20coccicheck: Allow for overriding spatch flagsDeepa Dinamani
Documentation/coccinelle.txt suggests using the SPFLAGS make variable to pass additional options to spatch. Reorder the way SPFLAGS is added to FLAGS, to allow for options in the SPFLAGS to override the default --very-quiet option. Similarly, rearrage the FLAGS for org or report mode. This allows for overriding of the default --no-show-diff option through SPFLAGS. Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com> Cc: Gilles Muller <Gilles.Muller@lip6.fr> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Acked-by: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-19coccicheck: Fix missing 0 index in kill loopKees Cook
By default, "seq" counts from 1, but processes were starting counting from 0, so when interrupted, coccicheck would leave the 0th process running. Signed-off-by: Kees Cook <keescook@chromium.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Acked-by: Julia Lawall <julia.lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2015-10-26Coccinelle: fix incorrect -include option transformationAndrzej Hajda
kbuild/gcc uses -include option to include files and -I to provide paths for #include <> directive. In case of spatch latter option should be prefixed with two -. Signed-off-by: Andrzej Hajda <a.hajda@samsung.com> Acked-by: Julia Lawall <julia.lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03Coccinelle: Update information about the minimal version requiredNicolas Palix
The naming convention of options has changed one year ago. The options have been recently updated in the cocci file and in scripts/coccicheck. This patch also adds this information in the documentation. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03Coccinelle: Update the options used to the new option schemeNicolas Palix
spatch has changed its option scheme. E.g., --no_show_diff is now --no-show-diff This patch updates: - scripts/coccicheck - Semantic patches under scripts/coccinelle/ Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-07-03coccicheck: span checks across CPUsKees Cook
This adds parallelism by default to the "coccicheck" target using spatch's "-max" and "-index" arguments. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-06-14Coccinelle: Make 'report' the default modeNicolas Palix
It appears that the 'report' mode is the one always provided by the semantic patches included in the kernel. It is thus more natural to select it by default. The 'chain' mode is however kept and the 'patch' mode is still the first tried in that case. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08Coccinelle: Fix patch output when coccicheck is used with M= and C=Nicolas Palix
When the M variable is used, the -patch option should be given to spatch. This patch fixes the case where C is used. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08Coccinelle: Add support to the SPFLAGS variableNicolas Palix
The SPFLAGS variable allows to pass additional options to spatch, e.g. -use_glimpse. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08Coccinelle: Cleanup the setting of the FLAGS and OPTIONS variablesNicolas Palix
The FLAGS variable is factorized independently of the ONLINE mode. The OPTIONS variable is now based on LINUXINCLUDE and explicit includes are thus removed. The format of the -I option differs between gcc and spatch. The COCCIINCLUDE is used to adapt the format. This rewritting needs bash. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-04-08Coccinelle: Restore coccicheck verbosity in ONLINE mode (C=1 or C=2)Nicolas Palix
A recent patch have introduce the VERBOSE variable and comments now depend on it. However, the message printed for each cocci file such not be printed when the ONLINE mode is active, whatever is the value of VERBOSE. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-02-22coccicheck: Allow to show the executed command lineBernd Schubert
On my system one of the tests failed with "Fatal error: exception Failure("No OCaml compiler found! Install either ocamlopt or ocamlopt.opt")". Investigating such issues is easier if the executed command line is being shown. Signed-off-by: Bernd Schubert <bernd.schubert@itwm.fraunhofer.de> CC: Julia Lawall <Julia.Lawall@lip6.fr> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2013-02-22coccicheck: Allow the user to give a V= (verbose) argumentBernd Schubert
Do not run with verbosity on/off depending on the ONLINE variable, which gets set with C=1 or C=2, but allow the user to set the verbosity using kernel default make V= paramemter. Verbosity is off by default now. Signed-off-by: Bernd Schubert <bernd.schubert@itwm.fraunhofer.de> CC: Julia Lawall <Julia.Lawall@lip6.fr> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-10-11coccicheck: Add the rep+ctxt modeNicolas Palix
This adds a 'rep+ctxt' mode which prints the warning message followed by the context. Signed-off-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-02-24coccicheck: change handling of C={1,2} when M= is setGreg Dietsche
This patch reverts a portion of d0bc1fb4 so that coccicheck will work properly when C=1 or C=2. Reported-and-tested-by: Brice Goglin <Brice.Goglin@inria.fr> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu> Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2012-01-14coccicheck: add M= option to control which dir is processedGreg Dietsche
Examples: make coccicheck M=drivers/net/wireless/ make coccicheck SUBDIRS=drivers/net/wireless/ Version 2: fix patch file names when using M= tell coccinelle where the include files are Version 3: Add second include option to support out of tree development Fix error message Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu> Acked-by: Julia Lawall <julia.lawall@lip6.fr> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-28Coccinelle: Add contextual messageNicolas Palix
Change the message displayed to the user according to the current mode used. Fix trailing white-space and spelling Signed-off-by: Nicolas Palix <npalix.work@gmail.com> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13Coccinelle: Use the -no_show_diff option for org and report modeNicolas Palix
This allows to write the semantic patches with code sharing for the matching parts. Signed-off-by: Nicolas Palix <npalix.work@gmail.com> Signed-off-by: Julia Lawall <julia@diku.dk> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13Coccinelle: Add a new mode named 'chain'Nicolas Palix
spatch now returns -1 when a virtual rule (given with -D on the command line) is not defined in the semantic patch. Using this spatch feature, coccicheck is now tries several modes by default, in the order: patch, report, context, org Signed-off-by: Nicolas Palix <npalix.work@gmail.com> Signed-off-by: Julia Lawall <julia@diku.dk> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-10-13Coccinelle: Improve user information with a new kind of commentNicolas Palix
Improve user information with a new kind of comment about semantic patch output. Fix spelling. Signed-off-by: Nicolas Palix <npalix.work@gmail.com> Signed-off-by: Julia Lawall <julia@diku.dk> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-06-28Add support for the C variable in the coccicheck scriptNicolas Palix
This patch makes it possible to use the Coccinelle checker with the C variable of the build system. To check only newly edited code, the following command may be used: 'make C={1,2} CHECK="scripts/coccicheck"' This runs every semantic patch in scripts/coccinelle by default. The COCCI variable may additionally be used to only apply a single semantic patch. Signed-off-by: Nicolas Palix <npalix@diku.dk> Signed-off-by: Julia Lawall <julia@diku.dk> Signed-off-by: Michal Marek <mmarek@suse.cz>
2010-06-12Add a target to use the Coccinelle checkerNicolas Palix
A 'coccicheck' target is added. It can be called with four different modes. Each one generates a different kind of output, i.e. context, patch, org, report, according to the corresponding mode to be activated. The new target calls the 'coccicheck' front-end in the 'scripts' directory with the MODE argument. Every SmPL file in the subdirectories of 'scripts/coccinelle' is then given to the front-end and applied to the entire source tree. The four modes behave as follows: 'report' generates a list in the following format: file:line:column-column: message 'patch' proposes a fix, when possible. 'context' highlights lines of interest and their context in a diff-like style. Lines of interest are indicated with '-'. 'org' generates a report in the Org mode format of Emacs. Signed-off-by: Nicolas Palix <npalix@diku.dk> Signed-off-by: Julia Lawall <julia@diku.dk> Acked-by: Sam Ravnborg <sam@ravnborg.org> Acked-by: Joerg Roedel <joerg.roedel@amd.com> Signed-off-by: Michal Marek <mmarek@suse.cz>