| Age | Commit message (Collapse) | Author |
|---|
|
This commit adds a kvm-again.sh script that, given the results directory
of a torture-test run, re-runs that test. This means that the kernels
need not be rebuilt, but it also is a step towards running torture tests
on remote systems.
This commit also adds a kvm-test-1-run-batch.sh script that runs one
batch out of the torture test. The idea is to copy a results directory
tree to remote systems, then use kvm-test-1-run-batch.sh to run batches
on these systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit creates a "batches" file in the res/$ds directory, where $ds
is the datestamp. This file contains the batches and the number of CPUs,
for example:
1 TREE03 16
1 SRCU-P 8
2 TREE07 16
2 TREE01 8
3 TREE02 8
3 TREE04 8
3 TREE05 8
4 SRCU-N 4
4 TRACE01 4
4 TRACE02 4
4 RUDE01 2
4 RUDE01.2 2
4 TASKS01 2
4 TASKS03 2
4 SRCU-t 1
4 SRCU-u 1
4 TASKS02 1
4 TINY01 1
5 TINY02 1
5 TREE09 1
The first column is the batch number, the second the scenario number
(possibly suffixed by a repetition number, as in "RUDE01.2"), and the
third is the number of CPUs required by that scenario. The last line
shows the number of CPUs expected by this batch file, which allows
the run to be re-batched if a different number of CPUs is available.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Although it might be unlikely that someone would name a scenario
"TORTURE_SUITE", they are within their rights to do so. This script
therefore renames the "TORTURE_SUITE" file in the top-level date-stamped
directory within "res" to "torture_suite" to avoid this name collision.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit enforces the defacto restriction on scenario names, which is
that they contain neither "/", ".", nor lowercase alphabetic characters.
This restriction avoids collisions between scenario names and the torture
scripting's files and directories.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The convention that scenario names are all uppercase has two exceptions,
SRCU-t and SRCU-u. This commit therefore renames them to SRCU-T and
SRCU-U, respectively, to bring them in line with this convention. This in
turn permits tighter argument checking in the torture-test scripting.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The cpus2use.sh script complains if the mpstat command is not available,
and instead uses all available CPUs. Unfortunately, this complaint
goes to stdout, where it confuses invokers who expect a single number.
This commit removes this error message in order to avoid this confusion.
The tendency of late has been to give rcutorture a full system, so this
should not cause issues.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit records the process IDs of the kvm-test-1-run.sh and
kvm-test-1-run-qemu.sh scripts to ease monitoring of remotely running
instances of these scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Distributed runs of rcutorture will need to start and stop jittering on
the remote hosts, which means that the commands must be communicated to
those hosts. The commit therefore causes kvm.sh to place these commands
in new TORTURE_JITTER_START and TORTURE_JITTER_STOP environment variables
to communicate them to the scripts that will set this up. In addition,
this commit causes kvm-test-1-run.sh to append these commands to each
generated qemu-cmd file, which allows any remotely executing script to
extract the needed commands from this file.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, kvm-test-1-run.sh both builds and runs an rcutorture kernel,
which is inconvenient when it is necessary to re-run an old run or to
carry out a run on a remote system. This commit therefore extracts the
portion of kvm-test-1-run.sh that invoke qemu to actually run rcutorture
and places it in kvm-test-1-run-qemu.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When re-running old rcutorture builds, if the original run involved
gdb, the re-run also needs to do so. This commit therefore records the
TORTURE_KCONFIG_GDB_ARG environment variable into the qemu-cmd file so
that the re-run can access it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit creates jitterstart.sh and jitterstop.sh scripts that handle
the starting and stopping of the jitter.sh scripts. These must be sourced
using the bash "." command to allow the generated script to wait on the
backgrounded jitter.sh scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Remote rcutorture testing requires that jitter.sh continue to be
invoked from the generated script for local runs, but that it instead
be invoked on the remote system for distributed runs. This argues
for common jitterstart and jitterstop scripts. But it would be good
for jitterstart and jitterstop to control the name and location of the
"jittering" file, while continuing to have the duration controlled by
the caller of these new scripts.
This commit therefore reverses the order of the jittering and duration
parameters for jitter.sh, so that the jittering parameter precedes the
duration parameter.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Now that there is a reliable way to convince the jitter.sh scripts to
stop, the jitter_pids file is not needed, nor is the code that kills all
the PIDs contained in this file. This commit therefore eliminates this
file and the code using it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, jitter.sh execution is controlled by a time limit and by the
"kill" command. The former allowed jitter.sh to run uselessly past
the end of a set of runs that panicked during boot, and the latter is
vulnerable to PID reuse. This commit therefore introduces a "jittering"
file in the date-stamp directory within "res" that must be present for
the jitter.sh scripts to continue executing. The time limit is still
in place in order to avoid disturbing runs featuring large trace dumps,
but the removal of the "jittering" file handles the panic-during-boot
scenario without relying on PIDs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, the script generated by kvm.sh does a "wait" to wait on both
the current batch's guest OSes and any jitter.sh scripts. This works,
but makes it hard to abstract the jittering so that common code can be
used for both local and distributed runs. This commit therefore uses
"build.run" files in scenario directories, and these files are removed
after the corresponding scenario's guest OS has completed.
Note that --build-only runs do not create build.run files because they
also do not create guest OSes and do not run any jitter.sh scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently the bN.ready and bN.wait files are placed in the
rcutorture directory, which really is not at all a good place
for run-specific files. This commit therefore renames these
files to build.ready and build.wait and then moves them into the
scenario directories within the "res" directory, for example, into
tools/testing/selftests/rcutorture/res/2021.02.10-15.08.23/TINY01.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Given large numbers of threads, the quantity of torture-test output is
sufficient to sometimes result in RCU CPU stall warnings. The probability
of these stall warnings was greatly reduced by batching the output,
but the warnings were not eliminated. However, the actual test only
depends on console output that is printed even when refscale.verbose=0.
This commit therefore causes this test to run with refscale.verbose=0.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Given large numbers of threads, the quantity of torture-test output is
sufficient to sometimes result in RCU CPU stall warnings. The probability
of these stall warnings was greatly reduced by batching the output,
but the warnings were not eliminated. However, the actual test only
depends on console output that is printed even when rcuscale.verbose=0.
This commit therefore causes this test to run with rcuscale.verbose=0.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The testid.txt file was intended for occasional in extremis use, but
now that the new "bare-metal" file references it, it might see more use.
This commit therefore labels sections of output and adds spacing to make
it easier to see what needs to be done to make a bare-metal build tree
match an rcutorture build tree.
Of course, you can avoid this whole issue by building your bare-metal
kernel in the same directory in which you ran rcutorture, but that might
not always be an option.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In some environments, the torture-testing use of virtualization is
inconvenient. In such cases, the modprobe and rmmod commands may be used
to do torture testing, but significant setup is required to build, boot,
and modprobe a kernel so as to match a given torture-test scenario.
This commit therefore creates a "bare-metal" file in each results
directory containing steps to run the corresponding scenario using the
modprobe command on bare metal. For example, the contents of this file
after using kvm.sh to build an rcutorture TREE01 kernel, perhaps with
the --buildonly argument, is as follows:
To run this scenario on bare metal:
1. Set your bare-metal build tree to the state shown in this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/testid.txt
2. Update your bare-metal build tree's .config based on this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/ConfigFragment
3. Make the bare-metal kernel's build system aware of your .config updates:
$ yes "" | make oldconfig
4. Build your bare-metal kernel.
5. Boot your bare-metal kernel with the following parameters:
maxcpus=8 nr_cpus=43 rcutree.gp_preinit_delay=3 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcu_nocbs=0-1,3-7
6. Start the test with the following command:
$ modprobe rcutorture nocbs_nthreads=8 nocbs_toggle=1000 fwd_progress=0 onoff_interval=1000 onoff_holdoff=30 n_barrier_cbs=4 stat_interval=15 shutdown_secs=120 test_no_idle_hz=1 verbose=1
7. After some time, end the test with the following command:
$ rmmod rcutorture
8. Copy your bare-metal kernel's .config file, overwriting this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/.config
9. Copy the console output from just before the modprobe to just after
the rmmod into this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/console.log
10. Check for runtime errors using the following command:
$ tools/testing/selftests/rcutorture/bin/kvm-recheck.sh /home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Yes, I do recall a time when 512MB of memory was a lot of mass storage,
much less main memory, but the rcuscale kvfree_rcu() testing invoked by
torture.sh can sometimes exceed it on large systems, resulting in OOM.
This commit therefore causes torture.sh to pase the "--memory 1G"
argument to kvm.sh to reserve a full gigabyte for this purpose.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
If the build fails when running multiple instances of a given rcutorture
scenario, for example, using the kvm.sh --configs "8*RUDE01" argument,
the build will be rerun an additional seven times. This is in some sense
correct, but it can waste significant time. This commit therefore checks
for a prior failed build and simply copies over that build's output.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The current jitter.sh script expects cpumask bits to fit into whatever
the awk interpreter uses for an integer, which clearly does not hold for
even medium-sized systems these days. This means that on a large system,
only the first 32 or 64 CPUs (depending) are subjected to jitter.sh
CPU-time perturbations. This commit therefore computes a given CPU's
cpumask using text manipulation rather than arithmetic shifts.
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
TREE03 tests RCU priority boosting, which is a real-time feature.
It would also be good if it tested something closer to what is
actually used by the real-time folks. This commit therefore adds
tree.use_softirq=0 to the TREE03 kernel boot parameters in TREE03.boot.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit uses the shiny new "all" and "N" cpumask options to decouple
the "nohz_full" and "rcu_nocbs" kernel boot parameters in the TREE04.boot
and TREE08.boot files from the CONFIG_NR_CPUS options in the TREE04 and
TREE08 files.
Reported-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
Pull nolibc fixes from Paul E. McKenney.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
'kfree_rcu.2021.01.04a', 'mmdumpobj.2021.01.22a', 'nocb.2021.01.06a', 'rt.2021.01.04a', 'stall.2021.01.06a', 'torture.2021.01.12a' and 'tortureall.2021.01.06a' into HEAD
doc.2021.01.06a: Documentation updates.
fixes.2021.01.04b: Miscellaneous fixes.
kfree_rcu.2021.01.04a: kfree_rcu() updates.
mmdumpobj.2021.01.22a: Dump allocation point for memory blocks.
nocb.2021.01.06a: RCU callback offload updates and cblist segment lengths.
rt.2021.01.04a: Real-time updates.
stall.2021.01.06a: RCU CPU stall warning updates.
torture.2021.01.12a: Torture-test updates and polling SRCU grace-period API.
tortureall.2021.01.06a: Torture-test script updates.
|
|
The -lgcc command-line argument is placed poorly in the build options,
which can result in build failures, for exapmle, on ARM when uidiv()
is required. This commit therefore places the -lgcc argument after the
source files.
Fixes: b94ec36896da ("rcutorture: Make use of nolibc when available")
Tested-by: Valentin Schneider <valentin.schneider@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com> [arm64]
Signed-off-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
RCU's rcutree.use_softirq=0 kernel boot parameter substitutes the per-CPU
rcuc kthreads for softirq, which is used in real-time installations.
However, none of the rcutorture scenarios test this parameter.
This commit therefore adds rcutree.use_softirq=0 to the RUDE01 and
TASKS01 rcutorture scenarios, both of which indirectly exercise RCU.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The sizes of vmlinux files built with KASAN enabled can approach a full
gigabyte, which can result in disk overflow sooner rather than later.
Fortunately, the xz command compresses them by almost an order of
magnitude. This commit therefore uses xz to compress vmlinux file built
by torture.sh with KASAN enabled.
However, xz is not the fastest thing in the world. In fact, it is way
slower than rotating-rust mass storage. This commit therefore also adds a
--compress-kasan-vmlinux argument to specify the degree of xz concurrency,
which defaults to using all available CPUs if there are that many files in
need of compression.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In 2020, running KCSAN often requires careful choice of compiler.
This commit therefore adds a --kcsan-kmake-arg parameter to torture.sh
to allow specifying (for example) "CC=clang" to the kernel build process
to correctly build a KCSAN-enabled kernel.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the command and arguments to the torture.sh log file, and
also outputs the results directory. This latter allows impatient users
to quickly find the results that are being generated by the current run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds --configs-rcutorture, --configs-locktorture, and
--configs-scftorture arguments to torture.sh, allowing the desired
set of scenarios to be passed to each. The default for each has been
changed from a large-system-appropriate set to just CFLIST for each.
Users are encouraged to create scripts that provide appropriate settings
for their specific systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Now that kvm.sh puts all the relevant details in the "log" file,
there is no need for torture.sh to generate a separate "log.long"
file. This commit therefore drops this from torture.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit causes torture.sh to check for zero-length runs and to take
the cowardly option of refusing to run them, logging its cowardice for
later inspection.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit causes torture.sh to use the torture.verbose_sleep_frequency
kernel boot parameter to throttle verbose refscale output on large systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit places "---" markers in the torture.sh script's allmodconfig
output, and uses "<<" to avoid overwriting earlier output from this
build test.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit saves a few lines of code by creating a doyesno helper bash
function for argument parsing.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
On large systems, the refscale printk() rate can overrun the file system's
ability to accept console log messages. This commit therefore uses the
new verbose_batched module parameter to rate-limit some of the higher-rate
printk() calls.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The .mod.c files created by allmodconfig builds interfers with the approach
torture.sh uses to enumerate types of rcuscale and refscale runs. This
commit therefore tightens the pattern matching to avoid this interference.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit uncomments the argument checking for the --duration argument
to torture.sh. While in the area, it also corrects the duration units
from seconds to minutes.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit improves torture.sh flexibility by autoscaling the number
of CPUs to be used in variable-CPUs torture tests, including scftorture,
refscale, rcuscale, and kvfree.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the ability to do "make allmodconfig" to torture.sh,
given that normal rcutorture runs do not normally catch missing exports.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The bash "eval" command enables Bobby Tables attacks, which might not
be a concern in torture testing by themselves, but one could imagine
these combined with a cut-and-paste attack. This commit therefore gets
rid of them. This comes at a price in terms of bash quoting not working
nicely, so the "--bootargs" argument lists are now passed to torture_one
via a bash-variable side channel. This might be a bit ugly, but it will
also allow torture.sh to grow its own --bootargs parameter.
While in the area, add proper header comments for the bash functions.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit makes torture.sh use the new bash functions get_starttime()
and get_starttime_duration() created for kvm.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Although tailoring a specific set of kvm.sh runs has served rcutorture
testing well over many years, it requires a relatively distraction-free
environment, which is not always available. This commit therefore
adds a prototype torture.sh script that by default tortures pretty much
everything the rcutorture scripting is designed to torture, and which
can be given command-line arguments to take a more focused approach.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds periodic toggling of 7 of 8 CPUs every second to TREE01
in order to test NOCB toggle code.
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Neeraj Upadhyay <neeraju@codeaurora.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, if a scenario is repeated as in "--configs '4*TREE01'",
the Kconfig analysis is performed for each occurrance (four times in
this example) and each analysis places the exact same data into the
exact same files. This is not really an issue in this repetition-four
example, but it can needlessly consume tens of seconds of wallclock time
for something like "--config '128*TINY01'".
This commit therefore does Kconfig analysis only once per set of
repeats of a given scenario, courtesy of the "sort -u" command and an
automatically generated awk script.
While in the area, this commit also wordsmiths a comment.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Normally, kvm-recheck.sh is run from kvm.sh, which provides the
TORTURE_TRUST_MAKE environment variable that, if a non-empty string,
indicates that the --trust-make command-line parameter has been passed
to kvm.sh. If there was no --trust-make, kvm-recheck.sh insists
that the Make.out file contain at least one "CC" command. Thus, when
kvm-recheck.sh is run standalone to evaluate a prior --trust-make run,
it will incorrectly insist that a proper kernel build did not happen.
This commit therefore causes kvm-recheck.sh to also search the "log"
file in the top-level results directory for the string "--trust-make".
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Commit 757055ae8ded ("init/console: Use ttynull as a fallback when
there is no console") results in the string "Warning: Failed to add
ttynull console. No stdin, stdout, and stderr for the init process!"
appearing on the console, which the rcutorture scripting interprets as
a warning, which causes every rcutorture run to be flagged. However,
the rcutorture init process never attempts to do any I/O, and thus does
not care that it has no stdin, stdout, or stderr.
This commit therefore causes the rcutorture scripting to ignore this
message.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
