| Age | Commit message (Collapse) | Author |
|---|
|
Distributed runs of rcutorture will need to start and stop jittering on
the remote hosts, which means that the commands must be communicated to
those hosts. The commit therefore causes kvm.sh to place these commands
in new TORTURE_JITTER_START and TORTURE_JITTER_STOP environment variables
to communicate them to the scripts that will set this up. In addition,
this commit causes kvm-test-1-run.sh to append these commands to each
generated qemu-cmd file, which allows any remotely executing script to
extract the needed commands from this file.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, kvm-test-1-run.sh both builds and runs an rcutorture kernel,
which is inconvenient when it is necessary to re-run an old run or to
carry out a run on a remote system. This commit therefore extracts the
portion of kvm-test-1-run.sh that invoke qemu to actually run rcutorture
and places it in kvm-test-1-run-qemu.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When re-running old rcutorture builds, if the original run involved
gdb, the re-run also needs to do so. This commit therefore records the
TORTURE_KCONFIG_GDB_ARG environment variable into the qemu-cmd file so
that the re-run can access it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit creates jitterstart.sh and jitterstop.sh scripts that handle
the starting and stopping of the jitter.sh scripts. These must be sourced
using the bash "." command to allow the generated script to wait on the
backgrounded jitter.sh scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Remote rcutorture testing requires that jitter.sh continue to be
invoked from the generated script for local runs, but that it instead
be invoked on the remote system for distributed runs. This argues
for common jitterstart and jitterstop scripts. But it would be good
for jitterstart and jitterstop to control the name and location of the
"jittering" file, while continuing to have the duration controlled by
the caller of these new scripts.
This commit therefore reverses the order of the jittering and duration
parameters for jitter.sh, so that the jittering parameter precedes the
duration parameter.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Now that there is a reliable way to convince the jitter.sh scripts to
stop, the jitter_pids file is not needed, nor is the code that kills all
the PIDs contained in this file. This commit therefore eliminates this
file and the code using it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, jitter.sh execution is controlled by a time limit and by the
"kill" command. The former allowed jitter.sh to run uselessly past
the end of a set of runs that panicked during boot, and the latter is
vulnerable to PID reuse. This commit therefore introduces a "jittering"
file in the date-stamp directory within "res" that must be present for
the jitter.sh scripts to continue executing. The time limit is still
in place in order to avoid disturbing runs featuring large trace dumps,
but the removal of the "jittering" file handles the panic-during-boot
scenario without relying on PIDs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, the script generated by kvm.sh does a "wait" to wait on both
the current batch's guest OSes and any jitter.sh scripts. This works,
but makes it hard to abstract the jittering so that common code can be
used for both local and distributed runs. This commit therefore uses
"build.run" files in scenario directories, and these files are removed
after the corresponding scenario's guest OS has completed.
Note that --build-only runs do not create build.run files because they
also do not create guest OSes and do not run any jitter.sh scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently the bN.ready and bN.wait files are placed in the
rcutorture directory, which really is not at all a good place
for run-specific files. This commit therefore renames these
files to build.ready and build.wait and then moves them into the
scenario directories within the "res" directory, for example, into
tools/testing/selftests/rcutorture/res/2021.02.10-15.08.23/TINY01.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The testid.txt file was intended for occasional in extremis use, but
now that the new "bare-metal" file references it, it might see more use.
This commit therefore labels sections of output and adds spacing to make
it easier to see what needs to be done to make a bare-metal build tree
match an rcutorture build tree.
Of course, you can avoid this whole issue by building your bare-metal
kernel in the same directory in which you ran rcutorture, but that might
not always be an option.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In some environments, the torture-testing use of virtualization is
inconvenient. In such cases, the modprobe and rmmod commands may be used
to do torture testing, but significant setup is required to build, boot,
and modprobe a kernel so as to match a given torture-test scenario.
This commit therefore creates a "bare-metal" file in each results
directory containing steps to run the corresponding scenario using the
modprobe command on bare metal. For example, the contents of this file
after using kvm.sh to build an rcutorture TREE01 kernel, perhaps with
the --buildonly argument, is as follows:
To run this scenario on bare metal:
1. Set your bare-metal build tree to the state shown in this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/testid.txt
2. Update your bare-metal build tree's .config based on this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/ConfigFragment
3. Make the bare-metal kernel's build system aware of your .config updates:
$ yes "" | make oldconfig
4. Build your bare-metal kernel.
5. Boot your bare-metal kernel with the following parameters:
maxcpus=8 nr_cpus=43 rcutree.gp_preinit_delay=3 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcu_nocbs=0-1,3-7
6. Start the test with the following command:
$ modprobe rcutorture nocbs_nthreads=8 nocbs_toggle=1000 fwd_progress=0 onoff_interval=1000 onoff_holdoff=30 n_barrier_cbs=4 stat_interval=15 shutdown_secs=120 test_no_idle_hz=1 verbose=1
7. After some time, end the test with the following command:
$ rmmod rcutorture
8. Copy your bare-metal kernel's .config file, overwriting this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/.config
9. Copy the console output from just before the modprobe to just after
the rmmod into this file:
/home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19/TREE01/console.log
10. Check for runtime errors using the following command:
$ tools/testing/selftests/rcutorture/bin/kvm-recheck.sh /home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2021.02.04-17.10.19
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Yes, I do recall a time when 512MB of memory was a lot of mass storage,
much less main memory, but the rcuscale kvfree_rcu() testing invoked by
torture.sh can sometimes exceed it on large systems, resulting in OOM.
This commit therefore causes torture.sh to pase the "--memory 1G"
argument to kvm.sh to reserve a full gigabyte for this purpose.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
If the build fails when running multiple instances of a given rcutorture
scenario, for example, using the kvm.sh --configs "8*RUDE01" argument,
the build will be rerun an additional seven times. This is in some sense
correct, but it can waste significant time. This commit therefore checks
for a prior failed build and simply copies over that build's output.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The current jitter.sh script expects cpumask bits to fit into whatever
the awk interpreter uses for an integer, which clearly does not hold for
even medium-sized systems these days. This means that on a large system,
only the first 32 or 64 CPUs (depending) are subjected to jitter.sh
CPU-time perturbations. This commit therefore computes a given CPU's
cpumask using text manipulation rather than arithmetic shifts.
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
Pull nolibc fixes from Paul E. McKenney.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
The -lgcc command-line argument is placed poorly in the build options,
which can result in build failures, for exapmle, on ARM when uidiv()
is required. This commit therefore places the -lgcc argument after the
source files.
Fixes: b94ec36896da ("rcutorture: Make use of nolibc when available")
Tested-by: Valentin Schneider <valentin.schneider@arm.com>
Tested-by: Mark Rutland <mark.rutland@arm.com> [arm64]
Signed-off-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The sizes of vmlinux files built with KASAN enabled can approach a full
gigabyte, which can result in disk overflow sooner rather than later.
Fortunately, the xz command compresses them by almost an order of
magnitude. This commit therefore uses xz to compress vmlinux file built
by torture.sh with KASAN enabled.
However, xz is not the fastest thing in the world. In fact, it is way
slower than rotating-rust mass storage. This commit therefore also adds a
--compress-kasan-vmlinux argument to specify the degree of xz concurrency,
which defaults to using all available CPUs if there are that many files in
need of compression.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In 2020, running KCSAN often requires careful choice of compiler.
This commit therefore adds a --kcsan-kmake-arg parameter to torture.sh
to allow specifying (for example) "CC=clang" to the kernel build process
to correctly build a KCSAN-enabled kernel.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the command and arguments to the torture.sh log file, and
also outputs the results directory. This latter allows impatient users
to quickly find the results that are being generated by the current run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds --configs-rcutorture, --configs-locktorture, and
--configs-scftorture arguments to torture.sh, allowing the desired
set of scenarios to be passed to each. The default for each has been
changed from a large-system-appropriate set to just CFLIST for each.
Users are encouraged to create scripts that provide appropriate settings
for their specific systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Now that kvm.sh puts all the relevant details in the "log" file,
there is no need for torture.sh to generate a separate "log.long"
file. This commit therefore drops this from torture.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit causes torture.sh to check for zero-length runs and to take
the cowardly option of refusing to run them, logging its cowardice for
later inspection.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit causes torture.sh to use the torture.verbose_sleep_frequency
kernel boot parameter to throttle verbose refscale output on large systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit places "---" markers in the torture.sh script's allmodconfig
output, and uses "<<" to avoid overwriting earlier output from this
build test.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit saves a few lines of code by creating a doyesno helper bash
function for argument parsing.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
On large systems, the refscale printk() rate can overrun the file system's
ability to accept console log messages. This commit therefore uses the
new verbose_batched module parameter to rate-limit some of the higher-rate
printk() calls.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The .mod.c files created by allmodconfig builds interfers with the approach
torture.sh uses to enumerate types of rcuscale and refscale runs. This
commit therefore tightens the pattern matching to avoid this interference.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit uncomments the argument checking for the --duration argument
to torture.sh. While in the area, it also corrects the duration units
from seconds to minutes.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit improves torture.sh flexibility by autoscaling the number
of CPUs to be used in variable-CPUs torture tests, including scftorture,
refscale, rcuscale, and kvfree.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the ability to do "make allmodconfig" to torture.sh,
given that normal rcutorture runs do not normally catch missing exports.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The bash "eval" command enables Bobby Tables attacks, which might not
be a concern in torture testing by themselves, but one could imagine
these combined with a cut-and-paste attack. This commit therefore gets
rid of them. This comes at a price in terms of bash quoting not working
nicely, so the "--bootargs" argument lists are now passed to torture_one
via a bash-variable side channel. This might be a bit ugly, but it will
also allow torture.sh to grow its own --bootargs parameter.
While in the area, add proper header comments for the bash functions.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit makes torture.sh use the new bash functions get_starttime()
and get_starttime_duration() created for kvm.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Although tailoring a specific set of kvm.sh runs has served rcutorture
testing well over many years, it requires a relatively distraction-free
environment, which is not always available. This commit therefore
adds a prototype torture.sh script that by default tortures pretty much
everything the rcutorture scripting is designed to torture, and which
can be given command-line arguments to take a more focused approach.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, if a scenario is repeated as in "--configs '4*TREE01'",
the Kconfig analysis is performed for each occurrance (four times in
this example) and each analysis places the exact same data into the
exact same files. This is not really an issue in this repetition-four
example, but it can needlessly consume tens of seconds of wallclock time
for something like "--config '128*TINY01'".
This commit therefore does Kconfig analysis only once per set of
repeats of a given scenario, courtesy of the "sort -u" command and an
automatically generated awk script.
While in the area, this commit also wordsmiths a comment.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Normally, kvm-recheck.sh is run from kvm.sh, which provides the
TORTURE_TRUST_MAKE environment variable that, if a non-empty string,
indicates that the --trust-make command-line parameter has been passed
to kvm.sh. If there was no --trust-make, kvm-recheck.sh insists
that the Make.out file contain at least one "CC" command. Thus, when
kvm-recheck.sh is run standalone to evaluate a prior --trust-make run,
it will incorrectly insist that a proper kernel build did not happen.
This commit therefore causes kvm-recheck.sh to also search the "log"
file in the top-level results directory for the string "--trust-make".
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Commit 757055ae8ded ("init/console: Use ttynull as a fallback when
there is no console") results in the string "Warning: Failed to add
ttynull console. No stdin, stdout, and stderr for the init process!"
appearing on the console, which the rcutorture scripting interprets as
a warning, which causes every rcutorture run to be flagged. However,
the rcutorture init process never attempts to do any I/O, and thus does
not care that it has no stdin, stdout, or stderr.
This commit therefore causes the rcutorture scripting to ignore this
message.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit simplifies exit-code plumbing. It makes kvm-recheck.sh return
the value 1 for a build error and 2 for a runtime error. It also makes
kvm-find-errors.sh avoid checking runtime files for --build-only runs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit changes the "STOP" file that is used to cleanly halt a running
rcutorture run to "STOP.1" because no scenario directory will ever end
with ".1". If there really was a scenario named "STOP", its directories
would instead be named "STOP", "STOP.2", "STOP.3", and so on. While in
the area, the commit also changes the kernel-run-time checks for this
file to look directly in the directory above $resdir, thus avoiding the
need to pass the TORTURE_STOPFILE environment variable to remote systems.
While in the area, move the STOP.1 file to the top-level directory
covering all of the scenarios.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When all of the remote systems have the same number of CPUs, one
approach is to use one "--buildonly" run and one "--dryrun sched" run,
and then distributing the batches out one per remote system. However,
the output of "--dryrun sched" is not made for parsing, so this commit
adds a "--dryrun batches" that provides the same information in easily
parsed form.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
By default, the "panic" kernel parameter is zero, which causes the kernel
to loop indefinitely after a panic(). The rcutorture scripting will
eventually kill the corresponding qemu process, but only after waiting
for the full run duration plus a few minutes. This works, but delays
notifying the developer of the failure.
This commit therefore causes the rcutorture scripting to pass the
"panic=-1" kernel parameter, which caused the kernel to instead
unceremoniously shut down immediately. This in turn causes qemu to
terminate, so that if all of the runs in a given batch panic(), the
rcutorture scripting can immediately proceed to the next batch.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the test summary to the end of the log in the top-level
directory containing the kvm.sh test artifacts. While in the area, it adds
the kvm.sh exit code to this test summary.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, passing something like "--kconfig CONFIG_NR_CPUS=2" to kvm.sh
has no effect on scenario scheduling. For scenarios that do not specify
the number of CPUs, this can result in kvm.sh wastefully scheduling only
one scenario at a time even when the --kconfig argument would allow
a number to be run concurrently. This commit therefore makes kvm.sh
consider the --kconfig arguments when scheduling scenarios across the
available CPUs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The kvm.sh script uses kvm-find-errors.sh to evaluate whether or not
a build failed. Unfortunately, kvm-find-errors.sh returns success if
there are no failed runs (including when there are no runs at all) even if
there are build failures. This commit therefore makes kvm-find-errors.sh
return failure in response to build failures.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Yes, you can mentally subtract the timestamps, but this commit makes
the computer do this work.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Given that kvm.sh in invoked from scripts, it is only natural for
different levels of scripting to provide their own Kconfig option values,
for example. Unfortunately, right now, the last such argument on the
command line wins.
This commit therefore makes the --bootargs, --configs, --kconfigs,
--kmake-args, and --qemu-args argument values accumulate. For example,
where "--configs TREE01 --configs TREE02" would previously have run only
scenario TREE02, now it will run both scenarios.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, the "date" command producing the output on the kvm.sh "Test
Summary" line is executed at the beginning of the test, which produces a
date that is less than helpful to someone wanting to know the duration
of the test. This commit therefore defers this command's execution to
the end of the test.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The rcutorture scripts' identify_qemu_vcpus() function expects `lscpu`
to have a "CPU: " line, for example:
CPU(s): 8
But different local language settings can give different results:
Processeur(s) : 8
As a result, identify_qemu_vcpus() may return an empty string, resulting
in the following warning (with the same local language settings):
kvm-test-1-run.sh: ligne 138 : test: : nombre entier attendu comme expression
This commit therefore changes identify_qemu_vcpus() to use getconf,
which produces local-language-independend output.
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: rcu@vger.kernel.org
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds a config2csv.sh script that converts the specified
torture-test scenarios' Kconfig options and kernel-boot parameters to
.csv format. This allows easier comparison of scenarios when one fails
and another does not.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Distributed execution of rcutorture is eased if the qemu execution can
be split from the building of the kernel, as this allows target systems
to be used that are not set up to build kernels. It also avoids issues
with toolchain version skew across the cluster, aside of course from
qemu and KVM version skew.
This commit therefore records needed data as comments in the qemu-cmd file
and moves recording of the starting time to just before qemu is launched.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Scripts like kvm-check-branches.sh group runs under a single directory
in resdir in order to allow easier retrospective analysis. However, they
do this by letting kvm.sh create a directory as usual and then moving it
after the run. This can be very confusing when looking at the results
while kvm-check-branches.sh is running. This commit therefore enables
--datestamp to hand subdirectories to kvm.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
