From ace3c5499e61ef7c0433a7a297227a9bdde54a55 Mon Sep 17 00:00:00 2001 From: tiozhang Date: Thu, 29 Jun 2023 11:50:50 +0800 Subject: workqueue: add cmdline parameter `workqueue.unbound_cpus` to further constrain wq_unbound_cpumask at boot time Motivation of doing this is to better improve boot times for devices when we want to prevent our workqueue works from running on some specific CPUs, e,g, some CPUs are busy with interrupts. Signed-off-by: tiozhang Signed-off-by: Tejun Heo --- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'Documentation/admin-guide/kernel-parameters.txt') diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a1457995fd41..d1edee0fd5ec 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6964,6 +6964,13 @@ disables both lockup detectors. Default is 10 seconds. + workqueue.unbound_cpus= + [KNL,SMP] Specify to constrain one or some CPUs + to use in unbound workqueues. + Format: + By default, all online CPUs are available for + unbound workqueues. + workqueue.watchdog_thresh= If CONFIG_WQ_WATCHDOG is configured, workqueue can warn stall conditions and dump internal state to -- cgit From fcecfa8f271acdf130acbb30842e7848a138af0f Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Mon, 7 Aug 2023 15:57:23 -1000 Subject: workqueue: Remove module param disable_numa and sysfs knobs pool_ids and numa Unbound workqueue CPU affinity is going to receive an overhaul and the NUMA specific knobs won't make sense anymore. Remove them. Also, the pool_ids knob was used for debugging and not really meaningful given that there is no visibility into the pools associated with those IDs. Remove it too. A future patch will improve overall visibility. Signed-off-by: Tejun Heo --- Documentation/admin-guide/kernel-parameters.txt | 9 --------- 1 file changed, 9 deletions(-) (limited to 'Documentation/admin-guide/kernel-parameters.txt') diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index d1edee0fd5ec..2b89cbc39713 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6992,15 +6992,6 @@ threshold repeatedly. They are likely good candidates for using WQ_UNBOUND workqueues instead. - workqueue.disable_numa - By default, all work items queued to unbound - workqueues are affine to the NUMA nodes they're - issued on, which results in better behavior in - general. If NUMA affinity needs to be disabled for - whatever reason, this option can be used. Note - that this also can be controlled per-workqueue for - workqueues visible under /sys/bus/workqueue/. - workqueue.power_efficient Per-cpu workqueues are generally preferred because they show better performance thanks to cache -- cgit From 63c5484e74952f60f5810256bd69814d167b8d22 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Mon, 7 Aug 2023 15:57:24 -1000 Subject: workqueue: Add multiple affinity scopes and interface to select them Add three more affinity scopes - WQ_AFFN_CPU, SMT and CACHE - and make CACHE the default. The code changes to actually add the additional scopes are trivial. Also add module parameter "workqueue.default_affinity_scope" to override the default scope and "affinity_scope" sysfs file to configure it per workqueue. wq_dump.py and documentations are updated accordingly. This enables significant flexibility in configuring how unbound workqueues behave. If affinity scope is set to "cpu", it'll behave close to a per-cpu workqueue. On the other hand, "system" removes all locality boundaries. Many modern machines have multiple L3 caches often while being mostly uniform in terms of memory access. Thus, workqueue's previous behavior of spreading work items in each NUMA node had negative performance implications from unncessarily crossing L3 boundaries between issue and execution. However, picking a finer grained affinity scope also has a downside in that an issuer in one group can't utilize CPUs in other groups. While dependent on the specifics of workload, there's usually a noticeable penalty in crossing L3 boundaries, so let's default to CACHE. This issue will be further addressed and documented with examples in future patches. Signed-off-by: Tejun Heo --- Documentation/admin-guide/kernel-parameters.txt | 12 ++++++++++++ 1 file changed, 12 insertions(+) (limited to 'Documentation/admin-guide/kernel-parameters.txt') diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 2b89cbc39713..732c5c7e3fa5 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7007,6 +7007,18 @@ The default value of this parameter is determined by the config option CONFIG_WQ_POWER_EFFICIENT_DEFAULT. + workqueue.default_affinity_scope= + Select the default affinity scope to use for unbound + workqueues. Can be one of "cpu", "smt", "cache", + "numa" and "system". Default is "cache". For more + information, see the Affinity Scopes section in + Documentation/core-api/workqueue.rst. + + This can be updated after boot through the matching + file under /sys/module/workqueue/parameters. + However, the changed default will only apply to + unbound workqueues created afterwards. + workqueue.debug_force_rr_cpu Workqueue used to implicitly guarantee that work items queued without explicit CPU specified are put -- cgit From 523a301e66afd1ea9856660bcf3cee3a7c84c6dd Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Mon, 7 Aug 2023 15:57:25 -1000 Subject: workqueue: Make default affinity_scope dynamically updatable While workqueue.default_affinity_scope is writable, it only affects workqueues which are created afterwards and isn't very useful. Instead, let's introduce explicit "default" scope and update the effective scope dynamically when workqueue.default_affinity_scope is changed. Signed-off-by: Tejun Heo --- Documentation/admin-guide/kernel-parameters.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'Documentation/admin-guide/kernel-parameters.txt') diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 732c5c7e3fa5..98ffce100a39 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7014,10 +7014,10 @@ information, see the Affinity Scopes section in Documentation/core-api/workqueue.rst. - This can be updated after boot through the matching - file under /sys/module/workqueue/parameters. - However, the changed default will only apply to - unbound workqueues created afterwards. + This can be changed after boot by writing to the + matching /sys/module/workqueue/parameters file. All + workqueues with the "default" affinity scope will be + updated accordignly. workqueue.debug_force_rr_cpu Workqueue used to implicitly guarantee that work -- cgit