From c4026b7b95a4b852e404afa2cd7720866159d118 Mon Sep 17 00:00:00 2001 From: Tony Luck Date: Mon, 3 Apr 2017 14:44:16 -0700 Subject: x86/intel_rdt: Implement "update" mode when writing schemata file The schemata file can have multiple lines and it is cumbersome to update all lines. Remove code that requires that the user provides values for every resource (in the right order). If the user provides values for just a few resources, update them and leave the rest unchanged. Side benefit: we now check which values were updated and only send IPIs to cpus that actually have updates. Signed-off-by: Tony Luck Signed-off-by: Vikas Shivappa Tested-by: Sai Praneeth Prakhya Cc: ravi.v.shankar@intel.com Cc: fenghua.yu@intel.com Cc: peterz@infradead.org Cc: vikas.shivappa@intel.com Cc: h.peter.anvin@intel.com Link: http://lkml.kernel.org/r/1491255857-17213-3-git-send-email-vikas.shivappa@linux.intel.com Signed-off-by: Thomas Gleixner --- Documentation/x86/intel_rdt_ui.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) (limited to 'Documentation/x86') diff --git a/Documentation/x86/intel_rdt_ui.txt b/Documentation/x86/intel_rdt_ui.txt index 51cf6fa5591f..3ea198460469 100644 --- a/Documentation/x86/intel_rdt_ui.txt +++ b/Documentation/x86/intel_rdt_ui.txt @@ -129,6 +129,20 @@ schemata format is always: L2:=;=;... +Reading/writing the schemata file +--------------------------------- +Reading the schemata file will show the state of all resources +on all domains. When writing you only need to specify those values +which you wish to change. E.g. + +# cat schemata +L3DATA:0=fffff;1=fffff;2=fffff;3=fffff +L3CODE:0=fffff;1=fffff;2=fffff;3=fffff +# echo "L3DATA:2=3c0;" > schemata +# cat schemata +L3DATA:0=fffff;1=fffff;2=3c0;3=fffff +L3CODE:0=fffff;1=fffff;2=fffff;3=fffff + Example 1 --------- On a two socket machine (one L3 cache per socket) with just four bits -- cgit From 4ffa3c977b5da2907eb294dc6d0259a02f2284aa Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 10 Apr 2017 16:52:32 +0200 Subject: x86/intel_rdt: Add cpus_list rdtgroup file The resource control filesystem provides only a bitmask based cpus file for assigning CPUs to a resource group. That's cumbersome with large cpumasks and non-intuitive when modifying the file from the command line. Range based cpu lists are commonly used along with bitmask based cpu files in various subsystems throughout the kernel. Add 'cpus_list' file which is CPU range based. # cd /sys/fs/resctrl/ # echo 1-10 > krava/cpus_list # cat krava/cpus_list 1-10 # cat krava/cpus 0007fe # cat cpus fffff9 # cat cpus_list 0,3-23 [ tglx: Massaged changelog and replaced "bitmask lists" by "CPU ranges" ] Signed-off-by: Jiri Olsa Cc: Fenghua Yu Cc: Peter Zijlstra Cc: Peter Zijlstra Cc: Mike Galbraith Cc: Shaohua Li Link: http://lkml.kernel.org/r/20170410145232.GF25354@krava Signed-off-by: Thomas Gleixner --- Documentation/x86/intel_rdt_ui.txt | 3 +++ 1 file changed, 3 insertions(+) (limited to 'Documentation/x86') diff --git a/Documentation/x86/intel_rdt_ui.txt b/Documentation/x86/intel_rdt_ui.txt index 3ea198460469..a1ace91c4b86 100644 --- a/Documentation/x86/intel_rdt_ui.txt +++ b/Documentation/x86/intel_rdt_ui.txt @@ -59,6 +59,9 @@ There are three files associated with each group: given to the default (root) group. You cannot remove CPUs from the default group. +"cpus_list": One or more CPU ranges of logical CPUs assigned to this + group. Same rules apply like for the "cpus" file. + "schemata": A list of all the resources available to this group. Each resource has its own line and format - see below for details. -- cgit From a9cad3d4f046bbd8f096b78d220c8d7074c2e93f Mon Sep 17 00:00:00 2001 From: Vikas Shivappa Date: Fri, 7 Apr 2017 17:33:50 -0700 Subject: Documentation, x86: Intel Memory bandwidth allocation Update the 'intel_rdt_ui' documentation to have Memory bandwidth(b/w) allocation interface usage. Signed-off-by: Vikas Shivappa Cc: ravi.v.shankar@intel.com Cc: tony.luck@intel.com Cc: fenghua.yu@intel.com Cc: vikas.shivappa@intel.com Link: http://lkml.kernel.org/r/1491611637-20417-2-git-send-email-vikas.shivappa@linux.intel.com Signed-off-by: Thomas Gleixner --- Documentation/x86/intel_rdt_ui.txt | 107 ++++++++++++++++++++++++++++++------- 1 file changed, 87 insertions(+), 20 deletions(-) (limited to 'Documentation/x86') diff --git a/Documentation/x86/intel_rdt_ui.txt b/Documentation/x86/intel_rdt_ui.txt index a1ace91c4b86..0f6d8477b66c 100644 --- a/Documentation/x86/intel_rdt_ui.txt +++ b/Documentation/x86/intel_rdt_ui.txt @@ -4,6 +4,7 @@ Copyright (C) 2016 Intel Corporation Fenghua Yu Tony Luck +Vikas Shivappa This feature is enabled by the CONFIG_INTEL_RDT_A Kconfig and the X86 /proc/cpuinfo flag bits "rdt", "cat_l3" and "cdp_l3". @@ -22,19 +23,34 @@ Info directory The 'info' directory contains information about the enabled resources. Each resource has its own subdirectory. The subdirectory -names reflect the resource names. Each subdirectory contains the -following files: +names reflect the resource names. +Cache resource(L3/L2) subdirectory contains the following files: -"num_closids": The number of CLOSIDs which are valid for this - resource. The kernel uses the smallest number of - CLOSIDs of all enabled resources as limit. +"num_closids": The number of CLOSIDs which are valid for this + resource. The kernel uses the smallest number of + CLOSIDs of all enabled resources as limit. -"cbm_mask": The bitmask which is valid for this resource. This - mask is equivalent to 100%. +"cbm_mask": The bitmask which is valid for this resource. + This mask is equivalent to 100%. -"min_cbm_bits": The minimum number of consecutive bits which must be - set when writing a mask. +"min_cbm_bits": The minimum number of consecutive bits which + must be set when writing a mask. +Memory bandwitdh(MB) subdirectory contains the following files: + +"min_bandwidth": The minimum memory bandwidth percentage which + user can request. + +"bandwidth_gran": The granularity in which the memory bandwidth + percentage is allocated. The allocated + b/w percentage is rounded off to the next + control step available on the hardware. The + available bandwidth control steps are: + min_bandwidth + N * bandwidth_gran. + +"delay_linear": Indicates if the delay scale is linear or + non-linear. This field is purely informational + only. Resource groups --------------- @@ -110,6 +126,22 @@ and 0xA are not. On a system with a 20-bit mask each bit represents 5% of the capacity of the cache. You could partition the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. +Memory bandwidth(b/w) percentage +-------------------------------- +For Memory b/w resource, user controls the resource by indicating the +percentage of total memory b/w. + +The minimum bandwidth percentage value for each cpu model is predefined +and can be looked up through "info/MB/min_bandwidth". The bandwidth +granularity that is allocated is also dependent on the cpu model and can +be looked up at "info/MB/bandwidth_gran". The available bandwidth +control steps are: min_bw + N * bw_gran. Intermediate values are rounded +to the next control step available on the hardware. + +The bandwidth throttling is a core specific mechanism on some of Intel +SKUs. Using a high bandwidth and a low bandwidth setting on two threads +sharing a core will result in both threads being throttled to use the +low bandwidth. L3 details (code and data prioritization disabled) -------------------------------------------------- @@ -132,6 +164,13 @@ schemata format is always: L2:=;=;... +Memory b/w Allocation details +----------------------------- + +Memory b/w domain is L3 cache. + + MB:=bandwidth0;=bandwidth1;... + Reading/writing the schemata file --------------------------------- Reading the schemata file will show the state of all resources @@ -149,13 +188,14 @@ L3CODE:0=fffff;1=fffff;2=fffff;3=fffff Example 1 --------- On a two socket machine (one L3 cache per socket) with just four bits -for cache bit masks +for cache bit masks, minimum b/w of 10% with a memory bandwidth +granularity of 10% # mount -t resctrl resctrl /sys/fs/resctrl # cd /sys/fs/resctrl # mkdir p0 p1 -# echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata -# echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata +# echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata +# echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata The default resource group is unmodified, so we have access to all parts of all caches (its schemata file reads "L3:0=f;1=f"). @@ -164,6 +204,14 @@ Tasks that are under the control of group "p0" may only allocate from the "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. Tasks in group "p1" use the "lower" 50% of cache on both sockets. +Similarly, tasks that are under the control of group "p0" may use a +maximum memory b/w of 50% on socket0 and 50% on socket 1. +Tasks in group "p1" may also use 50% memory b/w on both sockets. +Note that unlike cache masks, memory b/w cannot specify whether these +allocations can overlap or not. The allocations specifies the maximum +b/w that the group may be able to use and the system admin can configure +the b/w accordingly. + Example 2 --------- Again two sockets, but this time with a more realistic 20-bit mask. @@ -177,9 +225,10 @@ of L3 cache on socket 0. # cd /sys/fs/resctrl First we reset the schemata for the default group so that the "upper" -50% of the L3 cache on socket 0 cannot be used by ordinary tasks: +50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by +ordinary tasks: -# echo "L3:0=3ff;1=fffff" > schemata +# echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata Next we make a resource group for our first real time task and give it access to the "top" 25% of the cache on socket 0. @@ -202,6 +251,20 @@ Ditto for the second real time task (with the remaining 25% of cache): # echo 5678 > p1/tasks # taskset -cp 2 5678 +For the same 2 socket system with memory b/w resource and CAT L3 the +schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is +10): + +For our first real time task this would request 20% memory b/w on socket +0. + +# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata + +For our second real time task this would request an other 20% memory b/w +on socket 0. + +# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata + Example 3 --------- @@ -215,18 +278,22 @@ the tasks. # cd /sys/fs/resctrl First we reset the schemata for the default group so that the "upper" -50% of the L3 cache on socket 0 cannot be used by ordinary tasks: +50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 +cannot be used by ordinary tasks: -# echo "L3:0=3ff" > schemata +# echo "L3:0=3ff\nMB:0=50" > schemata -Next we make a resource group for our real time cores and give -it access to the "top" 50% of the cache on socket 0. +Next we make a resource group for our real time cores and give it access +to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on +socket 0. # mkdir p0 -# echo "L3:0=ffc00;" > p0/schemata +# echo "L3:0=ffc00\nMB:0=50" > p0/schemata Finally we move core 4-7 over to the new group and make sure that the -kernel and the tasks running there get 50% of the cache. +kernel and the tasks running there get 50% of the cache. They should +also get 50% of memory bandwidth assuming that the cores 4-7 are SMT +siblings and only the real time threads are scheduled on the cores 4-7. # echo C0 > p0/cpus -- cgit