| Age | Commit message (Collapse) | Author |
|---|
|
It appears that the intention of this snippet of code is to not exit
ocelot_xtr_irq_handler() while in the middle of extracting a frame.
The problem in extracting it word by word is that future extraction
attempts are really easy to get desynchronized, since the IRQ handler
assumes that the first 16 bytes are the IFH, which give further
information about the frame, such as frame length.
But during normal operation, "err" will not be 0, but 4, set from here:
for (i = 0; i < OCELOT_TAG_LEN / 4; i++) {
err = ocelot_rx_frame_word(ocelot, grp, true, &ifh[i]);
if (err != 4)
break;
}
if (err != 4)
break;
In that case, draining the extraction queue is a no-op. So explicitly
make this code execute only on negative err.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Since the xtr (extraction) IRQ of the ocelot switch is not shared, then
if it fired, it means that some data must be present in the queues of
the CPU port module. So simplify the code.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Michael Chan says:
====================
bnxt_en: Error recovery optimizations.
This series implements some optimizations to error recovery. One
patch adds an echo/reply mechanism with firmware to enhance error
detection. The other patches speed up the recovery process by
polling config space earlier and to selectively initialize
context memory during re-initialization.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
We currently only log the error recovery settings if it is enabled.
In some cases, firmware disables error recovery after it was
initially enabled. Without logging anything, the user will not be
aware of this change in setting.
Log it when error recovery is disabled. Also, change the reset count
value from hexadecimal to decimal.
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
This is a new async message that the firmware can send to check if it
can communicate with the driver. This is an added error detection
scheme that firmware can use if it suspects errors in the PCIe
interface. When the driver receives this async message, it will reply
back echoing some data in the async message. If the firmware is not
getting the reply with the proper data after some retries, error
recovery will kick in.
Reviewed-by: Andy Gospodarek <gospo@broadcom.com>
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Reviewed-by: Vasundhara Volam <vasundhara-v.volam@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
If firmware provides the offset to the "context kind" field of the
relevant context memory blocks, we'll initialize just that field for
each block instead of initializing all of context memory.
Populate the bnxt_mem_init structure with the proper offset returned
by firmware. If it is older firmware and the information is not
available, we set the offset to an invalid value and fall back to
the old behavior of initializing every byte. Otherwise, we initialize
only the "context kind" byte at the offset.
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Currently, the driver calls memset() to set all relevant context memory
used by the chip to the initial value. This can take many milliseconds
with the potentially large number of context pages allocated for the
chip.
To make this faster, we only need to initialize the "context kind" field
of each block of context memory. This patch sets up the infrastructure
to do that with the bnxt_mem_init structure. In the next patch, we'll
add the logic to obtain the offset of the "context kind" from the
firmware. This patch is not changing the current behavior of calling
memset() to initialize all relevant context memory.
Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
During some fatal firmware error conditions, the PCI config space
register 0x2e which normally contains the subsystem ID will become
0xffff. This register will revert back to the normal value after
the chip has completed core reset. If we detect this condition,
we can poll this config register immediately for the value to revert.
Because we use config read cycles to poll this register, there is no
possibility of Master Abort if we happen to read it during core reset.
This speeds up recovery significantly as we don't have to wait for the
conservative min_time before polling MMIO to see if the firmware has
come out of reset. As soon as this register changes value we can
proceed to re-initialize the device.
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Reviewed-by: Vasundhara Volam <vasundhara-v.volam@broadcom.com>
Reviewed-by: Andy Gospodarek <gospo@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Newer devices may have local context memory instead of relying on the
host for backing store. In these cases, HWRM_FUNC_BACKING_STORE_QCAPS
will return a zero entry size to indicate contexts for which the host
should not allocate backing store.
Selectively allocate context memory based on device capabilities and
only enable backing store for the appropriate contexts.
Signed-off-by: Edwin Peer <edwin.peer@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
The main changes are the echo request/response from firmware for error
detection and the NO_FCS feature to transmit frames without FCS.
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
|
|
dtschema for pca95xx expects GPIO hogs to end with 'hog' suffix.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
Link: https://lore.kernel.org/r/20210212162640.66677-2-krzk@kernel.org'
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
|
|
The PCA95xx GPIO expander requires GPIO controller properties to operate
properly.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
Acked-by: Joel Stanley <joel@jms.id.au>
Link: https://lore.kernel.org/r/20210212162640.66677-1-krzk@kernel.org'
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
|
|
Use macros to generate pll_mask to make code
more readable.
Signed-off-by: Wasim Khan <wasim.khan@nxp.com>
Link: https://lore.kernel.org/r/20210125142513.3919014-1-wasim.khan@oss.nxp.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add support for the global clock controller found on SC7280
based devices. This should allow most non-multimedia device
drivers to probe and control their clocks.
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Link: https://lore.kernel.org/r/1612981579-17391-3-git-send-email-tdas@codeaurora.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add device tree bindings for global clock subsystem clock
controller for Qualcomm Technology Inc's SC7280 SoCs.
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Link: https://lore.kernel.org/r/1612981579-17391-2-git-send-email-tdas@codeaurora.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add support for RPMH clocks on SC7280 SoCs.
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Link: https://lore.kernel.org/r/1612977230-11566-3-git-send-email-tdas@codeaurora.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add bindings and update documentation for clock rpmh driver on SC7280.
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Link: https://lore.kernel.org/r/1612977230-11566-2-git-send-email-tdas@codeaurora.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add the GDSC found in GCC for SM8350 SoC
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Link: https://lore.kernel.org/r/20210210161649.431741-1-vkoul@kernel.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add device tree bindings for graphics clock controller for
Qualcomm Technology Inc's SDM630 and SDM660 SoCs.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-10-angelogioacchino.delregno@somainline.org
Reviewed-by: Rob Herring <robh@kernel.org>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
The GPUCC manages the clocks for the Adreno GPU found on the
SDM630, SDM636, SDM660 SoCs.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-9-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
In the previous commit ("clk: qcom: rcg2: Stop hardcoding gfx3d pingpong
parent numbers") the gfx3d ping-pong ops (clk_gfx3d_ops) were
generalized in order to be able to reuse the same ops for more than just
one clock for one SoC: follow the change here in the MSM8996 MMCC.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-7-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
The function clk_gfx3d_determine_rate is selecting different PLLs
to manage the GFX3D clock source in a special way: this one needs
to be ping-pong'ed on different PLLs to ensure stability during
frequency switching (set a PLL rate, let it stabilize, switch the
RCG to the new PLL) and fast frequency transitions.
This technique is currently being used in the MSM8996 SoC and the
function was assuming that the parents were always at a specific
index in the parents list, which is TRUE, if we use this only on
the MSM8996 MMCC.
Unfortunately, MSM8996 is not the only SoC that needs to ping-pong
the graphics RCG, so choices are:
1. Make new special ops just to hardcode *again* other indexes,
creating code duplication for (imo) no reason; or
2. Generalize this function, so that it becomes usable for a range
of SoCs with slightly different ping-pong configuration.
In this commit, the second road was taken: define a new "special"
struct clk_rcg2_gfx3d, containing the ordered list of parents to
ping-pong the graphics clock on, and the "regular" rcg2 clock
structure in order to generalize the clk_gfx3d_determine_rate
function and make it working for other SoCs.
As for the function itself it is left with the assumption that we
need to ping-pong over three parents. The reasons for this are:
1. The initial model was MSM8996, which has 3 parents for the
graphics clock pingpong;
2. The other example that was taken into consideration is the
SDM630/636/660 SoC gpu clock controller, which is ping-ponging
over two dynamic clocked and one fixed clock PLL.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-6-angelogioacchino.delregno@somainline.org
[sboyd@kernel.org: Grow some local variables, drop do_div() usage in
favor of plain division, we're not dealing with a u64 here]
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Document the multimedia clock controller found on SDM630/660.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-5-angelogioacchino.delregno@somainline.org
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add a driver for the multimedia clock controller found on SDM660
based devices. This should allow most multimedia device drivers
to probe and control their clocks.
Signed-off-by: Martin Botka <martin.botka@somainline.org>
Co-developed-by: Konrad Dybcio <konrad.dybcio@somainline.org>
Signed-off-by: Konrad Dybcio <konrad.dybcio@somainline.org>
[angelogioacchino.delregno@somainline.org: Cleaned up SDM630 clock fixups]
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-4-angelogioacchino.delregno@somainline.org
[sboyd@kernel.org: Silence NULL pointer sparse warnings]
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
This clock is critical for any access to the GPU: gating it will
crash the system when the GPU has been initialized (so, you cannot
gate it unless you deinit the Adreno completely).
So, to achieve a working state with GPU on, set the CLK_IS_CRITICAL
flag to this clock.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-3-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Similarly to MSM8998, any access to the MMSS depends on this clock.
Gating it will crash the system when RPMCC inits mmssnoc_axi_rpm_clk.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-2-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
The GPU PLL0 is not a fixed PLL and the rate can be set on it:
this is necessary especially on boards which bootloader is setting
a very low rate on this PLL before booting Linux, which would be
unsuitable for postdividing to reach the maximum allowed Adreno GPU
frequency of 710MHz (or, actually, even 670MHz..) on this SoC.
To allow setting rates on the GPU PLL0, also define VCO boundaries
and set the CLK_SET_RATE_PARENT flag to the GPU PLL0 postdivider.
With this change, the Adreno GPU is now able to scale through all
the available frequencies.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210114221059.483390-12-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
The GPU GX GDSC has GPU_GX_BCR reset and gfx3d_clk CXC, as stated
on downstream kernels (and as verified upstream, because otherwise
random lockups happen).
Also, add PWRSTS_RET and NO_RET_PERIPH: also as found downstream,
and also as verified here, to avoid GPU related lockups it is
necessary to force retain mem, but *not* peripheral when enabling
this GDSC (and, of course, the inverse on disablement).
With this change, the GPU finally works flawlessly on my four
different MSM8998 devices from two different manufacturers.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210114221059.483390-11-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
In some rare occasions, we want to only set the RETAIN_MEM bit, but
not the RETAIN_PERIPH one: this is seen on at least SDM630/636/660's
GPU-GX GDSC, where unsetting and setting back the RETAIN_PERIPH bit
will generate chaos and panics during GPU suspend time (mainly, the
chaos is unaligned access).
For this reason, introduce a new NO_RET_PERIPH flag to the GDSC
driver to address this corner case.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-8-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
This adds a basic driver for the MPLL block found in MStar/SigmaStar
ARMv7 SoCs.
Currently this driver is only good for calculating the rates of it's
outputs and the actual configuration must be done before the kernel
boots. Usually this is done even before u-boot starts.
This driver targets the MPLL block found in the MSC313/MSC313E but
there is no documentation this chip so the register descriptions for
the another MStar chip the MST786 were used as they seem to match.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Link: https://lore.kernel.org/r/20210211052206.2955988-5-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add a devm helper for clk_hw_register_fixed_factor() so that drivers that internally
register fixed factor clocks for things like dividers don't need to manually unregister
them on remove or if probe fails.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Link: https://lore.kernel.org/r/20210211052206.2955988-4-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Add a binding description for the MStar/SigmaStar MPLL clock block.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Link: https://lore.kernel.org/r/20210211052206.2955988-3-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
Simple header to document the relationship between the MPLL outputs
and which divider they come from.
Output 0 is missing because it should not be consumed.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Acked-by: Rob Herring <robh@kernel.org>
Link: https://lore.kernel.org/r/20210211052206.2955988-2-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/pavel/linux-leds
Pull LED fix from Pavel Machek:
"One-liner fixing a build problem"
* 'for-rc8-5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/pavel/linux-leds:
leds: rt8515: add V4L2_FLASH_LED_CLASS dependency
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
Pull Kbuild fixes from Masahiro Yamada:
- Fix CONFIG_TRIM_UNUSED_KSYMS build for ppc64
- Use pkg-config for scripts/sign-file.c CFLAGS
* tag 'kbuild-fixes-v5.11-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
scripts: set proper OpenSSL include dir also for sign-file
sparc: remove wrong comment from arch/sparc/include/asm/Kbuild
kbuild: fix CONFIG_TRIM_UNUSED_KSYMS build for ppc64
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fixes from Borislav Petkov:
"I kinda knew while typing 'I hope this is the last batch of x86/urgent
updates' last week, Murphy was reading too and uttered 'Hold my
beer!'.
So here's more fixes... Thanks Murphy.
Anyway, three more x86/urgent fixes for 5.11 final. We should be
finally ready (famous last words). :-)
- An SGX use after free fix
- A fix for the fix to disable CET instrumentation generation for
kernel code. We forgot 32-bit, which we seem to do very often
nowadays
- A Xen PV fix to irqdomain init ordering"
* tag 'x86_urgent_for_v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/pci: Create PCI/MSI irqdomain after x86_init.pci.arch_init()
x86/build: Disable CET instrumentation in the kernel for 32-bit too
x86/sgx: Maintain encl->refcount for each encl->mm_list entry
|
|
pcpu_build_alloc_info() is an __init function that makes a call to
cpumask_clear_cpu(). With CONFIG_GCOV_PROFILE_ALL enabled, the inline
heuristics are modified and such cpumask_clear_cpu() which is marked
inline doesn't get inlined. Because it works on mask in __initdata,
modpost throws a section mismatch error.
Arnd sent a patch with the flatten attribute as an alternative [2]. I've
added it to compiler_attributes.h.
modpost complaint:
WARNING: modpost: vmlinux.o(.text+0x735425): Section mismatch in reference from the function cpumask_clear_cpu() to the variable .init.data:pcpu_build_alloc_info.mask
The function cpumask_clear_cpu() references
the variable __initdata pcpu_build_alloc_info.mask.
This is often because cpumask_clear_cpu lacks a __initdata
annotation or the annotation of pcpu_build_alloc_info.mask is wrong.
clang output:
mm/percpu.c:2724:5: remark: cpumask_clear_cpu not inlined into pcpu_build_alloc_info because too costly to inline (cost=725, threshold=325) [-Rpass-missed=inline]
[1] https://lore.kernel.org/linux-mm/202012220454.9F6Bkz9q-lkp@intel.com/
[2] https://lore.kernel.org/lkml/CAK8P3a2ZWfNeXKSm8K_SUhhwkor17jFo3xApLXjzfPqX0eUDUA@mail.gmail.com/
Reported-by: kernel test robot <lkp@intel.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
|
|
To build group_map[] and group_cnt[], we find out which group
CPUs belong to by comparing the distance of the cpu. However,
this includes cases where comparisons are not required.
This patch uses a bitmap to record CPUs that is not classified in
the group. CPUs that we know which group they belong to should be
cleared from the bitmap. In result, we can reduce the number of
unnecessary comparisons.
Signed-off-by: Wonhyuk Yang <vvghjk1234@gmail.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
[Dennis: added cpumask_clear() call and #include cpumask.h.]
|
|
The leds-rt8515 driver can optionall use the v4l2 flash led class,
but it causes a link error when that class is in a loadable module
and the rt8515 driver itself is built-in:
ld.lld: error: undefined symbol: v4l2_flash_init
>>> referenced by leds-rt8515.c
>>> leds/flash/leds-rt8515.o:(rt8515_probe) in archive
drivers/built-in.a
Adding 'depends on V4L2_FLASH_LED_CLASS' in Kconfig would avoid that,
but it would make it impossible to use the driver without the
v4l2 support.
Add the same dependency that the other users of this class have
instead, which just prevents the broken configuration.
Fixes: e1c6edcbea13 ("leds: rt8515: Add Richtek RT8515 LED driver")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Pavel Machek <pavel@ucw.cz>
|
|
Fixes: 2cea4a7a1885 ("scripts: use pkg-config to locate libcrypto")
Signed-off-by: Rolf Eike Beer <eb@emlix.com>
Cc: stable@vger.kernel.org # 5.6.x
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
|
|
These are NOT exported to userspace.
The headers listed in arch/sparc/include/uapi/asm/Kbuild are exported.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
|
|
BOSS GP-10 with 0582:0185 requires the similar quirk to make the
implicit feedback working like other BOSS devices.
Reported-by: Keith Milner <kamilner@superlative.org>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20210214154251.10750-1-tiwai@suse.de
Signed-off-by: Takashi Iwai <tiwai@suse.de>
|
|
Merely enabling CONFIG_COMPILE_TEST should not enable additional code.
To fix this, restrict the automatic enabling of IMX_INTMUX to ARCH_MXC,
and ask the user in case of compile-testing.
Fixes: 66968d7dfc3f5451 ("irqchip: Add COMPILE_TEST support for IMX_INTMUX")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210208145605.422943-1-geert+renesas@glider.be
|
|
Alexander Lobakin says:
====================
skbuff: introduce skbuff_heads bulking and reusing
Currently, all sorts of skb allocation always do allocate
skbuff_heads one by one via kmem_cache_alloc().
On the other hand, we have percpu napi_alloc_cache to store
skbuff_heads queued up for freeing and flush them by bulks.
We can use this cache not only for bulk-wiping, but also to obtain
heads for new skbs and avoid unconditional allocations, as well as
for bulk-allocating (like XDP's cpumap code and veth driver already
do).
As this might affect latencies, cache pressure and lots of hardware
and driver-dependent stuff, this new feature is mostly optional and
can be issued via:
- a new napi_build_skb() function (as a replacement for build_skb());
- existing {,__}napi_alloc_skb() and napi_get_frags() functions;
- __alloc_skb() with passing SKB_ALLOC_NAPI in flags.
iperf3 showed 35-70 Mbps bumps for both TCP and UDP while performing
VLAN NAT on 1.2 GHz MIPS board. The boost is likely to be bigger
on more powerful hosts and NICs with tens of Mpps.
Note on skbuff_heads from distant slabs or pfmemalloc'ed slabs:
- kmalloc()/kmem_cache_alloc() itself allows by default allocating
memory from the remote nodes to defragment their slabs. This is
controlled by sysctl, but according to this, skbuff_head from a
remote node is an OK case;
- The easiest way to check if the slab of skbuff_head is remote or
pfmemalloc'ed is:
if (!dev_page_is_reusable(virt_to_head_page(skb)))
/* drop it */;
...*but*, regarding that most slabs are built of compound pages,
virt_to_head_page() will hit unlikely-branch every single call.
This check costed at least 20 Mbps in test scenarios and seems
like it'd be better to _not_ do this.
Since v5 [4]:
- revert flags-to-bool conversion and simplify flags testing in
__alloc_skb() (Alexander Duyck).
Since v4 [3]:
- rebase on top of net-next and address kernel build robot issue;
- reorder checks a bit in __alloc_skb() to make new condition even
more harmless.
Since v3 [2]:
- make the feature mostly optional, so driver developers could
decide whether to use it or not (Paolo Abeni).
This reuses the old flag for __alloc_skb() and introduces
a new napi_build_skb();
- reduce bulk-allocation size from 32 to 16 elements (also Paolo).
This equals to the value of XDP's devmap and veth batch processing
(which were tested a lot) and should be sane enough;
- don't waste cycles on explicit in_serving_softirq() check.
Since v2 [1]:
- also cover {,__}alloc_skb() and {,__}build_skb() cases (became handy
after the changes that pass tiny skbs requests to kmalloc layer);
- cover the cache with KASAN instrumentation (suggested by Eric
Dumazet, help of Dmitry Vyukov);
- completely drop redundant __kfree_skb_flush() (also Eric);
- lots of code cleanups;
- expand the commit message with NUMA and pfmemalloc points (Jakub).
Since v1 [0]:
- use one unified cache instead of two separate to greatly simplify
the logics and reduce hotpath overhead (Edward Cree);
- new: recycle also GRO_MERGED_FREE skbs instead of immediate
freeing;
- correct performance numbers after optimizations and performing
lots of tests for different use cases.
[0] https://lore.kernel.org/netdev/20210111182655.12159-1-alobakin@pm.me
[1] https://lore.kernel.org/netdev/20210113133523.39205-1-alobakin@pm.me
[2] https://lore.kernel.org/netdev/20210209204533.327360-1-alobakin@pm.me
[3] https://lore.kernel.org/netdev/20210210162732.80467-1-alobakin@pm.me
[4] https://lore.kernel.org/netdev/20210211185220.9753-1-alobakin@pm.me
====================
Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
napi_frags_finish() and napi_skb_finish() can only be called inside
NAPI Rx context, so we can feed NAPI cache with skbuff_heads that
got NAPI_MERGED_FREE verdict instead of immediate freeing.
Replace __kfree_skb() with __kfree_skb_defer() in napi_skb_finish()
and move napi_skb_free_stolen_head() to skbuff.c, so it can drop skbs
to NAPI cache.
As many drivers call napi_alloc_skb()/napi_get_frags() on their
receive path, this becomes especially useful.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
{,__}napi_alloc_skb() is mostly used either for optional non-linear
receive methods (usually controlled via Ethtool private flags and off
by default) and/or for Rx copybreaks.
Use __napi_build_skb() here for obtaining skbuff_heads from NAPI cache
instead of inplace allocations. This includes both kmalloc and page
frag paths.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Reuse the old and forgotten SKB_ALLOC_NAPI to add an option to get
an skbuff_head from the NAPI cache instead of inplace allocation
inside __alloc_skb().
This implies that the function is called from softirq or BH-off
context, not for allocating a clone or from a distant node.
Cc: Alexander Duyck <alexander.duyck@gmail.com> # Simplified flags check
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
|
NAPI cache structures will be used for allocating skbuff_heads,
so move their declarations a bit upper.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
