diff options
272 files changed, 6981 insertions, 2135 deletions
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6cfa6e3996cf..29fb41e801ce 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -557,6 +557,7 @@ Format: <string> nosocket -- Disable socket memory accounting. nokmem -- Disable kernel memory accounting. + nobpf -- Disable BPF memory accounting. checkreqprot= [SELINUX] Set initial checkreqprot flag value. Format: { "0" | "1" } diff --git a/Documentation/bpf/graph_ds_impl.rst b/Documentation/bpf/graph_ds_impl.rst new file mode 100644 index 000000000000..61274622b71d --- /dev/null +++ b/Documentation/bpf/graph_ds_impl.rst @@ -0,0 +1,267 @@ +========================= +BPF Graph Data Structures +========================= + +This document describes implementation details of new-style "graph" data +structures (linked_list, rbtree), with particular focus on the verifier's +implementation of semantics specific to those data structures. + +Although no specific verifier code is referred to in this document, the document +assumes that the reader has general knowledge of BPF verifier internals, BPF +maps, and BPF program writing. + +Note that the intent of this document is to describe the current state of +these graph data structures. **No guarantees** of stability for either +semantics or APIs are made or implied here. + +.. contents:: + :local: + :depth: 2 + +Introduction +------------ + +The BPF map API has historically been the main way to expose data structures +of various types for use within BPF programs. Some data structures fit naturally +with the map API (HASH, ARRAY), others less so. Consequentially, programs +interacting with the latter group of data structures can be hard to parse +for kernel programmers without previous BPF experience. + +Luckily, some restrictions which necessitated the use of BPF map semantics are +no longer relevant. With the introduction of kfuncs, kptrs, and the any-context +BPF allocator, it is now possible to implement BPF data structures whose API +and semantics more closely match those exposed to the rest of the kernel. + +Two such data structures - linked_list and rbtree - have many verification +details in common. Because both have "root"s ("head" for linked_list) and +"node"s, the verifier code and this document refer to common functionality +as "graph_api", "graph_root", "graph_node", etc. + +Unless otherwise stated, examples and semantics below apply to both graph data +structures. + +Unstable API +------------ + +Data structures implemented using the BPF map API have historically used BPF +helper functions - either standard map API helpers like ``bpf_map_update_elem`` +or map-specific helpers. The new-style graph data structures instead use kfuncs +to define their manipulation helpers. Because there are no stability guarantees +for kfuncs, the API and semantics for these data structures can be evolved in +a way that breaks backwards compatibility if necessary. + +Root and node types for the new data structures are opaquely defined in the +``uapi/linux/bpf.h`` header. + +Locking +------- + +The new-style data structures are intrusive and are defined similarly to their +vanilla kernel counterparts: + +.. code-block:: c + + struct node_data { + long key; + long data; + struct bpf_rb_node node; + }; + + struct bpf_spin_lock glock; + struct bpf_rb_root groot __contains(node_data, node); + +The "root" type for both linked_list and rbtree expects to be in a map_value +which also contains a ``bpf_spin_lock`` - in the above example both global +variables are placed in a single-value arraymap. The verifier considers this +spin_lock to be associated with the ``bpf_rb_root`` by virtue of both being in +the same map_value and will enforce that the correct lock is held when +verifying BPF programs that manipulate the tree. Since this lock checking +happens at verification time, there is no runtime penalty. + +Non-owning references +--------------------- + +**Motivation** + +Consider the following BPF code: + +.. code-block:: c + + struct node_data *n = bpf_obj_new(typeof(*n)); /* ACQUIRED */ + + bpf_spin_lock(&lock); + + bpf_rbtree_add(&tree, n); /* PASSED */ + + bpf_spin_unlock(&lock); + +From the verifier's perspective, the pointer ``n`` returned from ``bpf_obj_new`` +has type ``PTR_TO_BTF_ID | MEM_ALLOC``, with a ``btf_id`` of +``struct node_data`` and a nonzero ``ref_obj_id``. Because it holds ``n``, the +program has ownership of the pointee's (object pointed to by ``n``) lifetime. +The BPF program must pass off ownership before exiting - either via +``bpf_obj_drop``, which ``free``'s the object, or by adding it to ``tree`` with +``bpf_rbtree_add``. + +(``ACQUIRED`` and ``PASSED`` comments in the example denote statements where +"ownership is acquired" and "ownership is passed", respectively) + +What should the verifier do with ``n`` after ownership is passed off? If the +object was ``free``'d with ``bpf_obj_drop`` the answer is obvious: the verifier +should reject programs which attempt to access ``n`` after ``bpf_obj_drop`` as +the object is no longer valid. The underlying memory may have been reused for +some other allocation, unmapped, etc. + +When ownership is passed to ``tree`` via ``bpf_rbtree_add`` the answer is less +obvious. The verifier could enforce the same semantics as for ``bpf_obj_drop``, +but that would result in programs with useful, common coding patterns being +rejected, e.g.: + +.. code-block:: c + + int x; + struct node_data *n = bpf_obj_new(typeof(*n)); /* ACQUIRED */ + + bpf_spin_lock(&lock); + + bpf_rbtree_add(&tree, n); /* PASSED */ + x = n->data; + n->data = 42; + + bpf_spin_unlock(&lock); + +Both the read from and write to ``n->data`` would be rejected. The verifier +can do better, though, by taking advantage of two details: + + * Graph data structure APIs can only be used when the ``bpf_spin_lock`` + associated with the graph root is held + + * Both graph data structures have pointer stability + + * Because graph nodes are allocated with ``bpf_obj_new`` and + adding / removing from the root involves fiddling with the + ``bpf_{list,rb}_node`` field of the node struct, a graph node will + remain at the same address after either operation. + +Because the associated ``bpf_spin_lock`` must be held by any program adding +or removing, if we're in the critical section bounded by that lock, we know +that no other program can add or remove until the end of the critical section. +This combined with pointer stability means that, until the critical section +ends, we can safely access the graph node through ``n`` even after it was used +to pass ownership. + +The verifier considers such a reference a *non-owning reference*. The ref +returned by ``bpf_obj_new`` is accordingly considered an *owning reference*. +Both terms currently only have meaning in the context of graph nodes and API. + +**Details** + +Let's enumerate the properties of both types of references. + +*owning reference* + + * This reference controls the lifetime of the pointee + + * Ownership of pointee must be 'released' by passing it to some graph API + kfunc, or via ``bpf_obj_drop``, which ``free``'s the pointee + + * If not released before program ends, verifier considers program invalid + + * Access to the pointee's memory will not page fault + +*non-owning reference* + + * This reference does not own the pointee + + * It cannot be used to add the graph node to a graph root, nor ``free``'d via + ``bpf_obj_drop`` + + * No explicit control of lifetime, but can infer valid lifetime based on + non-owning ref existence (see explanation below) + + * Access to the pointee's memory will not page fault + +From verifier's perspective non-owning references can only exist +between spin_lock and spin_unlock. Why? After spin_unlock another program +can do arbitrary operations on the data structure like removing and ``free``-ing +via bpf_obj_drop. A non-owning ref to some chunk of memory that was remove'd, +``free``'d, and reused via bpf_obj_new would point to an entirely different thing. +Or the memory could go away. + +To prevent this logic violation all non-owning references are invalidated by the +verifier after a critical section ends. This is necessary to ensure the "will +not page fault" property of non-owning references. So if the verifier hasn't +invalidated a non-owning ref, accessing it will not page fault. + +Currently ``bpf_obj_drop`` is not allowed in the critical section, so +if there's a valid non-owning ref, we must be in a critical section, and can +conclude that the ref's memory hasn't been dropped-and- ``free``'d or +dropped-and-reused. + +Any reference to a node that is in an rbtree _must_ be non-owning, since +the tree has control of the pointee's lifetime. Similarly, any ref to a node +that isn't in rbtree _must_ be owning. This results in a nice property: +graph API add / remove implementations don't need to check if a node +has already been added (or already removed), as the ownership model +allows the verifier to prevent such a state from being valid by simply checking +types. + +However, pointer aliasing poses an issue for the above "nice property". +Consider the following example: + +.. code-block:: c + + struct node_data *n, *m, *o, *p; + n = bpf_obj_new(typeof(*n)); /* 1 */ + + bpf_spin_lock(&lock); + + bpf_rbtree_add(&tree, n); /* 2 */ + m = bpf_rbtree_first(&tree); /* 3 */ + + o = bpf_rbtree_remove(&tree, n); /* 4 */ + p = bpf_rbtree_remove(&tree, m); /* 5 */ + + bpf_spin_unlock(&lock); + + bpf_obj_drop(o); + bpf_obj_drop(p); /* 6 */ + +Assume the tree is empty before this program runs. If we track verifier state +changes here using numbers in above comments: + + 1) n is an owning reference + + 2) n is a non-owning reference, it's been added to the tree + + 3) n and m are non-owning references, they both point to the same node + + 4) o is an owning reference, n and m non-owning, all point to same node + + 5) o and p are owning, n and m non-owning, all point to the same node + + 6) a double-free has occurred, since o and p point to same node and o was + ``free``'d in previous statement + +States 4 and 5 violate our "nice property", as there are non-owning refs to +a node which is not in an rbtree. Statement 5 will try to remove a node which +has already been removed as a result of this violation. State 6 is a dangerous +double-free. + +At a minimum we should prevent state 6 from being possible. If we can't also +prevent state 5 then we must abandon our "nice property" and check whether a +node has already been removed at runtime. + +We prevent both by generalizing the "invalidate non-owning references" behavior +of ``bpf_spin_unlock`` and doing similar invalidation after +``bpf_rbtree_remove``. The logic here being that any graph API kfunc which: + + * takes an arbitrary node argument + + * removes it from the data structure + + * returns an owning reference to the removed node + +May result in a state where some other non-owning reference points to the same +node. So ``remove``-type kfuncs must be considered a non-owning reference +invalidation point as well. diff --git a/Documentation/bpf/other.rst b/Documentation/bpf/other.rst index 3d61963403b4..7e6b12018802 100644 --- a/Documentation/bpf/other.rst +++ b/Documentation/bpf/other.rst @@ -6,4 +6,5 @@ Other :maxdepth: 1 ringbuf - llvm_reloc
\ No newline at end of file + llvm_reloc + graph_ds_impl diff --git a/MAINTAINERS b/MAINTAINERS index c93d65a3d616..20b25511e18d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4023,6 +4023,13 @@ L: bpf@vger.kernel.org S: Maintained F: tools/testing/selftests/bpf/ +BPF [DOCUMENTATION] (Related to Standardization) +R: David Vernet <void@manifault.com> +L: bpf@vger.kernel.org +L: bpf@ietf.org +S: Maintained +F: Documentation/bpf/instruction-set.rst + BPF [MISC] L: bpf@vger.kernel.org S: Odd Fixes @@ -4812,7 +4819,6 @@ L: linux-wpan@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/net/ieee802154/cc2520.txt F: drivers/net/ieee802154/cc2520.c -F: include/linux/spi/cc2520.h CCREE ARM TRUSTZONE CRYPTOCELL REE DRIVER M: Gilad Ben-Yossef <gilad@benyossef.com> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c index c4b1947ebf76..288003a9f0ca 100644 --- a/arch/loongarch/net/bpf_jit.c +++ b/arch/loongarch/net/bpf_jit.c @@ -841,7 +841,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext if (ret < 0) return ret; - move_imm(ctx, t1, func_addr, is32); + move_addr(ctx, t1, func_addr); emit_insn(ctx, jirl, t1, LOONGARCH_GPR_RA, 0); move_reg(ctx, regmap[BPF_REG_0], LOONGARCH_GPR_A0); break; diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h index ca708024fdd3..c335dc4eed37 100644 --- a/arch/loongarch/net/bpf_jit.h +++ b/arch/loongarch/net/bpf_jit.h @@ -82,6 +82,27 @@ static inline void emit_sext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, boo emit_insn(ctx, addiw, reg, reg, 0); } +static inline void move_addr(struct jit_ctx *ctx, enum loongarch_gpr rd, u64 addr) +{ + u64 imm_11_0, imm_31_12, imm_51_32, imm_63_52; + + /* lu12iw rd, imm_31_12 */ + imm_31_12 = (addr >> 12) & 0xfffff; + emit_insn(ctx, lu12iw, rd, imm_31_12); + + /* ori rd, rd, imm_11_0 */ + imm_11_0 = addr & 0xfff; + emit_insn(ctx, ori, rd, rd, imm_11_0); + + /* lu32id rd, imm_51_32 */ + imm_51_32 = (addr >> 32) & 0xfffff; + emit_insn(ctx, lu32id, rd, imm_51_32); + + /* lu52id rd, rd, imm_63_52 */ + imm_63_52 = (addr >> 52) & 0xfff; + emit_insn(ctx, lu52id, rd, rd, imm_63_52); +} + static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm, bool is32) { long imm_11_0, imm_31_12, imm_51_32, imm_63_52, imm_51_0, imm_51_31; diff --git a/arch/riscv/include/asm/patch.h b/arch/riscv/include/asm/patch.h index 9a7d7346001e..f433121774c0 100644 --- a/arch/riscv/include/asm/patch.h +++ b/arch/riscv/include/asm/patch.h @@ -7,6 +7,6 @@ #define _ASM_RISCV_PATCH_H int patch_text_nosync(void *addr, const void *insns, size_t len); -int patch_text(void *addr, u32 insn); +int patch_text(void *addr, u32 *insns, int ninsns); #endif /* _ASM_RISCV_PATCH_H */ diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 765004b60513..8086d1a281cd 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -15,7 +15,8 @@ struct patch_insn { void *addr; - u32 insn; + u32 *insns; + int ninsns; atomic_t cpu_count; }; @@ -102,12 +103,15 @@ NOKPROBE_SYMBOL(patch_text_nosync); static int patch_text_cb(void *data) { struct patch_insn *patch = data; - int ret = 0; + unsigned long len; + int i, ret = 0; if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) { - ret = - patch_text_nosync(patch->addr, &patch->insn, - GET_INSN_LENGTH(patch->insn)); + for (i = 0; ret == 0 && i < patch->ninsns; i++) { + len = GET_INSN_LENGTH(patch->insns[i]); + ret = patch_text_nosync(patch->addr + i * len, + &patch->insns[i], len); + } atomic_inc(&patch->cpu_count); } else { while (atomic_read(&patch->cpu_count) <= num_online_cpus()) @@ -119,11 +123,12 @@ static int patch_text_cb(void *data) } NOKPROBE_SYMBOL(patch_text_cb); -int patch_text(void *addr, u32 insn) +int patch_text(void *addr, u32 *insns, int ninsns) { struct patch_insn patch = { .addr = addr, - .insn = insn, + .insns = insns, + .ninsns = ninsns, .cpu_count = ATOMIC_INIT(0), }; diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 2bedec37d092..2f08c14a933d 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -23,13 +23,14 @@ post_kprobe_handler(struct kprobe *, struct kprobe_ctlblk *, struct pt_regs *); static void __kprobes arch_prepare_ss_slot(struct kprobe *p) { + u32 insn = __BUG_INSN_32; unsigned long offset = GET_INSN_LENGTH(p->opcode); p->ainsn.api.restore = (unsigned long)p->addr + offset; - patch_text(p->ainsn.api.insn, p->opcode); + patch_text(p->ainsn.api.insn, &p->opcode, 1); patch_text((void *)((unsigned long)(p->ainsn.api.insn) + offset), - __BUG_INSN_32); + &insn, 1); } static void __kprobes arch_prepare_simulate(struct kprobe *p) @@ -116,16 +117,16 @@ void *alloc_insn_page(void) /* install breakpoint in text */ void __kprobes arch_arm_kprobe(struct kprobe *p) { - if ((p->opcode & __INSN_LENGTH_MASK) == __INSN_LENGTH_32) - patch_text(p->addr, __BUG_INSN_32); - else - patch_text(p->addr, __BUG_INSN_16); + u32 insn = (p->opcode & __INSN_LENGTH_MASK) == __INSN_LENGTH_32 ? + __BUG_INSN_32 : __BUG_INSN_16; + + patch_text(p->addr, &insn, 1); } /* remove breakpoint from text */ void __kprobes arch_disarm_kprobe(struct kprobe *p) { - patch_text(p->addr, p->opcode); + patch_text(p->addr, &p->opcode, 1); } void __kprobes arch_remove_kprobe(struct kprobe *p) diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index d926e0f7ef57..bf9802a63061 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -573,6 +573,11 @@ static inline u32 rv_fence(u8 pred, u8 succ) return rv_i_insn(imm11_0, 0, 0, 0, 0xf); } +static inline u32 rv_nop(void) +{ + return rv_i_insn(0, 0, 0, 0, 0x13); +} + /* RVC instrutions. */ static inline u16 rvc_addi4spn(u8 rd, u32 imm10) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index f2417ac54edd..f5a668736c79 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -8,6 +8,8 @@ #include <linux/bitfield.h> #include <linux/bpf.h> #include <linux/filter.h> +#include <linux/memory.h> +#include <linux/stop_machine.h> #include "bpf_jit.h" #define RV_REG_TCC RV_REG_A6 @@ -238,7 +240,7 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) if (!is_tail_call) emit_mv(RV_REG_A0, RV_REG_A5, ctx); emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, - is_tail_call ? 4 : 0, /* skip TCC init */ + is_tail_call ? 20 : 0, /* skip reserved nops and TCC init */ ctx); } @@ -428,12 +430,12 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } -static int emit_jump_and_link(u8 rd, s64 rvoff, bool force_jalr, +static int emit_jump_and_link(u8 rd, s64 rvoff, bool fixed_addr, struct rv_jit_context *ctx) { s64 upper, lower; - if (rvoff && is_21b_int(rvoff) && !force_jalr) { + if (rvoff && fixed_addr && is_21b_int(rvoff)) { emit(rv_jal(rd, rvoff >> 1), ctx); return 0; } else if (in_auipc_jalr_range(rvoff)) { @@ -454,24 +456,17 @@ static bool is_signed_bpf_cond(u8 cond) cond == BPF_JSGE || cond == BPF_JSLE; } -static int emit_call(bool fixed, u64 addr, struct rv_jit_context *ctx) +static int emit_call(u64 addr, bool fixed_addr, struct rv_jit_context *ctx) { s64 off = 0; u64 ip; - u8 rd; - int ret; if (addr && ctx->insns) { ip = (u64)(long)(ctx->insns + ctx->ninsns); off = addr - ip; } - ret = emit_jump_and_link(RV_REG_RA, off, !fixed, ctx); - if (ret) - return ret; - rd = bpf_to_rv_reg(BPF_REG_0, ctx); - emit_mv(rd, RV_REG_A0, ctx); - return 0; + return emit_jump_and_link(RV_REG_RA, off, fixed_addr, ctx); } static void emit_atomic(u8 rd, u8 rs, s16 off, s32 imm, bool is64, @@ -622,6 +617,401 @@ static int add_exception_handler(const struct bpf_insn *insn, return 0; } +static int gen_call_or_nops(void *target, void *ip, u32 *insns) +{ + s64 rvoff; + int i, ret; + struct rv_jit_context ctx; + + ctx.ninsns = 0; + ctx.insns = (u16 *)insns; + + if (!target) { + for (i = 0; i < 4; i++) + emit(rv_nop(), &ctx); + return 0; + } + + rvoff = (s64)(target - (ip + 4)); + emit(rv_sd(RV_REG_SP, -8, RV_REG_RA), &ctx); + ret = emit_jump_and_link(RV_REG_RA, rvoff, false, &ctx); + if (ret) + return ret; + emit(rv_ld(RV_REG_RA, -8, RV_REG_SP), &ctx); + + return 0; +} + +static int gen_jump_or_nops(void *target, void *ip, u32 *insns) +{ + s64 rvoff; + struct rv_jit_context ctx; + + ctx.ninsns = 0; + ctx.insns = (u16 *)insns; + + if (!target) { + emit(rv_nop(), &ctx); + emit(rv_nop(), &ctx); + return 0; + } + + rvoff = (s64)(target - ip); + return emit_jump_and_link(RV_REG_ZERO, rvoff, false, &ctx); +} + +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, + void *old_addr, void *new_addr) +{ + u32 old_insns[4], new_insns[4]; + bool is_call = poke_type == BPF_MOD_CALL; + int (*gen_insns)(void *target, void *ip, u32 *insns); + int ninsns = is_call ? 4 : 2; + int ret; + + if (!is_bpf_text_address((unsigned long)ip)) + return -ENOTSUPP; + + gen_insns = is_call ? gen_call_or_nops : gen_jump_or_nops; + + ret = gen_insns(old_addr, ip, old_insns); + if (ret) + return ret; + + if (memcmp(ip, old_insns, ninsns * 4)) + return -EFAULT; + + ret = gen_insns(new_addr, ip, new_insns); + if (ret) + return ret; + + cpus_read_lock(); + mutex_lock(&text_mutex); + if (memcmp(ip, new_insns, ninsns * 4)) + ret = patch_text(ip, new_insns, ninsns); + mutex_unlock(&text_mutex); + cpus_read_unlock(); + + return ret; +} + +static void store_args(int nregs, int args_off, struct rv_jit_context *ctx) +{ + int i; + + for (i = 0; i < nregs; i++) { + emit_sd(RV_REG_FP, -args_off, RV_REG_A0 + i, ctx); + args_off -= 8; + } +} + +static void restore_args(int nregs, int args_off, struct rv_jit_context *ctx) +{ + int i; + + for (i = 0; i < nregs; i++) { + emit_ld(RV_REG_A0 + i, -args_off, RV_REG_FP, ctx); + args_off -= 8; + } +} + +static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off, + int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) +{ + int ret, branch_off; + struct bpf_prog *p = l->link.prog; + int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); + + if (l->cookie) { + emit_imm(RV_REG_T1, l->cookie, ctx); + emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_T1, ctx); + } else { + emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx); + } + + /* arg1: prog */ + emit_imm(RV_REG_A0, (const s64)p, ctx); + /* arg2: &run_ctx */ + emit_addi(RV_REG_A1, RV_REG_FP, -run_ctx_off, ctx); + ret = emit_call((const u64)bpf_trampoline_enter(p), true, ctx); + if (ret) + return ret; + + /* if (__bpf_prog_enter(prog) == 0) + * goto skip_exec_of_prog; + */ + branch_off = ctx->ninsns; + /* nop reserved for conditional jump */ + emit(rv_nop(), ctx); + + /* store prog start time */ + emit_mv(RV_REG_S1, RV_REG_A0, ctx); + + /* arg1: &args_off */ + emit_addi(RV_REG_A0, RV_REG_FP, -args_off, ctx); + if (!p->jited) + /* arg2: progs[i]->insnsi for interpreter */ + emit_imm(RV_REG_A1, (const s64)p->insnsi, ctx); + ret = emit_call((const u64)p->bpf_func, true, ctx); + if (ret) + return ret; + + if (save_ret) + emit_sd(RV_REG_FP, -retval_off, regmap[BPF_REG_0], ctx); + + /* update branch with beqz */ + if (ctx->insns) { + int offset = ninsns_rvoff(ctx->ninsns - branch_off); + u32 insn = rv_beq(RV_REG_A0, RV_REG_ZERO, offset >> 1); + *(u32 *)(ctx->insns + branch_off) = insn; + } + + /* arg1: prog */ + emit_imm(RV_REG_A0, (const s64)p, ctx); + /* arg2: prog start time */ + emit_mv(RV_REG_A1, RV_REG_S1, ctx); + /* arg3: &run_ctx */ + emit_addi(RV_REG_A2, RV_REG_FP, -run_ctx_off, ctx); + ret = emit_call((const u64)bpf_trampoline_exit(p), true, ctx); + + return ret; +} + +static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, + const struct btf_func_model *m, + struct bpf_tramp_links *tlinks, + void *func_addr, u32 flags, + struct rv_jit_context *ctx) +{ + int i, ret, offset; + int *branches_off = NULL; + int stack_size = 0, nregs = m->nr_args; + int retaddr_off, fp_off, retval_off, args_off; + int nregs_off, ip_off, run_ctx_off, sreg_off; + struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; + struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; + struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; + void *orig_call = func_addr; + bool save_ret; + u32 insn; + + /* Generated trampoline stack layout: + * + * FP - 8 [ RA of parent func ] return address of parent + * function + * FP - retaddr_off [ RA of traced func ] return address of traced + * function + * FP - fp_off [ FP of parent func ] + * + * FP - retval_off [ return value ] BPF_TRAMP_F_CALL_ORIG or + * BPF_TRAMP_F_RET_FENTRY_RET + * [ argN ] + * [ ... ] + * FP - args_off [ arg1 ] + * + * FP - nregs_off [ regs count ] + * + * FP - ip_off [ traced func ] BPF_TRAMP_F_IP_ARG + * + * FP - run_ctx_off [ bpf_tramp_run_ctx ] + * + * FP - sreg_off [ callee saved reg ] + * + * [ pads ] pads for 16 bytes alignment + */ + + if (flags & (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY)) + return -ENOTSUPP; + + /* extra regiters for struct arguments */ + for (i = 0; i < m->nr_args; i++) + if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) + nregs += round_up(m->arg_size[i], 8) / 8 - 1; + + /* 8 arguments passed by registers */ + if (nregs > 8) + return -ENOTSUPP; + + /* room for parent function return address */ + stack_size += 8; + + stack_size += 8; + retaddr_off = stack_size; + + stack_size += 8; + fp_off = stack_size; + + save_ret = flags & (BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_RET_FENTRY_RET); + if (save_ret) { + stack_size += 8; + retval_off = stack_size; + } + + stack_size += nregs * 8; + args_off = stack_size; + + stack_size += 8; + nregs_off = stack_size; + + if (flags & BPF_TRAMP_F_IP_ARG) { + stack_size += 8; + ip_off = stack_size; + } + + stack_size += round_up(sizeof(struct bpf_tramp_run_ctx), 8); + run_ctx_off = stack_size; + + stack_size += 8; + sreg_off = stack_size; + + stack_size = round_up(stack_size, 16); + + emit_addi(RV_REG_SP, RV_REG_SP, -stack_size, ctx); + + emit_sd(RV_REG_SP, stack_size - retaddr_off, RV_REG_RA, ctx); + emit_sd(RV_REG_SP, stack_size - fp_off, RV_REG_FP, ctx); + + emit_addi(RV_REG_FP, RV_REG_SP, stack_size, ctx); + + /* callee saved register S1 to pass start time */ + emit_sd(RV_REG_FP, -sreg_off, RV_REG_S1, ctx); + + /* store ip address of the traced function */ + if (flags & BPF_TRAMP_F_IP_ARG) { + emit_imm(RV_REG_T1, (const s64)func_addr, ctx); + emit_sd(RV_REG_FP, -ip_off, RV_REG_T1, ctx); + } + + emit_li(RV_REG_T1, nregs, ctx); + emit_sd(RV_REG_FP, -nregs_off, RV_REG_T1, ctx); + + store_args(nregs, args_off, ctx); + + /* skip to actual body of traced function */ + if (flags & BPF_TRAMP_F_SKIP_FRAME) + orig_call += 16; + + if (flags & BPF_TRAMP_F_CALL_ORIG) { + emit_imm(RV_REG_A0, (const s64)im, ctx); + ret = emit_call((const u64)__bpf_tramp_enter, true, ctx); + if (ret) + return ret; + } + + for (i = 0; i < fentry->nr_links; i++) { + ret = invoke_bpf_prog(fentry->links[i], args_off, retval_off, run_ctx_off, + flags & BPF_TRAMP_F_RET_FENTRY_RET, ctx); + if (ret) + return ret; + } + + if (fmod_ret->nr_links) { + branches_off = kcalloc(fmod_ret->nr_links, sizeof(int), GFP_KERNEL); + if (!branches_off) + return -ENOMEM; + + /* cleanup to avoid garbage return value confusion */ + emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx); + for (i = 0; i < fmod_ret->nr_links; i++) { + ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off, + run_ctx_off, true, ctx); + if (ret) + goto out; + emit_ld(RV_REG_T1, -retval_off, RV_REG_FP, ctx); + branches_off[i] = ctx->ninsns; + /* nop reserved for conditional jump */ + emit(rv_nop(), ctx); + } + } + + if (flags & BPF_TRAMP_F_CALL_ORIG) { + restore_args(nregs, args_off, ctx); + ret = emit_call((const u64)orig_call, true, ctx); + if (ret) + goto out; + emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx); + im->ip_after_call = ctx->insns + ctx->ninsns; + /* 2 nops reserved for auipc+jalr pair */ + emit(rv_nop(), ctx); + emit(rv_nop(), ctx); + } + + /* update branches saved in invoke_bpf_mod_ret with bnez */ + for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) { + offset = ninsns_rvoff(ctx->ninsns - branches_off[i]); + insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1); + *(u32 *)(ctx->insns + branches_off[i]) = insn; + } + + for (i = 0; i < fexit->nr_links; i++) { + ret = invoke_bpf_prog(fexit->links[i], args_off, retval_off, + run_ctx_off, false, ctx); + if (ret) + goto out; + } + + if (flags & BPF_TRAMP_F_CALL_ORIG) { + im->ip_epilogue = ctx->insns + ctx->ninsns; + emit_imm(RV_REG_A0, (const s64)im, ctx); + ret = emit_call((const u64)__bpf_tramp_exit, true, ctx); + if (ret) + goto out; + } + + if (flags & BPF_TRAMP_F_RESTORE_REGS) + restore_args(nregs, args_off, ctx); + + if (save_ret) + emit_ld(RV_REG_A0, -retval_off, RV_REG_FP, ctx); + + emit_ld(RV_REG_S1, -sreg_off, RV_REG_FP, ctx); + + if (flags & BPF_TRAMP_F_SKIP_FRAME) + /* return address of parent function */ + emit_ld(RV_REG_RA, stack_size - 8, RV_REG_SP, ctx); + else + /* return address of traced function */ + emit_ld(RV_REG_RA, stack_size - retaddr_off, RV_REG_SP, ctx); + + emit_ld(RV_REG_FP, stack_size - fp_off, RV_REG_SP, ctx); + emit_addi(RV_REG_SP, RV_REG_SP, stack_size, ctx); + + emit_jalr(RV_REG_ZERO, RV_REG_RA, 0, ctx); + + ret = ctx->ninsns; +out: + kfree(branches_off); + return ret; +} + +int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, + void *image_end, const struct btf_func_model *m, + u32 flags, struct bpf_tramp_links *tlinks, + void *func_addr) +{ + int ret; + struct rv_jit_context ctx; + + ctx.ninsns = 0; + ctx.insns = NULL; + ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); + if (ret < 0) + return ret; + + if (ninsns_rvoff(ret) > (long)image_end - (long)image) + return -EFBIG; + + ctx.ninsns = 0; + ctx.insns = image; + ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); + if (ret < 0) + return ret; + + bpf_flush_icache(ctx.insns, ctx.insns + ctx.ninsns); + + return ninsns_rvoff(ret); +} + int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, bool extra_pass) { @@ -913,7 +1303,7 @@ out_be: /* JUMP off */ case BPF_JMP | BPF_JA: rvoff = rv_offset(i, off, ctx); - ret = emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); + ret = emit_jump_and_link(RV_REG_ZERO, rvoff, true, ctx); if (ret) return ret; break; @@ -1032,17 +1422,20 @@ out_be: /* function call */ case BPF_JMP | BPF_CALL: { - bool fixed; + bool fixed_addr; u64 addr; mark_call(ctx); - ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass, &addr, - &fixed); + ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass, + &addr, &fixed_addr); if (ret < 0) return ret; - ret = emit_call(fixed, addr, ctx); + + ret = emit_call(addr, fixed_addr, ctx); if (ret) return ret; + + emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_A0, ctx); break; } /* tail call */ @@ -1057,7 +1450,7 @@ out_be: break; rvoff = epilogue_offset(ctx); - ret = emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); + ret = emit_jump_and_link(RV_REG_ZERO, rvoff, true, ctx); if (ret) return ret; break; @@ -1270,7 +1663,7 @@ out_be: void bpf_jit_build_prologue(struct rv_jit_context *ctx) { - int stack_adjust = 0, store_offset, bpf_stack_adjust; + int i, stack_adjust = 0, store_offset, bpf_stack_adjust; bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); if (bpf_stack_adjust) @@ -1297,6 +1690,10 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx) store_offset = stack_adjust - 8; + /* reserve 4 nop insns */ + for (i = 0; i < 4; i++) + emit(rv_nop(), ctx); + /* First instruction is always setting the tail-call-counter * (TCC) register. This instruction is skipped for tail calls. * Force using a 4-byte (non-compressed) instruction. diff --git a/drivers/net/can/ctucanfd/ctucanfd_platform.c b/drivers/net/can/ctucanfd/ctucanfd_platform.c index f83684f006ea..a17561d97192 100644 --- a/drivers/net/can/ctucanfd/ctucanfd_platform.c +++ b/drivers/net/can/ctucanfd/ctucanfd_platform.c @@ -47,7 +47,6 @@ static void ctucan_platform_set_drvdata(struct device *dev, */ static int ctucan_platform_probe(struct platform_device *pdev) { - struct resource *res; /* IO mem resources */ struct device *dev = &pdev->dev; void __iomem *addr; int ret; @@ -55,8 +54,7 @@ static int ctucan_platform_probe(struct platform_device *pdev) int irq; /* Get the virtual base address for the device */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - addr = devm_ioremap_resource(dev, res); + addr = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(addr)) { ret = PTR_ERR(addr); goto err; diff --git a/drivers/net/can/usb/esd_usb.c b/drivers/net/can/usb/esd_usb.c index 42323f5e6f3a..55b36973952d 100644 --- a/drivers/net/can/usb/esd_usb.c +++ b/drivers/net/can/usb/esd_usb.c @@ -127,7 +127,15 @@ struct rx_msg { u8 dlc; __le32 ts; __le32 id; /* upper 3 bits contain flags */ - u8 data[8]; + union { + u8 data[8]; + struct { + u8 status; /* CAN Controller Status */ + u8 ecc; /* Error Capture Register */ + u8 rec; /* RX Error Counter */ + u8 tec; /* TX Error Counter */ + } ev_can_err_ext; /* For ESD_EV_CAN_ERROR_EXT */ + }; }; struct tx_msg { @@ -229,51 +237,52 @@ static void esd_usb_rx_event(struct esd_usb_net_priv *priv, u32 id = le32_to_cpu(msg->msg.rx.id) & ESD_IDMASK; if (id == ESD_EV_CAN_ERROR_EXT) { - u8 state = msg->msg.rx.data[0]; - u8 ecc = msg->msg.rx.data[1]; - u8 rxerr = msg->msg.rx.data[2]; - u8 txerr = msg->msg.rx.data[3]; + u8 state = msg->msg.rx.ev_can_err_ext.status; + u8 ecc = msg->msg.rx.ev_can_err_ext.ecc; + u8 rxerr = msg->msg.rx.ev_can_err_ext.rec; + u8 txerr = msg->msg.rx.ev_can_err_ext.tec; netdev_dbg(priv->netdev, "CAN_ERR_EV_EXT: dlc=%#02x state=%02x ecc=%02x rec=%02x tec=%02x\n", msg->msg.rx.dlc, state, ecc, rxerr, txerr); skb = alloc_can_err_skb(priv->netdev, &cf); - if (skb == NULL) { - stats->rx_dropped++; - return; - } if (state != priv->old_state) { + enum can_state tx_state, rx_state; + enum can_state new_state = CAN_STATE_ERROR_ACTIVE; + priv->old_state = state; switch (state & ESD_BUSSTATE_MASK) { case ESD_BUSSTATE_BUSOFF: - priv->can.state = CAN_STATE_BUS_OFF; - cf->can_id |= CAN_ERR_BUSOFF; - priv->can.can_stats.bus_off++; + new_state = CAN_STATE_BUS_OFF; can_bus_off(priv->netdev); break; case ESD_BUSSTATE_WARN: - priv->can.state = CAN_STATE_ERROR_WARNING; - priv->can.can_stats.error_warning++; + new_state = CAN_STATE_ERROR_WARNING; break; case ESD_BUSSTATE_ERRPASSIVE: - priv->can.state = CAN_STATE_ERROR_PASSIVE; - priv->can.can_stats.error_passive++; + new_state = CAN_STATE_ERROR_PASSIVE; break; default: - priv->can.state = CAN_STATE_ERROR_ACTIVE; + new_state = CAN_STATE_ERROR_ACTIVE; txerr = 0; rxerr = 0; break; } - } else { + + if (new_state != priv->can.state) { + tx_state = (txerr >= rxerr) ? new_state : 0; + rx_state = (txerr <= rxerr) ? new_state : 0; + can_change_state(priv->netdev, cf, + tx_state, rx_state); + } + } else if (skb) { priv->can.can_stats.bus_error++; stats->rx_errors++; - cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR | - CAN_ERR_CNT; + cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR; switch (ecc & SJA1000_ECC_MASK) { case SJA1000_ECC_BIT: @@ -286,7 +295,6 @@ static void esd_usb_rx_event(struct esd_usb_net_priv *priv, cf->data[2] |= CAN_ERR_PROT_STUFF; break; default: - cf->data[3] = ecc & SJA1000_ECC_SEG; break; } @@ -294,20 +302,22 @@ static void esd_usb_rx_event(struct esd_usb_net_priv *priv, if (!(ecc & SJA1000_ECC_DIR)) cf->data[2] |= CAN_ERR_PROT_TX; - if (priv->can.state == CAN_STATE_ERROR_WARNING || - priv->can.state == CAN_STATE_ERROR_PASSIVE) { - cf->data[1] = (txerr > rxerr) ? - CAN_ERR_CRTL_TX_PASSIVE : - CAN_ERR_CRTL_RX_PASSIVE; - } - cf->data[6] = txerr; - cf->data[7] = rxerr; + /* Bit stream position in CAN frame as the error was detected */ + cf->data[3] = ecc & SJA1000_ECC_SEG; } priv->bec.txerr = txerr; priv->bec.rxerr = rxerr; - netif_rx(skb); + if (skb) { + cf->can_id |= CAN_ERR_CNT; + cf->data[6] = txerr; + cf->data[7] = rxerr; + + netif_rx(skb); + } else { + stats->rx_dropped++; + } } } diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c index f55d9d9c01a8..3a4b6cb7b7b9 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c @@ -77,14 +77,18 @@ int bcmgenet_set_wol(struct net_device *dev, struct ethtool_wolinfo *wol) if (wol->wolopts) { device_set_wakeup_enable(kdev, 1); /* Avoid unbalanced enable_irq_wake calls */ - if (priv->wol_irq_disabled) + if (priv->wol_irq_disabled) { enable_irq_wake(priv->wol_irq); + enable_irq_wake(priv->irq0); + } priv->wol_irq_disabled = false; } else { device_set_wakeup_enable(kdev, 0); /* Avoid unbalanced disable_irq_wake calls */ - if (!priv->wol_irq_disabled) + if (!priv->wol_irq_disabled) { disable_irq_wake(priv->wol_irq); + disable_irq_wake(priv->irq0); + } priv->wol_irq_disabled = true; } diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c index 746ccfde7255..a62cffaf6ff1 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c @@ -4598,8 +4598,10 @@ static int dpaa2_eth_netdev_init(struct net_device *net_dev) net_dev->hw_features = net_dev->features; net_dev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | - NETDEV_XDP_ACT_XSK_ZEROCOPY | NETDEV_XDP_ACT_NDO_XMIT; + if (priv->dpni_attrs.wriop_version >= DPAA2_WRIOP_VERSION(3, 0, 0) && + priv->dpni_attrs.num_queues <= 8) + net_dev->xdp_features |= NETDEV_XDP_ACT_XSK_ZEROCOPY; if (priv->dpni_attrs.vlan_filter_entries) net_dev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER; diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c index 8cfc30fc9840..781475480ff2 100644 --- a/drivers/net/ethernet/intel/ice/ice_lib.c +++ b/drivers/net/ethernet/intel/ice/ice_lib.c @@ -627,6 +627,7 @@ ice_vsi_alloc_def(struct ice_vsi *vsi, struct ice_channel *ch) vsi->next_base_q = ch->base_q; break; case ICE_VSI_VF: + case ICE_VSI_LB: break; default: ice_vsi_free_arrays(vsi); diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c index c036be5eb35d..dfd22862e926 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.c +++ b/drivers/net/ethernet/intel/ice/ice_txrx.c @@ -85,7 +85,7 @@ ice_prgm_fdir_fltr(struct ice_vsi *vsi, struct ice_fltr_desc *fdir_desc, td_cmd = ICE_TXD_LAST_DESC_CMD | ICE_TX_DESC_CMD_DUMMY | ICE_TX_DESC_CMD_RE; - tx_buf->tx_flags = ICE_TX_FLAGS_DUMMY_PKT; + tx_buf->type = ICE_TX_BUF_DUMMY; tx_buf->raw_buf = raw_packet; tx_desc->cmd_type_offset_bsz = @@ -112,31 +112,29 @@ ice_prgm_fdir_fltr(struct ice_vsi *vsi, struct ice_fltr_desc *fdir_desc, static void ice_unmap_and_free_tx_buf(struct ice_tx_ring *ring, struct ice_tx_buf *tx_buf) { - if (tx_buf->skb) { - if (tx_buf->tx_flags & ICE_TX_FLAGS_DUMMY_PKT) { - devm_kfree(ring->dev, tx_buf->raw_buf); - } else if (ice_ring_is_xdp(ring)) { - if (ring->xsk_pool) - xsk_buff_free(tx_buf->xdp); - else - page_frag_free(tx_buf->raw_buf); - } else { - dev_kfree_skb_any(tx_buf->skb); - } - if (dma_unmap_len(tx_buf, len)) - dma_unmap_single(ring->dev, - dma_unmap_addr(tx_buf, dma), - dma_unmap_len(tx_buf, len), - DMA_TO_DEVICE); - } else if (dma_unmap_len(tx_buf, len)) { + if (dma_unmap_len(tx_buf, len)) dma_unmap_page(ring->dev, dma_unmap_addr(tx_buf, dma), dma_unmap_len(tx_buf, len), DMA_TO_DEVICE); + + switch (tx_buf->type) { + case ICE_TX_BUF_DUMMY: + devm_kfree(ring->dev, tx_buf->raw_buf); + break; + case ICE_TX_BUF_SKB: + dev_kfree_skb_any(tx_buf->skb); + break; + case ICE_TX_BUF_XDP_TX: + page_frag_free(tx_buf->raw_buf); + break; + case ICE_TX_BUF_XDP_XMIT: + xdp_return_frame(tx_buf->xdpf); + break; } tx_buf->next_to_watch = NULL; - tx_buf->skb = NULL; + tx_buf->type = ICE_TX_BUF_EMPTY; dma_unmap_len_set(tx_buf, len, 0); /* tx_buf must be completely set up in the transmit path */ } @@ -269,7 +267,7 @@ static bool ice_clean_tx_irq(struct ice_tx_ring *tx_ring, int napi_budget) DMA_TO_DEVICE); /* clear tx_buf data */ - tx_buf->skb = NULL; + tx_buf->type = ICE_TX_BUF_EMPTY; dma_unmap_len_set(tx_buf, len, 0); /* unmap remaining buffers */ @@ -580,7 +578,7 @@ ice_run_xdp(struct ice_rx_ring *rx_ring, struct xdp_buff *xdp, case XDP_TX: if (static_branch_unlikely(&ice_xdp_locking_key)) spin_lock(&xdp_ring->tx_lock); - ret = __ice_xmit_xdp_ring(xdp, xdp_ring); + ret = __ice_xmit_xdp_ring(xdp, xdp_ring, false); if (static_branch_unlikely(&ice_xdp_locking_key)) spin_unlock(&xdp_ring->tx_lock); if (ret == ICE_XDP_CONSUMED) @@ -608,6 +606,25 @@ exit: } /** + * ice_xmit_xdp_ring - submit frame to XDP ring for transmission + * @xdpf: XDP frame that will be converted to XDP buff + * @xdp_ring: XDP ring for transmission + */ +static int ice_xmit_xdp_ring(const struct xdp_frame *xdpf, + struct ice_tx_ring *xdp_ring) +{ + struct xdp_buff xdp; + + xdp.data_hard_start = (void *)xdpf; + xdp.data = xdpf->data; + xdp.data_end = xdp.data + xdpf->len; + xdp.frame_sz = xdpf->frame_sz; + xdp.flags = xdpf->flags; + + return __ice_xmit_xdp_ring(&xdp, xdp_ring, true); +} + +/** * ice_xdp_xmit - submit packets to XDP ring for transmission * @dev: netdev * @n: number of XDP frames to be transmitted @@ -652,7 +669,7 @@ ice_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, tx_buf = &xdp_ring->tx_buf[xdp_ring->next_to_use]; for (i = 0; i < n; i++) { - struct xdp_frame *xdpf = frames[i]; + const struct xdp_frame *xdpf = frames[i]; int err; err = ice_xmit_xdp_ring(xdpf, xdp_ring); @@ -1712,6 +1729,7 @@ ice_tx_map(struct ice_tx_ring *tx_ring, struct ice_tx_buf *first, DMA_TO_DEVICE); tx_buf = &tx_ring->tx_buf[i]; + tx_buf->type = ICE_TX_BUF_FRAG; } /* record SW timestamp if HW timestamp is not available */ @@ -2358,6 +2376,7 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring) /* record the location of the first descriptor for this packet */ first = &tx_ring->tx_buf[tx_ring->next_to_use]; first->skb = skb; + first->type = ICE_TX_BUF_SKB; first->bytecount = max_t(unsigned int, skb->len, ETH_ZLEN); first->gso_segs = 1; first->tx_flags = 0; @@ -2530,11 +2549,11 @@ void ice_clean_ctrl_tx_irq(struct ice_tx_ring *tx_ring) dma_unmap_addr(tx_buf, dma), dma_unmap_len(tx_buf, len), DMA_TO_DEVICE); - if (tx_buf->tx_flags & ICE_TX_FLAGS_DUMMY_PKT) + if (tx_buf->type == ICE_TX_BUF_DUMMY) devm_kfree(tx_ring->dev, tx_buf->raw_buf); /* clear next_to_watch to prevent false hangs */ - tx_buf->raw_buf = NULL; + tx_buf->type = ICE_TX_BUF_EMPTY; tx_buf->tx_flags = 0; tx_buf->next_to_watch = NULL; dma_unmap_len_set(tx_buf, len, 0); diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h index efa3d378f19e..fff0efe28373 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.h +++ b/drivers/net/ethernet/intel/ice/ice_txrx.h @@ -121,10 +121,7 @@ static inline int ice_skb_pad(void) #define ICE_TX_FLAGS_TSO BIT(0) #define ICE_TX_FLAGS_HW_VLAN BIT(1) #define ICE_TX_FLAGS_SW_VLAN BIT(2) -/* ICE_TX_FLAGS_DUMMY_PKT is used to mark dummy packets that should be - * freed instead of returned like skb packets. - */ -#define ICE_TX_FLAGS_DUMMY_PKT BIT(3) +/* Free, was ICE_TX_FLAGS_DUMMY_PKT */ #define ICE_TX_FLAGS_TSYN BIT(4) #define ICE_TX_FLAGS_IPV4 BIT(5) #define ICE_TX_FLAGS_IPV6 BIT(6) @@ -149,22 +146,44 @@ static inline int ice_skb_pad(void) #define ICE_TXD_LAST_DESC_CMD (ICE_TX_DESC_CMD_EOP | ICE_TX_DESC_CMD_RS) +/** + * enum ice_tx_buf_type - type of &ice_tx_buf to act on Tx completion + * @ICE_TX_BUF_EMPTY: unused OR XSk frame, no action required + * @ICE_TX_BUF_DUMMY: dummy Flow Director packet, unmap and kfree() + * @ICE_TX_BUF_FRAG: mapped skb OR &xdp_buff frag, only unmap DMA + * @ICE_TX_BUF_SKB: &sk_buff, unmap and consume_skb(), update stats + * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free(), stats + * @ICE_TX_BUF_XDP_XMIT: &xdp_frame, unmap and xdp_return_frame(), stats + * @ICE_TX_BUF_XSK_TX: &xdp_buff on XSk queue, xsk_buff_free(), stats + */ +enum ice_tx_buf_type { + ICE_TX_BUF_EMPTY = 0U, + ICE_TX_BUF_DUMMY, + ICE_TX_BUF_FRAG, + ICE_TX_BUF_SKB, + ICE_TX_BUF_XDP_TX, + ICE_TX_BUF_XDP_XMIT, + ICE_TX_BUF_XSK_TX, +}; + struct ice_tx_buf { union { struct ice_tx_desc *next_to_watch; u32 rs_idx; }; union { - struct sk_buff *skb; - void *raw_buf; /* used for XDP */ - struct xdp_buff *xdp; /* used for XDP_TX ZC */ + void *raw_buf; /* used for XDP_TX and FDir rules */ + struct sk_buff *skb; /* used for .ndo_start_xmit() */ + struct xdp_frame *xdpf; /* used for .ndo_xdp_xmit() */ + struct xdp_buff *xdp; /* used for XDP_TX ZC */ }; unsigned int bytecount; union { unsigned int gso_segs; - unsigned int nr_frags; /* used for mbuf XDP */ + unsigned int nr_frags; /* used for mbuf XDP */ }; - u32 tx_flags; + u32 type:16; /* &ice_tx_buf_type */ + u32 tx_flags:16; DEFINE_DMA_UNMAP_LEN(len); DEFINE_DMA_UNMAP_ADDR(dma); }; diff --git a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c index 9bbed3f14e42..7bc5aa340c7d 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c +++ b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c @@ -222,18 +222,28 @@ ice_receive_skb(struct ice_rx_ring *rx_ring, struct sk_buff *skb, u16 vlan_tag) /** * ice_clean_xdp_tx_buf - Free and unmap XDP Tx buffer - * @xdp_ring: XDP Tx ring + * @dev: device for DMA mapping * @tx_buf: Tx buffer to clean + * @bq: XDP bulk flush struct */ static void -ice_clean_xdp_tx_buf(struct ice_tx_ring *xdp_ring, struct ice_tx_buf *tx_buf) +ice_clean_xdp_tx_buf(struct device *dev, struct ice_tx_buf *tx_buf, + struct xdp_frame_bulk *bq) { - dma_unmap_single(xdp_ring->dev, dma_unmap_addr(tx_buf, dma), + dma_unmap_single(dev, dma_unmap_addr(tx_buf, dma), dma_unmap_len(tx_buf, len), DMA_TO_DEVICE); dma_unmap_len_set(tx_buf, len, 0); - xdp_ring->xdp_tx_active--; - page_frag_free(tx_buf->raw_buf); - tx_buf->raw_buf = NULL; + + switch (tx_buf->type) { + case ICE_TX_BUF_XDP_TX: + page_frag_free(tx_buf->raw_buf); + break; + case ICE_TX_BUF_XDP_XMIT: + xdp_return_frame_bulk(tx_buf->xdpf, bq); + break; + } + + tx_buf->type = ICE_TX_BUF_EMPTY; } /** @@ -243,11 +253,13 @@ ice_clean_xdp_tx_buf(struct ice_tx_ring *xdp_ring, struct ice_tx_buf *tx_buf) static u32 ice_clean_xdp_irq(struct ice_tx_ring *xdp_ring) { int total_bytes = 0, total_pkts = 0; + struct device *dev = xdp_ring->dev; u32 ntc = xdp_ring->next_to_clean; struct ice_tx_desc *tx_desc; u32 cnt = xdp_ring->count; + struct xdp_frame_bulk bq; + u32 frags, xdp_tx = 0; u32 ready_frames = 0; - u32 frags; u32 idx; u32 ret; @@ -261,12 +273,16 @@ static u32 ice_clean_xdp_irq(struct ice_tx_ring *xdp_ring) ready_frames = idx + cnt - ntc + 1; } - if (!ready_frames) + if (unlikely(!ready_frames)) return 0; ret = ready_frames; + xdp_frame_bulk_init(&bq); + rcu_read_lock(); /* xdp_return_frame_bulk() */ + while (ready_frames) { struct ice_tx_buf *tx_buf = &xdp_ring->tx_buf[ntc]; + struct ice_tx_buf *head = tx_buf; /* bytecount holds size of head + frags */ total_bytes += tx_buf->bytecount; @@ -274,11 +290,8 @@ static u32 ice_clean_xdp_irq(struct ice_tx_ring *xdp_ring) total_pkts++; /* count head + frags */ ready_frames -= frags + 1; + xdp_tx++; - if (xdp_ring->xsk_pool) - xsk_buff_free(tx_buf->xdp); - else - ice_clean_xdp_tx_buf(xdp_ring, tx_buf); ntc++; if (ntc == cnt) ntc = 0; @@ -286,15 +299,21 @@ static u32 ice_clean_xdp_irq(struct ice_tx_ring *xdp_ring) for (int i = 0; i < frags; i++) { tx_buf = &xdp_ring->tx_buf[ntc]; - ice_clean_xdp_tx_buf(xdp_ring, tx_buf); + ice_clean_xdp_tx_buf(dev, tx_buf, &bq); ntc++; if (ntc == cnt) ntc = 0; } + + ice_clean_xdp_tx_buf(dev, head, &bq); } + xdp_flush_frame_bulk(&bq); + rcu_read_unlock(); + tx_desc->cmd_type_offset_bsz = 0; xdp_ring->next_to_clean = ntc; + xdp_ring->xdp_tx_active -= xdp_tx; ice_update_tx_ring_stats(xdp_ring, total_pkts, total_bytes); return ret; @@ -304,8 +323,10 @@ static u32 ice_clean_xdp_irq(struct ice_tx_ring *xdp_ring) * __ice_xmit_xdp_ring - submit frame to XDP ring for transmission * @xdp: XDP buffer to be placed onto Tx descriptors * @xdp_ring: XDP ring for transmission + * @frame: whether this comes from .ndo_xdp_xmit() */ -int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring) +int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring, + bool frame) { struct skb_shared_info *sinfo = NULL; u32 size = xdp->data_end - xdp->data; @@ -321,17 +342,17 @@ int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring) u32 frag = 0; free_space = ICE_DESC_UNUSED(xdp_ring); - - if (ICE_DESC_UNUSED(xdp_ring) < ICE_RING_QUARTER(xdp_ring)) + if (free_space < ICE_RING_QUARTER(xdp_ring)) free_space += ice_clean_xdp_irq(xdp_ring); + if (unlikely(!free_space)) + goto busy; + if (unlikely(xdp_buff_has_frags(xdp))) { sinfo = xdp_get_shared_info_from_buff(xdp); nr_frags = sinfo->nr_frags; - if (free_space < nr_frags + 1) { - xdp_ring->ring_stats->tx_stats.tx_busy++; - return ICE_XDP_CONSUMED; - } + if (free_space < nr_frags + 1) + goto busy; } tx_desc = ICE_TX_DESC(xdp_ring, ntu); @@ -349,9 +370,15 @@ int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring) dma_unmap_len_set(tx_buf, len, size); dma_unmap_addr_set(tx_buf, dma, dma); + if (frame) { + tx_buf->type = ICE_TX_BUF_FRAG; + } else { + tx_buf->type = ICE_TX_BUF_XDP_TX; + tx_buf->raw_buf = data; + } + tx_desc->buf_addr = cpu_to_le64(dma); tx_desc->cmd_type_offset_bsz = ice_build_ctob(0, 0, size, 0); - tx_buf->raw_buf = data; ntu++; if (ntu == cnt) @@ -372,6 +399,11 @@ int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring) tx_head->bytecount = xdp_get_buff_len(xdp); tx_head->nr_frags = nr_frags; + if (frame) { + tx_head->type = ICE_TX_BUF_XDP_XMIT; + tx_head->xdpf = xdp->data_hard_start; + } + /* update last descriptor from a frame with EOP */ tx_desc->cmd_type_offset_bsz |= cpu_to_le64(ICE_TX_DESC_CMD_EOP << ICE_TXD_QW1_CMD_S); @@ -395,19 +427,11 @@ dma_unmap: ntu--; } return ICE_XDP_CONSUMED; -} -/** - * ice_xmit_xdp_ring - submit frame to XDP ring for transmission - * @xdpf: XDP frame that will be converted to XDP buff - * @xdp_ring: XDP ring for transmission - */ -int ice_xmit_xdp_ring(struct xdp_frame *xdpf, struct ice_tx_ring *xdp_ring) -{ - struct xdp_buff xdp; +busy: + xdp_ring->ring_stats->tx_stats.tx_busy++; - xdp_convert_frame_to_buff(xdpf, &xdp); - return __ice_xmit_xdp_ring(&xdp, xdp_ring); + return ICE_XDP_CONSUMED; } /** diff --git a/drivers/net/ethernet/intel/ice/ice_txrx_lib.h b/drivers/net/ethernet/intel/ice/ice_txrx_lib.h index ea977f283c22..115969ecdf7b 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx_lib.h +++ b/drivers/net/ethernet/intel/ice/ice_txrx_lib.h @@ -142,8 +142,8 @@ static inline u32 ice_set_rs_bit(const struct ice_tx_ring *xdp_ring) void ice_finalize_xdp_rx(struct ice_tx_ring *xdp_ring, unsigned int xdp_res, u32 first_idx); int ice_xmit_xdp_buff(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring); -int ice_xmit_xdp_ring(struct xdp_frame *xdpf, struct ice_tx_ring *xdp_ring); -int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring); +int __ice_xmit_xdp_ring(struct xdp_buff *xdp, struct ice_tx_ring *xdp_ring, + bool frame); void ice_release_rx_desc(struct ice_rx_ring *rx_ring, u16 val); void ice_process_skb_fields(struct ice_rx_ring *rx_ring, diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c index b2d96ae5668c..31565bbafa22 100644 --- a/drivers/net/ethernet/intel/ice/ice_xsk.c +++ b/drivers/net/ethernet/intel/ice/ice_xsk.c @@ -598,21 +598,6 @@ ice_construct_skb_zc(struct ice_rx_ring *rx_ring, struct xdp_buff *xdp) } /** - * ice_clean_xdp_tx_buf - Free and unmap XDP Tx buffer - * @xdp_ring: XDP Tx ring - * @tx_buf: Tx buffer to clean - */ -static void -ice_clean_xdp_tx_buf(struct ice_tx_ring *xdp_ring, struct ice_tx_buf *tx_buf) -{ - page_frag_free(tx_buf->raw_buf); - xdp_ring->xdp_tx_active--; - dma_unmap_single(xdp_ring->dev, dma_unmap_addr(tx_buf, dma), - dma_unmap_len(tx_buf, len), DMA_TO_DEVICE); - dma_unmap_len_set(tx_buf, len, 0); -} - -/** * ice_clean_xdp_irq_zc - produce AF_XDP descriptors to CQ * @xdp_ring: XDP Tx ring */ @@ -629,8 +614,8 @@ static void ice_clean_xdp_irq_zc(struct ice_tx_ring *xdp_ring) last_rs = xdp_ring->next_to_use ? xdp_ring->next_to_use - 1 : cnt - 1; tx_desc = ICE_TX_DESC(xdp_ring, last_rs); - if ((tx_desc->cmd_type_offset_bsz & - cpu_to_le64(ICE_TX_DESC_DTYPE_DESC_DONE))) { + if (tx_desc->cmd_type_offset_bsz & + cpu_to_le64(ICE_TX_DESC_DTYPE_DESC_DONE)) { if (last_rs >= ntc) completed_frames = last_rs - ntc + 1; else @@ -649,9 +634,10 @@ static void ice_clean_xdp_irq_zc(struct ice_tx_ring *xdp_ring) for (i = 0; i < completed_frames; i++) { tx_buf = &xdp_ring->tx_buf[ntc]; - if (tx_buf->raw_buf) { - ice_clean_xdp_tx_buf(xdp_ring, tx_buf); - tx_buf->raw_buf = NULL; + if (tx_buf->type == ICE_TX_BUF_XSK_TX) { + tx_buf->type = ICE_TX_BUF_EMPTY; + xsk_buff_free(tx_buf->xdp); + xdp_ring->xdp_tx_active--; } else { xsk_frames++; } @@ -703,6 +689,7 @@ static int ice_xmit_xdp_tx_zc(struct xdp_buff *xdp, tx_buf = &xdp_ring->tx_buf[ntu]; tx_buf->xdp = xdp; + tx_buf->type = ICE_TX_BUF_XSK_TX; tx_desc = ICE_TX_DESC(xdp_ring, ntu); tx_desc->buf_addr = cpu_to_le64(dma); tx_desc->cmd_type_offset_bsz = ice_build_ctob(ICE_TX_DESC_CMD_EOP, @@ -1101,12 +1088,12 @@ void ice_xsk_clean_xdp_ring(struct ice_tx_ring *xdp_ring) while (ntc != ntu) { struct ice_tx_buf *tx_buf = &xdp_ring->tx_buf[ntc]; - if (tx_buf->xdp) + if (tx_buf->type == ICE_TX_BUF_XSK_TX) { + tx_buf->type = ICE_TX_BUF_EMPTY; xsk_buff_free(tx_buf->xdp); - else + } else { xsk_frames++; - - tx_buf->raw_buf = NULL; + } ntc++; if (ntc >= xdp_ring->count) diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h index 5eea2b6cf6bd..389663a13d1d 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h @@ -888,6 +888,9 @@ int rvu_cpt_init(struct rvu *rvu); int rvu_set_channels_base(struct rvu *rvu); void rvu_program_channels(struct rvu *rvu); +/* CN10K NIX */ +void rvu_nix_block_cn10k_init(struct rvu *rvu, struct nix_hw *nix_hw); + /* CN10K RVU - LMT*/ void rvu_reset_lmt_map_tbl(struct rvu *rvu, u16 pcifunc); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c index 7dbbc115cde4..4ad9ff025c96 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c @@ -538,3 +538,21 @@ void rvu_program_channels(struct rvu *rvu) rvu_lbk_set_channels(rvu); rvu_rpm_set_channels(rvu); } + +void rvu_nix_block_cn10k_init(struct rvu *rvu, struct nix_hw *nix_hw) +{ + int blkaddr = nix_hw->blkaddr; + u64 cfg; + + /* Set AF vWQE timer interval to a LF configurable range of + * 6.4us to 1.632ms. + */ + rvu_write64(rvu, blkaddr, NIX_AF_VWQE_TIMER, 0x3FULL); + + /* Enable NIX RX stream and global conditional clock to + * avoild multiple free of NPA buffers. + */ + cfg = rvu_read64(rvu, blkaddr, NIX_AF_CFG); + cfg |= BIT_ULL(1) | BIT_ULL(2); + rvu_write64(rvu, blkaddr, NIX_AF_CFG, cfg); +} diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c index 89e94569e74c..26e639e57dae 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c @@ -2058,6 +2058,13 @@ static int nix_smq_flush(struct rvu *rvu, int blkaddr, int err, restore_tx_en = 0; u64 cfg; + if (!is_rvu_otx2(rvu)) { + /* Skip SMQ flush if pkt count is zero */ + cfg = rvu_read64(rvu, blkaddr, NIX_AF_MDQX_IN_MD_COUNT(smq)); + if (!cfg) + return 0; + } + /* enable cgx tx if disabled */ if (is_pf_cgxmapped(rvu, pf)) { rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id); @@ -4309,6 +4316,9 @@ static int rvu_nix_block_init(struct rvu *rvu, struct nix_hw *nix_hw) rvu_write64(rvu, blkaddr, NIX_AF_SEB_CFG, cfg); + if (!is_rvu_otx2(rvu)) + rvu_nix_block_cn10k_init(rvu, nix_hw); + if (is_block_implemented(hw, blkaddr)) { err = nix_setup_txschq(rvu, nix_hw, blkaddr); if (err) diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_reg.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu_reg.h index 5437bd20c719..1729b22580ce 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_reg.h +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_reg.h @@ -189,6 +189,7 @@ #define NIX_AF_RX_CFG (0x00D0) #define NIX_AF_AVG_DELAY (0x00E0) #define NIX_AF_CINT_DELAY (0x00F0) +#define NIX_AF_VWQE_TIMER (0x00F8) #define NIX_AF_RX_MCAST_BASE (0x0100) #define NIX_AF_RX_MCAST_CFG (0x0110) #define NIX_AF_RX_MCAST_BUF_BASE (0x0120) @@ -426,6 +427,7 @@ #define NIX_AF_RX_NPC_MIRROR_DROP (0x4730) #define NIX_AF_RX_ACTIVE_CYCLES_PCX(a) (0x4800 | (a) << 16) #define NIX_AF_LINKX_CFG(a) (0x4010 | (a) << 17) +#define NIX_AF_MDQX_IN_MD_COUNT(a) (0x14e0 | (a) << 16) #define NIX_PRIV_AF_INT_CFG (0x8000000) #define NIX_PRIV_LFX_CFG (0x8000010) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index c5758637b7be..2f79378fbf6e 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -699,32 +699,32 @@ static void build_inline_wqe(struct mlx4_en_tx_desc *tx_desc, inl->byte_count = cpu_to_be32(1 << 31 | skb->len); } else { inl->byte_count = cpu_to_be32(1 << 31 | MIN_PKT_LEN); - memset(((void *)(inl + 1)) + skb->len, 0, + memset(inl->data + skb->len, 0, MIN_PKT_LEN - skb->len); } - skb_copy_from_linear_data(skb, inl + 1, hlen); + skb_copy_from_linear_data(skb, inl->data, hlen); if (shinfo->nr_frags) - memcpy(((void *)(inl + 1)) + hlen, fragptr, + memcpy(inl->data + hlen, fragptr, skb_frag_size(&shinfo->frags[0])); } else { inl->byte_count = cpu_to_be32(1 << 31 | spc); if (hlen <= spc) { - skb_copy_from_linear_data(skb, inl + 1, hlen); + skb_copy_from_linear_data(skb, inl->data, hlen); if (hlen < spc) { - memcpy(((void *)(inl + 1)) + hlen, + memcpy(inl->data + hlen, fragptr, spc - hlen); fragptr += spc - hlen; } - inl = (void *) (inl + 1) + spc; - memcpy(((void *)(inl + 1)), fragptr, skb->len - spc); + inl = (void *)inl->data + spc; + memcpy(inl->data, fragptr, skb->len - spc); } else { - skb_copy_from_linear_data(skb, inl + 1, spc); - inl = (void *) (inl + 1) + spc; - skb_copy_from_linear_data_offset(skb, spc, inl + 1, + skb_copy_from_linear_data(skb, inl->data, spc); + inl = (void *)inl->data + spc; + skb_copy_from_linear_data_offset(skb, spc, inl->data, hlen - spc); if (shinfo->nr_frags) - memcpy(((void *)(inl + 1)) + hlen - spc, + memcpy(inl->data + hlen - spc, fragptr, skb_frag_size(&shinfo->frags[0])); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig index 26685fd0fdaa..bb1d7b039a7e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig @@ -85,7 +85,7 @@ config MLX5_BRIDGE config MLX5_CLS_ACT bool "MLX5 TC classifier action support" - depends on MLX5_ESWITCH && NET_CLS_ACT + depends on MLX5_ESWITCH && NET_CLS_ACT && NET_TC_SKB_EXT default y help mlx5 ConnectX offloads support for TC classifier action (NET_CLS_ACT), @@ -100,7 +100,7 @@ config MLX5_CLS_ACT config MLX5_TC_CT bool "MLX5 TC connection tracking offload support" - depends on MLX5_CLS_ACT && NF_FLOW_TABLE && NET_ACT_CT && NET_TC_SKB_EXT + depends on MLX5_CLS_ACT && NF_FLOW_TABLE && NET_ACT_CT default y help Say Y here if you want to support offloading connection tracking rules diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/tc.c index 3b590cfe33b8..e24b46953542 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/tc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/tc.c @@ -1,7 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB /* Copyright (c) 2020 Mellanox Technologies. */ -#include <net/dst_metadata.h> #include <linux/netdevice.h> #include <linux/if_macvlan.h> #include <linux/list.h> @@ -665,232 +664,54 @@ void mlx5e_rep_tc_netdevice_event_unregister(struct mlx5e_rep_priv *rpriv) mlx5e_rep_indr_block_unbind); } -static bool mlx5e_restore_tunnel(struct mlx5e_priv *priv, struct sk_buff *skb, - struct mlx5e_tc_update_priv *tc_priv, - u32 tunnel_id) -{ - struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; - struct tunnel_match_enc_opts enc_opts = {}; - struct mlx5_rep_uplink_priv *uplink_priv; - struct mlx5e_rep_priv *uplink_rpriv; - struct metadata_dst *tun_dst; - struct tunnel_match_key key; - u32 tun_id, enc_opts_id; - struct net_device *dev; - int err; - - enc_opts_id = tunnel_id & ENC_OPTS_BITS_MASK; - tun_id = tunnel_id >> ENC_OPTS_BITS; - - if (!tun_id) - return true; - - uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); - uplink_priv = &uplink_rpriv->uplink_priv; - - err = mapping_find(uplink_priv->tunnel_mapping, tun_id, &key); - if (err) { - netdev_dbg(priv->netdev, - "Couldn't find tunnel for tun_id: %d, err: %d\n", - tun_id, err); - return false; - } - - if (enc_opts_id) { - err = mapping_find(uplink_priv->tunnel_enc_opts_mapping, - enc_opts_id, &enc_opts); - if (err) { - netdev_dbg(priv->netdev, - "Couldn't find tunnel (opts) for tun_id: %d, err: %d\n", - enc_opts_id, err); - return false; - } - } - - if (key.enc_control.addr_type == FLOW_DISSECTOR_KEY_IPV4_ADDRS) { - tun_dst = __ip_tun_set_dst(key.enc_ipv4.src, key.enc_ipv4.dst, - key.enc_ip.tos, key.enc_ip.ttl, - key.enc_tp.dst, TUNNEL_KEY, - key32_to_tunnel_id(key.enc_key_id.keyid), - enc_opts.key.len); - } else if (key.enc_control.addr_type == FLOW_DISSECTOR_KEY_IPV6_ADDRS) { - tun_dst = __ipv6_tun_set_dst(&key.enc_ipv6.src, &key.enc_ipv6.dst, - key.enc_ip.tos, key.enc_ip.ttl, - key.enc_tp.dst, 0, TUNNEL_KEY, - key32_to_tunnel_id(key.enc_key_id.keyid), - enc_opts.key.len); - } else { - netdev_dbg(priv->netdev, - "Couldn't restore tunnel, unsupported addr_type: %d\n", - key.enc_control.addr_type); - return false; - } - - if (!tun_dst) { - netdev_dbg(priv->netdev, "Couldn't restore tunnel, no tun_dst\n"); - return false; - } - - tun_dst->u.tun_info.key.tp_src = key.enc_tp.src; - - if (enc_opts.key.len) - ip_tunnel_info_opts_set(&tun_dst->u.tun_info, - enc_opts.key.data, - enc_opts.key.len, - enc_opts.key.dst_opt_type); - - skb_dst_set(skb, (struct dst_entry *)tun_dst); - dev = dev_get_by_index(&init_net, key.filter_ifindex); - if (!dev) { - netdev_dbg(priv->netdev, - "Couldn't find tunnel device with ifindex: %d\n", - key.filter_ifindex); - return false; - } - - /* Set fwd_dev so we do dev_put() after datapath */ - tc_priv->fwd_dev = dev; - - skb->dev = dev; - - return true; -} - -static bool mlx5e_restore_skb_chain(struct sk_buff *skb, u32 chain, u32 reg_c1, - struct mlx5e_tc_update_priv *tc_priv) -{ - struct mlx5e_priv *priv = netdev_priv(skb->dev); - u32 tunnel_id = (reg_c1 >> ESW_TUN_OFFSET) & TUNNEL_ID_MASK; - -#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT) - if (chain) { - struct mlx5_rep_uplink_priv *uplink_priv; - struct mlx5e_rep_priv *uplink_rpriv; - struct tc_skb_ext *tc_skb_ext; - struct mlx5_eswitch *esw; - u32 zone_restore_id; - - tc_skb_ext = tc_skb_ext_alloc(skb); - if (!tc_skb_ext) { - WARN_ON(1); - return false; - } - tc_skb_ext->chain = chain; - zone_restore_id = reg_c1 & ESW_ZONE_ID_MASK; - esw = priv->mdev->priv.eswitch; - uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); - uplink_priv = &uplink_rpriv->uplink_priv; - if (!mlx5e_tc_ct_restore_flow(uplink_priv->ct_priv, skb, - zone_restore_id)) - return false; - } -#endif /* CONFIG_NET_TC_SKB_EXT */ - - return mlx5e_restore_tunnel(priv, skb, tc_priv, tunnel_id); -} - -static void mlx5_rep_tc_post_napi_receive(struct mlx5e_tc_update_priv *tc_priv) -{ - if (tc_priv->fwd_dev) - dev_put(tc_priv->fwd_dev); -} - -static void mlx5e_restore_skb_sample(struct mlx5e_priv *priv, struct sk_buff *skb, - struct mlx5_mapped_obj *mapped_obj, - struct mlx5e_tc_update_priv *tc_priv) -{ - if (!mlx5e_restore_tunnel(priv, skb, tc_priv, mapped_obj->sample.tunnel_id)) { - netdev_dbg(priv->netdev, - "Failed to restore tunnel info for sampled packet\n"); - return; - } - mlx5e_tc_sample_skb(skb, mapped_obj); - mlx5_rep_tc_post_napi_receive(tc_priv); -} - -static bool mlx5e_restore_skb_int_port(struct mlx5e_priv *priv, struct sk_buff *skb, - struct mlx5_mapped_obj *mapped_obj, - struct mlx5e_tc_update_priv *tc_priv, - bool *forward_tx, - u32 reg_c1) -{ - u32 tunnel_id = (reg_c1 >> ESW_TUN_OFFSET) & TUNNEL_ID_MASK; - struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; - struct mlx5_rep_uplink_priv *uplink_priv; - struct mlx5e_rep_priv *uplink_rpriv; - - /* Tunnel restore takes precedence over int port restore */ - if (tunnel_id) - return mlx5e_restore_tunnel(priv, skb, tc_priv, tunnel_id); - - uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); - uplink_priv = &uplink_rpriv->uplink_priv; - - if (mlx5e_tc_int_port_dev_fwd(uplink_priv->int_port_priv, skb, - mapped_obj->int_port_metadata, forward_tx)) { - /* Set fwd_dev for future dev_put */ - tc_priv->fwd_dev = skb->dev; - - return true; - } - - return false; -} - void mlx5e_rep_tc_receive(struct mlx5_cqe64 *cqe, struct mlx5e_rq *rq, struct sk_buff *skb) { - u32 reg_c1 = be32_to_cpu(cqe->ft_metadata); + u32 reg_c0, reg_c1, zone_restore_id, tunnel_id; struct mlx5e_tc_update_priv tc_priv = {}; - struct mlx5_mapped_obj mapped_obj; + struct mlx5_rep_uplink_priv *uplink_priv; + struct mlx5e_rep_priv *uplink_rpriv; + struct mlx5_tc_ct_priv *ct_priv; + struct mapping_ctx *mapping_ctx; struct mlx5_eswitch *esw; - bool forward_tx = false; struct mlx5e_priv *priv; - u32 reg_c0; - int err; reg_c0 = (be32_to_cpu(cqe->sop_drop_qpn) & MLX5E_TC_FLOW_ID_MASK); if (!reg_c0 || reg_c0 == MLX5_FS_DEFAULT_FLOW_TAG) goto forward; - /* If reg_c0 is not equal to the default flow tag then skb->mark + /* If mapped_obj_id is not equal to the default flow tag then skb->mark * is not supported and must be reset back to 0. */ skb->mark = 0; priv = netdev_priv(skb->dev); esw = priv->mdev->priv.eswitch; - err = mapping_find(esw->offloads.reg_c0_obj_pool, reg_c0, &mapped_obj); - if (err) { - netdev_dbg(priv->netdev, - "Couldn't find mapped object for reg_c0: %d, err: %d\n", - reg_c0, err); - goto free_skb; - } + mapping_ctx = esw->offloads.reg_c0_obj_pool; + reg_c1 = be32_to_cpu(cqe->ft_metadata); + zone_restore_id = reg_c1 & ESW_ZONE_ID_MASK; + tunnel_id = (reg_c1 >> ESW_TUN_OFFSET) & TUNNEL_ID_MASK; - if (mapped_obj.type == MLX5_MAPPED_OBJ_CHAIN) { - if (!mlx5e_restore_skb_chain(skb, mapped_obj.chain, reg_c1, &tc_priv) && - !mlx5_ipsec_is_rx_flow(cqe)) - goto free_skb; - } else if (mapped_obj.type == MLX5_MAPPED_OBJ_SAMPLE) { - mlx5e_restore_skb_sample(priv, skb, &mapped_obj, &tc_priv); - goto free_skb; - } else if (mapped_obj.type == MLX5_MAPPED_OBJ_INT_PORT_METADATA) { - if (!mlx5e_restore_skb_int_port(priv, skb, &mapped_obj, &tc_priv, - &forward_tx, reg_c1)) - goto free_skb; - } else { - netdev_dbg(priv->netdev, "Invalid mapped object type: %d\n", mapped_obj.type); + uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); + uplink_priv = &uplink_rpriv->uplink_priv; + ct_priv = uplink_priv->ct_priv; + + if (!mlx5_ipsec_is_rx_flow(cqe) && + !mlx5e_tc_update_skb(cqe, skb, mapping_ctx, reg_c0, ct_priv, zone_restore_id, tunnel_id, + &tc_priv)) goto free_skb; - } forward: - if (forward_tx) + if (tc_priv.skb_done) + goto free_skb; + + if (tc_priv.forward_tx) dev_queue_xmit(skb); else napi_gro_receive(rq->cq.napi, skb); - mlx5_rep_tc_post_napi_receive(&tc_priv); + if (tc_priv.fwd_dev) + dev_put(tc_priv.fwd_dev); return; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/sample.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/sample.c index f2c2c752bd1c..558a776359af 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/sample.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/sample.c @@ -237,7 +237,7 @@ sample_modify_hdr_get(struct mlx5_core_dev *mdev, u32 obj_id, int err; err = mlx5e_tc_match_to_reg_set(mdev, mod_acts, MLX5_FLOW_NAMESPACE_FDB, - CHAIN_TO_REG, obj_id); + MAPPED_OBJ_TO_REG, obj_id); if (err) goto err_set_regc0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c index f0acb02ffc76..314983bc6f08 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c @@ -60,6 +60,7 @@ struct mlx5_tc_ct_debugfs { struct mlx5_tc_ct_priv { struct mlx5_core_dev *dev; + struct mlx5e_priv *priv; const struct net_device *netdev; struct mod_hdr_tbl *mod_hdr_tbl; struct xarray tuple_ids; @@ -86,7 +87,6 @@ struct mlx5_ct_flow { struct mlx5_flow_attr *pre_ct_attr; struct mlx5_flow_handle *pre_ct_rule; struct mlx5_ct_ft *ft; - u32 chain_mapping; }; struct mlx5_ct_zone_rule { @@ -1558,6 +1558,7 @@ mlx5_tc_ct_parse_action(struct mlx5_tc_ct_priv *priv, attr->ct_attr.zone = act->ct.zone; attr->ct_attr.ct_action = act->ct.action; attr->ct_attr.nf_ft = act->ct.flow_table; + attr->ct_attr.act_miss_cookie = act->miss_cookie; return 0; } @@ -1895,7 +1896,7 @@ mlx5_tc_ct_del_ft_cb(struct mlx5_tc_ct_priv *ct_priv, struct mlx5_ct_ft *ft) * + ft prio (tc chain) + * + original match + * +---------------------+ - * | set chain miss mapping + * | set act_miss_cookie mapping * | set fte_id * | set tunnel_id * | do decap @@ -1940,7 +1941,7 @@ __mlx5_tc_ct_flow_offload(struct mlx5_tc_ct_priv *ct_priv, struct mlx5_flow_attr *pre_ct_attr; struct mlx5_modify_hdr *mod_hdr; struct mlx5_ct_flow *ct_flow; - int chain_mapping = 0, err; + int act_miss_mapping = 0, err; struct mlx5_ct_ft *ft; u16 zone; @@ -1975,22 +1976,18 @@ __mlx5_tc_ct_flow_offload(struct mlx5_tc_ct_priv *ct_priv, pre_ct_attr->action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST | MLX5_FLOW_CONTEXT_ACTION_MOD_HDR; - /* Write chain miss tag for miss in ct table as we - * don't go though all prios of this chain as normal tc rules - * miss. - */ - err = mlx5_chains_get_chain_mapping(ct_priv->chains, attr->chain, - &chain_mapping); + err = mlx5e_tc_action_miss_mapping_get(ct_priv->priv, attr, attr->ct_attr.act_miss_cookie, + &act_miss_mapping); if (err) { - ct_dbg("Failed to get chain register mapping for chain"); - goto err_get_chain; + ct_dbg("Failed to get register mapping for act miss"); + goto err_get_act_miss; } - ct_flow->chain_mapping = chain_mapping; + attr->ct_attr.act_miss_mapping = act_miss_mapping; err = mlx5e_tc_match_to_reg_set(priv->mdev, pre_mod_acts, ct_priv->ns_type, - CHAIN_TO_REG, chain_mapping); + MAPPED_OBJ_TO_REG, act_miss_mapping); if (err) { - ct_dbg("Failed to set chain register mapping"); + ct_dbg("Failed to set act miss register mapping"); goto err_mapping; } @@ -2054,8 +2051,8 @@ err_insert_orig: mlx5_modify_header_dealloc(priv->mdev, pre_ct_attr->modify_hdr); err_mapping: mlx5e_mod_hdr_dealloc(pre_mod_acts); - mlx5_chains_put_chain_mapping(ct_priv->chains, ct_flow->chain_mapping); -err_get_chain: + mlx5e_tc_action_miss_mapping_put(ct_priv->priv, attr, act_miss_mapping); +err_get_act_miss: kfree(ct_flow->pre_ct_attr); err_alloc_pre: mlx5_tc_ct_del_ft_cb(ct_priv, ft); @@ -2094,7 +2091,7 @@ __mlx5_tc_ct_delete_flow(struct mlx5_tc_ct_priv *ct_priv, mlx5_tc_rule_delete(priv, ct_flow->pre_ct_rule, pre_ct_attr); mlx5_modify_header_dealloc(priv->mdev, pre_ct_attr->modify_hdr); - mlx5_chains_put_chain_mapping(ct_priv->chains, ct_flow->chain_mapping); + mlx5e_tc_action_miss_mapping_put(ct_priv->priv, attr, attr->ct_attr.act_miss_mapping); mlx5_tc_ct_del_ft_cb(ct_priv, ct_flow->ft); kfree(ct_flow->pre_ct_attr); @@ -2191,13 +2188,6 @@ mlx5_tc_ct_init_check_support(struct mlx5e_priv *priv, const char *err_msg = NULL; int err = 0; -#if !IS_ENABLED(CONFIG_NET_TC_SKB_EXT) - /* cannot restore chain ID on HW miss */ - - err_msg = "tc skb extension missing"; - err = -EOPNOTSUPP; - goto out_err; -#endif if (IS_ERR_OR_NULL(post_act)) { /* Ignore_flow_level support isn't supported by default for VFs and so post_act * won't be supported. Skip showing error msg. @@ -2274,6 +2264,7 @@ mlx5_tc_ct_init(struct mlx5e_priv *priv, struct mlx5_fs_chains *chains, } spin_lock_init(&ct_priv->ht_lock); + ct_priv->priv = priv; ct_priv->ns_type = ns_type; ct_priv->chains = chains; ct_priv->netdev = priv->netdev; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.h b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.h index 5bbd6b92840f..5c5ddaa83055 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.h @@ -28,6 +28,8 @@ struct mlx5_ct_attr { struct mlx5_ct_flow *ct_flow; struct nf_flowtable *nf_ft; u32 ct_labels_id; + u32 act_miss_mapping; + u64 act_miss_cookie; }; #define zone_to_reg_ct {\ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 15d9932f741d..3f7b63d6616b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -1790,7 +1790,7 @@ static void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb); if (mlx5e_cqe_regb_chain(cqe)) - if (!mlx5e_tc_update_skb(cqe, skb)) { + if (!mlx5e_tc_update_skb_nic(cqe, skb)) { dev_kfree_skb_any(skb); goto free_wqe; } @@ -2257,7 +2257,7 @@ static void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cq mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb); if (mlx5e_cqe_regb_chain(cqe)) - if (!mlx5e_tc_update_skb(cqe, skb)) { + if (!mlx5e_tc_update_skb_nic(cqe, skb)) { dev_kfree_skb_any(skb); goto mpwrq_cqe_out; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c index 9bbd31e304be..e34d9b5fb504 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c @@ -43,6 +43,7 @@ #include <net/ipv6_stubs.h> #include <net/bareudp.h> #include <net/bonding.h> +#include <net/dst_metadata.h> #include "en.h" #include "en/tc/post_act.h" #include "en/tc/act_stats.h" @@ -108,7 +109,7 @@ struct mlx5e_tc_table { }; struct mlx5e_tc_attr_to_reg_mapping mlx5e_tc_attr_to_reg_mappings[] = { - [CHAIN_TO_REG] = { + [MAPPED_OBJ_TO_REG] = { .mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_0, .moffset = 0, .mlen = 16, @@ -135,7 +136,7 @@ struct mlx5e_tc_attr_to_reg_mapping mlx5e_tc_attr_to_reg_mappings[] = { * into reg_b that is passed to SW since we don't * jump between steering domains. */ - [NIC_CHAIN_TO_REG] = { + [NIC_MAPPED_OBJ_TO_REG] = { .mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_B, .moffset = 0, .mlen = 16, @@ -1604,7 +1605,7 @@ mlx5e_tc_offload_to_slow_path(struct mlx5_eswitch *esw, goto err_get_chain; err = mlx5e_tc_match_to_reg_set(esw->dev, &mod_acts, MLX5_FLOW_NAMESPACE_FDB, - CHAIN_TO_REG, chain_mapping); + MAPPED_OBJ_TO_REG, chain_mapping); if (err) goto err_reg_set; @@ -3815,6 +3816,7 @@ mlx5e_clone_flow_attr_for_post_act(struct mlx5_flow_attr *attr, attr2->parse_attr = parse_attr; attr2->dest_chain = 0; attr2->dest_ft = NULL; + attr2->act_id_restore_rule = NULL; if (ns_type == MLX5_FLOW_NAMESPACE_FDB) { attr2->esw_attr->out_count = 0; @@ -4176,7 +4178,7 @@ parse_tc_actions(struct mlx5e_tc_act_parse_state *parse_state, parse_state->actions |= attr->action; if (!tc_act->stats_action) - attr->tc_act_cookies[attr->tc_act_cookies_count++] = act->act_cookie; + attr->tc_act_cookies[attr->tc_act_cookies_count++] = act->cookie; /* Split attr for multi table act if not the last act. */ if (jump_state.jump_target || @@ -5604,48 +5606,268 @@ int mlx5e_setup_tc_block_cb(enum tc_setup_type type, void *type_data, } } -bool mlx5e_tc_update_skb(struct mlx5_cqe64 *cqe, - struct sk_buff *skb) +static bool mlx5e_tc_restore_tunnel(struct mlx5e_priv *priv, struct sk_buff *skb, + struct mlx5e_tc_update_priv *tc_priv, + u32 tunnel_id) { -#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT) - u32 chain = 0, chain_tag, reg_b, zone_restore_id; - struct mlx5e_priv *priv = netdev_priv(skb->dev); - struct mlx5_mapped_obj mapped_obj; - struct tc_skb_ext *tc_skb_ext; - struct mlx5e_tc_table *tc; + struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; + struct tunnel_match_enc_opts enc_opts = {}; + struct mlx5_rep_uplink_priv *uplink_priv; + struct mlx5e_rep_priv *uplink_rpriv; + struct metadata_dst *tun_dst; + struct tunnel_match_key key; + u32 tun_id, enc_opts_id; + struct net_device *dev; int err; - reg_b = be32_to_cpu(cqe->ft_metadata); - tc = mlx5e_fs_get_tc(priv->fs); - chain_tag = reg_b & MLX5E_TC_TABLE_CHAIN_TAG_MASK; + enc_opts_id = tunnel_id & ENC_OPTS_BITS_MASK; + tun_id = tunnel_id >> ENC_OPTS_BITS; + + if (!tun_id) + return true; - err = mapping_find(tc->mapping, chain_tag, &mapped_obj); + uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); + uplink_priv = &uplink_rpriv->uplink_priv; + + err = mapping_find(uplink_priv->tunnel_mapping, tun_id, &key); if (err) { netdev_dbg(priv->netdev, - "Couldn't find chain for chain tag: %d, err: %d\n", - chain_tag, err); + "Couldn't find tunnel for tun_id: %d, err: %d\n", + tun_id, err); return false; } - if (mapped_obj.type == MLX5_MAPPED_OBJ_CHAIN) { - chain = mapped_obj.chain; - tc_skb_ext = tc_skb_ext_alloc(skb); - if (WARN_ON(!tc_skb_ext)) + if (enc_opts_id) { + err = mapping_find(uplink_priv->tunnel_enc_opts_mapping, + enc_opts_id, &enc_opts); + if (err) { + netdev_dbg(priv->netdev, + "Couldn't find tunnel (opts) for tun_id: %d, err: %d\n", + enc_opts_id, err); return false; + } + } + + switch (key.enc_control.addr_type) { + case FLOW_DISSECTOR_KEY_IPV4_ADDRS: + tun_dst = __ip_tun_set_dst(key.enc_ipv4.src, key.enc_ipv4.dst, + key.enc_ip.tos, key.enc_ip.ttl, + key.enc_tp.dst, TUNNEL_KEY, + key32_to_tunnel_id(key.enc_key_id.keyid), + enc_opts.key.len); + break; + case FLOW_DISSECTOR_KEY_IPV6_ADDRS: + tun_dst = __ipv6_tun_set_dst(&key.enc_ipv6.src, &key.enc_ipv6.dst, + key.enc_ip.tos, key.enc_ip.ttl, + key.enc_tp.dst, 0, TUNNEL_KEY, + key32_to_tunnel_id(key.enc_key_id.keyid), + enc_opts.key.len); + break; + default: + netdev_dbg(priv->netdev, + "Couldn't restore tunnel, unsupported addr_type: %d\n", + key.enc_control.addr_type); + return false; + } + + if (!tun_dst) { + netdev_dbg(priv->netdev, "Couldn't restore tunnel, no tun_dst\n"); + return false; + } + + tun_dst->u.tun_info.key.tp_src = key.enc_tp.src; + + if (enc_opts.key.len) + ip_tunnel_info_opts_set(&tun_dst->u.tun_info, + enc_opts.key.data, + enc_opts.key.len, + enc_opts.key.dst_opt_type); - tc_skb_ext->chain = chain; + skb_dst_set(skb, (struct dst_entry *)tun_dst); + dev = dev_get_by_index(&init_net, key.filter_ifindex); + if (!dev) { + netdev_dbg(priv->netdev, + "Couldn't find tunnel device with ifindex: %d\n", + key.filter_ifindex); + return false; + } + + /* Set fwd_dev so we do dev_put() after datapath */ + tc_priv->fwd_dev = dev; - zone_restore_id = (reg_b >> MLX5_REG_MAPPING_MOFFSET(NIC_ZONE_RESTORE_TO_REG)) & - ESW_ZONE_ID_MASK; + skb->dev = dev; - if (!mlx5e_tc_ct_restore_flow(tc->ct, skb, - zone_restore_id)) + return true; +} + +static bool mlx5e_tc_restore_skb_tc_meta(struct sk_buff *skb, struct mlx5_tc_ct_priv *ct_priv, + struct mlx5_mapped_obj *mapped_obj, u32 zone_restore_id, + u32 tunnel_id, struct mlx5e_tc_update_priv *tc_priv) +{ + struct mlx5e_priv *priv = netdev_priv(skb->dev); + struct tc_skb_ext *tc_skb_ext; + u64 act_miss_cookie; + u32 chain; + + chain = mapped_obj->type == MLX5_MAPPED_OBJ_CHAIN ? mapped_obj->chain : 0; + act_miss_cookie = mapped_obj->type == MLX5_MAPPED_OBJ_ACT_MISS ? + mapped_obj->act_miss_cookie : 0; + if (chain || act_miss_cookie) { + if (!mlx5e_tc_ct_restore_flow(ct_priv, skb, zone_restore_id)) return false; - } else { + + tc_skb_ext = tc_skb_ext_alloc(skb); + if (!tc_skb_ext) { + WARN_ON(1); + return false; + } + + if (act_miss_cookie) { + tc_skb_ext->act_miss_cookie = act_miss_cookie; + tc_skb_ext->act_miss = 1; + } else { + tc_skb_ext->chain = chain; + } + } + + if (tc_priv) + return mlx5e_tc_restore_tunnel(priv, skb, tc_priv, tunnel_id); + + return true; +} + +static void mlx5e_tc_restore_skb_sample(struct mlx5e_priv *priv, struct sk_buff *skb, + struct mlx5_mapped_obj *mapped_obj, + struct mlx5e_tc_update_priv *tc_priv) +{ + if (!mlx5e_tc_restore_tunnel(priv, skb, tc_priv, mapped_obj->sample.tunnel_id)) { + netdev_dbg(priv->netdev, + "Failed to restore tunnel info for sampled packet\n"); + return; + } + mlx5e_tc_sample_skb(skb, mapped_obj); +} + +static bool mlx5e_tc_restore_skb_int_port(struct mlx5e_priv *priv, struct sk_buff *skb, + struct mlx5_mapped_obj *mapped_obj, + struct mlx5e_tc_update_priv *tc_priv, + u32 tunnel_id) +{ + struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; + struct mlx5_rep_uplink_priv *uplink_priv; + struct mlx5e_rep_priv *uplink_rpriv; + bool forward_tx = false; + + /* Tunnel restore takes precedence over int port restore */ + if (tunnel_id) + return mlx5e_tc_restore_tunnel(priv, skb, tc_priv, tunnel_id); + + uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH); + uplink_priv = &uplink_rpriv->uplink_priv; + + if (mlx5e_tc_int_port_dev_fwd(uplink_priv->int_port_priv, skb, + mapped_obj->int_port_metadata, &forward_tx)) { + /* Set fwd_dev for future dev_put */ + tc_priv->fwd_dev = skb->dev; + tc_priv->forward_tx = forward_tx; + + return true; + } + + return false; +} + +bool mlx5e_tc_update_skb(struct mlx5_cqe64 *cqe, struct sk_buff *skb, + struct mapping_ctx *mapping_ctx, u32 mapped_obj_id, + struct mlx5_tc_ct_priv *ct_priv, + u32 zone_restore_id, u32 tunnel_id, + struct mlx5e_tc_update_priv *tc_priv) +{ + struct mlx5e_priv *priv = netdev_priv(skb->dev); + struct mlx5_mapped_obj mapped_obj; + int err; + + err = mapping_find(mapping_ctx, mapped_obj_id, &mapped_obj); + if (err) { + netdev_dbg(skb->dev, + "Couldn't find mapped object for mapped_obj_id: %d, err: %d\n", + mapped_obj_id, err); + return false; + } + + switch (mapped_obj.type) { + case MLX5_MAPPED_OBJ_CHAIN: + case MLX5_MAPPED_OBJ_ACT_MISS: + return mlx5e_tc_restore_skb_tc_meta(skb, ct_priv, &mapped_obj, zone_restore_id, + tunnel_id, tc_priv); + case MLX5_MAPPED_OBJ_SAMPLE: + mlx5e_tc_restore_skb_sample(priv, skb, &mapped_obj, tc_priv); + tc_priv->skb_done = true; + return true; + case MLX5_MAPPED_OBJ_INT_PORT_METADATA: + return mlx5e_tc_restore_skb_int_port(priv, skb, &mapped_obj, tc_priv, tunnel_id); + default: netdev_dbg(priv->netdev, "Invalid mapped object type: %d\n", mapped_obj.type); return false; } -#endif /* CONFIG_NET_TC_SKB_EXT */ - return true; + return false; +} + +bool mlx5e_tc_update_skb_nic(struct mlx5_cqe64 *cqe, struct sk_buff *skb) +{ + struct mlx5e_priv *priv = netdev_priv(skb->dev); + u32 mapped_obj_id, reg_b, zone_restore_id; + struct mlx5_tc_ct_priv *ct_priv; + struct mapping_ctx *mapping_ctx; + struct mlx5e_tc_table *tc; + + reg_b = be32_to_cpu(cqe->ft_metadata); + tc = mlx5e_fs_get_tc(priv->fs); + mapped_obj_id = reg_b & MLX5E_TC_TABLE_CHAIN_TAG_MASK; + zone_restore_id = (reg_b >> MLX5_REG_MAPPING_MOFFSET(NIC_ZONE_RESTORE_TO_REG)) & + ESW_ZONE_ID_MASK; + ct_priv = tc->ct; + mapping_ctx = tc->mapping; + + return mlx5e_tc_update_skb(cqe, skb, mapping_ctx, mapped_obj_id, ct_priv, zone_restore_id, + 0, NULL); +} + +int mlx5e_tc_action_miss_mapping_get(struct mlx5e_priv *priv, struct mlx5_flow_attr *attr, + u64 act_miss_cookie, u32 *act_miss_mapping) +{ + struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; + struct mlx5_mapped_obj mapped_obj = {}; + struct mapping_ctx *ctx; + int err; + + ctx = esw->offloads.reg_c0_obj_pool; + + mapped_obj.type = MLX5_MAPPED_OBJ_ACT_MISS; + mapped_obj.act_miss_cookie = act_miss_cookie; + err = mapping_add(ctx, &mapped_obj, act_miss_mapping); + if (err) + return err; + + attr->act_id_restore_rule = esw_add_restore_rule(esw, *act_miss_mapping); + if (IS_ERR(attr->act_id_restore_rule)) + goto err_rule; + + return 0; + +err_rule: + mapping_remove(ctx, *act_miss_mapping); + return err; +} + +void mlx5e_tc_action_miss_mapping_put(struct mlx5e_priv *priv, struct mlx5_flow_attr *attr, + u32 act_miss_mapping) +{ + struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; + struct mapping_ctx *ctx; + + ctx = esw->offloads.reg_c0_obj_pool; + mlx5_del_flow_rules(attr->act_id_restore_rule); + mapping_remove(ctx, act_miss_mapping); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h index f6b10bd3368b..adb39e30f90f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h @@ -59,6 +59,8 @@ int mlx5e_tc_num_filters(struct mlx5e_priv *priv, unsigned long flags); struct mlx5e_tc_update_priv { struct net_device *fwd_dev; + bool skb_done; + bool forward_tx; }; struct mlx5_nic_flow_attr { @@ -95,6 +97,7 @@ struct mlx5_flow_attr { struct mlx5_flow_attr *branch_true; struct mlx5_flow_attr *branch_false; struct mlx5_flow_attr *jumping_attr; + struct mlx5_flow_handle *act_id_restore_rule; /* keep this union last */ union { DECLARE_FLEX_ARRAY(struct mlx5_esw_flow_attr, esw_attr); @@ -225,7 +228,7 @@ void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe); void mlx5e_tc_reoffload_flows_work(struct work_struct *work); enum mlx5e_tc_attr_to_reg { - CHAIN_TO_REG, + MAPPED_OBJ_TO_REG, VPORT_TO_REG, TUNNEL_TO_REG, CTSTATE_TO_REG, @@ -234,7 +237,7 @@ enum mlx5e_tc_attr_to_reg { MARK_TO_REG, LABELS_TO_REG, FTEID_TO_REG, - NIC_CHAIN_TO_REG, + NIC_MAPPED_OBJ_TO_REG, NIC_ZONE_RESTORE_TO_REG, PACKET_COLOR_TO_REG, }; @@ -368,7 +371,6 @@ struct mlx5e_tc_table *mlx5e_tc_table_alloc(void); void mlx5e_tc_table_free(struct mlx5e_tc_table *tc); static inline bool mlx5e_cqe_regb_chain(struct mlx5_cqe64 *cqe) { -#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT) u32 chain, reg_b; reg_b = be32_to_cpu(cqe->ft_metadata); @@ -379,20 +381,29 @@ static inline bool mlx5e_cqe_regb_chain(struct mlx5_cqe64 *cqe) chain = reg_b & MLX5E_TC_TABLE_CHAIN_TAG_MASK; if (chain) return true; -#endif return false; } -bool mlx5e_tc_update_skb(struct mlx5_cqe64 *cqe, struct sk_buff *skb); +bool mlx5e_tc_update_skb_nic(struct mlx5_cqe64 *cqe, struct sk_buff *skb); +bool mlx5e_tc_update_skb(struct mlx5_cqe64 *cqe, struct sk_buff *skb, + struct mapping_ctx *mapping_ctx, u32 mapped_obj_id, + struct mlx5_tc_ct_priv *ct_priv, + u32 zone_restore_id, u32 tunnel_id, + struct mlx5e_tc_update_priv *tc_priv); #else /* CONFIG_MLX5_CLS_ACT */ static inline struct mlx5e_tc_table *mlx5e_tc_table_alloc(void) { return NULL; } static inline void mlx5e_tc_table_free(struct mlx5e_tc_table *tc) {} static inline bool mlx5e_cqe_regb_chain(struct mlx5_cqe64 *cqe) { return false; } static inline bool -mlx5e_tc_update_skb(struct mlx5_cqe64 *cqe, struct sk_buff *skb) +mlx5e_tc_update_skb_nic(struct mlx5_cqe64 *cqe, struct sk_buff *skb) { return true; } #endif +int mlx5e_tc_action_miss_mapping_get(struct mlx5e_priv *priv, struct mlx5_flow_attr *attr, + u64 act_miss_cookie, u32 *act_miss_mapping); +void mlx5e_tc_action_miss_mapping_put(struct mlx5e_priv *priv, struct mlx5_flow_attr *attr, + u32 act_miss_mapping); + #endif /* __MLX5_EN_TC_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index fd03f076551b..19e9a77c4633 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -52,12 +52,14 @@ enum mlx5_mapped_obj_type { MLX5_MAPPED_OBJ_CHAIN, MLX5_MAPPED_OBJ_SAMPLE, MLX5_MAPPED_OBJ_INT_PORT_METADATA, + MLX5_MAPPED_OBJ_ACT_MISS, }; struct mlx5_mapped_obj { enum mlx5_mapped_obj_type type; union { u32 chain; + u64 act_miss_cookie; struct { u32 group_id; u32 rate; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_chains.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_chains.c index df58cba37930..81ed91fee59b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_chains.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_chains.c @@ -214,7 +214,7 @@ create_chain_restore(struct fs_chain *chain) struct mlx5_eswitch *esw = chain->chains->dev->priv.eswitch; u8 modact[MLX5_UN_SZ_BYTES(set_add_copy_action_in_auto)] = {}; struct mlx5_fs_chains *chains = chain->chains; - enum mlx5e_tc_attr_to_reg chain_to_reg; + enum mlx5e_tc_attr_to_reg mapped_obj_to_reg; struct mlx5_modify_hdr *mod_hdr; u32 index; int err; @@ -242,7 +242,7 @@ create_chain_restore(struct fs_chain *chain) chain->id = index; if (chains->ns == MLX5_FLOW_NAMESPACE_FDB) { - chain_to_reg = CHAIN_TO_REG; + mapped_obj_to_reg = MAPPED_OBJ_TO_REG; chain->restore_rule = esw_add_restore_rule(esw, chain->id); if (IS_ERR(chain->restore_rule)) { err = PTR_ERR(chain->restore_rule); @@ -253,7 +253,7 @@ create_chain_restore(struct fs_chain *chain) * since we write the metadata to reg_b * that is passed to SW directly. */ - chain_to_reg = NIC_CHAIN_TO_REG; + mapped_obj_to_reg = NIC_MAPPED_OBJ_TO_REG; } else { err = -EINVAL; goto err_rule; @@ -261,12 +261,12 @@ create_chain_restore(struct fs_chain *chain) MLX5_SET(set_action_in, modact, action_type, MLX5_ACTION_TYPE_SET); MLX5_SET(set_action_in, modact, field, - mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mfield); + mlx5e_tc_attr_to_reg_mappings[mapped_obj_to_reg].mfield); MLX5_SET(set_action_in, modact, offset, - mlx5e_tc_attr_to_reg_mappings[chain_to_reg].moffset); + mlx5e_tc_attr_to_reg_mappings[mapped_obj_to_reg].moffset); MLX5_SET(set_action_in, modact, length, - mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mlen == 32 ? - 0 : mlx5e_tc_attr_to_reg_mappings[chain_to_reg].mlen); + mlx5e_tc_attr_to_reg_mappings[mapped_obj_to_reg].mlen == 32 ? + 0 : mlx5e_tc_attr_to_reg_mappings[mapped_obj_to_reg].mlen); MLX5_SET(set_action_in, modact, data, chain->id); mod_hdr = mlx5_modify_header_alloc(chains->dev, chains->ns, 1, modact); diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c index e91fb205e0b4..594cdcb90b3d 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c @@ -103,7 +103,7 @@ static int mlxsw_sp_flower_parse_actions(struct mlxsw_sp *mlxsw_sp, } ingress = mlxsw_sp_flow_block_is_ingress_bound(block); err = mlxsw_sp_acl_rulei_act_drop(rulei, ingress, - act->cookie, extack); + act->user_cookie, extack); if (err) { NL_SET_ERR_MSG_MOD(extack, "Cannot append drop action"); return err; diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_ptp.c b/drivers/net/ethernet/microchip/lan966x/lan966x_ptp.c index a8348437dd87..ded9ab79ccc2 100644 --- a/drivers/net/ethernet/microchip/lan966x/lan966x_ptp.c +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_ptp.c @@ -83,8 +83,7 @@ static int lan966x_ptp_add_trap(struct lan966x_port *port, if (err) goto free_rule; - err = vcap_set_rule_set_actionset(vrule, VCAP_AFS_BASE_TYPE); - err |= vcap_rule_add_action_bit(vrule, VCAP_AF_CPU_COPY_ENA, VCAP_BIT_1); + err = vcap_rule_add_action_bit(vrule, VCAP_AF_CPU_COPY_ENA, VCAP_BIT_1); err |= vcap_rule_add_action_u32(vrule, VCAP_AF_MASK_MODE, LAN966X_PMM_REPLACE); err |= vcap_val_rule(vrule, proto); if (err) diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_tc_flower.c b/drivers/net/ethernet/microchip/lan966x/lan966x_tc_flower.c index bd10a7189741..f960727ecaee 100644 --- a/drivers/net/ethernet/microchip/lan966x/lan966x_tc_flower.c +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_tc_flower.c @@ -261,8 +261,6 @@ static int lan966x_tc_flower_add(struct lan966x_port *port, 0); err |= vcap_rule_add_action_u32(vrule, VCAP_AF_MASK_MODE, LAN966X_PMM_REPLACE); - err |= vcap_set_rule_set_actionset(vrule, - VCAP_AFS_BASE_TYPE); if (err) goto out; diff --git a/drivers/net/ethernet/microchip/vcap/vcap_api_kunit.c b/drivers/net/ethernet/microchip/vcap/vcap_api_kunit.c index 0a1d4d740567..c07f25e791c7 100644 --- a/drivers/net/ethernet/microchip/vcap/vcap_api_kunit.c +++ b/drivers/net/ethernet/microchip/vcap/vcap_api_kunit.c @@ -1876,53 +1876,51 @@ static void vcap_api_next_lookup_basic_test(struct kunit *test) static void vcap_api_next_lookup_advanced_test(struct kunit *test) { - struct vcap_admin admin1 = { + struct vcap_admin admin[] = { + { .vtype = VCAP_TYPE_IS0, .vinst = 0, .first_cid = 1000000, .last_cid = 1199999, .lookups = 6, .lookups_per_instance = 2, - }; - struct vcap_admin admin2 = { + }, { .vtype = VCAP_TYPE_IS0, .vinst = 1, .first_cid = 1200000, .last_cid = 1399999, .lookups = 6, .lookups_per_instance = 2, - }; - struct vcap_admin admin3 = { + }, { .vtype = VCAP_TYPE_IS0, .vinst = 2, .first_cid = 1400000, .last_cid = 1599999, .lookups = 6, .lookups_per_instance = 2, - }; - struct vcap_admin admin4 = { + }, { .vtype = VCAP_TYPE_IS2, .vinst = 0, .first_cid = 8000000, .last_cid = 8199999, .lookups = 4, .lookups_per_instance = 2, - }; - struct vcap_admin admin5 = { + }, { .vtype = VCAP_TYPE_IS2, .vinst = 1, .first_cid = 8200000, .last_cid = 8399999, .lookups = 4, .lookups_per_instance = 2, + } }; bool ret; - vcap_test_api_init(&admin1); - list_add_tail(&admin2.list, &test_vctrl.list); - list_add_tail(&admin3.list, &test_vctrl.list); - list_add_tail(&admin4.list, &test_vctrl.list); - list_add_tail(&admin5.list, &test_vctrl.list); + vcap_test_api_init(&admin[0]); + list_add_tail(&admin[1].list, &test_vctrl.list); + list_add_tail(&admin[2].list, &test_vctrl.list); + list_add_tail(&admin[3].list, &test_vctrl.list); + list_add_tail(&admin[4].list, &test_vctrl.list); ret = vcap_is_next_lookup(&test_vctrl, 1000000, 1001000); KUNIT_EXPECT_EQ(test, false, ret); diff --git a/drivers/net/ethernet/sfc/ef100_nic.c b/drivers/net/ethernet/sfc/ef100_nic.c index becd21c2325d..4dc643b0d2db 100644 --- a/drivers/net/ethernet/sfc/ef100_nic.c +++ b/drivers/net/ethernet/sfc/ef100_nic.c @@ -399,14 +399,14 @@ static int ef100_filter_table_up(struct efx_nic *efx) * filter insertion will need to take the lock for read. */ up_write(&efx->filter_sem); -#ifdef CONFIG_SFC_SRIOV - rc = efx_tc_insert_rep_filters(efx); + if (IS_ENABLED(CONFIG_SFC_SRIOV)) + rc = efx_tc_insert_rep_filters(efx); + /* Rep filter failure is nonfatal */ if (rc) netif_warn(efx, drv, efx->net_dev, "Failed to insert representor filters, rc %d\n", rc); -#endif return 0; fail_vlan0: @@ -419,9 +419,8 @@ fail_unspec: static void ef100_filter_table_down(struct efx_nic *efx) { -#ifdef CONFIG_SFC_SRIOV - efx_tc_remove_rep_filters(efx); -#endif + if (IS_ENABLED(CONFIG_SFC_SRIOV)) + efx_tc_remove_rep_filters(efx); down_write(&efx->filter_sem); efx_mcdi_filter_del_vlan(efx, 0); efx_mcdi_filter_del_vlan(efx, EFX_FILTER_VID_UNSPEC); @@ -737,7 +736,6 @@ static unsigned int efx_ef100_recycle_ring_size(const struct efx_nic *efx) return 10 * EFX_RECYCLE_RING_SIZE_10G; } -#ifdef CONFIG_SFC_SRIOV static int efx_ef100_get_base_mport(struct efx_nic *efx) { struct ef100_nic_data *nic_data = efx->nic_data; @@ -773,7 +771,6 @@ static int efx_ef100_get_base_mport(struct efx_nic *efx) return 0; } -#endif static int compare_versions(const char *a, const char *b) { @@ -1155,10 +1152,9 @@ int ef100_probe_netdev_pf(struct efx_nic *efx) struct net_device *net_dev = efx->net_dev; int rc; - if (!nic_data->grp_mae) + if (!IS_ENABLED(CONFIG_SFC_SRIOV) || !nic_data->grp_mae) return 0; -#ifdef CONFIG_SFC_SRIOV rc = efx_init_struct_tc(efx); if (rc) return rc; @@ -1193,7 +1189,6 @@ int ef100_probe_netdev_pf(struct efx_nic *efx) net_dev->features |= NETIF_F_HW_TC; efx->fixed_features |= NETIF_F_HW_TC; } -#endif return rc; } @@ -1206,12 +1201,11 @@ void ef100_remove(struct efx_nic *efx) { struct ef100_nic_data *nic_data = efx->nic_data; -#ifdef CONFIG_SFC_SRIOV - if (efx->mae) { + if (IS_ENABLED(CONFIG_SFC_SRIOV) && efx->mae) { efx_ef100_fini_reps(efx); efx_fini_mae(efx); } -#endif + efx_mcdi_detach(efx); efx_mcdi_fini(efx); if (nic_data) @@ -1304,9 +1298,8 @@ const struct efx_nic_type ef100_pf_nic_type = { .update_stats = ef100_update_stats, .pull_stats = efx_mcdi_mac_pull_stats, .stop_stats = efx_mcdi_mac_stop_stats, -#ifdef CONFIG_SFC_SRIOV - .sriov_configure = efx_ef100_sriov_configure, -#endif + .sriov_configure = IS_ENABLED(CONFIG_SFC_SRIOV) ? + efx_ef100_sriov_configure : NULL, /* Per-type bar/size configuration not used on ef100. Location of * registers is defined by extended capabilities. diff --git a/drivers/net/ethernet/sfc/efx_devlink.c b/drivers/net/ethernet/sfc/efx_devlink.c index d2eb6712ba35..381b805659d3 100644 --- a/drivers/net/ethernet/sfc/efx_devlink.c +++ b/drivers/net/ethernet/sfc/efx_devlink.c @@ -323,7 +323,7 @@ static void efx_devlink_info_running_v2(struct efx_nic *efx, GET_VERSION_V2_OUT_SUCFW_BUILD_DATE); rtc_time64_to_tm(tstamp, &build_date); #else - memset(&build_date, 0, sizeof(build_date) + memset(&build_date, 0, sizeof(build_date)); #endif build_id = MCDI_DWORD(outbuf, GET_VERSION_V2_OUT_SUCFW_CHIP_ID); @@ -655,7 +655,7 @@ static struct devlink_port *ef100_set_devlink_port(struct efx_nic *efx, u32 idx) "devlink port creation for PF failed.\n"); else pci_warn(efx->pci_dev, - "devlink_port creationg for VF %u failed.\n", + "devlink_port creation for VF %u failed.\n", idx); return NULL; } diff --git a/drivers/net/ethernet/sfc/mae.c b/drivers/net/ethernet/sfc/mae.c index 6321fd393fc3..2d32abe5f478 100644 --- a/drivers/net/ethernet/sfc/mae.c +++ b/drivers/net/ethernet/sfc/mae.c @@ -654,8 +654,8 @@ int efx_mae_enumerate_mports(struct efx_nic *efx) MAE_MPORT_DESC_VNIC_FUNCTION_INTERFACE); d->pf_idx = MCDI_STRUCT_WORD(desc, MAE_MPORT_DESC_VNIC_FUNCTION_PF_IDX); - d->vf_idx = MCDI_STRUCT_WORD(desc, - MAE_MPORT_DESC_VNIC_FUNCTION_VF_IDX); + d->vf_idx = MCDI_STRUCT_WORD(desc, + MAE_MPORT_DESC_VNIC_FUNCTION_VF_IDX); break; default: /* Unknown mport_type, just accept it */ diff --git a/drivers/net/ieee802154/at86rf230.c b/drivers/net/ieee802154/at86rf230.c index 15f283b26721..62b984f84d9f 100644 --- a/drivers/net/ieee802154/at86rf230.c +++ b/drivers/net/ieee802154/at86rf230.c @@ -17,8 +17,8 @@ #include <linux/irq.h> #include <linux/gpio.h> #include <linux/delay.h> +#include <linux/property.h> #include <linux/spi/spi.h> -#include <linux/spi/at86rf230.h> #include <linux/regmap.h> #include <linux/skbuff.h> #include <linux/of_gpio.h> @@ -82,7 +82,7 @@ struct at86rf230_local { struct ieee802154_hw *hw; struct at86rf2xx_chip_data *data; struct regmap *regmap; - int slp_tr; + struct gpio_desc *slp_tr; bool sleep; struct completion state_complete; @@ -107,8 +107,8 @@ at86rf230_async_state_change(struct at86rf230_local *lp, static inline void at86rf230_sleep(struct at86rf230_local *lp) { - if (gpio_is_valid(lp->slp_tr)) { - gpio_set_value(lp->slp_tr, 1); + if (lp->slp_tr) { + gpiod_set_value(lp->slp_tr, 1); usleep_range(lp->data->t_off_to_sleep, lp->data->t_off_to_sleep + 10); lp->sleep = true; @@ -118,8 +118,8 @@ at86rf230_sleep(struct at86rf230_local *lp) static inline void at86rf230_awake(struct at86rf230_local *lp) { - if (gpio_is_valid(lp->slp_tr)) { - gpio_set_value(lp->slp_tr, 0); + if (lp->slp_tr) { + gpiod_set_value(lp->slp_tr, 0); usleep_range(lp->data->t_sleep_to_off, lp->data->t_sleep_to_off + 100); lp->sleep = false; @@ -204,9 +204,9 @@ at86rf230_write_subreg(struct at86rf230_local *lp, static inline void at86rf230_slp_tr_rising_edge(struct at86rf230_local *lp) { - gpio_set_value(lp->slp_tr, 1); + gpiod_set_value(lp->slp_tr, 1); udelay(1); - gpio_set_value(lp->slp_tr, 0); + gpiod_set_value(lp->slp_tr, 0); } static bool @@ -819,7 +819,7 @@ at86rf230_write_frame_complete(void *context) ctx->trx.len = 2; - if (gpio_is_valid(lp->slp_tr)) + if (lp->slp_tr) at86rf230_slp_tr_rising_edge(lp); else at86rf230_async_write_reg(lp, RG_TRX_STATE, STATE_BUSY_TX, ctx, @@ -1416,32 +1416,6 @@ static int at86rf230_hw_init(struct at86rf230_local *lp, u8 xtal_trim) } static int -at86rf230_get_pdata(struct spi_device *spi, int *rstn, int *slp_tr, - u8 *xtal_trim) -{ - struct at86rf230_platform_data *pdata = spi->dev.platform_data; - int ret; - - if (!IS_ENABLED(CONFIG_OF) || !spi->dev.of_node) { - if (!pdata) - return -ENOENT; - - *rstn = pdata->rstn; - *slp_tr = pdata->slp_tr; - *xtal_trim = pdata->xtal_trim; - return 0; - } - - *rstn = of_get_named_gpio(spi->dev.of_node, "reset-gpio", 0); - *slp_tr = of_get_named_gpio(spi->dev.of_node, "sleep-gpio", 0); - ret = of_property_read_u8(spi->dev.of_node, "xtal-trim", xtal_trim); - if (ret < 0 && ret != -EINVAL) - return ret; - - return 0; -} - -static int at86rf230_detect_device(struct at86rf230_local *lp) { unsigned int part, version, val; @@ -1546,41 +1520,47 @@ static int at86rf230_probe(struct spi_device *spi) { struct ieee802154_hw *hw; struct at86rf230_local *lp; + struct gpio_desc *slp_tr; + struct gpio_desc *rstn; unsigned int status; - int rc, irq_type, rstn, slp_tr; - u8 xtal_trim = 0; + int rc, irq_type; + u8 xtal_trim; if (!spi->irq) { dev_err(&spi->dev, "no IRQ specified\n"); return -EINVAL; } - rc = at86rf230_get_pdata(spi, &rstn, &slp_tr, &xtal_trim); + rc = device_property_read_u8(&spi->dev, "xtal-trim", &xtal_trim); if (rc < 0) { - dev_err(&spi->dev, "failed to parse platform_data: %d\n", rc); - return rc; - } - - if (gpio_is_valid(rstn)) { - rc = devm_gpio_request_one(&spi->dev, rstn, - GPIOF_OUT_INIT_HIGH, "rstn"); - if (rc) + if (rc != -EINVAL) { + dev_err(&spi->dev, + "failed to parse xtal-trim: %d\n", rc); return rc; + } + xtal_trim = 0; } - if (gpio_is_valid(slp_tr)) { - rc = devm_gpio_request_one(&spi->dev, slp_tr, - GPIOF_OUT_INIT_LOW, "slp_tr"); - if (rc) - return rc; - } + rstn = devm_gpiod_get_optional(&spi->dev, "reset", GPIOD_OUT_LOW); + rc = PTR_ERR_OR_ZERO(rstn); + if (rc) + return rc; + + gpiod_set_consumer_name(rstn, "rstn"); + + slp_tr = devm_gpiod_get_optional(&spi->dev, "sleep", GPIOD_OUT_LOW); + rc = PTR_ERR_OR_ZERO(slp_tr); + if (rc) + return rc; + + gpiod_set_consumer_name(slp_tr, "slp_tr"); /* Reset */ - if (gpio_is_valid(rstn)) { + if (rstn) { udelay(1); - gpio_set_value_cansleep(rstn, 0); + gpiod_set_value_cansleep(rstn, 1); udelay(1); - gpio_set_value_cansleep(rstn, 1); + gpiod_set_value_cansleep(rstn, 0); usleep_range(120, 240); } diff --git a/drivers/net/ieee802154/cc2520.c b/drivers/net/ieee802154/cc2520.c index edc769daad07..a94d8dd71aad 100644 --- a/drivers/net/ieee802154/cc2520.c +++ b/drivers/net/ieee802154/cc2520.c @@ -7,14 +7,13 @@ */ #include <linux/kernel.h> #include <linux/module.h> -#include <linux/gpio.h> +#include <linux/gpio/consumer.h> #include <linux/delay.h> #include <linux/spi/spi.h> -#include <linux/spi/cc2520.h> +#include <linux/property.h> #include <linux/workqueue.h> #include <linux/interrupt.h> #include <linux/skbuff.h> -#include <linux/of_gpio.h> #include <linux/ieee802154.h> #include <linux/crc-ccitt.h> #include <asm/unaligned.h> @@ -206,7 +205,7 @@ struct cc2520_private { struct mutex buffer_mutex; /* SPI buffer mutex */ bool is_tx; /* Flag for sync b/w Tx and Rx */ bool amplified; /* Flag for CC2591 */ - int fifo_pin; /* FIFO GPIO pin number */ + struct gpio_desc *fifo_pin; /* FIFO GPIO pin number */ struct work_struct fifop_irqwork;/* Workqueue for FIFOP */ spinlock_t lock; /* Lock for is_tx*/ struct completion tx_complete; /* Work completion for Tx */ @@ -875,7 +874,7 @@ static void cc2520_fifop_irqwork(struct work_struct *work) dev_dbg(&priv->spi->dev, "fifop interrupt received\n"); - if (gpio_get_value(priv->fifo_pin)) + if (gpiod_get_value(priv->fifo_pin)) cc2520_rx(priv); else dev_dbg(&priv->spi->dev, "rxfifo overflow\n"); @@ -912,49 +911,11 @@ static irqreturn_t cc2520_sfd_isr(int irq, void *data) return IRQ_HANDLED; } -static int cc2520_get_platform_data(struct spi_device *spi, - struct cc2520_platform_data *pdata) -{ - struct device_node *np = spi->dev.of_node; - struct cc2520_private *priv = spi_get_drvdata(spi); - - if (!np) { - struct cc2520_platform_data *spi_pdata = spi->dev.platform_data; - - if (!spi_pdata) - return -ENOENT; - *pdata = *spi_pdata; - priv->fifo_pin = pdata->fifo; - return 0; - } - - pdata->fifo = of_get_named_gpio(np, "fifo-gpio", 0); - priv->fifo_pin = pdata->fifo; - - pdata->fifop = of_get_named_gpio(np, "fifop-gpio", 0); - - pdata->sfd = of_get_named_gpio(np, "sfd-gpio", 0); - pdata->cca = of_get_named_gpio(np, "cca-gpio", 0); - pdata->vreg = of_get_named_gpio(np, "vreg-gpio", 0); - pdata->reset = of_get_named_gpio(np, "reset-gpio", 0); - - /* CC2591 front end for CC2520 */ - if (of_property_read_bool(np, "amplified")) - priv->amplified = true; - - return 0; -} - static int cc2520_hw_init(struct cc2520_private *priv) { u8 status = 0, state = 0xff; int ret; int timeout = 100; - struct cc2520_platform_data pdata; - - ret = cc2520_get_platform_data(priv->spi, &pdata); - if (ret) - goto err_ret; ret = cc2520_read_register(priv, CC2520_FSMSTAT1, &state); if (ret) @@ -1071,7 +1032,11 @@ err_ret: static int cc2520_probe(struct spi_device *spi) { struct cc2520_private *priv; - struct cc2520_platform_data pdata; + struct gpio_desc *fifop; + struct gpio_desc *cca; + struct gpio_desc *sfd; + struct gpio_desc *reset; + struct gpio_desc *vreg; int ret; priv = devm_kzalloc(&spi->dev, sizeof(*priv), GFP_KERNEL); @@ -1080,11 +1045,11 @@ static int cc2520_probe(struct spi_device *spi) spi_set_drvdata(spi, priv); - ret = cc2520_get_platform_data(spi, &pdata); - if (ret < 0) { - dev_err(&spi->dev, "no platform data\n"); - return -EINVAL; - } + /* CC2591 front end for CC2520 */ + /* Assumption that CC2591 is not connected */ + priv->amplified = false; + if (device_property_read_bool(&spi->dev, "amplified")) + priv->amplified = true; priv->spi = spi; @@ -1098,80 +1063,53 @@ static int cc2520_probe(struct spi_device *spi) spin_lock_init(&priv->lock); init_completion(&priv->tx_complete); - /* Assumption that CC2591 is not connected */ - priv->amplified = false; - /* Request all the gpio's */ - if (!gpio_is_valid(pdata.fifo)) { + priv->fifo_pin = devm_gpiod_get(&spi->dev, "fifo", GPIOD_IN); + if (IS_ERR(priv->fifo_pin)) { dev_err(&spi->dev, "fifo gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(priv->fifo_pin); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.fifo, - GPIOF_IN, "fifo"); - if (ret) - goto err_hw_init; - - if (!gpio_is_valid(pdata.cca)) { + cca = devm_gpiod_get(&spi->dev, "cca", GPIOD_IN); + if (IS_ERR(cca)) { dev_err(&spi->dev, "cca gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(cca); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.cca, - GPIOF_IN, "cca"); - if (ret) - goto err_hw_init; - - if (!gpio_is_valid(pdata.fifop)) { + fifop = devm_gpiod_get(&spi->dev, "fifop", GPIOD_IN); + if (IS_ERR(fifop)) { dev_err(&spi->dev, "fifop gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(fifop); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.fifop, - GPIOF_IN, "fifop"); - if (ret) - goto err_hw_init; - - if (!gpio_is_valid(pdata.sfd)) { + sfd = devm_gpiod_get(&spi->dev, "sfd", GPIOD_IN); + if (IS_ERR(sfd)) { dev_err(&spi->dev, "sfd gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(sfd); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.sfd, - GPIOF_IN, "sfd"); - if (ret) - goto err_hw_init; - - if (!gpio_is_valid(pdata.reset)) { + reset = devm_gpiod_get(&spi->dev, "reset", GPIOD_OUT_LOW); + if (IS_ERR(reset)) { dev_err(&spi->dev, "reset gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(reset); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.reset, - GPIOF_OUT_INIT_LOW, "reset"); - if (ret) - goto err_hw_init; - - if (!gpio_is_valid(pdata.vreg)) { + vreg = devm_gpiod_get(&spi->dev, "vreg", GPIOD_OUT_LOW); + if (IS_ERR(vreg)) { dev_err(&spi->dev, "vreg gpio is not valid\n"); - ret = -EINVAL; + ret = PTR_ERR(vreg); goto err_hw_init; } - ret = devm_gpio_request_one(&spi->dev, pdata.vreg, - GPIOF_OUT_INIT_LOW, "vreg"); - if (ret) - goto err_hw_init; - - gpio_set_value(pdata.vreg, HIGH); + gpiod_set_value(vreg, HIGH); usleep_range(100, 150); - gpio_set_value(pdata.reset, HIGH); + gpiod_set_value(reset, HIGH); usleep_range(200, 250); ret = cc2520_hw_init(priv); @@ -1180,7 +1118,7 @@ static int cc2520_probe(struct spi_device *spi) /* Set up fifop interrupt */ ret = devm_request_irq(&spi->dev, - gpio_to_irq(pdata.fifop), + gpiod_to_irq(fifop), cc2520_fifop_isr, IRQF_TRIGGER_RISING, dev_name(&spi->dev), @@ -1192,7 +1130,7 @@ static int cc2520_probe(struct spi_device *spi) /* Set up sfd interrupt */ ret = devm_request_irq(&spi->dev, - gpio_to_irq(pdata.sfd), + gpiod_to_irq(sfd), cc2520_sfd_isr, IRQF_TRIGGER_FALLING, dev_name(&spi->dev), @@ -1241,7 +1179,7 @@ MODULE_DEVICE_TABLE(of, cc2520_of_ids); static struct spi_driver cc2520_driver = { .driver = { .name = "cc2520", - .of_match_table = of_match_ptr(cc2520_of_ids), + .of_match_table = cc2520_of_ids, }, .id_table = cc2520_ids, .probe = cc2520_probe, diff --git a/drivers/net/ipa/gsi.c b/drivers/net/ipa/gsi.c index f44d2d843de1..9a0b1fe4a93a 100644 --- a/drivers/net/ipa/gsi.c +++ b/drivers/net/ipa/gsi.c @@ -185,7 +185,7 @@ static u32 ch_c_cntxt_0_type_encode(enum ipa_version version, u32 val; val = reg_encode(reg, CHTYPE_PROTOCOL, type); - if (version < IPA_VERSION_4_5) + if (version < IPA_VERSION_4_5 || version >= IPA_VERSION_5_0) return val; type >>= hweight32(reg_fmask(reg, CHTYPE_PROTOCOL)); @@ -193,17 +193,6 @@ static u32 ch_c_cntxt_0_type_encode(enum ipa_version version, return val | reg_encode(reg, CHTYPE_PROTOCOL_MSB, type); } -/* Encode the length of the event channel ring buffer for the - * EV_CH_E_CNTXT_1 register. - */ -static u32 ev_ch_e_cntxt_1_length_encode(enum ipa_version version, u32 length) -{ - if (version < IPA_VERSION_4_9) - return u32_encode_bits(length, GENMASK(15, 0)); - - return u32_encode_bits(length, GENMASK(19, 0)); -} - /* Update the GSI IRQ type register with the cached value */ static void gsi_irq_type_update(struct gsi *gsi, u32 val) { @@ -731,7 +720,6 @@ static void gsi_evt_ring_program(struct gsi *gsi, u32 evt_ring_id) struct gsi_evt_ring *evt_ring = &gsi->evt_ring[evt_ring_id]; struct gsi_ring *ring = &evt_ring->ring; const struct reg *reg; - size_t size; u32 val; reg = gsi_reg(gsi, EV_CH_E_CNTXT_0); @@ -743,8 +731,7 @@ static void gsi_evt_ring_program(struct gsi *gsi, u32 evt_ring_id) iowrite32(val, gsi->virt + reg_n_offset(reg, evt_ring_id)); reg = gsi_reg(gsi, EV_CH_E_CNTXT_1); - size = ring->count * GSI_RING_ELEMENT_SIZE; - val = ev_ch_e_cntxt_1_length_encode(gsi->version, size); + val = reg_encode(reg, R_LENGTH, ring->count * GSI_RING_ELEMENT_SIZE); iowrite32(val, gsi->virt + reg_n_offset(reg, evt_ring_id)); /* The context 2 and 3 registers store the low-order and @@ -762,7 +749,7 @@ static void gsi_evt_ring_program(struct gsi *gsi, u32 evt_ring_id) /* Enable interrupt moderation by setting the moderation delay */ reg = gsi_reg(gsi, EV_CH_E_CNTXT_8); val = reg_encode(reg, EV_MODT, GSI_EVT_RING_INT_MODT); - val = reg_encode(reg, EV_MODC, 1); /* comes from channel */ + val |= reg_encode(reg, EV_MODC, 1); /* comes from channel */ /* EV_MOD_CNT is 0 (no counter-based interrupt coalescing) */ iowrite32(val, gsi->virt + reg_n_offset(reg, evt_ring_id)); @@ -853,12 +840,15 @@ static void gsi_channel_program(struct gsi_channel *channel, bool doorbell) val = ch_c_cntxt_0_type_encode(gsi->version, reg, GSI_CHANNEL_TYPE_GPI); if (channel->toward_ipa) val |= reg_bit(reg, CHTYPE_DIR); - val |= reg_encode(reg, ERINDEX, channel->evt_ring_id); + if (gsi->version < IPA_VERSION_5_0) + val |= reg_encode(reg, ERINDEX, channel->evt_ring_id); val |= reg_encode(reg, ELEMENT_SIZE, GSI_RING_ELEMENT_SIZE); iowrite32(val, gsi->virt + reg_n_offset(reg, channel_id)); reg = gsi_reg(gsi, CH_C_CNTXT_1); val = reg_encode(reg, CH_R_LENGTH, size); + if (gsi->version >= IPA_VERSION_5_0) + val |= reg_encode(reg, CH_ERINDEX, channel->evt_ring_id); iowrite32(val, gsi->virt + reg_n_offset(reg, channel_id)); /* The context 2 and 3 registers store the low-order and @@ -1999,12 +1989,11 @@ static int gsi_irq_setup(struct gsi *gsi) /* The inter-EE interrupts are not supported for IPA v3.0-v3.1 */ if (gsi->version > IPA_VERSION_3_1) { - /* These registers are in the non-adjusted address range */ reg = gsi_reg(gsi, INTER_EE_SRC_CH_IRQ_MSK); - iowrite32(0, gsi->virt_raw + reg_offset(reg)); + iowrite32(0, gsi->virt + reg_offset(reg)); reg = gsi_reg(gsi, INTER_EE_SRC_EV_CH_IRQ_MSK); - iowrite32(0, gsi->virt_raw + reg_offset(reg)); + iowrite32(0, gsi->virt + reg_offset(reg)); } reg = gsi_reg(gsi, CNTXT_GSI_IRQ_EN); @@ -2053,7 +2042,12 @@ static int gsi_ring_setup(struct gsi *gsi) } gsi->channel_count = count; - count = reg_decode(reg, NUM_EV_PER_EE, val); + if (gsi->version < IPA_VERSION_5_0) { + count = reg_decode(reg, NUM_EV_PER_EE, val); + } else { + reg = gsi_reg(gsi, HW_PARAM_4); + count = reg_decode(reg, EV_PER_EE, val); + } if (!count) { dev_err(dev, "GSI reports zero event rings supported\n"); return -EINVAL; diff --git a/drivers/net/ipa/gsi.h b/drivers/net/ipa/gsi.h index bc5ff617341a..50bc80cb167c 100644 --- a/drivers/net/ipa/gsi.h +++ b/drivers/net/ipa/gsi.h @@ -140,8 +140,7 @@ struct gsi_evt_ring { struct gsi { struct device *dev; /* Same as IPA device */ enum ipa_version version; - void __iomem *virt_raw; /* I/O mapped address range */ - void __iomem *virt; /* Adjusted for most registers */ + void __iomem *virt; /* I/O mapped registers */ const struct regs *regs; u32 irq; diff --git a/drivers/net/ipa/gsi_reg.c b/drivers/net/ipa/gsi_reg.c index 0bb70a7ef4e6..1412b67304c8 100644 --- a/drivers/net/ipa/gsi_reg.c +++ b/drivers/net/ipa/gsi_reg.c @@ -9,20 +9,6 @@ #include "reg.h" #include "gsi_reg.h" -/* GSI EE registers as a group are shifted downward by a fixed constant amount - * for IPA versions 4.5 and beyond. This applies to all GSI registers we use - * *except* the ones that disable inter-EE interrupts for channels and event - * channels. - * - * The "raw" (not adjusted) GSI register range is mapped, and a pointer to - * the mapped range is held in gsi->virt_raw. The inter-EE interrupt - * registers are accessed using that pointer. - * - * Most registers are accessed using gsi->virt, which is a copy of the "raw" - * pointer, adjusted downward by the fixed amount. - */ -#define GSI_EE_REG_ADJUST 0x0000d000 /* IPA v4.5+ */ - /* Is this register ID valid for the current GSI version? */ static bool gsi_reg_id_valid(struct gsi *gsi, enum gsi_reg_id reg_id) { @@ -121,13 +107,12 @@ static const struct regs *gsi_regs(struct gsi *gsi) } } -/* Sets gsi->virt_raw and gsi->virt, and I/O maps the "gsi" memory range */ +/* Sets gsi->virt and I/O maps the "gsi" memory range for registers */ int gsi_reg_init(struct gsi *gsi, struct platform_device *pdev) { struct device *dev = &pdev->dev; struct resource *res; resource_size_t size; - u32 adjust; /* Get GSI memory range and map it */ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "gsi"); @@ -142,27 +127,17 @@ int gsi_reg_init(struct gsi *gsi, struct platform_device *pdev) return -EINVAL; } - /* Make sure we can make our pointer adjustment if necessary */ - adjust = gsi->version < IPA_VERSION_4_5 ? 0 : GSI_EE_REG_ADJUST; - if (res->start < adjust) { - dev_err(dev, "DT memory resource \"gsi\" too low (< %u)\n", - adjust); - return -EINVAL; - } - gsi->regs = gsi_regs(gsi); if (!gsi->regs) { dev_err(dev, "unsupported IPA version %u (?)\n", gsi->version); return -EINVAL; } - gsi->virt_raw = ioremap(res->start, size); - if (!gsi->virt_raw) { + gsi->virt = ioremap(res->start, size); + if (!gsi->virt) { dev_err(dev, "unable to remap \"gsi\" memory\n"); return -ENOMEM; } - /* Most registers are accessed using an adjusted register range */ - gsi->virt = gsi->virt_raw - adjust; return 0; } @@ -170,7 +145,7 @@ int gsi_reg_init(struct gsi *gsi, struct platform_device *pdev) /* Inverse of gsi_reg_init() */ void gsi_reg_exit(struct gsi *gsi) { + iounmap(gsi->virt); gsi->virt = NULL; - iounmap(gsi->virt_raw); - gsi->virt_raw = NULL; + gsi->regs = NULL; } diff --git a/drivers/net/ipa/gsi_reg.h b/drivers/net/ipa/gsi_reg.h index 5eda4def7ac4..f62f0a5c653d 100644 --- a/drivers/net/ipa/gsi_reg.h +++ b/drivers/net/ipa/gsi_reg.h @@ -71,6 +71,7 @@ enum gsi_reg_id { EV_CH_CMD, GENERIC_CMD, HW_PARAM_2, /* IPA v3.5.1+ */ + HW_PARAM_4, /* IPA v5.0+ */ CNTXT_TYPE_IRQ, CNTXT_TYPE_IRQ_MSK, CNTXT_SRC_CH_IRQ, @@ -101,8 +102,8 @@ enum gsi_reg_ch_c_cntxt_0_field_id { CHTYPE_DIR, CH_EE, CHID, - CHTYPE_PROTOCOL_MSB, /* IPA v4.9+ */ - ERINDEX, + CHTYPE_PROTOCOL_MSB, /* IPA v4.5-4.11 */ + ERINDEX, /* Not IPA v5.0+ */ CHSTATE, ELEMENT_SIZE, }; @@ -124,6 +125,7 @@ enum gsi_channel_type { /* CH_C_CNTXT_1 register */ enum gsi_reg_ch_c_cntxt_1_field_id { CH_R_LENGTH, + CH_ERINDEX, /* IPA v5.0+ */ }; /* CH_C_QOS register */ @@ -135,6 +137,7 @@ enum gsi_reg_ch_c_qos_field_id { PREFETCH_MODE, /* IPA v4.5+ */ EMPTY_LVL_THRSHOLD, /* IPA v4.5+ */ DB_IN_BYTES, /* IPA v4.9+ */ + LOW_LATENCY_EN, /* IPA v5.0+ */ }; /** enum gsi_prefetch_mode - PREFETCH_MODE field in CH_C_QOS */ @@ -155,6 +158,11 @@ enum gsi_reg_ch_c_ev_ch_e_cntxt_0_field_id { EV_ELEMENT_SIZE, }; +/* EV_CH_E_CNTXT_1 register */ +enum gsi_reg_ev_ch_c_cntxt_1_field_id { + R_LENGTH, +}; + /* EV_CH_E_CNTXT_8 register */ enum gsi_reg_ch_c_ev_ch_e_cntxt_8_field_id { EV_MODT, @@ -217,7 +225,7 @@ enum gsi_generic_cmd_opcode { enum gsi_hw_param_2_field_id { IRAM_SIZE, NUM_CH_PER_EE, - NUM_EV_PER_EE, + NUM_EV_PER_EE, /* Not IPA v5.0+ */ GSI_CH_PEND_TRANSLATE, GSI_CH_FULL_LOGIC, GSI_USE_SDMA, /* IPA v4.0+ */ @@ -240,6 +248,12 @@ enum gsi_iram_size { IRAM_SIZE_FOUR_KB = 0x5, }; +/* HW_PARAM_4 register */ /* IPA v5.0+ */ +enum gsi_hw_param_4_field_id { + EV_PER_EE, + IRAM_PROTOCOL_COUNT, +}; + /** * enum gsi_irq_type_id: GSI IRQ types * @GSI_CH_CTRL: Channel allocation, deallocation, etc. @@ -351,8 +365,7 @@ const struct reg *gsi_reg(struct gsi *gsi, enum gsi_reg_id reg_id); * @pdev: GSI (IPA) platform device * * Initialize GSI registers, including looking up and I/O mapping - * the "gsi" memory space. This function sets gsi->virt_raw and - * gsi->virt. + * the "gsi" memory space. */ int gsi_reg_init(struct gsi *gsi, struct platform_device *pdev); diff --git a/drivers/net/ipa/reg/gsi_reg-v3.1.c b/drivers/net/ipa/reg/gsi_reg-v3.1.c index 651c8a7ed611..e036805a7882 100644 --- a/drivers/net/ipa/reg/gsi_reg-v3.1.c +++ b/drivers/net/ipa/reg/gsi_reg-v3.1.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -66,10 +62,6 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); @@ -95,8 +87,12 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(15, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); @@ -152,6 +148,7 @@ REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; @@ -159,6 +156,7 @@ REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; @@ -220,6 +218,10 @@ static const u32 reg_cntxt_intset_fmask[] = { REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); + static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), /* Bits 3-4 reserved */ diff --git a/drivers/net/ipa/reg/gsi_reg-v3.5.1.c b/drivers/net/ipa/reg/gsi_reg-v3.5.1.c index 0b39f8374ec1..8c3ab3a5288e 100644 --- a/drivers/net/ipa/reg/gsi_reg-v3.5.1.c +++ b/drivers/net/ipa/reg/gsi_reg-v3.5.1.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -66,10 +62,6 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); @@ -95,8 +87,12 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(15, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); @@ -152,6 +148,7 @@ REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; @@ -159,6 +156,7 @@ REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; @@ -231,6 +229,10 @@ static const u32 reg_cntxt_intset_fmask[] = { REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); + static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), /* Bits 3-4 reserved */ diff --git a/drivers/net/ipa/reg/gsi_reg-v4.0.c b/drivers/net/ipa/reg/gsi_reg-v4.0.c index 5a979ef4caad..7cc7a21d07f9 100644 --- a/drivers/net/ipa/reg/gsi_reg-v4.0.c +++ b/drivers/net/ipa/reg/gsi_reg-v4.0.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -67,10 +63,6 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); @@ -96,8 +88,12 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(15, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); @@ -153,6 +149,7 @@ REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; @@ -160,6 +157,7 @@ REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; @@ -236,6 +234,10 @@ static const u32 reg_cntxt_intset_fmask[] = { REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); + static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), /* Bits 3-4 reserved */ diff --git a/drivers/net/ipa/reg/gsi_reg-v4.11.c b/drivers/net/ipa/reg/gsi_reg-v4.11.c index d97597330659..01696519032f 100644 --- a/drivers/net/ipa/reg/gsi_reg-v4.11.c +++ b/drivers/net/ipa/reg/gsi_reg-v4.11.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -31,7 +27,7 @@ static const u32 reg_ch_c_cntxt_0_fmask[] = { }; REG_STRIDE_FIELDS(CH_C_CNTXT_0, ch_c_cntxt_0, - 0x0001c000 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f000 + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ch_c_cntxt_1_fmask[] = { [CH_R_LENGTH] = GENMASK(19, 0), @@ -39,11 +35,11 @@ static const u32 reg_ch_c_cntxt_1_fmask[] = { }; REG_STRIDE_FIELDS(CH_C_CNTXT_1, ch_c_cntxt_1, - 0x0001c004 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f004 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(CH_C_CNTXT_2, ch_c_cntxt_2, 0x0001c008 + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE(CH_C_CNTXT_2, ch_c_cntxt_2, 0x0000f008 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(CH_C_CNTXT_3, ch_c_cntxt_3, 0x0001c00c + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE(CH_C_CNTXT_3, ch_c_cntxt_3, 0x0000f00c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ch_c_qos_fmask[] = { [WRR_WEIGHT] = GENMASK(3, 0), @@ -57,7 +53,7 @@ static const u32 reg_ch_c_qos_fmask[] = { /* Bits 25-31 reserved */ }; -REG_STRIDE_FIELDS(CH_C_QOS, ch_c_qos, 0x0001c05c + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE_FIELDS(CH_C_QOS, ch_c_qos, 0x0000f05c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_error_log_fmask[] = { [ERR_ARG3] = GENMASK(3, 0), @@ -70,21 +66,17 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, - 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f060 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_1, ch_c_scratch_1, - 0x0001c064 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f064 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_2, ch_c_scratch_2, - 0x0001c068 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f068 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_3, ch_c_scratch_3, - 0x0001c06c + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f06c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { [EV_CHTYPE] = GENMASK(3, 0), @@ -97,19 +89,23 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { }; REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, - 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(19, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x00010004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, - 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010008 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_3, ev_ch_e_cntxt_3, - 0x0001d00c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001000c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_4, ev_ch_e_cntxt_4, - 0x0001d010 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010010 + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ev_ch_e_cntxt_8_fmask[] = { [EV_MODT] = GENMASK(15, 0), @@ -118,55 +114,57 @@ static const u32 reg_ev_ch_e_cntxt_8_fmask[] = { }; REG_STRIDE_FIELDS(EV_CH_E_CNTXT_8, ev_ch_e_cntxt_8, - 0x0001d020 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010020 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_9, ev_ch_e_cntxt_9, - 0x0001d024 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010024 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_10, ev_ch_e_cntxt_10, - 0x0001d028 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010028 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_11, ev_ch_e_cntxt_11, - 0x0001d02c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001002c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_12, ev_ch_e_cntxt_12, - 0x0001d030 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010030 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_13, ev_ch_e_cntxt_13, - 0x0001d034 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010034 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_SCRATCH_0, ev_ch_e_scratch_0, - 0x0001d048 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010048 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_SCRATCH_1, ev_ch_e_scratch_1, - 0x0001d04c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001004c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_DOORBELL_0, ch_c_doorbell_0, - 0x0001e000 + 0x4000 * GSI_EE_AP, 0x08); + 0x00011000 + 0x4000 * GSI_EE_AP, 0x08); REG_STRIDE(EV_CH_E_DOORBELL_0, ev_ch_e_doorbell_0, - 0x0001e100 + 0x4000 * GSI_EE_AP, 0x08); + 0x00011100 + 0x4000 * GSI_EE_AP, 0x08); static const u32 reg_gsi_status_fmask[] = { [ENABLED] = BIT(0), /* Bits 1-31 reserved */ }; -REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); +REG_FIELDS(GSI_STATUS, gsi_status, 0x00012000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; -REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CH_CMD, ch_cmd, 0x00012008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; -REG_FIELDS(EV_CH_CMD, ev_ch_cmd, 0x0001f010 + 0x4000 * GSI_EE_AP); +REG_FIELDS(EV_CH_CMD, ev_ch_cmd, 0x00012010 + 0x4000 * GSI_EE_AP); static const u32 reg_generic_cmd_fmask[] = { [GENERIC_OPCODE] = GENMASK(4, 0), @@ -176,7 +174,7 @@ static const u32 reg_generic_cmd_fmask[] = { [GENERIC_PARAMS] = GENMASK(31, 24), }; -REG_FIELDS(GENERIC_CMD, generic_cmd, 0x0001f018 + 0x4000 * GSI_EE_AP); +REG_FIELDS(GENERIC_CMD, generic_cmd, 0x00012018 + 0x4000 * GSI_EE_AP); static const u32 reg_hw_param_2_fmask[] = { [IRAM_SIZE] = GENMASK(2, 0), @@ -192,54 +190,58 @@ static const u32 reg_hw_param_2_fmask[] = { [GSI_USE_INTER_EE] = BIT(31), }; -REG_FIELDS(HW_PARAM_2, hw_param_2, 0x0001f040 + 0x4000 * GSI_EE_AP); +REG_FIELDS(HW_PARAM_2, hw_param_2, 0x00012040 + 0x4000 * GSI_EE_AP); -REG(CNTXT_TYPE_IRQ, cntxt_type_irq, 0x0001f080 + 0x4000 * GSI_EE_AP); +REG(CNTXT_TYPE_IRQ, cntxt_type_irq, 0x00012080 + 0x4000 * GSI_EE_AP); -REG(CNTXT_TYPE_IRQ_MSK, cntxt_type_irq_msk, 0x0001f088 + 0x4000 * GSI_EE_AP); +REG(CNTXT_TYPE_IRQ_MSK, cntxt_type_irq_msk, 0x00012088 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_CH_IRQ, cntxt_src_ch_irq, 0x0001f090 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_CH_IRQ, cntxt_src_ch_irq, 0x00012090 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_EV_CH_IRQ, cntxt_src_ev_ch_irq, 0x0001f094 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_EV_CH_IRQ, cntxt_src_ev_ch_irq, 0x00012094 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_CH_IRQ_MSK, cntxt_src_ch_irq_msk, - 0x0001f098 + 0x4000 * GSI_EE_AP); + 0x00012098 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_EV_CH_IRQ_MSK, cntxt_src_ev_ch_irq_msk, - 0x0001f09c + 0x4000 * GSI_EE_AP); + 0x0001209c + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_CH_IRQ_CLR, cntxt_src_ch_irq_clr, - 0x0001f0a0 + 0x4000 * GSI_EE_AP); + 0x000120a0 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_EV_CH_IRQ_CLR, cntxt_src_ev_ch_irq_clr, - 0x0001f0a4 + 0x4000 * GSI_EE_AP); + 0x000120a4 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_IEOB_IRQ, cntxt_src_ieob_irq, 0x0001f0b0 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_IEOB_IRQ, cntxt_src_ieob_irq, 0x000120b0 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_IEOB_IRQ_MSK, cntxt_src_ieob_irq_msk, - 0x0001f0b8 + 0x4000 * GSI_EE_AP); + 0x000120b8 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_IEOB_IRQ_CLR, cntxt_src_ieob_irq_clr, - 0x0001f0c0 + 0x4000 * GSI_EE_AP); + 0x000120c0 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_STTS, cntxt_glob_irq_stts, 0x0001f100 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_STTS, cntxt_glob_irq_stts, 0x00012100 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_EN, cntxt_glob_irq_en, 0x0001f108 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_EN, cntxt_glob_irq_en, 0x00012108 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_CLR, cntxt_glob_irq_clr, 0x0001f110 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_CLR, cntxt_glob_irq_clr, 0x00012110 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_STTS, cntxt_gsi_irq_stts, 0x0001f118 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_STTS, cntxt_gsi_irq_stts, 0x00012118 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_EN, cntxt_gsi_irq_en, 0x0001f120 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_EN, cntxt_gsi_irq_en, 0x00012120 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_CLR, cntxt_gsi_irq_clr, 0x0001f128 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_CLR, cntxt_gsi_irq_clr, 0x00012128 + 0x4000 * GSI_EE_AP); static const u32 reg_cntxt_intset_fmask[] = { [INTYPE] = BIT(0) /* Bits 1-31 reserved */ }; -REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x00012180 + 0x4000 * GSI_EE_AP); + +REG_FIELDS(ERROR_LOG, error_log, 0x00012200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x00012210 + 0x4000 * GSI_EE_AP); static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), @@ -248,7 +250,7 @@ static const u32 reg_cntxt_scratch_0_fmask[] = { /* Bits 8-31 reserved */ }; -REG_FIELDS(CNTXT_SCRATCH_0, cntxt_scratch_0, 0x0001f400 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CNTXT_SCRATCH_0, cntxt_scratch_0, 0x00012400 + 0x4000 * GSI_EE_AP); static const struct reg *reg_array[] = { [INTER_EE_SRC_CH_IRQ_MSK] = ®_inter_ee_src_ch_irq_msk, diff --git a/drivers/net/ipa/reg/gsi_reg-v4.5.c b/drivers/net/ipa/reg/gsi_reg-v4.5.c index 13c66b29840e..648b51b88d4e 100644 --- a/drivers/net/ipa/reg/gsi_reg-v4.5.c +++ b/drivers/net/ipa/reg/gsi_reg-v4.5.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -31,7 +27,7 @@ static const u32 reg_ch_c_cntxt_0_fmask[] = { }; REG_STRIDE_FIELDS(CH_C_CNTXT_0, ch_c_cntxt_0, - 0x0001c000 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f000 + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ch_c_cntxt_1_fmask[] = { [CH_R_LENGTH] = GENMASK(15, 0), @@ -39,11 +35,11 @@ static const u32 reg_ch_c_cntxt_1_fmask[] = { }; REG_STRIDE_FIELDS(CH_C_CNTXT_1, ch_c_cntxt_1, - 0x0001c004 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f004 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(CH_C_CNTXT_2, ch_c_cntxt_2, 0x0001c008 + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE(CH_C_CNTXT_2, ch_c_cntxt_2, 0x0000f008 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(CH_C_CNTXT_3, ch_c_cntxt_3, 0x0001c00c + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE(CH_C_CNTXT_3, ch_c_cntxt_3, 0x0000f00c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ch_c_qos_fmask[] = { [WRR_WEIGHT] = GENMASK(3, 0), @@ -56,7 +52,7 @@ static const u32 reg_ch_c_qos_fmask[] = { /* Bits 24-31 reserved */ }; -REG_STRIDE_FIELDS(CH_C_QOS, ch_c_qos, 0x0001c05c + 0x4000 * GSI_EE_AP, 0x80); +REG_STRIDE_FIELDS(CH_C_QOS, ch_c_qos, 0x0000f05c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_error_log_fmask[] = { [ERR_ARG3] = GENMASK(3, 0), @@ -69,21 +65,17 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, - 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f060 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_1, ch_c_scratch_1, - 0x0001c064 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f064 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_2, ch_c_scratch_2, - 0x0001c068 + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f068 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_SCRATCH_3, ch_c_scratch_3, - 0x0001c06c + 0x4000 * GSI_EE_AP, 0x80); + 0x0000f06c + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { [EV_CHTYPE] = GENMASK(3, 0), @@ -96,19 +88,23 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { }; REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, - 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(15, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x00010004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, - 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010008 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_3, ev_ch_e_cntxt_3, - 0x0001d00c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001000c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_4, ev_ch_e_cntxt_4, - 0x0001d010 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010010 + 0x4000 * GSI_EE_AP, 0x80); static const u32 reg_ev_ch_e_cntxt_8_fmask[] = { [EV_MODT] = GENMASK(15, 0), @@ -117,28 +113,28 @@ static const u32 reg_ev_ch_e_cntxt_8_fmask[] = { }; REG_STRIDE_FIELDS(EV_CH_E_CNTXT_8, ev_ch_e_cntxt_8, - 0x0001d020 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010020 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_9, ev_ch_e_cntxt_9, - 0x0001d024 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010024 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_10, ev_ch_e_cntxt_10, - 0x0001d028 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010028 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_11, ev_ch_e_cntxt_11, - 0x0001d02c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001002c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_12, ev_ch_e_cntxt_12, - 0x0001d030 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010030 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_13, ev_ch_e_cntxt_13, - 0x0001d034 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010034 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_SCRATCH_0, ev_ch_e_scratch_0, - 0x0001d048 + 0x4000 * GSI_EE_AP, 0x80); + 0x00010048 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_SCRATCH_1, ev_ch_e_scratch_1, - 0x0001d04c + 0x4000 * GSI_EE_AP, 0x80); + 0x0001004c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_DOORBELL_0, ch_c_doorbell_0, 0x0001e000 + 0x4000 * GSI_EE_AP, 0x08); @@ -155,6 +151,7 @@ REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; @@ -162,6 +159,7 @@ REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; @@ -239,6 +237,10 @@ static const u32 reg_cntxt_intset_fmask[] = { REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); + static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), /* Bits 3-4 reserved */ diff --git a/drivers/net/ipa/reg/gsi_reg-v4.9.c b/drivers/net/ipa/reg/gsi_reg-v4.9.c index a7d5732b72e9..4bf45d264d6b 100644 --- a/drivers/net/ipa/reg/gsi_reg-v4.9.c +++ b/drivers/net/ipa/reg/gsi_reg-v4.9.c @@ -8,16 +8,12 @@ #include "../reg.h" #include "../gsi_reg.h" -/* The inter-EE IRQ registers are relative to gsi->virt_raw (IPA v3.5+) */ - REG(INTER_EE_SRC_CH_IRQ_MSK, inter_ee_src_ch_irq_msk, 0x0000c020 + 0x1000 * GSI_EE_AP); REG(INTER_EE_SRC_EV_CH_IRQ_MSK, inter_ee_src_ev_ch_irq_msk, 0x0000c024 + 0x1000 * GSI_EE_AP); -/* All other register offsets are relative to gsi->virt */ - static const u32 reg_ch_c_cntxt_0_fmask[] = { [CHTYPE_PROTOCOL] = GENMASK(2, 0), [CHTYPE_DIR] = BIT(3), @@ -70,10 +66,6 @@ static const u32 reg_error_log_fmask[] = { [ERR_EE] = GENMASK(31, 28), }; -REG_FIELDS(ERROR_LOG, error_log, 0x0001f200 + 0x4000 * GSI_EE_AP); - -REG(ERROR_LOG_CLR, error_log_clr, 0x0001f210 + 0x4000 * GSI_EE_AP); - REG_STRIDE(CH_C_SCRATCH_0, ch_c_scratch_0, 0x0001c060 + 0x4000 * GSI_EE_AP, 0x80); @@ -99,8 +91,12 @@ static const u32 reg_ev_ch_e_cntxt_0_fmask[] = { REG_STRIDE_FIELDS(EV_CH_E_CNTXT_0, ev_ch_e_cntxt_0, 0x0001d000 + 0x4000 * GSI_EE_AP, 0x80); -REG_STRIDE(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, - 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); +static const u32 reg_ev_ch_e_cntxt_1_fmask[] = { + [R_LENGTH] = GENMASK(15, 0), +}; + +REG_STRIDE_FIELDS(EV_CH_E_CNTXT_1, ev_ch_e_cntxt_1, + 0x0001d004 + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(EV_CH_E_CNTXT_2, ev_ch_e_cntxt_2, 0x0001d008 + 0x4000 * GSI_EE_AP, 0x80); @@ -142,31 +138,33 @@ REG_STRIDE(EV_CH_E_SCRATCH_1, ev_ch_e_scratch_1, 0x0001d04c + 0x4000 * GSI_EE_AP, 0x80); REG_STRIDE(CH_C_DOORBELL_0, ch_c_doorbell_0, - 0x0001e000 + 0x4000 * GSI_EE_AP, 0x08); + 0x00011000 + 0x4000 * GSI_EE_AP, 0x08); REG_STRIDE(EV_CH_E_DOORBELL_0, ev_ch_e_doorbell_0, - 0x0001e100 + 0x4000 * GSI_EE_AP, 0x08); + 0x00011100 + 0x4000 * GSI_EE_AP, 0x08); static const u32 reg_gsi_status_fmask[] = { [ENABLED] = BIT(0), /* Bits 1-31 reserved */ }; -REG_FIELDS(GSI_STATUS, gsi_status, 0x0001f000 + 0x4000 * GSI_EE_AP); +REG_FIELDS(GSI_STATUS, gsi_status, 0x00012000 + 0x4000 * GSI_EE_AP); static const u32 reg_ch_cmd_fmask[] = { [CH_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [CH_OPCODE] = GENMASK(31, 24), }; -REG_FIELDS(CH_CMD, ch_cmd, 0x0001f008 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CH_CMD, ch_cmd, 0x00012008 + 0x4000 * GSI_EE_AP); static const u32 reg_ev_ch_cmd_fmask[] = { [EV_CHID] = GENMASK(7, 0), + /* Bits 8-23 reserved */ [EV_OPCODE] = GENMASK(31, 24), }; -REG_FIELDS(EV_CH_CMD, ev_ch_cmd, 0x0001f010 + 0x4000 * GSI_EE_AP); +REG_FIELDS(EV_CH_CMD, ev_ch_cmd, 0x00012010 + 0x4000 * GSI_EE_AP); static const u32 reg_generic_cmd_fmask[] = { [GENERIC_OPCODE] = GENMASK(4, 0), @@ -175,7 +173,7 @@ static const u32 reg_generic_cmd_fmask[] = { /* Bits 14-31 reserved */ }; -REG_FIELDS(GENERIC_CMD, generic_cmd, 0x0001f018 + 0x4000 * GSI_EE_AP); +REG_FIELDS(GENERIC_CMD, generic_cmd, 0x00012018 + 0x4000 * GSI_EE_AP); static const u32 reg_hw_param_2_fmask[] = { [IRAM_SIZE] = GENMASK(2, 0), @@ -191,54 +189,58 @@ static const u32 reg_hw_param_2_fmask[] = { [GSI_USE_INTER_EE] = BIT(31), }; -REG_FIELDS(HW_PARAM_2, hw_param_2, 0x0001f040 + 0x4000 * GSI_EE_AP); +REG_FIELDS(HW_PARAM_2, hw_param_2, 0x00012040 + 0x4000 * GSI_EE_AP); -REG(CNTXT_TYPE_IRQ, cntxt_type_irq, 0x0001f080 + 0x4000 * GSI_EE_AP); +REG(CNTXT_TYPE_IRQ, cntxt_type_irq, 0x00012080 + 0x4000 * GSI_EE_AP); -REG(CNTXT_TYPE_IRQ_MSK, cntxt_type_irq_msk, 0x0001f088 + 0x4000 * GSI_EE_AP); +REG(CNTXT_TYPE_IRQ_MSK, cntxt_type_irq_msk, 0x00012088 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_CH_IRQ, cntxt_src_ch_irq, 0x0001f090 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_CH_IRQ, cntxt_src_ch_irq, 0x00012090 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_EV_CH_IRQ, cntxt_src_ev_ch_irq, 0x0001f094 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_EV_CH_IRQ, cntxt_src_ev_ch_irq, 0x00012094 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_CH_IRQ_MSK, cntxt_src_ch_irq_msk, - 0x0001f098 + 0x4000 * GSI_EE_AP); + 0x00012098 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_EV_CH_IRQ_MSK, cntxt_src_ev_ch_irq_msk, - 0x0001f09c + 0x4000 * GSI_EE_AP); + 0x0001209c + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_CH_IRQ_CLR, cntxt_src_ch_irq_clr, - 0x0001f0a0 + 0x4000 * GSI_EE_AP); + 0x000120a0 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_EV_CH_IRQ_CLR, cntxt_src_ev_ch_irq_clr, - 0x0001f0a4 + 0x4000 * GSI_EE_AP); + 0x000120a4 + 0x4000 * GSI_EE_AP); -REG(CNTXT_SRC_IEOB_IRQ, cntxt_src_ieob_irq, 0x0001f0b0 + 0x4000 * GSI_EE_AP); +REG(CNTXT_SRC_IEOB_IRQ, cntxt_src_ieob_irq, 0x000120b0 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_IEOB_IRQ_MSK, cntxt_src_ieob_irq_msk, - 0x0001f0b8 + 0x4000 * GSI_EE_AP); + 0x000120b8 + 0x4000 * GSI_EE_AP); REG(CNTXT_SRC_IEOB_IRQ_CLR, cntxt_src_ieob_irq_clr, - 0x0001f0c0 + 0x4000 * GSI_EE_AP); + 0x000120c0 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_STTS, cntxt_glob_irq_stts, 0x0001f100 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_STTS, cntxt_glob_irq_stts, 0x00012100 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_EN, cntxt_glob_irq_en, 0x0001f108 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_EN, cntxt_glob_irq_en, 0x00012108 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GLOB_IRQ_CLR, cntxt_glob_irq_clr, 0x0001f110 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GLOB_IRQ_CLR, cntxt_glob_irq_clr, 0x00012110 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_STTS, cntxt_gsi_irq_stts, 0x0001f118 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_STTS, cntxt_gsi_irq_stts, 0x00012118 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_EN, cntxt_gsi_irq_en, 0x0001f120 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_EN, cntxt_gsi_irq_en, 0x00012120 + 0x4000 * GSI_EE_AP); -REG(CNTXT_GSI_IRQ_CLR, cntxt_gsi_irq_clr, 0x0001f128 + 0x4000 * GSI_EE_AP); +REG(CNTXT_GSI_IRQ_CLR, cntxt_gsi_irq_clr, 0x00012128 + 0x4000 * GSI_EE_AP); static const u32 reg_cntxt_intset_fmask[] = { [INTYPE] = BIT(0) /* Bits 1-31 reserved */ }; -REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x0001f180 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CNTXT_INTSET, cntxt_intset, 0x00012180 + 0x4000 * GSI_EE_AP); + +REG_FIELDS(ERROR_LOG, error_log, 0x00012200 + 0x4000 * GSI_EE_AP); + +REG(ERROR_LOG_CLR, error_log_clr, 0x00012210 + 0x4000 * GSI_EE_AP); static const u32 reg_cntxt_scratch_0_fmask[] = { [INTER_EE_RESULT] = GENMASK(2, 0), @@ -247,7 +249,7 @@ static const u32 reg_cntxt_scratch_0_fmask[] = { /* Bits 8-31 reserved */ }; -REG_FIELDS(CNTXT_SCRATCH_0, cntxt_scratch_0, 0x0001f400 + 0x4000 * GSI_EE_AP); +REG_FIELDS(CNTXT_SCRATCH_0, cntxt_scratch_0, 0x00012400 + 0x4000 * GSI_EE_AP); static const struct reg *reg_array[] = { [INTER_EE_SRC_CH_IRQ_MSK] = ®_inter_ee_src_ch_irq_msk, diff --git a/drivers/net/phy/marvell.c b/drivers/net/phy/marvell.c index 0d706ee266af..63a3644d86c9 100644 --- a/drivers/net/phy/marvell.c +++ b/drivers/net/phy/marvell.c @@ -1467,7 +1467,7 @@ static int m88e1540_set_fld(struct phy_device *phydev, const u8 *msecs) /* According to the Marvell data sheet EEE must be disabled for * Fast Link Down detection to work properly */ - ret = phy_ethtool_get_eee(phydev, &eee); + ret = genphy_c45_ethtool_get_eee(phydev, &eee); if (!ret && eee.eee_enabled) { phydev_warn(phydev, "Fast Link Down detection requires EEE to be disabled!\n"); return -EBUSY; diff --git a/drivers/net/phy/phy.c b/drivers/net/phy/phy.c index 2f1041a7211e..b33e55a7364e 100644 --- a/drivers/net/phy/phy.c +++ b/drivers/net/phy/phy.c @@ -1069,27 +1069,35 @@ EXPORT_SYMBOL(phy_ethtool_ksettings_set); int phy_speed_down(struct phy_device *phydev, bool sync) { __ETHTOOL_DECLARE_LINK_MODE_MASK(adv_tmp); - int ret; + int ret = 0; + + mutex_lock(&phydev->lock); if (phydev->autoneg != AUTONEG_ENABLE) - return 0; + goto out; linkmode_copy(adv_tmp, phydev->advertising); ret = phy_speed_down_core(phydev); if (ret) - return ret; + goto out; linkmode_copy(phydev->adv_old, adv_tmp); - if (linkmode_equal(phydev->advertising, adv_tmp)) - return 0; + if (linkmode_equal(phydev->advertising, adv_tmp)) { + ret = 0; + goto out; + } ret = phy_config_aneg(phydev); if (ret) - return ret; + goto out; + + ret = sync ? phy_poll_aneg_done(phydev) : 0; +out: + mutex_unlock(&phydev->lock); - return sync ? phy_poll_aneg_done(phydev) : 0; + return ret; } EXPORT_SYMBOL_GPL(phy_speed_down); @@ -1102,21 +1110,28 @@ EXPORT_SYMBOL_GPL(phy_speed_down); int phy_speed_up(struct phy_device *phydev) { __ETHTOOL_DECLARE_LINK_MODE_MASK(adv_tmp); + int ret = 0; + + mutex_lock(&phydev->lock); if (phydev->autoneg != AUTONEG_ENABLE) - return 0; + goto out; if (linkmode_empty(phydev->adv_old)) - return 0; + goto out; linkmode_copy(adv_tmp, phydev->advertising); linkmode_copy(phydev->advertising, phydev->adv_old); linkmode_zero(phydev->adv_old); if (linkmode_equal(phydev->advertising, adv_tmp)) - return 0; + goto out; + + ret = phy_config_aneg(phydev); +out: + mutex_unlock(&phydev->lock); - return phy_config_aneg(phydev); + return ret; } EXPORT_SYMBOL_GPL(phy_speed_up); @@ -1500,10 +1515,16 @@ EXPORT_SYMBOL(phy_init_eee); */ int phy_get_eee_err(struct phy_device *phydev) { + int ret; + if (!phydev->drv) return -EIO; - return phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_WK_ERR); + mutex_lock(&phydev->lock); + ret = phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_WK_ERR); + mutex_unlock(&phydev->lock); + + return ret; } EXPORT_SYMBOL(phy_get_eee_err); @@ -1517,10 +1538,16 @@ EXPORT_SYMBOL(phy_get_eee_err); */ int phy_ethtool_get_eee(struct phy_device *phydev, struct ethtool_eee *data) { + int ret; + if (!phydev->drv) return -EIO; - return genphy_c45_ethtool_get_eee(phydev, data); + mutex_lock(&phydev->lock); + ret = genphy_c45_ethtool_get_eee(phydev, data); + mutex_unlock(&phydev->lock); + + return ret; } EXPORT_SYMBOL(phy_ethtool_get_eee); @@ -1533,10 +1560,16 @@ EXPORT_SYMBOL(phy_ethtool_get_eee); */ int phy_ethtool_set_eee(struct phy_device *phydev, struct ethtool_eee *data) { + int ret; + if (!phydev->drv) return -EIO; - return genphy_c45_ethtool_set_eee(phydev, data); + mutex_lock(&phydev->lock); + ret = genphy_c45_ethtool_set_eee(phydev, data); + mutex_unlock(&phydev->lock); + + return ret; } EXPORT_SYMBOL(phy_ethtool_set_eee); @@ -1548,8 +1581,15 @@ EXPORT_SYMBOL(phy_ethtool_set_eee); */ int phy_ethtool_set_wol(struct phy_device *phydev, struct ethtool_wolinfo *wol) { - if (phydev->drv && phydev->drv->set_wol) - return phydev->drv->set_wol(phydev, wol); + int ret; + + if (phydev->drv && phydev->drv->set_wol) { + mutex_lock(&phydev->lock); + ret = phydev->drv->set_wol(phydev, wol); + mutex_unlock(&phydev->lock); + + return ret; + } return -EOPNOTSUPP; } @@ -1563,8 +1603,11 @@ EXPORT_SYMBOL(phy_ethtool_set_wol); */ void phy_ethtool_get_wol(struct phy_device *phydev, struct ethtool_wolinfo *wol) { - if (phydev->drv && phydev->drv->get_wol) + if (phydev->drv && phydev->drv->get_wol) { + mutex_lock(&phydev->lock); phydev->drv->get_wol(phydev, wol); + mutex_unlock(&phydev->lock); + } } EXPORT_SYMBOL(phy_ethtool_get_wol); @@ -1601,6 +1644,7 @@ EXPORT_SYMBOL(phy_ethtool_set_link_ksettings); int phy_ethtool_nway_reset(struct net_device *ndev) { struct phy_device *phydev = ndev->phydev; + int ret; if (!phydev) return -ENODEV; @@ -1608,6 +1652,10 @@ int phy_ethtool_nway_reset(struct net_device *ndev) if (!phydev->drv) return -EIO; - return phy_restart_aneg(phydev); + mutex_lock(&phydev->lock); + ret = phy_restart_aneg(phydev); + mutex_unlock(&phydev->lock); + + return ret; } EXPORT_SYMBOL(phy_ethtool_nway_reset); diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c index 8d927c5e3bf8..71becceb8764 100644 --- a/drivers/net/phy/phy_device.c +++ b/drivers/net/phy/phy_device.c @@ -3113,8 +3113,10 @@ static int phy_probe(struct device *dev) * a controller will attach, and may modify one * or both of these values */ - if (phydrv->features) + if (phydrv->features) { linkmode_copy(phydev->supported, phydrv->features); + genphy_c45_read_eee_abilities(phydev); + } else if (phydrv->get_features) err = phydrv->get_features(phydev); else if (phydev->is_c45) diff --git a/drivers/net/wireless/intel/iwlegacy/4965-mac.c b/drivers/net/wireless/intel/iwlegacy/4965-mac.c index 4d3c544ff2e6..0a4aa3c678c1 100644 --- a/drivers/net/wireless/intel/iwlegacy/4965-mac.c +++ b/drivers/net/wireless/intel/iwlegacy/4965-mac.c @@ -4020,7 +4020,7 @@ il4965_hdl_alive(struct il_priv *il, struct il_rx_buf *rxb) if (palive->ver_subtype == INITIALIZE_SUBTYPE) { D_INFO("Initialization Alive received.\n"); - memcpy(&il->card_alive_init, &pkt->u.alive_frame, + memcpy(&il->card_alive_init, &pkt->u.raw, sizeof(struct il_init_alive_resp)); pwork = &il->init_alive_start; } else { diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c index 78d8b37eb71a..3779ac040ba0 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c @@ -438,13 +438,6 @@ static ssize_t iwl_dbgfs_bf_params_read(struct file *file, return simple_read_from_buffer(user_buf, count, ppos, buf, pos); } -static inline char *iwl_dbgfs_is_match(char *name, char *buf) -{ - int len = strlen(name); - - return !strncmp(name, buf, len) ? buf + len : NULL; -} - static ssize_t iwl_dbgfs_os_device_timediff_read(struct file *file, char __user *user_buf, size_t count, loff_t *ppos) diff --git a/drivers/net/wireless/realtek/rtl8xxxu/Kconfig b/drivers/net/wireless/realtek/rtl8xxxu/Kconfig index 091d3ad98093..2eed20b0988c 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/Kconfig +++ b/drivers/net/wireless/realtek/rtl8xxxu/Kconfig @@ -5,6 +5,7 @@ config RTL8XXXU tristate "Realtek 802.11n USB wireless chips support" depends on MAC80211 && USB + depends on LEDS_CLASS help This is an alternative driver for various Realtek RTL8XXX parts written to utilize the Linux mac80211 stack. diff --git a/drivers/net/wireless/realtek/rtw88/coex.c b/drivers/net/wireless/realtek/rtw88/coex.c index 38697237ee5f..86467d2f8888 100644 --- a/drivers/net/wireless/realtek/rtw88/coex.c +++ b/drivers/net/wireless/realtek/rtw88/coex.c @@ -4056,7 +4056,7 @@ void rtw_coex_display_coex_info(struct rtw_dev *rtwdev, struct seq_file *m) rtwdev->stats.tx_throughput, rtwdev->stats.rx_throughput); seq_printf(m, "%-40s = %u/ %u/ %u\n", "IPS/ Low Power/ PS mode", - test_bit(RTW_FLAG_INACTIVE_PS, rtwdev->flags), + !test_bit(RTW_FLAG_POWERON, rtwdev->flags), test_bit(RTW_FLAG_LEISURE_PS_DEEP, rtwdev->flags), rtwdev->lps_conf.mode); diff --git a/drivers/net/wireless/realtek/rtw88/mac.c b/drivers/net/wireless/realtek/rtw88/mac.c index 4e5c194aac29..dae64901bac5 100644 --- a/drivers/net/wireless/realtek/rtw88/mac.c +++ b/drivers/net/wireless/realtek/rtw88/mac.c @@ -273,6 +273,11 @@ static int rtw_mac_power_switch(struct rtw_dev *rtwdev, bool pwr_on) if (rtw_pwr_seq_parser(rtwdev, pwr_seq)) return -EINVAL; + if (pwr_on) + set_bit(RTW_FLAG_POWERON, rtwdev->flags); + else + clear_bit(RTW_FLAG_POWERON, rtwdev->flags); + return 0; } @@ -335,6 +340,11 @@ int rtw_mac_power_on(struct rtw_dev *rtwdev) ret = rtw_mac_power_switch(rtwdev, true); if (ret == -EALREADY) { rtw_mac_power_switch(rtwdev, false); + + ret = rtw_mac_pre_system_cfg(rtwdev); + if (ret) + goto err; + ret = rtw_mac_power_switch(rtwdev, true); if (ret) goto err; diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h index 165f299e8e1f..d4a53d556745 100644 --- a/drivers/net/wireless/realtek/rtw88/main.h +++ b/drivers/net/wireless/realtek/rtw88/main.h @@ -356,7 +356,7 @@ enum rtw_flags { RTW_FLAG_RUNNING, RTW_FLAG_FW_RUNNING, RTW_FLAG_SCANNING, - RTW_FLAG_INACTIVE_PS, + RTW_FLAG_POWERON, RTW_FLAG_LEISURE_PS, RTW_FLAG_LEISURE_PS_DEEP, RTW_FLAG_DIG_DISABLE, diff --git a/drivers/net/wireless/realtek/rtw88/ps.c b/drivers/net/wireless/realtek/rtw88/ps.c index 11594940d6b0..996365575f44 100644 --- a/drivers/net/wireless/realtek/rtw88/ps.c +++ b/drivers/net/wireless/realtek/rtw88/ps.c @@ -25,7 +25,7 @@ static int rtw_ips_pwr_up(struct rtw_dev *rtwdev) int rtw_enter_ips(struct rtw_dev *rtwdev) { - if (test_and_set_bit(RTW_FLAG_INACTIVE_PS, rtwdev->flags)) + if (!test_bit(RTW_FLAG_POWERON, rtwdev->flags)) return 0; rtw_coex_ips_notify(rtwdev, COEX_IPS_ENTER); @@ -50,7 +50,7 @@ int rtw_leave_ips(struct rtw_dev *rtwdev) { int ret; - if (!test_and_clear_bit(RTW_FLAG_INACTIVE_PS, rtwdev->flags)) + if (test_bit(RTW_FLAG_POWERON, rtwdev->flags)) return 0; rtw_hci_link_ps(rtwdev, false); diff --git a/drivers/net/wireless/realtek/rtw88/wow.c b/drivers/net/wireless/realtek/rtw88/wow.c index 89dc595094d5..16ddee577efe 100644 --- a/drivers/net/wireless/realtek/rtw88/wow.c +++ b/drivers/net/wireless/realtek/rtw88/wow.c @@ -592,7 +592,7 @@ static int rtw_wow_leave_no_link_ps(struct rtw_dev *rtwdev) if (rtw_get_lps_deep_mode(rtwdev) != LPS_DEEP_MODE_NONE) rtw_leave_lps_deep(rtwdev); } else { - if (test_bit(RTW_FLAG_INACTIVE_PS, rtwdev->flags)) { + if (!test_bit(RTW_FLAG_POWERON, rtwdev->flags)) { rtw_wow->ips_enabled = true; ret = rtw_leave_ips(rtwdev); if (ret) diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c index 3ed2f3a96635..f09361bc4a4d 100644 --- a/drivers/net/wireless/realtek/rtw89/core.c +++ b/drivers/net/wireless/realtek/rtw89/core.c @@ -2435,6 +2435,7 @@ int rtw89_core_sta_add(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif = (struct rtw89_vif *)vif->drv_priv; struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; int i; + int ret; rtwsta->rtwdev = rtwdev; rtwsta->rtwvif = rtwvif; @@ -2459,6 +2460,21 @@ int rtw89_core_sta_add(struct rtw89_dev *rtwdev, RTW89_MAX_MAC_ID_NUM); if (rtwsta->mac_id == RTW89_MAX_MAC_ID_NUM) return -ENOSPC; + + ret = rtw89_mac_set_macid_pause(rtwdev, rtwsta->mac_id, false); + if (ret) { + rtw89_core_release_bit_map(rtwdev->mac_id_map, rtwsta->mac_id); + rtw89_warn(rtwdev, "failed to send h2c macid pause\n"); + return ret; + } + + ret = rtw89_fw_h2c_role_maintain(rtwdev, rtwvif, rtwsta, + RTW89_ROLE_CREATE); + if (ret) { + rtw89_core_release_bit_map(rtwdev->mac_id_map, rtwsta->mac_id); + rtw89_warn(rtwdev, "failed to send h2c role info\n"); + return ret; + } } return 0; @@ -2513,14 +2529,6 @@ int rtw89_core_sta_disconnect(struct rtw89_dev *rtwdev, return ret; } - if (vif->type == NL80211_IFTYPE_AP || sta->tdls) { - ret = rtw89_fw_h2c_role_maintain(rtwdev, rtwvif, rtwsta, RTW89_ROLE_REMOVE); - if (ret) { - rtw89_warn(rtwdev, "failed to send h2c role info\n"); - return ret; - } - } - /* update cam aid mac_id net_type */ ret = rtw89_fw_h2c_cam(rtwdev, rtwvif, rtwsta, NULL); if (ret) { @@ -2541,18 +2549,6 @@ int rtw89_core_sta_assoc(struct rtw89_dev *rtwdev, int ret; if (vif->type == NL80211_IFTYPE_AP || sta->tdls) { - ret = rtw89_mac_set_macid_pause(rtwdev, rtwsta->mac_id, false); - if (ret) { - rtw89_warn(rtwdev, "failed to send h2c macid pause\n"); - return ret; - } - - ret = rtw89_fw_h2c_role_maintain(rtwdev, rtwvif, rtwsta, RTW89_ROLE_CREATE); - if (ret) { - rtw89_warn(rtwdev, "failed to send h2c role info\n"); - return ret; - } - if (sta->tdls) { ret = rtw89_cam_init_bssid_cam(rtwdev, rtwvif, bssid_cam, sta->addr); if (ret) { @@ -2622,13 +2618,22 @@ int rtw89_core_sta_remove(struct rtw89_dev *rtwdev, { struct rtw89_vif *rtwvif = (struct rtw89_vif *)vif->drv_priv; struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; + int ret; if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls) rtw89_btc_ntfy_role_info(rtwdev, rtwvif, rtwsta, BTC_ROLE_MSTS_STA_DIS_CONN); - else if (vif->type == NL80211_IFTYPE_AP || sta->tdls) + else if (vif->type == NL80211_IFTYPE_AP || sta->tdls) { rtw89_core_release_bit_map(rtwdev->mac_id_map, rtwsta->mac_id); + ret = rtw89_fw_h2c_role_maintain(rtwdev, rtwvif, rtwsta, + RTW89_ROLE_REMOVE); + if (ret) { + rtw89_warn(rtwdev, "failed to send h2c role info\n"); + return ret; + } + } + return 0; } diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 35c18a98c21a..520b238abd5a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -28,6 +28,7 @@ #include <linux/btf.h> #include <linux/rcupdate_trace.h> #include <linux/static_call.h> +#include <linux/memcontrol.h> struct bpf_verifier_env; struct bpf_verifier_log; @@ -180,6 +181,10 @@ enum btf_field_type { BPF_KPTR = BPF_KPTR_UNREF | BPF_KPTR_REF, BPF_LIST_HEAD = (1 << 4), BPF_LIST_NODE = (1 << 5), + BPF_RB_ROOT = (1 << 6), + BPF_RB_NODE = (1 << 7), + BPF_GRAPH_NODE_OR_ROOT = BPF_LIST_NODE | BPF_LIST_HEAD | + BPF_RB_NODE | BPF_RB_ROOT, }; struct btf_field_kptr { @@ -283,6 +288,10 @@ static inline const char *btf_field_type_name(enum btf_field_type type) return "bpf_list_head"; case BPF_LIST_NODE: return "bpf_list_node"; + case BPF_RB_ROOT: + return "bpf_rb_root"; + case BPF_RB_NODE: + return "bpf_rb_node"; default: WARN_ON_ONCE(1); return "unknown"; @@ -303,6 +312,10 @@ static inline u32 btf_field_type_size(enum btf_field_type type) return sizeof(struct bpf_list_head); case BPF_LIST_NODE: return sizeof(struct bpf_list_node); + case BPF_RB_ROOT: + return sizeof(struct bpf_rb_root); + case BPF_RB_NODE: + return sizeof(struct bpf_rb_node); default: WARN_ON_ONCE(1); return 0; @@ -323,6 +336,10 @@ static inline u32 btf_field_type_align(enum btf_field_type type) return __alignof__(struct bpf_list_head); case BPF_LIST_NODE: return __alignof__(struct bpf_list_node); + case BPF_RB_ROOT: + return __alignof__(struct bpf_rb_root); + case BPF_RB_NODE: + return __alignof__(struct bpf_rb_node); default: WARN_ON_ONCE(1); return 0; @@ -346,6 +363,13 @@ static inline void bpf_obj_init(const struct btf_field_offs *foffs, void *obj) memset(obj + foffs->field_off[i], 0, foffs->field_sz[i]); } +/* 'dst' must be a temporary buffer and should not point to memory that is being + * used in parallel by a bpf program or bpf syscall, otherwise the access from + * the bpf program or bpf syscall may be corrupted by the reinitialization, + * leading to weird problems. Even 'dst' is newly-allocated from bpf memory + * allocator, it is still possible for 'dst' to be used in parallel by a bpf + * program or bpf syscall. + */ static inline void check_and_init_map_value(struct bpf_map *map, void *dst) { bpf_obj_init(map->field_offs, dst); @@ -433,6 +457,9 @@ void copy_map_value_locked(struct bpf_map *map, void *dst, void *src, void bpf_timer_cancel_and_free(void *timer); void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock); +void bpf_rb_root_free(const struct btf_field *field, void *rb_root, + struct bpf_spin_lock *spin_lock); + int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); @@ -575,6 +602,11 @@ enum bpf_type_flag { /* MEM is tagged with rcu and memory access needs rcu_read_lock protection. */ MEM_RCU = BIT(13 + BPF_BASE_TYPE_BITS), + /* Used to tag PTR_TO_BTF_ID | MEM_ALLOC references which are non-owning. + * Currently only valid for linked-list and rbtree nodes. + */ + NON_OWN_REF = BIT(14 + BPF_BASE_TYPE_BITS), + __BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, }; @@ -1886,6 +1918,8 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); #else @@ -1902,6 +1936,12 @@ bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return kzalloc(size, flags); } +static inline void * +bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) +{ + return kvcalloc(n, size, flags); +} + static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) @@ -2925,4 +2965,11 @@ static inline bool type_is_alloc(u32 type) return type & MEM_ALLOC; } +static inline gfp_t bpf_memcg_flags(gfp_t flags) +{ + if (memcg_bpf_enabled()) + return flags | __GFP_ACCOUNT; + return flags; +} + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index aa83de1fe755..cf1bb1cf4a7b 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -43,6 +43,22 @@ enum bpf_reg_liveness { REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ }; +/* For every reg representing a map value or allocated object pointer, + * we consider the tuple of (ptr, id) for them to be unique in verifier + * context and conside them to not alias each other for the purposes of + * tracking lock state. + */ +struct bpf_active_lock { + /* This can either be reg->map_ptr or reg->btf. If ptr is NULL, + * there's no active lock held, and other fields have no + * meaning. If non-NULL, it indicates that a lock is held and + * id member has the reg->id of the register which can be >= 0. + */ + void *ptr; + /* This will be reg->id */ + u32 id; +}; + struct bpf_reg_state { /* Ordering of fields matters. See states_equal() */ enum bpf_reg_type type; @@ -226,11 +242,6 @@ struct bpf_reference_state { * exiting a callback function. */ int callback_ref; - /* Mark the reference state to release the registers sharing the same id - * on bpf_spin_unlock (for nodes that we will lose ownership to but are - * safe to access inside the critical section). - */ - bool release_on_unlock; }; /* state of the program: @@ -331,21 +342,8 @@ struct bpf_verifier_state { u32 branches; u32 insn_idx; u32 curframe; - /* For every reg representing a map value or allocated object pointer, - * we consider the tuple of (ptr, id) for them to be unique in verifier - * context and conside them to not alias each other for the purposes of - * tracking lock state. - */ - struct { - /* This can either be reg->map_ptr or reg->btf. If ptr is NULL, - * there's no active lock held, and other fields have no - * meaning. If non-NULL, it indicates that a lock is held and - * id member has the reg->id of the register which can be >= 0. - */ - void *ptr; - /* This will be reg->id */ - u32 id; - } active_lock; + + struct bpf_active_lock active_lock; bool speculative; bool active_rcu_lock; diff --git a/include/linux/ieee802154.h b/include/linux/ieee802154.h index 0303eb84d596..140f61ec0f5f 100644 --- a/include/linux/ieee802154.h +++ b/include/linux/ieee802154.h @@ -44,6 +44,13 @@ #define IEEE802154_SHORT_ADDR_LEN 2 #define IEEE802154_PAN_ID_LEN 2 +/* Duration in superframe order */ +#define IEEE802154_MAX_SCAN_DURATION 14 +#define IEEE802154_ACTIVE_SCAN_DURATION 15 +/* Superframe duration in slots */ +#define IEEE802154_SUPERFRAME_PERIOD 16 +/* Various periods expressed in symbols */ +#define IEEE802154_SLOT_PERIOD 60 #define IEEE802154_LIFS_PERIOD 40 #define IEEE802154_SIFS_PERIOD 12 #define IEEE802154_MAX_SIFS_FRAME_SIZE 18 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 85dc9b88ea37..1e38e99998c7 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1754,6 +1754,12 @@ struct obj_cgroup *get_obj_cgroup_from_page(struct page *page); int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); +extern struct static_key_false memcg_bpf_enabled_key; +static inline bool memcg_bpf_enabled(void) +{ + return static_branch_likely(&memcg_bpf_enabled_key); +} + extern struct static_key_false memcg_kmem_enabled_key; static inline bool memcg_kmem_enabled(void) @@ -1832,6 +1838,11 @@ static inline struct obj_cgroup *get_obj_cgroup_from_page(struct page *page) return NULL; } +static inline bool memcg_bpf_enabled(void) +{ + return false; +} + static inline bool memcg_kmem_enabled(void) { return false; diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h index 9db93e487496..b6b626157b03 100644 --- a/include/linux/mlx4/qp.h +++ b/include/linux/mlx4/qp.h @@ -446,6 +446,7 @@ enum { struct mlx4_wqe_inline_seg { __be32 byte_count; + __u8 data[]; }; enum mlx4_update_qp_attr { diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index efbee940bb03..6a14b7b11766 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -224,7 +224,6 @@ struct net_device_core_stats { #include <linux/static_key.h> extern struct static_key_false rps_needed; extern struct static_key_false rfs_needed; -extern struct cpumask rps_default_mask; #endif struct neighbour; diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h index d8817d381c14..6863e271a9de 100644 --- a/include/linux/netfilter.h +++ b/include/linux/netfilter.h @@ -437,11 +437,13 @@ nf_nat_decode_session(struct sk_buff *skb, struct flowi *fl, u_int8_t family) #include <linux/netfilter/nf_conntrack_zones_common.h> void nf_ct_attach(struct sk_buff *, const struct sk_buff *); +void nf_ct_set_closing(struct nf_conntrack *nfct); struct nf_conntrack_tuple; bool nf_ct_get_tuple_skb(struct nf_conntrack_tuple *dst_tuple, const struct sk_buff *skb); #else static inline void nf_ct_attach(struct sk_buff *new, struct sk_buff *skb) {} +static inline void nf_ct_set_closing(struct nf_conntrack *nfct) {} struct nf_conntrack_tuple; static inline bool nf_ct_get_tuple_skb(struct nf_conntrack_tuple *dst_tuple, const struct sk_buff *skb) @@ -459,6 +461,7 @@ struct nf_ct_hook { bool (*get_tuple_skb)(struct nf_conntrack_tuple *, const struct sk_buff *); void (*attach)(struct sk_buff *nskb, const struct sk_buff *skb); + void (*set_closing)(struct nf_conntrack *nfct); }; extern const struct nf_ct_hook __rcu *nf_ct_hook; diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index d5602b15c714..ff7ad331fb82 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -319,12 +319,16 @@ struct nf_bridge_info { * and read by ovs to recirc_id. */ struct tc_skb_ext { - __u32 chain; + union { + u64 act_miss_cookie; + __u32 chain; + }; __u16 mru; __u16 zone; u8 post_ct:1; u8 post_ct_snat:1; u8 post_ct_dnat:1; + u8 act_miss:1; /* Set if act_miss_cookie is used */ }; #endif diff --git a/include/linux/spi/at86rf230.h b/include/linux/spi/at86rf230.h deleted file mode 100644 index d278576ab692..000000000000 --- a/include/linux/spi/at86rf230.h +++ /dev/null @@ -1,20 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * AT86RF230/RF231 driver - * - * Copyright (C) 2009-2012 Siemens AG - * - * Written by: - * Dmitry Eremin-Solenikov <dmitry.baryshkov@siemens.com> - */ -#ifndef AT86RF230_H -#define AT86RF230_H - -struct at86rf230_platform_data { - int rstn; - int slp_tr; - int dig2; - u8 xtal_trim; -}; - -#endif diff --git a/include/linux/spi/cc2520.h b/include/linux/spi/cc2520.h deleted file mode 100644 index 449bacf10700..000000000000 --- a/include/linux/spi/cc2520.h +++ /dev/null @@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* Header file for cc2520 radio driver - * - * Copyright (C) 2014 Varka Bhadram <varkab@cdac.in> - * Md.Jamal Mohiuddin <mjmohiuddin@cdac.in> - * P Sowjanya <sowjanyap@cdac.in> - */ - -#ifndef __CC2520_H -#define __CC2520_H - -struct cc2520_platform_data { - int fifo; - int fifop; - int cca; - int sfd; - int reset; - int vreg; -}; - -#endif diff --git a/include/net/act_api.h b/include/net/act_api.h index 2a6f443f0ef6..4ae0580b63ca 100644 --- a/include/net/act_api.h +++ b/include/net/act_api.h @@ -39,7 +39,7 @@ struct tc_action { struct gnet_stats_basic_sync __percpu *cpu_bstats; struct gnet_stats_basic_sync __percpu *cpu_bstats_hw; struct gnet_stats_queue __percpu *cpu_qstats; - struct tc_cookie __rcu *act_cookie; + struct tc_cookie __rcu *user_cookie; struct tcf_chain __rcu *goto_chain; u32 tcfa_flags; u8 hw_stats; diff --git a/include/net/cfg802154.h b/include/net/cfg802154.h index d09c393d229f..0c2778a836db 100644 --- a/include/net/cfg802154.h +++ b/include/net/cfg802154.h @@ -18,6 +18,8 @@ struct wpan_phy; struct wpan_phy_cca; +struct cfg802154_scan_request; +struct cfg802154_beacon_request; #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL struct ieee802154_llsec_device_key; @@ -67,6 +69,14 @@ struct cfg802154_ops { struct wpan_dev *wpan_dev, bool mode); int (*set_ackreq_default)(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, bool ackreq); + int (*trigger_scan)(struct wpan_phy *wpan_phy, + struct cfg802154_scan_request *request); + int (*abort_scan)(struct wpan_phy *wpan_phy, + struct wpan_dev *wpan_dev); + int (*send_beacons)(struct wpan_phy *wpan_phy, + struct cfg802154_beacon_request *request); + int (*stop_beacons)(struct wpan_phy *wpan_phy, + struct wpan_dev *wpan_dev); #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL void (*get_llsec_table)(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, @@ -241,6 +251,17 @@ static inline void wpan_phy_net_set(struct wpan_phy *wpan_phy, struct net *net) write_pnet(&wpan_phy->_net, net); } +static inline bool ieee802154_chan_is_valid(struct wpan_phy *phy, + u8 page, u8 channel) +{ + if (page > IEEE802154_MAX_PAGE || + channel > IEEE802154_MAX_CHANNEL || + !(phy->supported.channels[page] & BIT(channel))) + return false; + + return true; +} + /** * struct ieee802154_addr - IEEE802.15.4 device address * @mode: Address mode from frame header. Can be one of: @@ -278,6 +299,60 @@ struct ieee802154_coord_desc { bool gts_permit; }; +/** + * struct cfg802154_scan_request - Scan request + * + * @type: type of scan to be performed + * @page: page on which to perform the scan + * @channels: channels in te %page to be scanned + * @duration: time spent on each channel, calculated with: + * aBaseSuperframeDuration * (2 ^ duration + 1) + * @wpan_dev: the wpan device on which to perform the scan + * @wpan_phy: the wpan phy on which to perform the scan + */ +struct cfg802154_scan_request { + enum nl802154_scan_types type; + u8 page; + u32 channels; + u8 duration; + struct wpan_dev *wpan_dev; + struct wpan_phy *wpan_phy; +}; + +/** + * struct cfg802154_beacon_request - Beacon request descriptor + * + * @interval: interval n between sendings, in multiple order of the super frame + * duration: aBaseSuperframeDuration * (2^n) unless the interval + * order is greater or equal to 15, in this case beacons won't be + * passively sent out at a fixed rate but instead inform the device + * that it should answer beacon requests as part of active scan + * procedures + * @wpan_dev: the concerned wpan device + * @wpan_phy: the wpan phy this was for + */ +struct cfg802154_beacon_request { + u8 interval; + struct wpan_dev *wpan_dev; + struct wpan_phy *wpan_phy; +}; + +/** + * struct cfg802154_mac_pkt - MAC packet descriptor (beacon/command) + * @node: MAC packets to process list member + * @skb: the received sk_buff + * @sdata: the interface on which @skb was received + * @page: page configuration when @skb was received + * @channel: channel configuration when @skb was received + */ +struct cfg802154_mac_pkt { + struct list_head node; + struct sk_buff *skb; + struct ieee802154_sub_if_data *sdata; + u8 page; + u8 channel; +}; + struct ieee802154_llsec_key_id { u8 mode; u8 id; @@ -447,6 +522,7 @@ static inline const char *wpan_phy_name(struct wpan_phy *phy) return dev_name(&phy->dev); } -void ieee802154_configure_durations(struct wpan_phy *phy); +void ieee802154_configure_durations(struct wpan_phy *phy, + unsigned int page, unsigned int channel); #endif /* __NET_CFG802154_H */ diff --git a/include/net/dropreason.h b/include/net/dropreason.h index ef3f65d135d3..c0a3ea806cd5 100644 --- a/include/net/dropreason.h +++ b/include/net/dropreason.h @@ -76,6 +76,8 @@ FN(IPV6_NDISC_FRAG) \ FN(IPV6_NDISC_HOP_LIMIT) \ FN(IPV6_NDISC_BAD_CODE) \ + FN(IPV6_NDISC_BAD_OPTIONS) \ + FN(IPV6_NDISC_NS_OTHERHOST) \ FNe(MAX) /** @@ -330,6 +332,12 @@ enum skb_drop_reason { SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT, /** @SKB_DROP_REASON_IPV6_NDISC_BAD_CODE: invalid NDISC icmp6 code. */ SKB_DROP_REASON_IPV6_NDISC_BAD_CODE, + /** @SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS: invalid NDISC options. */ + SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS, + /** @SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST: NEIGHBOUR SOLICITATION + * for another host. + */ + SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST, /** * @SKB_DROP_REASON_MAX: the maximum of drop reason, which shouldn't be * used as a real 'reason' diff --git a/include/net/flow_offload.h b/include/net/flow_offload.h index 8c05455b1e34..118082eae48c 100644 --- a/include/net/flow_offload.h +++ b/include/net/flow_offload.h @@ -228,7 +228,8 @@ void flow_action_cookie_destroy(struct flow_action_cookie *cookie); struct flow_action_entry { enum flow_action_id id; u32 hw_index; - unsigned long act_cookie; + unsigned long cookie; + u64 miss_cookie; enum flow_action_hw_stats hw_stats; action_destr destructor; void *destructor_priv; @@ -321,7 +322,7 @@ struct flow_action_entry { u16 sid; } pppoe; }; - struct flow_action_cookie *cookie; /* user defined action cookie */ + struct flow_action_cookie *user_cookie; /* user defined action cookie */ }; struct flow_action { diff --git a/include/net/ieee802154_netdev.h b/include/net/ieee802154_netdev.h index 4c33a20ea57f..da8a3e648c7a 100644 --- a/include/net/ieee802154_netdev.h +++ b/include/net/ieee802154_netdev.h @@ -38,6 +38,42 @@ #include <net/cfg802154.h> +struct ieee802154_beacon_hdr { +#if defined(__LITTLE_ENDIAN_BITFIELD) + u16 beacon_order:4, + superframe_order:4, + final_cap_slot:4, + battery_life_ext:1, + reserved0:1, + pan_coordinator:1, + assoc_permit:1; + u8 gts_count:3, + gts_reserved:4, + gts_permit:1; + u8 pend_short_addr_count:3, + reserved1:1, + pend_ext_addr_count:3, + reserved2:1; +#elif defined(__BIG_ENDIAN_BITFIELD) + u16 assoc_permit:1, + pan_coordinator:1, + reserved0:1, + battery_life_ext:1, + final_cap_slot:4, + superframe_order:4, + beacon_order:4; + u8 gts_permit:1, + gts_reserved:4, + gts_count:3; + u8 reserved2:1, + pend_ext_addr_count:3, + reserved1:1, + pend_short_addr_count:3; +#else +#error "Please fix <asm/byteorder.h>" +#endif +} __packed; + struct ieee802154_sechdr { #if defined(__LITTLE_ENDIAN_BITFIELD) u8 level:3, @@ -93,6 +129,13 @@ enum ieee802154_frame_version { IEEE802154_MULTIPURPOSE_STD = IEEE802154_2003_STD, }; +enum ieee802154_addressing_mode { + IEEE802154_NO_ADDRESSING, + IEEE802154_RESERVED, + IEEE802154_SHORT_ADDRESSING, + IEEE802154_EXTENDED_ADDRESSING, +}; + struct ieee802154_hdr { struct ieee802154_hdr_fc fc; u8 seq; @@ -101,6 +144,11 @@ struct ieee802154_hdr { struct ieee802154_sechdr sec; }; +struct ieee802154_beacon_frame { + struct ieee802154_hdr mhr; + struct ieee802154_beacon_hdr mac_pl; +}; + /* pushes hdr onto the skb. fields of hdr->fc that can be calculated from * the contents of hdr will be, and the actual value of those bits in * hdr->fc will be ignored. this includes the INTRA_PAN bit and the frame @@ -126,6 +174,10 @@ int ieee802154_hdr_peek_addrs(const struct sk_buff *skb, */ int ieee802154_hdr_peek(const struct sk_buff *skb, struct ieee802154_hdr *hdr); +/* pushes a beacon frame into an skb */ +int ieee802154_beacon_push(struct sk_buff *skb, + struct ieee802154_beacon_frame *beacon); + int ieee802154_max_payload(const struct ieee802154_hdr *hdr); static inline int diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h index c6c61100d244..6d71a5ff52df 100644 --- a/include/net/ip_vs.h +++ b/include/net/ip_vs.h @@ -461,6 +461,7 @@ void ip_vs_stats_free(struct ip_vs_stats *stats); /* Multiple chains processed in same tick */ struct ip_vs_est_tick_data { + struct rcu_head rcu_head; struct hlist_head chains[IPVS_EST_TICK_CHAINS]; DECLARE_BITMAP(present, IPVS_EST_TICK_CHAINS); DECLARE_BITMAP(full, IPVS_EST_TICK_CHAINS); diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h index 7bbab8f2b73d..a72028dbef0c 100644 --- a/include/net/netfilter/nf_conntrack.h +++ b/include/net/netfilter/nf_conntrack.h @@ -126,6 +126,12 @@ struct nf_conn { }; static inline struct nf_conn * +nf_ct_to_nf_conn(const struct nf_conntrack *nfct) +{ + return container_of(nfct, struct nf_conn, ct_general); +} + +static inline struct nf_conn * nf_ct_tuplehash_to_ctrack(const struct nf_conntrack_tuple_hash *hash) { return container_of(hash, struct nf_conn, @@ -175,6 +181,8 @@ nf_ct_get(const struct sk_buff *skb, enum ip_conntrack_info *ctinfo) void nf_ct_destroy(struct nf_conntrack *nfct); +void nf_conntrack_tcp_set_closing(struct nf_conn *ct); + /* decrement reference count on a conntrack */ static inline void nf_ct_put(struct nf_conn *ct) { diff --git a/include/net/netns/core.h b/include/net/netns/core.h index 8249060cf5d0..a91ef9f8de60 100644 --- a/include/net/netns/core.h +++ b/include/net/netns/core.h @@ -6,6 +6,7 @@ struct ctl_table_header; struct prot_inuse; +struct cpumask; struct netns_core { /* core sysctls */ @@ -17,6 +18,10 @@ struct netns_core { #ifdef CONFIG_PROC_FS struct prot_inuse __percpu *prot_inuse; #endif + +#if IS_ENABLED(CONFIG_RPS) && IS_ENABLED(CONFIG_SYSCTL) + struct cpumask *rps_default_mask; +#endif }; #endif diff --git a/include/net/nl802154.h b/include/net/nl802154.h index b79a89d5207c..8cd9d141f5af 100644 --- a/include/net/nl802154.h +++ b/include/net/nl802154.h @@ -73,6 +73,11 @@ enum nl802154_commands { NL802154_CMD_DEL_SEC_LEVEL, NL802154_CMD_SCAN_EVENT, + NL802154_CMD_TRIGGER_SCAN, + NL802154_CMD_ABORT_SCAN, + NL802154_CMD_SCAN_DONE, + NL802154_CMD_SEND_BEACONS, + NL802154_CMD_STOP_BEACONS, /* add new commands above here */ @@ -134,6 +139,14 @@ enum nl802154_attrs { NL802154_ATTR_NETNS_FD, NL802154_ATTR_COORDINATOR, + NL802154_ATTR_SCAN_TYPE, + NL802154_ATTR_SCAN_FLAGS, + NL802154_ATTR_SCAN_CHANNELS, + NL802154_ATTR_SCAN_PREAMBLE_CODES, + NL802154_ATTR_SCAN_MEAN_PRF, + NL802154_ATTR_SCAN_DURATION, + NL802154_ATTR_SCAN_DONE_REASON, + NL802154_ATTR_BEACON_INTERVAL, /* add attributes here, update the policy in nl802154.c */ @@ -260,6 +273,54 @@ enum nl802154_coord { }; /** + * enum nl802154_scan_types - Scan types + * + * @__NL802154_SCAN_INVALID: scan type number 0 is reserved + * @NL802154_SCAN_ED: An ED scan allows a device to obtain a measure of the peak + * energy in each requested channel + * @NL802154_SCAN_ACTIVE: Locate any coordinator transmitting Beacon frames using + * a Beacon Request command + * @NL802154_SCAN_PASSIVE: Locate any coordinator transmitting Beacon frames + * @NL802154_SCAN_ORPHAN: Relocate coordinator following a loss of synchronisation + * @NL802154_SCAN_ENHANCED_ACTIVE: Same as Active using Enhanced Beacon Request + * command instead of Beacon Request command + * @NL802154_SCAN_RIT_PASSIVE: Passive scan for RIT Data Request command frames + * instead of Beacon frames + * @NL802154_SCAN_ATTR_MAX: Maximum SCAN attribute number + */ +enum nl802154_scan_types { + __NL802154_SCAN_INVALID, + NL802154_SCAN_ED, + NL802154_SCAN_ACTIVE, + NL802154_SCAN_PASSIVE, + NL802154_SCAN_ORPHAN, + NL802154_SCAN_ENHANCED_ACTIVE, + NL802154_SCAN_RIT_PASSIVE, + + /* keep last */ + NL802154_SCAN_ATTR_MAX, +}; + +/** + * enum nl802154_scan_done_reasons - End of scan reasons + * + * @__NL802154_SCAN_DONE_REASON_INVALID: scan done reason number 0 is reserved. + * @NL802154_SCAN_DONE_REASON_FINISHED: The scan just finished naturally after + * going through all the requested and possible (complex) channels. + * @NL802154_SCAN_DONE_REASON_ABORTED: The scan was aborted upon user request. + * a Beacon Request command + * @NL802154_SCAN_DONE_REASON_MAX: Maximum scan done reason attribute number. + */ +enum nl802154_scan_done_reasons { + __NL802154_SCAN_DONE_REASON_INVALID, + NL802154_SCAN_DONE_REASON_FINISHED, + NL802154_SCAN_DONE_REASON_ABORTED, + + /* keep last */ + NL802154_SCAN_DONE_REASON_MAX, +}; + +/** * enum nl802154_cca_modes - cca modes * * @__NL802154_CCA_INVALID: cca mode number 0 is reserved diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h index ace437c6754b..b3b5b0b62f16 100644 --- a/include/net/pkt_cls.h +++ b/include/net/pkt_cls.h @@ -59,6 +59,8 @@ int tcf_block_get_ext(struct tcf_block **p_block, struct Qdisc *q, void tcf_block_put(struct tcf_block *block); void tcf_block_put_ext(struct tcf_block *block, struct Qdisc *q, struct tcf_block_ext_info *ei); +int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action, + int police, struct tcf_proto *tp, u32 handle, bool used_action_miss); static inline bool tcf_block_shared(struct tcf_block *block) { @@ -229,6 +231,7 @@ struct tcf_exts { struct tc_action **actions; struct net *net; netns_tracker ns_tracker; + struct tcf_exts_miss_cookie_node *miss_cookie_node; #endif /* Map to export classifier specific extension TLV types to the * generic extensions API. Unsupported extensions must be set to 0. @@ -240,21 +243,11 @@ struct tcf_exts { static inline int tcf_exts_init(struct tcf_exts *exts, struct net *net, int action, int police) { -#ifdef CONFIG_NET_CLS_ACT - exts->type = 0; - exts->nr_actions = 0; - /* Note: we do not own yet a reference on net. - * This reference might be taken later from tcf_exts_get_net(). - */ - exts->net = net; - exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *), - GFP_KERNEL); - if (!exts->actions) - return -ENOMEM; +#ifdef CONFIG_NET_CLS + return tcf_exts_init_ex(exts, net, action, police, NULL, 0, false); +#else + return -EOPNOTSUPP; #endif - exts->action = action; - exts->police = police; - return 0; } /* Return false if the netns is being destroyed in cleanup_net(). Callers @@ -360,6 +353,18 @@ tcf_exts_exec(struct sk_buff *skb, struct tcf_exts *exts, return TC_ACT_OK; } +static inline int +tcf_exts_exec_ex(struct sk_buff *skb, struct tcf_exts *exts, int act_index, + struct tcf_result *res) +{ +#ifdef CONFIG_NET_CLS_ACT + return tcf_action_exec(skb, exts->actions + act_index, + exts->nr_actions - act_index, res); +#else + return TC_ACT_OK; +#endif +} + int tcf_exts_validate(struct net *net, struct tcf_proto *tp, struct nlattr **tb, struct nlattr *rate_tlv, struct tcf_exts *exts, u32 flags, @@ -584,6 +589,7 @@ int tc_setup_offload_action(struct flow_action *flow_action, void tc_cleanup_offload_action(struct flow_action *flow_action); int tc_setup_action(struct flow_action *flow_action, struct tc_action *actions[], + u32 miss_cookie_base, struct netlink_ext_ack *extack); int tc_setup_cb_call(struct tcf_block *block, enum tc_setup_type type, diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index af4aa66aaa4e..fab5ba3e61b7 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -369,6 +369,8 @@ struct tcf_proto_ops { struct nlattr **tca, struct netlink_ext_ack *extack); void (*tmplt_destroy)(void *tmplt_priv); + struct tcf_exts * (*get_exts)(const struct tcf_proto *tp, + u32 handle); /* rtnetlink specific */ int (*dump)(struct net*, struct tcf_proto*, void *, diff --git a/include/trace/events/rxrpc.h b/include/trace/events/rxrpc.h index c3c0b0aa8381..4c53a5ef6257 100644 --- a/include/trace/events/rxrpc.h +++ b/include/trace/events/rxrpc.h @@ -318,6 +318,7 @@ EM(rxrpc_recvmsg_return, "RETN") \ EM(rxrpc_recvmsg_terminal, "TERM") \ EM(rxrpc_recvmsg_to_be_accepted, "TBAC") \ + EM(rxrpc_recvmsg_unqueue, "UNQU") \ E_(rxrpc_recvmsg_wait, "WAIT") #define rxrpc_rtt_tx_traces \ diff --git a/include/trace/events/skb.h b/include/trace/events/skb.h index 25ab1ff9423d..07e0715628ec 100644 --- a/include/trace/events/skb.h +++ b/include/trace/events/skb.h @@ -53,19 +53,21 @@ TRACE_EVENT(kfree_skb, TRACE_EVENT(consume_skb, - TP_PROTO(struct sk_buff *skb), + TP_PROTO(struct sk_buff *skb, void *location), - TP_ARGS(skb), + TP_ARGS(skb, location), TP_STRUCT__entry( - __field( void *, skbaddr ) + __field( void *, skbaddr) + __field( void *, location) ), TP_fast_assign( __entry->skbaddr = skb; + __entry->location = location; ), - TP_printk("skbaddr=%p", __entry->skbaddr) + TP_printk("skbaddr=%p location=%pS", __entry->skbaddr, __entry->location) ); TRACE_EVENT(skb_copy_datagram_iovec, diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 17afd2b35ee5..62ce1f5d1b1d 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3134,6 +3134,11 @@ union bpf_attr { * **BPF_FIB_LOOKUP_OUTPUT** * Perform lookup from an egress perspective (default is * ingress). + * **BPF_FIB_LOOKUP_SKIP_NEIGH** + * Skip the neighbour table lookup. *params*->dmac + * and *params*->smac will not be set as output. A common + * use case is to call **bpf_redirect_neigh**\ () after + * doing **bpf_fib_lookup**\ (). * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -6750,6 +6755,7 @@ struct bpf_raw_tracepoint_args { enum { BPF_FIB_LOOKUP_DIRECT = (1U << 0), BPF_FIB_LOOKUP_OUTPUT = (1U << 1), + BPF_FIB_LOOKUP_SKIP_NEIGH = (1U << 2), }; enum { @@ -6917,6 +6923,17 @@ struct bpf_list_node { __u64 :64; } __attribute__((aligned(8))); +struct bpf_rb_root { + __u64 :64; + __u64 :64; +} __attribute__((aligned(8))); + +struct bpf_rb_node { + __u64 :64; + __u64 :64; + __u64 :64; +} __attribute__((aligned(8))); + struct bpf_sysctl { __u32 write; /* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 373c3c2c75bc..35f4138a54dc 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -568,8 +568,8 @@ static struct bpf_local_storage_map *__bpf_local_storage_map_alloc(union bpf_att nbuckets = max_t(u32, 2, nbuckets); smap->bucket_log = ilog2(nbuckets); - smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets), + nbuckets, GFP_USER | __GFP_NOWARN); if (!smap->buckets) { bpf_map_area_free(smap); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 740bdb045b14..fa22ec79ac0e 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3324,12 +3324,14 @@ static const char *btf_find_decl_tag_value(const struct btf *btf, return NULL; } -static int btf_find_list_head(const struct btf *btf, const struct btf_type *pt, - const struct btf_type *t, int comp_idx, - u32 off, int sz, struct btf_field_info *info) +static int +btf_find_graph_root(const struct btf *btf, const struct btf_type *pt, + const struct btf_type *t, int comp_idx, u32 off, + int sz, struct btf_field_info *info, + enum btf_field_type head_type) { + const char *node_field_name; const char *value_type; - const char *list_node; s32 id; if (!__btf_type_is_struct(t)) @@ -3339,26 +3341,32 @@ static int btf_find_list_head(const struct btf *btf, const struct btf_type *pt, value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:"); if (!value_type) return -EINVAL; - list_node = strstr(value_type, ":"); - if (!list_node) + node_field_name = strstr(value_type, ":"); + if (!node_field_name) return -EINVAL; - value_type = kstrndup(value_type, list_node - value_type, GFP_KERNEL | __GFP_NOWARN); + value_type = kstrndup(value_type, node_field_name - value_type, GFP_KERNEL | __GFP_NOWARN); if (!value_type) return -ENOMEM; id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT); kfree(value_type); if (id < 0) return id; - list_node++; - if (str_is_empty(list_node)) + node_field_name++; + if (str_is_empty(node_field_name)) return -EINVAL; - info->type = BPF_LIST_HEAD; + info->type = head_type; info->off = off; info->graph_root.value_btf_id = id; - info->graph_root.node_name = list_node; + info->graph_root.node_name = node_field_name; return BTF_FIELD_FOUND; } +#define field_mask_test_name(field_type, field_type_str) \ + if (field_mask & field_type && !strcmp(name, field_type_str)) { \ + type = field_type; \ + goto end; \ + } + static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, int *align, int *sz) { @@ -3382,18 +3390,11 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, goto end; } } - if (field_mask & BPF_LIST_HEAD) { - if (!strcmp(name, "bpf_list_head")) { - type = BPF_LIST_HEAD; - goto end; - } - } - if (field_mask & BPF_LIST_NODE) { - if (!strcmp(name, "bpf_list_node")) { - type = BPF_LIST_NODE; - goto end; - } - } + field_mask_test_name(BPF_LIST_HEAD, "bpf_list_head"); + field_mask_test_name(BPF_LIST_NODE, "bpf_list_node"); + field_mask_test_name(BPF_RB_ROOT, "bpf_rb_root"); + field_mask_test_name(BPF_RB_NODE, "bpf_rb_node"); + /* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask & BPF_KPTR) { type = BPF_KPTR_REF; @@ -3406,6 +3407,8 @@ end: return type; } +#undef field_mask_test_name + static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt) @@ -3438,6 +3441,7 @@ static int btf_find_struct_field(const struct btf *btf, case BPF_SPIN_LOCK: case BPF_TIMER: case BPF_LIST_NODE: + case BPF_RB_NODE: ret = btf_find_struct(btf, member_type, off, sz, field_type, idx < info_cnt ? &info[idx] : &tmp); if (ret < 0) @@ -3451,8 +3455,11 @@ static int btf_find_struct_field(const struct btf *btf, return ret; break; case BPF_LIST_HEAD: - ret = btf_find_list_head(btf, t, member_type, i, off, sz, - idx < info_cnt ? &info[idx] : &tmp); + case BPF_RB_ROOT: + ret = btf_find_graph_root(btf, t, member_type, + i, off, sz, + idx < info_cnt ? &info[idx] : &tmp, + field_type); if (ret < 0) return ret; break; @@ -3499,6 +3506,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, case BPF_SPIN_LOCK: case BPF_TIMER: case BPF_LIST_NODE: + case BPF_RB_NODE: ret = btf_find_struct(btf, var_type, off, sz, field_type, idx < info_cnt ? &info[idx] : &tmp); if (ret < 0) @@ -3512,8 +3520,11 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, return ret; break; case BPF_LIST_HEAD: - ret = btf_find_list_head(btf, var, var_type, -1, off, sz, - idx < info_cnt ? &info[idx] : &tmp); + case BPF_RB_ROOT: + ret = btf_find_graph_root(btf, var, var_type, + -1, off, sz, + idx < info_cnt ? &info[idx] : &tmp, + field_type); if (ret < 0) return ret; break; @@ -3615,8 +3626,11 @@ end_btf: return ret; } -static int btf_parse_list_head(const struct btf *btf, struct btf_field *field, - struct btf_field_info *info) +static int btf_parse_graph_root(const struct btf *btf, + struct btf_field *field, + struct btf_field_info *info, + const char *node_type_name, + size_t node_type_align) { const struct btf_type *t, *n = NULL; const struct btf_member *member; @@ -3638,13 +3652,13 @@ static int btf_parse_list_head(const struct btf *btf, struct btf_field *field, n = btf_type_by_id(btf, member->type); if (!__btf_type_is_struct(n)) return -EINVAL; - if (strcmp("bpf_list_node", __btf_name_by_offset(btf, n->name_off))) + if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off))) return -EINVAL; offset = __btf_member_bit_offset(n, member); if (offset % 8) return -EINVAL; offset /= 8; - if (offset % __alignof__(struct bpf_list_node)) + if (offset % node_type_align) return -EINVAL; field->graph_root.btf = (struct btf *)btf; @@ -3656,6 +3670,20 @@ static int btf_parse_list_head(const struct btf *btf, struct btf_field *field, return 0; } +static int btf_parse_list_head(const struct btf *btf, struct btf_field *field, + struct btf_field_info *info) +{ + return btf_parse_graph_root(btf, field, info, "bpf_list_node", + __alignof__(struct bpf_list_node)); +} + +static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field, + struct btf_field_info *info) +{ + return btf_parse_graph_root(btf, field, info, "bpf_rb_node", + __alignof__(struct bpf_rb_node)); +} + struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t, u32 field_mask, u32 value_size) { @@ -3718,7 +3746,13 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type if (ret < 0) goto end; break; + case BPF_RB_ROOT: + ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]); + if (ret < 0) + goto end; + break; case BPF_LIST_NODE: + case BPF_RB_NODE: break; default: ret = -EFAULT; @@ -3727,8 +3761,33 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type rec->cnt++; } - /* bpf_list_head requires bpf_spin_lock */ - if (btf_record_has_field(rec, BPF_LIST_HEAD) && rec->spin_lock_off < 0) { + /* bpf_{list_head, rb_node} require bpf_spin_lock */ + if ((btf_record_has_field(rec, BPF_LIST_HEAD) || + btf_record_has_field(rec, BPF_RB_ROOT)) && rec->spin_lock_off < 0) { + ret = -EINVAL; + goto end; + } + + /* need collection identity for non-owning refs before allowing this + * + * Consider a node type w/ both list and rb_node fields: + * struct node { + * struct bpf_list_node l; + * struct bpf_rb_node r; + * } + * + * Used like so: + * struct node *n = bpf_obj_new(....); + * bpf_list_push_front(&list_head, &n->l); + * bpf_rbtree_remove(&rb_root, &n->r); + * + * It should not be possible to rbtree_remove the node since it hasn't + * been added to a tree. But push_front converts n to a non-owning + * reference, and rbtree_remove accepts the non-owning reference to + * a type w/ bpf_rb_node field. + */ + if (btf_record_has_field(rec, BPF_LIST_NODE) && + btf_record_has_field(rec, BPF_RB_NODE)) { ret = -EINVAL; goto end; } @@ -3739,22 +3798,28 @@ end: return ERR_PTR(ret); } +#define GRAPH_ROOT_MASK (BPF_LIST_HEAD | BPF_RB_ROOT) +#define GRAPH_NODE_MASK (BPF_LIST_NODE | BPF_RB_NODE) + int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec) { int i; - /* There are two owning types, kptr_ref and bpf_list_head. The former - * only supports storing kernel types, which can never store references - * to program allocated local types, atleast not yet. Hence we only need - * to ensure that bpf_list_head ownership does not form cycles. + /* There are three types that signify ownership of some other type: + * kptr_ref, bpf_list_head, bpf_rb_root. + * kptr_ref only supports storing kernel types, which can't store + * references to program allocated local types. + * + * Hence we only need to ensure that bpf_{list_head,rb_root} ownership + * does not form cycles. */ - if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & BPF_LIST_HEAD)) + if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & GRAPH_ROOT_MASK)) return 0; for (i = 0; i < rec->cnt; i++) { struct btf_struct_meta *meta; u32 btf_id; - if (!(rec->fields[i].type & BPF_LIST_HEAD)) + if (!(rec->fields[i].type & GRAPH_ROOT_MASK)) continue; btf_id = rec->fields[i].graph_root.value_btf_id; meta = btf_find_struct_meta(btf, btf_id); @@ -3762,39 +3827,47 @@ int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec) return -EFAULT; rec->fields[i].graph_root.value_rec = meta->record; - if (!(rec->field_mask & BPF_LIST_NODE)) + /* We need to set value_rec for all root types, but no need + * to check ownership cycle for a type unless it's also a + * node type. + */ + if (!(rec->field_mask & GRAPH_NODE_MASK)) continue; /* We need to ensure ownership acyclicity among all types. The * proper way to do it would be to topologically sort all BTF * IDs based on the ownership edges, since there can be multiple - * bpf_list_head in a type. Instead, we use the following - * reasoning: + * bpf_{list_head,rb_node} in a type. Instead, we use the + * following resaoning: * * - A type can only be owned by another type in user BTF if it - * has a bpf_list_node. + * has a bpf_{list,rb}_node. Let's call these node types. * - A type can only _own_ another type in user BTF if it has a - * bpf_list_head. + * bpf_{list_head,rb_root}. Let's call these root types. * - * We ensure that if a type has both bpf_list_head and - * bpf_list_node, its element types cannot be owning types. + * We ensure that if a type is both a root and node, its + * element types cannot be root types. * * To ensure acyclicity: * - * When A only has bpf_list_head, ownership chain can be: + * When A is an root type but not a node, its ownership + * chain can be: * A -> B -> C * Where: - * - B has both bpf_list_head and bpf_list_node. - * - C only has bpf_list_node. + * - A is an root, e.g. has bpf_rb_root. + * - B is both a root and node, e.g. has bpf_rb_node and + * bpf_list_head. + * - C is only an root, e.g. has bpf_list_node * - * When A has both bpf_list_head and bpf_list_node, some other - * type already owns it in the BTF domain, hence it can not own - * another owning type through any of the bpf_list_head edges. + * When A is both a root and node, some other type already + * owns it in the BTF domain, hence it can not own + * another root type through any of the ownership edges. * A -> B * Where: - * - B only has bpf_list_node. + * - A is both an root and node. + * - B is only an node. */ - if (meta->record->field_mask & BPF_LIST_HEAD) + if (meta->record->field_mask & GRAPH_ROOT_MASK) return -ELOOP; } return 0; @@ -5256,6 +5329,8 @@ static const char *alloc_obj_fields[] = { "bpf_spin_lock", "bpf_list_head", "bpf_list_node", + "bpf_rb_root", + "bpf_rb_node", }; static struct btf_struct_metas * @@ -5329,7 +5404,8 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf) type = &tab->types[tab->cnt]; type->btf_id = i; - record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE, t->size); + record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE | + BPF_RB_ROOT | BPF_RB_NODE, t->size); /* The record cannot be unset, treat it as an error if so */ if (IS_ERR_OR_NULL(record)) { ret = PTR_ERR_OR_ZERO(record) ?: -EFAULT; @@ -5593,6 +5669,7 @@ btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf, if (!ctx_struct) /* should not happen */ return NULL; +again: ctx_tname = btf_name_by_offset(btf_vmlinux, ctx_struct->name_off); if (!ctx_tname) { /* should not happen */ @@ -5606,8 +5683,16 @@ btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf, * int socket_filter_bpf_prog(struct __sk_buff *skb) * { // no fields of skb are ever used } */ - if (strcmp(ctx_tname, tname)) - return NULL; + if (strcmp(ctx_tname, tname)) { + /* bpf_user_pt_regs_t is a typedef, so resolve it to + * underlying struct and check name again + */ + if (!btf_type_is_modifier(ctx_struct)) + return NULL; + while (btf_type_is_modifier(ctx_struct)) + ctx_struct = btf_type_by_id(btf_vmlinux, ctx_struct->type); + goto again; + } return ctx_type; } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 16da51093aff..3390961c4e10 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -35,6 +35,7 @@ #include <linux/bpf_verifier.h> #include <linux/nodemask.h> #include <linux/bpf_mem_alloc.h> +#include <linux/memcontrol.h> #include <asm/barrier.h> #include <asm/unaligned.h> @@ -87,7 +88,7 @@ void *bpf_internal_load_pointer_neg_helper(const struct sk_buff *skb, int k, uns struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags) { - gfp_t gfp_flags = GFP_KERNEL_ACCOUNT | __GFP_ZERO | gfp_extra_flags; + gfp_t gfp_flags = bpf_memcg_flags(GFP_KERNEL | __GFP_ZERO | gfp_extra_flags); struct bpf_prog_aux *aux; struct bpf_prog *fp; @@ -96,12 +97,12 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag if (fp == NULL) return NULL; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT | gfp_extra_flags); + aux = kzalloc(sizeof(*aux), bpf_memcg_flags(GFP_KERNEL | gfp_extra_flags)); if (aux == NULL) { vfree(fp); return NULL; } - fp->active = alloc_percpu_gfp(int, GFP_KERNEL_ACCOUNT | gfp_extra_flags); + fp->active = alloc_percpu_gfp(int, bpf_memcg_flags(GFP_KERNEL | gfp_extra_flags)); if (!fp->active) { vfree(fp); kfree(aux); @@ -126,7 +127,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag struct bpf_prog *bpf_prog_alloc(unsigned int size, gfp_t gfp_extra_flags) { - gfp_t gfp_flags = GFP_KERNEL_ACCOUNT | __GFP_ZERO | gfp_extra_flags; + gfp_t gfp_flags = bpf_memcg_flags(GFP_KERNEL | __GFP_ZERO | gfp_extra_flags); struct bpf_prog *prog; int cpu; @@ -159,7 +160,7 @@ int bpf_prog_alloc_jited_linfo(struct bpf_prog *prog) prog->aux->jited_linfo = kvcalloc(prog->aux->nr_linfo, sizeof(*prog->aux->jited_linfo), - GFP_KERNEL_ACCOUNT | __GFP_NOWARN); + bpf_memcg_flags(GFP_KERNEL | __GFP_NOWARN)); if (!prog->aux->jited_linfo) return -ENOMEM; @@ -234,7 +235,7 @@ void bpf_prog_fill_jited_linfo(struct bpf_prog *prog, struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, gfp_t gfp_extra_flags) { - gfp_t gfp_flags = GFP_KERNEL_ACCOUNT | __GFP_ZERO | gfp_extra_flags; + gfp_t gfp_flags = bpf_memcg_flags(GFP_KERNEL | __GFP_ZERO | gfp_extra_flags); struct bpf_prog *fp; u32 pages; diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 66bded144377..5dfcb5ad0d06 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -1004,8 +1004,6 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, l_new = ERR_PTR(-ENOMEM); goto dec_count; } - check_and_init_map_value(&htab->map, - l_new->key + round_up(key_size, 8)); } memcpy(l_new->key, key, key_size); @@ -1592,6 +1590,7 @@ static int __htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, else copy_map_value(map, value, l->key + roundup_key_size); + /* Zeroing special fields in the temp buffer */ check_and_init_map_value(map, value); } @@ -1792,6 +1791,7 @@ again_nocopy: true); else copy_map_value(map, dst_val, value); + /* Zeroing special fields in the temp buffer */ check_and_init_map_value(map, dst_val); } if (do_delete) { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 2dae44581922..5b278a38ae58 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1772,6 +1772,46 @@ unlock: } } +/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are + * 'rb_node *', so field name of rb_node within containing struct is not + * needed. + * + * Since bpf_rb_tree's node type has a corresponding struct btf_field with + * graph_root.node_offset, it's not necessary to know field name + * or type of node struct + */ +#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \ + for (pos = rb_first_postorder(root); \ + pos && ({ n = rb_next_postorder(pos); 1; }); \ + pos = n) + +void bpf_rb_root_free(const struct btf_field *field, void *rb_root, + struct bpf_spin_lock *spin_lock) +{ + struct rb_root_cached orig_root, *root = rb_root; + struct rb_node *pos, *n; + void *obj; + + BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root)); + BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root)); + + __bpf_spin_lock_irqsave(spin_lock); + orig_root = *root; + *root = RB_ROOT_CACHED; + __bpf_spin_unlock_irqrestore(spin_lock); + + bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { + obj = pos; + obj -= field->graph_root.node_offset; + + bpf_obj_free_fields(field->graph_root.value_rec, obj); + + migrate_disable(); + bpf_mem_free(&bpf_global_ma, obj); + migrate_enable(); + } +} + __diag_push(); __diag_ignore_all("-Wmissing-prototypes", "Global functions as their definitions will be in vmlinux BTF"); @@ -1844,6 +1884,56 @@ __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) return __bpf_list_del(head, true); } +__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, + struct bpf_rb_node *node) +{ + struct rb_root_cached *r = (struct rb_root_cached *)root; + struct rb_node *n = (struct rb_node *)node; + + rb_erase_cached(n, r); + RB_CLEAR_NODE(n); + return (struct bpf_rb_node *)n; +} + +/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF + * program + */ +static void __bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + void *less) +{ + struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node; + bpf_callback_t cb = (bpf_callback_t)less; + struct rb_node *parent = NULL; + bool leftmost = true; + + while (*link) { + parent = *link; + if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) { + link = &parent->rb_left; + } else { + link = &parent->rb_right; + leftmost = false; + } + } + + rb_link_node((struct rb_node *)node, parent, link); + rb_insert_color_cached((struct rb_node *)node, + (struct rb_root_cached *)root, leftmost); +} + +__bpf_kfunc void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) +{ + __bpf_rbtree_add(root, node, (void *)less); +} + +__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) +{ + struct rb_root_cached *r = (struct rb_root_cached *)root; + + return (struct bpf_rb_node *)rb_first_cached(r); +} + /** * bpf_task_acquire - Acquire a reference to a task. A task acquired by this * kfunc which is not stored in a map as a kptr, must be released by calling @@ -2068,6 +2158,10 @@ BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_task_acquire_not_zero, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_kptr_get, KF_ACQUIRE | KF_KPTR_GET | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE) +BTF_ID_FLAGS(func, bpf_rbtree_add) +BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL) + #ifdef CONFIG_CGROUPS BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_cgroup_kptr_get, KF_ACQUIRE | KF_KPTR_GET | KF_RET_NULL) diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c index 1db156405b68..5fcdacbb8439 100644 --- a/kernel/bpf/memalloc.c +++ b/kernel/bpf/memalloc.c @@ -143,7 +143,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node) return obj; } - return kmalloc_node(c->unit_size, flags, node); + return kmalloc_node(c->unit_size, flags | __GFP_ZERO, node); } static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c) @@ -395,7 +395,8 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) unit_size = size; #ifdef CONFIG_MEMCG_KMEM - objcg = get_obj_cgroup_from_current(); + if (memcg_bpf_enabled()) + objcg = get_obj_cgroup_from_current(); #endif for_each_possible_cpu(cpu) { c = per_cpu_ptr(pc, cpu); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index bcc97613de76..e3fcdc9836a6 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -309,7 +309,7 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) * __GFP_RETRY_MAYFAIL to avoid such situations. */ - const gfp_t gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_ACCOUNT; + gfp_t gfp = bpf_memcg_flags(__GFP_NOWARN | __GFP_ZERO); unsigned int flags = 0; unsigned long align = 1; void *area; @@ -418,7 +418,8 @@ static void bpf_map_save_memcg(struct bpf_map *map) * So we have to check map->objcg for being NULL each time it's * being used. */ - map->objcg = get_obj_cgroup_from_current(); + if (memcg_bpf_enabled()) + map->objcg = get_obj_cgroup_from_current(); } static void bpf_map_release_memcg(struct bpf_map *map) @@ -464,6 +465,21 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return ptr; } +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags) +{ + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + set_active_memcg(old_memcg); + mem_cgroup_put(memcg); + + return ptr; +} + void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { @@ -521,9 +537,6 @@ void btf_record_free(struct btf_record *rec) return; for (i = 0; i < rec->cnt; i++) { switch (rec->fields[i].type) { - case BPF_SPIN_LOCK: - case BPF_TIMER: - break; case BPF_KPTR_UNREF: case BPF_KPTR_REF: if (rec->fields[i].kptr.module) @@ -532,7 +545,11 @@ void btf_record_free(struct btf_record *rec) break; case BPF_LIST_HEAD: case BPF_LIST_NODE: - /* Nothing to release for bpf_list_head */ + case BPF_RB_ROOT: + case BPF_RB_NODE: + case BPF_SPIN_LOCK: + case BPF_TIMER: + /* Nothing to release */ break; default: WARN_ON_ONCE(1); @@ -565,9 +582,6 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) new_rec->cnt = 0; for (i = 0; i < rec->cnt; i++) { switch (fields[i].type) { - case BPF_SPIN_LOCK: - case BPF_TIMER: - break; case BPF_KPTR_UNREF: case BPF_KPTR_REF: btf_get(fields[i].kptr.btf); @@ -578,7 +592,11 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) break; case BPF_LIST_HEAD: case BPF_LIST_NODE: - /* Nothing to acquire for bpf_list_head */ + case BPF_RB_ROOT: + case BPF_RB_NODE: + case BPF_SPIN_LOCK: + case BPF_TIMER: + /* Nothing to acquire */ break; default: ret = -EFAULT; @@ -658,7 +676,13 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) continue; bpf_list_head_free(field, field_ptr, obj + rec->spin_lock_off); break; + case BPF_RB_ROOT: + if (WARN_ON_ONCE(rec->spin_lock_off < 0)) + continue; + bpf_rb_root_free(field, field_ptr, obj + rec->spin_lock_off); + break; case BPF_LIST_NODE: + case BPF_RB_NODE: break; default: WARN_ON_ONCE(1); @@ -994,7 +1018,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, return -EINVAL; map->record = btf_parse_fields(btf, value_type, - BPF_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD, + BPF_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD | + BPF_RB_ROOT, map->value_size); if (!IS_ERR_OR_NULL(map->record)) { int i; @@ -1042,6 +1067,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, } break; case BPF_LIST_HEAD: + case BPF_RB_ROOT: if (map->map_type != BPF_MAP_TYPE_HASH && map->map_type != BPF_MAP_TYPE_LRU_HASH && map->map_type != BPF_MAP_TYPE_ARRAY) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 388245e8826e..272563a0b770 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -190,6 +190,10 @@ struct bpf_verifier_stack_elem { static int acquire_reference_state(struct bpf_verifier_env *env, int insn_idx); static int release_reference(struct bpf_verifier_env *env, int ref_obj_id); +static void invalidate_non_owning_refs(struct bpf_verifier_env *env); +static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); +static int ref_set_non_owning(struct bpf_verifier_env *env, + struct bpf_reg_state *reg); static bool bpf_map_ptr_poisoned(const struct bpf_insn_aux_data *aux) { @@ -457,6 +461,11 @@ static bool type_is_ptr_alloc_obj(u32 type) return base_type(type) == PTR_TO_BTF_ID && type_flag(type) & MEM_ALLOC; } +static bool type_is_non_owning_ref(u32 type) +{ + return type_is_ptr_alloc_obj(type) && type_flag(type) & NON_OWN_REF; +} + static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg) { struct btf_record *rec = NULL; @@ -1073,6 +1082,8 @@ static void print_verifier_state(struct bpf_verifier_env *env, verbose_a("id=%d", reg->id); if (reg->ref_obj_id) verbose_a("ref_obj_id=%d", reg->ref_obj_id); + if (type_is_non_owning_ref(reg->type)) + verbose_a("%s", "non_own_ref"); if (t != SCALAR_VALUE) verbose_a("off=%d", reg->off); if (type_is_pkt_pointer(t)) @@ -1632,6 +1643,16 @@ static void mark_ptr_not_null_reg(struct bpf_reg_state *reg) reg->type &= ~PTR_MAYBE_NULL; } +static void mark_reg_graph_node(struct bpf_reg_state *regs, u32 regno, + struct btf_field_graph_root *ds_head) +{ + __mark_reg_known_zero(®s[regno]); + regs[regno].type = PTR_TO_BTF_ID | MEM_ALLOC; + regs[regno].btf = ds_head->btf; + regs[regno].btf_id = ds_head->value_btf_id; + regs[regno].off = ds_head->node_offset; +} + static bool reg_is_pkt_pointer(const struct bpf_reg_state *reg) { return type_is_pkt_pointer(reg->type); @@ -3452,6 +3473,11 @@ static void save_register_state(struct bpf_func_state *state, scrub_spilled_slot(&state->stack[spi].slot_type[i - 1]); } +static bool is_bpf_st_mem(struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_ST && BPF_MODE(insn->code) == BPF_MEM; +} + /* check_stack_{read,write}_fixed_off functions track spill/fill of registers, * stack boundary and alignment are checked in check_mem_access() */ @@ -3463,8 +3489,9 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, { struct bpf_func_state *cur; /* state of the current function */ int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err; - u32 dst_reg = env->prog->insnsi[insn_idx].dst_reg; + struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; struct bpf_reg_state *reg = NULL; + u32 dst_reg = insn->dst_reg; err = grow_stack_state(state, round_up(slot + 1, BPF_REG_SIZE)); if (err) @@ -3517,6 +3544,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, return err; } save_register_state(state, spi, reg, size); + } else if (!reg && !(off % BPF_REG_SIZE) && is_bpf_st_mem(insn) && + insn->imm != 0 && env->bpf_capable) { + struct bpf_reg_state fake_reg = {}; + + __mark_reg_known(&fake_reg, (u32)insn->imm); + fake_reg.type = SCALAR_VALUE; + save_register_state(state, spi, &fake_reg, size); } else if (reg && is_spillable_regtype(reg->type)) { /* register containing pointer is being spilled into stack */ if (size != BPF_REG_SIZE) { @@ -3551,7 +3585,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN; /* when we zero initialize stack slots mark them as such */ - if (reg && register_is_null(reg)) { + if ((reg && register_is_null(reg)) || + (!reg && is_bpf_st_mem(insn) && insn->imm == 0)) { /* backtracking doesn't work for STACK_ZERO yet. */ err = mark_chain_precision(env, value_regno); if (err) @@ -3596,6 +3631,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, int min_off, max_off; int i, err; struct bpf_reg_state *ptr_reg = NULL, *value_reg = NULL; + struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; bool writing_zero = false; /* set if the fact that we're writing a zero is used to let any * stack slots remain STACK_ZERO @@ -3608,7 +3644,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, max_off = ptr_reg->smax_value + off + size; if (value_regno >= 0) value_reg = &cur->regs[value_regno]; - if (value_reg && register_is_null(value_reg)) + if ((value_reg && register_is_null(value_reg)) || + (!value_reg && is_bpf_st_mem(insn) && insn->imm == 0)) writing_zero = true; err = grow_stack_state(state, round_up(-min_off, BPF_REG_SIZE)); @@ -5052,7 +5089,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, return -EACCES; } - if (type_is_alloc(reg->type) && !reg->ref_obj_id) { + if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) && + !reg->ref_obj_id) { verbose(env, "verifier internal error: ref_obj_id for allocated object must be non-zero\n"); return -EFAULT; } @@ -6042,9 +6080,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, cur->active_lock.ptr = btf; cur->active_lock.id = reg->id; } else { - struct bpf_func_state *fstate = cur_func(env); void *ptr; - int i; if (map) ptr = map; @@ -6060,25 +6096,11 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, verbose(env, "bpf_spin_unlock of different lock\n"); return -EINVAL; } - cur->active_lock.ptr = NULL; - cur->active_lock.id = 0; - for (i = fstate->acquired_refs - 1; i >= 0; i--) { - int err; + invalidate_non_owning_refs(env); - /* Complain on error because this reference state cannot - * be freed before this point, as bpf_spin_lock critical - * section does not allow functions that release the - * allocated object immediately. - */ - if (!fstate->refs[i].release_on_unlock) - continue; - err = release_reference(env, fstate->refs[i].id); - if (err) { - verbose(env, "failed to release release_on_unlock reference"); - return err; - } - } + cur->active_lock.ptr = NULL; + cur->active_lock.id = 0; } return 0; } @@ -6546,6 +6568,23 @@ found: return 0; } +static struct btf_field * +reg_find_field_offset(const struct bpf_reg_state *reg, s32 off, u32 fields) +{ + struct btf_field *field; + struct btf_record *rec; + + rec = reg_btf_record(reg); + if (!rec) + return NULL; + + field = btf_record_find(rec, off, fields); + if (!field) + return NULL; + + return field; +} + int check_func_arg_reg_off(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, int regno, enum bpf_arg_type arg_type) @@ -6567,6 +6606,18 @@ int check_func_arg_reg_off(struct bpf_verifier_env *env, */ if (arg_type_is_dynptr(arg_type) && type == PTR_TO_STACK) return 0; + + if ((type_is_ptr_alloc_obj(type) || type_is_non_owning_ref(type)) && reg->off) { + if (reg_find_field_offset(reg, reg->off, BPF_GRAPH_NODE_OR_ROOT)) + return __check_ptr_off_reg(env, reg, regno, true); + + verbose(env, "R%d must have zero offset when passed to release func\n", + regno); + verbose(env, "No graph node or root found at R%d type:%s off:%d\n", regno, + kernel_type_name(reg->btf, reg->btf_id), reg->off); + return -EINVAL; + } + /* Doing check_ptr_off_reg check for the offset will catch this * because fixed_off_ok is false, but checking here allows us * to give the user a better error message. @@ -6601,6 +6652,7 @@ int check_func_arg_reg_off(struct bpf_verifier_env *env, case PTR_TO_BTF_ID | PTR_TRUSTED: case PTR_TO_BTF_ID | MEM_RCU: case PTR_TO_BTF_ID | MEM_ALLOC | PTR_TRUSTED: + case PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF: /* When referenced PTR_TO_BTF_ID is passed to release function, * its fixed offset must be 0. In the other cases, fixed offset * can be non-zero. This was already checked above. So pass @@ -6812,6 +6864,10 @@ skip_type_check: meta->ret_btf_id = reg->btf_id; break; case ARG_PTR_TO_SPIN_LOCK: + if (in_rbtree_lock_required_cb(env)) { + verbose(env, "can't spin_{lock,unlock} in rbtree cb\n"); + return -EACCES; + } if (meta->func_id == BPF_FUNC_spin_lock) { err = process_spin_lock(env, regno, true); if (err) @@ -7363,6 +7419,17 @@ static int release_reference(struct bpf_verifier_env *env, return 0; } +static void invalidate_non_owning_refs(struct bpf_verifier_env *env) +{ + struct bpf_func_state *unused; + struct bpf_reg_state *reg; + + bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({ + if (type_is_non_owning_ref(reg->type)) + __mark_reg_unknown(env, reg); + })); +} + static void clear_caller_saved_regs(struct bpf_verifier_env *env, struct bpf_reg_state *regs) { @@ -7384,6 +7451,8 @@ static int set_callee_state(struct bpf_verifier_env *env, struct bpf_func_state *caller, struct bpf_func_state *callee, int insn_idx); +static bool is_callback_calling_kfunc(u32 btf_id); + static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx, int subprog, set_callee_state_fn set_callee_state_cb) @@ -7438,10 +7507,18 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn * interested in validating only BPF helpers that can call subprogs as * callbacks */ - if (set_callee_state_cb != set_callee_state && !is_callback_calling_function(insn->imm)) { - verbose(env, "verifier bug: helper %s#%d is not marked as callback-calling\n", - func_id_name(insn->imm), insn->imm); - return -EFAULT; + if (set_callee_state_cb != set_callee_state) { + if (bpf_pseudo_kfunc_call(insn) && + !is_callback_calling_kfunc(insn->imm)) { + verbose(env, "verifier bug: kfunc %s#%d not marked as callback-calling\n", + func_id_name(insn->imm), insn->imm); + return -EFAULT; + } else if (!bpf_pseudo_kfunc_call(insn) && + !is_callback_calling_function(insn->imm)) { /* helper */ + verbose(env, "verifier bug: helper %s#%d not marked as callback-calling\n", + func_id_name(insn->imm), insn->imm); + return -EFAULT; + } } if (insn->code == (BPF_JMP | BPF_CALL) && @@ -7706,6 +7783,63 @@ static int set_user_ringbuf_callback_state(struct bpf_verifier_env *env, return 0; } +static int set_rbtree_add_callback_state(struct bpf_verifier_env *env, + struct bpf_func_state *caller, + struct bpf_func_state *callee, + int insn_idx) +{ + /* void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + * bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)); + * + * 'struct bpf_rb_node *node' arg to bpf_rbtree_add is the same PTR_TO_BTF_ID w/ offset + * that 'less' callback args will be receiving. However, 'node' arg was release_reference'd + * by this point, so look at 'root' + */ + struct btf_field *field; + + field = reg_find_field_offset(&caller->regs[BPF_REG_1], caller->regs[BPF_REG_1].off, + BPF_RB_ROOT); + if (!field || !field->graph_root.value_btf_id) + return -EFAULT; + + mark_reg_graph_node(callee->regs, BPF_REG_1, &field->graph_root); + ref_set_non_owning(env, &callee->regs[BPF_REG_1]); + mark_reg_graph_node(callee->regs, BPF_REG_2, &field->graph_root); + ref_set_non_owning(env, &callee->regs[BPF_REG_2]); + + __mark_reg_not_init(env, &callee->regs[BPF_REG_3]); + __mark_reg_not_init(env, &callee->regs[BPF_REG_4]); + __mark_reg_not_init(env, &callee->regs[BPF_REG_5]); + callee->in_callback_fn = true; + callee->callback_ret_range = tnum_range(0, 1); + return 0; +} + +static bool is_rbtree_lock_required_kfunc(u32 btf_id); + +/* Are we currently verifying the callback for a rbtree helper that must + * be called with lock held? If so, no need to complain about unreleased + * lock + */ +static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) +{ + struct bpf_verifier_state *state = env->cur_state; + struct bpf_insn *insn = env->prog->insnsi; + struct bpf_func_state *callee; + int kfunc_btf_id; + + if (!state->curframe) + return false; + + callee = state->frame[state->curframe]; + + if (!callee->in_callback_fn) + return false; + + kfunc_btf_id = insn[callee->callsite].imm; + return is_rbtree_lock_required_kfunc(kfunc_btf_id); +} + static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) { struct bpf_verifier_state *state = env->cur_state; @@ -8474,6 +8608,7 @@ struct bpf_kfunc_call_arg_meta { bool r0_rdonly; u32 ret_btf_id; u64 r0_size; + u32 subprogno; struct { u64 value; bool found; @@ -8485,6 +8620,9 @@ struct bpf_kfunc_call_arg_meta { struct { struct btf_field *field; } arg_list_head; + struct { + struct btf_field *field; + } arg_rbtree_root; }; static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta) @@ -8596,12 +8734,16 @@ enum { KF_ARG_DYNPTR_ID, KF_ARG_LIST_HEAD_ID, KF_ARG_LIST_NODE_ID, + KF_ARG_RB_ROOT_ID, + KF_ARG_RB_NODE_ID, }; BTF_ID_LIST(kf_arg_btf_ids) BTF_ID(struct, bpf_dynptr_kern) BTF_ID(struct, bpf_list_head) BTF_ID(struct, bpf_list_node) +BTF_ID(struct, bpf_rb_root) +BTF_ID(struct, bpf_rb_node) static bool __is_kfunc_ptr_arg_type(const struct btf *btf, const struct btf_param *arg, int type) @@ -8635,6 +8777,28 @@ static bool is_kfunc_arg_list_node(const struct btf *btf, const struct btf_param return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_LIST_NODE_ID); } +static bool is_kfunc_arg_rbtree_root(const struct btf *btf, const struct btf_param *arg) +{ + return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_RB_ROOT_ID); +} + +static bool is_kfunc_arg_rbtree_node(const struct btf *btf, const struct btf_param *arg) +{ + return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_RB_NODE_ID); +} + +static bool is_kfunc_arg_callback(struct bpf_verifier_env *env, const struct btf *btf, + const struct btf_param *arg) +{ + const struct btf_type *t; + + t = btf_type_resolve_func_ptr(btf, arg->type, NULL); + if (!t) + return false; + + return true; +} + /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf, @@ -8694,6 +8858,9 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_BTF_ID, /* Also covers reg2btf_ids conversions */ KF_ARG_PTR_TO_MEM, KF_ARG_PTR_TO_MEM_SIZE, /* Size derived from next argument, skip it */ + KF_ARG_PTR_TO_CALLBACK, + KF_ARG_PTR_TO_RB_ROOT, + KF_ARG_PTR_TO_RB_NODE, }; enum special_kfunc_type { @@ -8707,6 +8874,9 @@ enum special_kfunc_type { KF_bpf_rdonly_cast, KF_bpf_rcu_read_lock, KF_bpf_rcu_read_unlock, + KF_bpf_rbtree_remove, + KF_bpf_rbtree_add, + KF_bpf_rbtree_first, }; BTF_SET_START(special_kfunc_set) @@ -8718,6 +8888,9 @@ BTF_ID(func, bpf_list_pop_front) BTF_ID(func, bpf_list_pop_back) BTF_ID(func, bpf_cast_to_kern_ctx) BTF_ID(func, bpf_rdonly_cast) +BTF_ID(func, bpf_rbtree_remove) +BTF_ID(func, bpf_rbtree_add) +BTF_ID(func, bpf_rbtree_first) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -8731,6 +8904,9 @@ BTF_ID(func, bpf_cast_to_kern_ctx) BTF_ID(func, bpf_rdonly_cast) BTF_ID(func, bpf_rcu_read_lock) BTF_ID(func, bpf_rcu_read_unlock) +BTF_ID(func, bpf_rbtree_remove) +BTF_ID(func, bpf_rbtree_add) +BTF_ID(func, bpf_rbtree_first) static bool is_kfunc_bpf_rcu_read_lock(struct bpf_kfunc_call_arg_meta *meta) { @@ -8792,6 +8968,12 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_list_node(meta->btf, &args[argno])) return KF_ARG_PTR_TO_LIST_NODE; + if (is_kfunc_arg_rbtree_root(meta->btf, &args[argno])) + return KF_ARG_PTR_TO_RB_ROOT; + + if (is_kfunc_arg_rbtree_node(meta->btf, &args[argno])) + return KF_ARG_PTR_TO_RB_NODE; + if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { if (!btf_type_is_struct(ref_t)) { verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", @@ -8801,6 +8983,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, return KF_ARG_PTR_TO_BTF_ID; } + if (is_kfunc_arg_callback(env, meta->btf, &args[argno])) + return KF_ARG_PTR_TO_CALLBACK; + if (argno + 1 < nargs && is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1])) arg_mem_size = true; @@ -8915,38 +9100,54 @@ static int process_kf_arg_ptr_to_kptr(struct bpf_verifier_env *env, return 0; } -static int ref_set_release_on_unlock(struct bpf_verifier_env *env, u32 ref_obj_id) +static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg) { - struct bpf_func_state *state = cur_func(env); + struct bpf_verifier_state *state = env->cur_state; + + if (!state->active_lock.ptr) { + verbose(env, "verifier internal error: ref_set_non_owning w/o active lock\n"); + return -EFAULT; + } + + if (type_flag(reg->type) & NON_OWN_REF) { + verbose(env, "verifier internal error: NON_OWN_REF already set\n"); + return -EFAULT; + } + + reg->type |= NON_OWN_REF; + return 0; +} + +static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 ref_obj_id) +{ + struct bpf_func_state *state, *unused; struct bpf_reg_state *reg; int i; - /* bpf_spin_lock only allows calling list_push and list_pop, no BPF - * subprogs, no global functions. This means that the references would - * not be released inside the critical section but they may be added to - * the reference state, and the acquired_refs are never copied out for a - * different frame as BPF to BPF calls don't work in bpf_spin_lock - * critical sections. - */ + state = cur_func(env); + if (!ref_obj_id) { - verbose(env, "verifier internal error: ref_obj_id is zero for release_on_unlock\n"); + verbose(env, "verifier internal error: ref_obj_id is zero for " + "owning -> non-owning conversion\n"); return -EFAULT; } + for (i = 0; i < state->acquired_refs; i++) { - if (state->refs[i].id == ref_obj_id) { - if (state->refs[i].release_on_unlock) { - verbose(env, "verifier internal error: expected false release_on_unlock"); - return -EFAULT; + if (state->refs[i].id != ref_obj_id) + continue; + + /* Clear ref_obj_id here so release_reference doesn't clobber + * the whole reg + */ + bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({ + if (reg->ref_obj_id == ref_obj_id) { + reg->ref_obj_id = 0; + ref_set_non_owning(env, reg); } - state->refs[i].release_on_unlock = true; - /* Now mark everyone sharing same ref_obj_id as untrusted */ - bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ - if (reg->ref_obj_id == ref_obj_id) - reg->type |= PTR_UNTRUSTED; - })); - return 0; - } + })); + return 0; } + verbose(env, "verifier internal error: ref state missing for ref_obj_id\n"); return -EFAULT; } @@ -9032,102 +9233,226 @@ static bool is_bpf_list_api_kfunc(u32 btf_id) btf_id == special_kfunc_list[KF_bpf_list_pop_back]; } -static int process_kf_arg_ptr_to_list_head(struct bpf_verifier_env *env, - struct bpf_reg_state *reg, u32 regno, - struct bpf_kfunc_call_arg_meta *meta) +static bool is_bpf_rbtree_api_kfunc(u32 btf_id) +{ + return btf_id == special_kfunc_list[KF_bpf_rbtree_add] || + btf_id == special_kfunc_list[KF_bpf_rbtree_remove] || + btf_id == special_kfunc_list[KF_bpf_rbtree_first]; +} + +static bool is_bpf_graph_api_kfunc(u32 btf_id) +{ + return is_bpf_list_api_kfunc(btf_id) || is_bpf_rbtree_api_kfunc(btf_id); +} + +static bool is_callback_calling_kfunc(u32 btf_id) +{ + return btf_id == special_kfunc_list[KF_bpf_rbtree_add]; +} + +static bool is_rbtree_lock_required_kfunc(u32 btf_id) +{ + return is_bpf_rbtree_api_kfunc(btf_id); +} + +static bool check_kfunc_is_graph_root_api(struct bpf_verifier_env *env, + enum btf_field_type head_field_type, + u32 kfunc_btf_id) +{ + bool ret; + + switch (head_field_type) { + case BPF_LIST_HEAD: + ret = is_bpf_list_api_kfunc(kfunc_btf_id); + break; + case BPF_RB_ROOT: + ret = is_bpf_rbtree_api_kfunc(kfunc_btf_id); + break; + default: + verbose(env, "verifier internal error: unexpected graph root argument type %s\n", + btf_field_type_name(head_field_type)); + return false; + } + + if (!ret) + verbose(env, "verifier internal error: %s head arg for unknown kfunc\n", + btf_field_type_name(head_field_type)); + return ret; +} + +static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env, + enum btf_field_type node_field_type, + u32 kfunc_btf_id) +{ + bool ret; + + switch (node_field_type) { + case BPF_LIST_NODE: + ret = (kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_front] || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_back]); + break; + case BPF_RB_NODE: + ret = (kfunc_btf_id == special_kfunc_list[KF_bpf_rbtree_remove] || + kfunc_btf_id == special_kfunc_list[KF_bpf_rbtree_add]); + break; + default: + verbose(env, "verifier internal error: unexpected graph node argument type %s\n", + btf_field_type_name(node_field_type)); + return false; + } + + if (!ret) + verbose(env, "verifier internal error: %s node arg for unknown kfunc\n", + btf_field_type_name(node_field_type)); + return ret; +} + +static int +__process_kf_arg_ptr_to_graph_root(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, u32 regno, + struct bpf_kfunc_call_arg_meta *meta, + enum btf_field_type head_field_type, + struct btf_field **head_field) { + const char *head_type_name; struct btf_field *field; struct btf_record *rec; - u32 list_head_off; + u32 head_off; - if (meta->btf != btf_vmlinux || !is_bpf_list_api_kfunc(meta->func_id)) { - verbose(env, "verifier internal error: bpf_list_head argument for unknown kfunc\n"); + if (meta->btf != btf_vmlinux) { + verbose(env, "verifier internal error: unexpected btf mismatch in kfunc call\n"); return -EFAULT; } + if (!check_kfunc_is_graph_root_api(env, head_field_type, meta->func_id)) + return -EFAULT; + + head_type_name = btf_field_type_name(head_field_type); if (!tnum_is_const(reg->var_off)) { verbose(env, - "R%d doesn't have constant offset. bpf_list_head has to be at the constant offset\n", - regno); + "R%d doesn't have constant offset. %s has to be at the constant offset\n", + regno, head_type_name); return -EINVAL; } rec = reg_btf_record(reg); - list_head_off = reg->off + reg->var_off.value; - field = btf_record_find(rec, list_head_off, BPF_LIST_HEAD); + head_off = reg->off + reg->var_off.value; + field = btf_record_find(rec, head_off, head_field_type); if (!field) { - verbose(env, "bpf_list_head not found at offset=%u\n", list_head_off); + verbose(env, "%s not found at offset=%u\n", head_type_name, head_off); return -EINVAL; } /* All functions require bpf_list_head to be protected using a bpf_spin_lock */ if (check_reg_allocation_locked(env, reg)) { - verbose(env, "bpf_spin_lock at off=%d must be held for bpf_list_head\n", - rec->spin_lock_off); + verbose(env, "bpf_spin_lock at off=%d must be held for %s\n", + rec->spin_lock_off, head_type_name); return -EINVAL; } - if (meta->arg_list_head.field) { - verbose(env, "verifier internal error: repeating bpf_list_head arg\n"); + if (*head_field) { + verbose(env, "verifier internal error: repeating %s arg\n", head_type_name); return -EFAULT; } - meta->arg_list_head.field = field; + *head_field = field; return 0; } -static int process_kf_arg_ptr_to_list_node(struct bpf_verifier_env *env, +static int process_kf_arg_ptr_to_list_head(struct bpf_verifier_env *env, struct bpf_reg_state *reg, u32 regno, struct bpf_kfunc_call_arg_meta *meta) { + return __process_kf_arg_ptr_to_graph_root(env, reg, regno, meta, BPF_LIST_HEAD, + &meta->arg_list_head.field); +} + +static int process_kf_arg_ptr_to_rbtree_root(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, u32 regno, + struct bpf_kfunc_call_arg_meta *meta) +{ + return __process_kf_arg_ptr_to_graph_root(env, reg, regno, meta, BPF_RB_ROOT, + &meta->arg_rbtree_root.field); +} + +static int +__process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, u32 regno, + struct bpf_kfunc_call_arg_meta *meta, + enum btf_field_type head_field_type, + enum btf_field_type node_field_type, + struct btf_field **node_field) +{ + const char *node_type_name; const struct btf_type *et, *t; struct btf_field *field; - struct btf_record *rec; - u32 list_node_off; + u32 node_off; - if (meta->btf != btf_vmlinux || - (meta->func_id != special_kfunc_list[KF_bpf_list_push_front] && - meta->func_id != special_kfunc_list[KF_bpf_list_push_back])) { - verbose(env, "verifier internal error: bpf_list_node argument for unknown kfunc\n"); + if (meta->btf != btf_vmlinux) { + verbose(env, "verifier internal error: unexpected btf mismatch in kfunc call\n"); return -EFAULT; } + if (!check_kfunc_is_graph_node_api(env, node_field_type, meta->func_id)) + return -EFAULT; + + node_type_name = btf_field_type_name(node_field_type); if (!tnum_is_const(reg->var_off)) { verbose(env, - "R%d doesn't have constant offset. bpf_list_node has to be at the constant offset\n", - regno); + "R%d doesn't have constant offset. %s has to be at the constant offset\n", + regno, node_type_name); return -EINVAL; } - rec = reg_btf_record(reg); - list_node_off = reg->off + reg->var_off.value; - field = btf_record_find(rec, list_node_off, BPF_LIST_NODE); - if (!field || field->offset != list_node_off) { - verbose(env, "bpf_list_node not found at offset=%u\n", list_node_off); + node_off = reg->off + reg->var_off.value; + field = reg_find_field_offset(reg, node_off, node_field_type); + if (!field || field->offset != node_off) { + verbose(env, "%s not found at offset=%u\n", node_type_name, node_off); return -EINVAL; } - field = meta->arg_list_head.field; + field = *node_field; et = btf_type_by_id(field->graph_root.btf, field->graph_root.value_btf_id); t = btf_type_by_id(reg->btf, reg->btf_id); if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, 0, field->graph_root.btf, field->graph_root.value_btf_id, true)) { - verbose(env, "operation on bpf_list_head expects arg#1 bpf_list_node at offset=%d " + verbose(env, "operation on %s expects arg#1 %s at offset=%d " "in struct %s, but arg is at offset=%d in struct %s\n", + btf_field_type_name(head_field_type), + btf_field_type_name(node_field_type), field->graph_root.node_offset, btf_name_by_offset(field->graph_root.btf, et->name_off), - list_node_off, btf_name_by_offset(reg->btf, t->name_off)); + node_off, btf_name_by_offset(reg->btf, t->name_off)); return -EINVAL; } - if (list_node_off != field->graph_root.node_offset) { - verbose(env, "arg#1 offset=%d, but expected bpf_list_node at offset=%d in struct %s\n", - list_node_off, field->graph_root.node_offset, + if (node_off != field->graph_root.node_offset) { + verbose(env, "arg#1 offset=%d, but expected %s at offset=%d in struct %s\n", + node_off, btf_field_type_name(node_field_type), + field->graph_root.node_offset, btf_name_by_offset(field->graph_root.btf, et->name_off)); return -EINVAL; } - /* Set arg#1 for expiration after unlock */ - return ref_set_release_on_unlock(env, reg->ref_obj_id); + + return 0; +} + +static int process_kf_arg_ptr_to_list_node(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, u32 regno, + struct bpf_kfunc_call_arg_meta *meta) +{ + return __process_kf_arg_ptr_to_graph_node(env, reg, regno, meta, + BPF_LIST_HEAD, BPF_LIST_NODE, + &meta->arg_list_head.field); +} + +static int process_kf_arg_ptr_to_rbtree_node(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, u32 regno, + struct bpf_kfunc_call_arg_meta *meta) +{ + return __process_kf_arg_ptr_to_graph_node(env, reg, regno, meta, + BPF_RB_ROOT, BPF_RB_NODE, + &meta->arg_rbtree_root.field); } static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta) @@ -9264,8 +9589,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ case KF_ARG_PTR_TO_DYNPTR: case KF_ARG_PTR_TO_LIST_HEAD: case KF_ARG_PTR_TO_LIST_NODE: + case KF_ARG_PTR_TO_RB_ROOT: + case KF_ARG_PTR_TO_RB_NODE: case KF_ARG_PTR_TO_MEM: case KF_ARG_PTR_TO_MEM_SIZE: + case KF_ARG_PTR_TO_CALLBACK: /* Trusted by default */ break; default: @@ -9342,6 +9670,20 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (ret < 0) return ret; break; + case KF_ARG_PTR_TO_RB_ROOT: + if (reg->type != PTR_TO_MAP_VALUE && + reg->type != (PTR_TO_BTF_ID | MEM_ALLOC)) { + verbose(env, "arg#%d expected pointer to map value or allocated object\n", i); + return -EINVAL; + } + if (reg->type == (PTR_TO_BTF_ID | MEM_ALLOC) && !reg->ref_obj_id) { + verbose(env, "allocated object must be referenced\n"); + return -EINVAL; + } + ret = process_kf_arg_ptr_to_rbtree_root(env, reg, regno, meta); + if (ret < 0) + return ret; + break; case KF_ARG_PTR_TO_LIST_NODE: if (reg->type != (PTR_TO_BTF_ID | MEM_ALLOC)) { verbose(env, "arg#%d expected pointer to allocated object\n", i); @@ -9355,6 +9697,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (ret < 0) return ret; break; + case KF_ARG_PTR_TO_RB_NODE: + if (meta->func_id == special_kfunc_list[KF_bpf_rbtree_remove]) { + if (!type_is_non_owning_ref(reg->type) || reg->ref_obj_id) { + verbose(env, "rbtree_remove node input must be non-owning ref\n"); + return -EINVAL; + } + if (in_rbtree_lock_required_cb(env)) { + verbose(env, "rbtree_remove not allowed in rbtree cb\n"); + return -EINVAL; + } + } else { + if (reg->type != (PTR_TO_BTF_ID | MEM_ALLOC)) { + verbose(env, "arg#%d expected pointer to allocated object\n", i); + return -EINVAL; + } + if (!reg->ref_obj_id) { + verbose(env, "allocated object must be referenced\n"); + return -EINVAL; + } + } + + ret = process_kf_arg_ptr_to_rbtree_node(env, reg, regno, meta); + if (ret < 0) + return ret; + break; case KF_ARG_PTR_TO_BTF_ID: /* Only base_type is checked, further checks are done here */ if ((base_type(reg->type) != PTR_TO_BTF_ID || @@ -9390,6 +9757,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ /* Skip next '__sz' argument */ i++; break; + case KF_ARG_PTR_TO_CALLBACK: + meta->subprogno = reg->subprogno; + break; } } @@ -9406,11 +9776,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx_p) { const struct btf_type *t, *func, *func_proto, *ptr_type; + u32 i, nargs, func_id, ptr_type_id, release_ref_obj_id; struct bpf_reg_state *regs = cur_regs(env); const char *func_name, *ptr_type_name; bool sleepable, rcu_lock, rcu_unlock; struct bpf_kfunc_call_arg_meta meta; - u32 i, nargs, func_id, ptr_type_id; int err, insn_idx = *insn_idx_p; const struct btf_param *args; const struct btf_type *ret_t; @@ -9505,6 +9875,35 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } + if (meta.func_id == special_kfunc_list[KF_bpf_list_push_front] || + meta.func_id == special_kfunc_list[KF_bpf_list_push_back] || + meta.func_id == special_kfunc_list[KF_bpf_rbtree_add]) { + release_ref_obj_id = regs[BPF_REG_2].ref_obj_id; + err = ref_convert_owning_non_owning(env, release_ref_obj_id); + if (err) { + verbose(env, "kfunc %s#%d conversion of owning ref to non-owning failed\n", + func_name, func_id); + return err; + } + + err = release_reference(env, release_ref_obj_id); + if (err) { + verbose(env, "kfunc %s#%d reference has not been acquired before\n", + func_name, func_id); + return err; + } + } + + if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_add]) { + err = __check_func_call(env, insn, insn_idx_p, meta.subprogno, + set_rbtree_add_callback_state); + if (err) { + verbose(env, "kfunc %s#%d failed callback verification\n", + func_name, func_id); + return err; + } + } + for (i = 0; i < CALLER_SAVED_REGS; i++) mark_reg_not_init(env, regs, caller_saved[i]); @@ -9569,11 +9968,12 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, meta.func_id == special_kfunc_list[KF_bpf_list_pop_back]) { struct btf_field *field = meta.arg_list_head.field; - mark_reg_known_zero(env, regs, BPF_REG_0); - regs[BPF_REG_0].type = PTR_TO_BTF_ID | MEM_ALLOC; - regs[BPF_REG_0].btf = field->graph_root.btf; - regs[BPF_REG_0].btf_id = field->graph_root.value_btf_id; - regs[BPF_REG_0].off = field->graph_root.node_offset; + mark_reg_graph_node(regs, BPF_REG_0, &field->graph_root); + } else if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_remove] || + meta.func_id == special_kfunc_list[KF_bpf_rbtree_first]) { + struct btf_field *field = meta.arg_rbtree_root.field; + + mark_reg_graph_node(regs, BPF_REG_0, &field->graph_root); } else if (meta.func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx]) { mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_BTF_ID | PTR_TRUSTED; @@ -9639,7 +10039,13 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (is_kfunc_ret_null(&meta)) regs[BPF_REG_0].id = id; regs[BPF_REG_0].ref_obj_id = id; + } else if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_first]) { + ref_set_non_owning(env, ®s[BPF_REG_0]); } + + if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_remove]) + invalidate_non_owning_refs(env); + if (reg_may_point_to_spin_lock(®s[BPF_REG_0]) && !regs[BPF_REG_0].id) regs[BPF_REG_0].id = ++env->id_gen; } /* else { add_kfunc_call() ensures it is btf_type_is_void(t) } */ @@ -11825,8 +12231,10 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, */ if (WARN_ON_ONCE(reg->smin_value || reg->smax_value || !tnum_equals_const(reg->var_off, 0))) return; - if (reg->type != (PTR_TO_BTF_ID | MEM_ALLOC | PTR_MAYBE_NULL) && WARN_ON_ONCE(reg->off)) + if (!(type_is_ptr_alloc_obj(reg->type) || type_is_non_owning_ref(reg->type)) && + WARN_ON_ONCE(reg->off)) return; + if (is_null) { reg->type = SCALAR_VALUE; /* We don't need id and ref_obj_id from this point @@ -14335,7 +14743,7 @@ static int do_check(struct bpf_verifier_env *env) if ((insn->src_reg == BPF_REG_0 && insn->imm != BPF_FUNC_spin_unlock) || (insn->src_reg == BPF_PSEUDO_CALL) || (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && - (insn->off != 0 || !is_bpf_list_api_kfunc(insn->imm)))) { + (insn->off != 0 || !is_bpf_graph_api_kfunc(insn->imm)))) { verbose(env, "function calls are not allowed while holding a lock\n"); return -EINVAL; } @@ -14371,7 +14779,8 @@ static int do_check(struct bpf_verifier_env *env) return -EINVAL; } - if (env->cur_state->active_lock.ptr) { + if (env->cur_state->active_lock.ptr && + !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_spin_unlock is missing\n"); return -EINVAL; } @@ -14633,9 +15042,10 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, { enum bpf_prog_type prog_type = resolve_prog_type(prog); - if (btf_record_has_field(map->record, BPF_LIST_HEAD)) { + if (btf_record_has_field(map->record, BPF_LIST_HEAD) || + btf_record_has_field(map->record, BPF_RB_ROOT)) { if (is_tracing_prog_type(prog_type)) { - verbose(env, "tracing progs cannot use bpf_list_head yet\n"); + verbose(env, "tracing progs cannot use bpf_{list_head,rb_root} yet\n"); return -EINVAL; } } diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 1233affc106c..1a9ee8fcd477 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2079,14 +2079,19 @@ struct __cmp_key { static int hop_cmp(const void *a, const void *b) { - struct cpumask **prev_hop = *((struct cpumask ***)b - 1); - struct cpumask **cur_hop = *(struct cpumask ***)b; + struct cpumask **prev_hop, **cur_hop = *(struct cpumask ***)b; struct __cmp_key *k = (struct __cmp_key *)a; if (cpumask_weight_and(k->cpus, cur_hop[k->node]) <= k->cpu) return 1; - k->w = (b == k->masks) ? 0 : cpumask_weight_and(k->cpus, prev_hop[k->node]); + if (b == k->masks) { + k->w = 0; + return 0; + } + + prev_hop = *((struct cpumask ***)b - 1); + k->w = cpumask_weight_and(k->cpus, prev_hop[k->node]); if (k->w <= k->cpu) return 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 73afff8062f9..49f40730e711 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -88,6 +88,9 @@ static bool cgroup_memory_nosocket __ro_after_init; /* Kernel memory accounting disabled? */ static bool cgroup_memory_nokmem __ro_after_init; +/* BPF memory accounting disabled? */ +static bool cgroup_memory_nobpf __ro_after_init; + #ifdef CONFIG_CGROUP_WRITEBACK static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); #endif @@ -347,6 +350,9 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, */ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); + +DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key); +EXPORT_SYMBOL(memcg_bpf_enabled_key); #endif /** @@ -5357,6 +5363,11 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) if (cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_nosocket) static_branch_inc(&memcg_sockets_enabled_key); +#if defined(CONFIG_MEMCG_KMEM) + if (!cgroup_memory_nobpf) + static_branch_inc(&memcg_bpf_enabled_key); +#endif + return &memcg->css; } @@ -5441,6 +5452,11 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->tcpmem_active) static_branch_dec(&memcg_sockets_enabled_key); +#if defined(CONFIG_MEMCG_KMEM) + if (!cgroup_memory_nobpf) + static_branch_dec(&memcg_bpf_enabled_key); +#endif + vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); @@ -7269,6 +7285,8 @@ static int __init cgroup_memory(char *s) cgroup_memory_nosocket = true; if (!strcmp(token, "nokmem")) cgroup_memory_nokmem = true; + if (!strcmp(token, "nobpf")) + cgroup_memory_nobpf = true; } return 1; } diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index b766a84c8536..6f3d654b3339 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -396,10 +396,12 @@ static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, old_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); do { run_ctx.prog_item = &item; + local_bh_disable(); if (xdp) *retval = bpf_prog_run_xdp(prog, ctx); else *retval = bpf_prog_run(prog, ctx); + local_bh_enable(); } while (bpf_test_timer_continue(&t, 1, repeat, &ret, time)); bpf_reset_run_ctx(old_ctx); bpf_test_timer_leave(&t); diff --git a/net/core/dev.c b/net/core/dev.c index 5687b528d4c1..18dc8d75ead9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5019,7 +5019,7 @@ static __latent_entropy void net_tx_action(struct softirq_action *h) WARN_ON(refcount_read(&skb->users)); if (likely(get_kfree_skb_cb(skb)->reason == SKB_REASON_CONSUMED)) - trace_consume_skb(skb); + trace_consume_skb(skb, net_tx_action); else trace_kfree_skb(skb, net_tx_action, SKB_DROP_REASON_NOT_SPECIFIED); diff --git a/net/core/filter.c b/net/core/filter.c index 2ce06a72a5ba..1d6f165923bf 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -5722,12 +5722,8 @@ static const struct bpf_func_proto bpf_skb_get_xfrm_state_proto = { #endif #if IS_ENABLED(CONFIG_INET) || IS_ENABLED(CONFIG_IPV6) -static int bpf_fib_set_fwd_params(struct bpf_fib_lookup *params, - const struct neighbour *neigh, - const struct net_device *dev, u32 mtu) +static int bpf_fib_set_fwd_params(struct bpf_fib_lookup *params, u32 mtu) { - memcpy(params->dmac, neigh->ha, ETH_ALEN); - memcpy(params->smac, dev->dev_addr, ETH_ALEN); params->h_vlan_TCI = 0; params->h_vlan_proto = 0; if (mtu) @@ -5838,21 +5834,29 @@ static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, if (likely(nhc->nhc_gw_family != AF_INET6)) { if (nhc->nhc_gw_family) params->ipv4_dst = nhc->nhc_gw.ipv4; - - neigh = __ipv4_neigh_lookup_noref(dev, - (__force u32)params->ipv4_dst); } else { struct in6_addr *dst = (struct in6_addr *)params->ipv6_dst; params->family = AF_INET6; *dst = nhc->nhc_gw.ipv6; - neigh = __ipv6_neigh_lookup_noref_stub(dev, dst); } - if (!neigh) + if (flags & BPF_FIB_LOOKUP_SKIP_NEIGH) + goto set_fwd_params; + + if (likely(nhc->nhc_gw_family != AF_INET6)) + neigh = __ipv4_neigh_lookup_noref(dev, + (__force u32)params->ipv4_dst); + else + neigh = __ipv6_neigh_lookup_noref_stub(dev, params->ipv6_dst); + + if (!neigh || !(neigh->nud_state & NUD_VALID)) return BPF_FIB_LKUP_RET_NO_NEIGH; + memcpy(params->dmac, neigh->ha, ETH_ALEN); + memcpy(params->smac, dev->dev_addr, ETH_ALEN); - return bpf_fib_set_fwd_params(params, neigh, dev, mtu); +set_fwd_params: + return bpf_fib_set_fwd_params(params, mtu); } #endif @@ -5960,24 +5964,33 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, params->rt_metric = res.f6i->fib6_metric; params->ifindex = dev->ifindex; + if (flags & BPF_FIB_LOOKUP_SKIP_NEIGH) + goto set_fwd_params; + /* xdp and cls_bpf programs are run in RCU-bh so rcu_read_lock_bh is * not needed here. */ neigh = __ipv6_neigh_lookup_noref_stub(dev, dst); - if (!neigh) + if (!neigh || !(neigh->nud_state & NUD_VALID)) return BPF_FIB_LKUP_RET_NO_NEIGH; + memcpy(params->dmac, neigh->ha, ETH_ALEN); + memcpy(params->smac, dev->dev_addr, ETH_ALEN); - return bpf_fib_set_fwd_params(params, neigh, dev, mtu); +set_fwd_params: + return bpf_fib_set_fwd_params(params, mtu); } #endif +#define BPF_FIB_LOOKUP_MASK (BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT | \ + BPF_FIB_LOOKUP_SKIP_NEIGH) + BPF_CALL_4(bpf_xdp_fib_lookup, struct xdp_buff *, ctx, struct bpf_fib_lookup *, params, int, plen, u32, flags) { if (plen < sizeof(*params)) return -EINVAL; - if (flags & ~(BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT)) + if (flags & ~BPF_FIB_LOOKUP_MASK) return -EINVAL; switch (params->family) { @@ -6015,7 +6028,7 @@ BPF_CALL_4(bpf_skb_fib_lookup, struct sk_buff *, skb, if (plen < sizeof(*params)) return -EINVAL; - if (flags & ~(BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT)) + if (flags & ~BPF_FIB_LOOKUP_MASK) return -EINVAL; if (params->tot_len) diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index e20784b6f873..15e3f4606b5f 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -1060,6 +1060,18 @@ static const struct kobj_type rx_queue_ktype = { .get_ownership = rx_queue_get_ownership, }; +static int rx_queue_default_mask(struct net_device *dev, + struct netdev_rx_queue *queue) +{ +#if IS_ENABLED(CONFIG_RPS) && IS_ENABLED(CONFIG_SYSCTL) + struct cpumask *rps_default_mask = READ_ONCE(dev_net(dev)->core.rps_default_mask); + + if (rps_default_mask && !cpumask_empty(rps_default_mask)) + return netdev_rx_queue_set_rps_mask(queue, rps_default_mask); +#endif + return 0; +} + static int rx_queue_add_kobject(struct net_device *dev, int index) { struct netdev_rx_queue *queue = dev->_rx + index; @@ -1083,13 +1095,10 @@ static int rx_queue_add_kobject(struct net_device *dev, int index) goto err; } -#if IS_ENABLED(CONFIG_RPS) && IS_ENABLED(CONFIG_SYSCTL) - if (!cpumask_empty(&rps_default_mask)) { - error = netdev_rx_queue_set_rps_mask(queue, &rps_default_mask); - if (error) - goto err; - } -#endif + error = rx_queue_default_mask(dev, queue); + if (error) + goto err; + kobject_uevent(kobj, KOBJ_ADD); return error; diff --git a/net/core/scm.c b/net/core/scm.c index 5c356f0dee30..acb7d776fa6e 100644 --- a/net/core/scm.c +++ b/net/core/scm.c @@ -229,6 +229,8 @@ int put_cmsg(struct msghdr * msg, int level, int type, int len, void *data) if (msg->msg_control_is_user) { struct cmsghdr __user *cm = msg->msg_control_user; + check_object_size(data, cmlen - sizeof(*cm), true); + if (!user_write_access_begin(cm, cmlen)) goto efault; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 98ebce9f6a51..eb7d33b41e71 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -991,7 +991,7 @@ bool __kfree_skb_reason(struct sk_buff *skb, enum skb_drop_reason reason) DEBUG_NET_WARN_ON_ONCE(reason <= 0 || reason >= SKB_DROP_REASON_MAX); if (reason == SKB_CONSUMED) - trace_consume_skb(skb); + trace_consume_skb(skb, __builtin_return_address(0)); else trace_kfree_skb(skb, __builtin_return_address(0), reason); return true; @@ -1189,7 +1189,7 @@ void consume_skb(struct sk_buff *skb) if (!skb_unref(skb)) return; - trace_consume_skb(skb); + trace_consume_skb(skb, __builtin_return_address(0)); __kfree_skb(skb); } EXPORT_SYMBOL(consume_skb); @@ -1204,7 +1204,7 @@ EXPORT_SYMBOL(consume_skb); */ void __consume_stateless_skb(struct sk_buff *skb) { - trace_consume_skb(skb); + trace_consume_skb(skb, __builtin_return_address(0)); skb_release_data(skb, SKB_CONSUMED); kfree_skbmem(skb); } @@ -1260,7 +1260,7 @@ void napi_consume_skb(struct sk_buff *skb, int budget) return; /* if reaching here SKB is ready to free */ - trace_consume_skb(skb); + trace_consume_skb(skb, __builtin_return_address(0)); /* if SKB is a clone, don't handle this case */ if (skb->fclone != SKB_FCLONE_UNAVAILABLE) { diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 7130e6d9e263..74842b453407 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -74,24 +74,47 @@ static void dump_cpumask(void *buffer, size_t *lenp, loff_t *ppos, #endif #ifdef CONFIG_RPS -struct cpumask rps_default_mask; + +static struct cpumask *rps_default_mask_cow_alloc(struct net *net) +{ + struct cpumask *rps_default_mask; + + if (net->core.rps_default_mask) + return net->core.rps_default_mask; + + rps_default_mask = kzalloc(cpumask_size(), GFP_KERNEL); + if (!rps_default_mask) + return NULL; + + /* pairs with READ_ONCE in rx_queue_default_mask() */ + WRITE_ONCE(net->core.rps_default_mask, rps_default_mask); + return rps_default_mask; +} static int rps_default_mask_sysctl(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { + struct net *net = (struct net *)table->data; int err = 0; rtnl_lock(); if (write) { - err = cpumask_parse(buffer, &rps_default_mask); + struct cpumask *rps_default_mask = rps_default_mask_cow_alloc(net); + + err = -ENOMEM; + if (!rps_default_mask) + goto done; + + err = cpumask_parse(buffer, rps_default_mask); if (err) goto done; - err = rps_cpumask_housekeeping(&rps_default_mask); + err = rps_cpumask_housekeeping(rps_default_mask); if (err) goto done; } else { - dump_cpumask(buffer, lenp, ppos, &rps_default_mask); + dump_cpumask(buffer, lenp, ppos, + net->core.rps_default_mask ? : cpu_none_mask); } done: @@ -508,11 +531,6 @@ static struct ctl_table net_core_table[] = { .mode = 0644, .proc_handler = rps_sock_flow_sysctl }, - { - .procname = "rps_default_mask", - .mode = 0644, - .proc_handler = rps_default_mask_sysctl - }, #endif #ifdef CONFIG_NET_FLOW_LIMIT { @@ -639,6 +657,14 @@ static struct ctl_table net_core_table[] = { }; static struct ctl_table netns_core_table[] = { +#if IS_ENABLED(CONFIG_RPS) + { + .procname = "rps_default_mask", + .data = &init_net, + .mode = 0644, + .proc_handler = rps_default_mask_sysctl + }, +#endif { .procname = "somaxconn", .data = &init_net.core.sysctl_somaxconn, @@ -706,6 +732,9 @@ static __net_exit void sysctl_core_net_exit(struct net *net) tbl = net->core.sysctl_hdr->ctl_table_arg; unregister_net_sysctl_table(net->core.sysctl_hdr); BUG_ON(tbl == netns_core_table); +#if IS_ENABLED(CONFIG_RPS) + kfree(net->core.rps_default_mask); +#endif kfree(tbl); } @@ -716,10 +745,6 @@ static __net_initdata struct pernet_operations sysctl_core_ops = { static __init int sysctl_core_init(void) { -#if IS_ENABLED(CONFIG_RPS) - cpumask_copy(&rps_default_mask, cpu_none_mask); -#endif - register_net_sysctl(&init_net, "net/core", net_core_table); return register_pernet_subsys(&sysctl_core_ops); } diff --git a/net/devlink/leftover.c b/net/devlink/leftover.c index 8fc85ba3f6e0..dffca2f9bfa7 100644 --- a/net/devlink/leftover.c +++ b/net/devlink/leftover.c @@ -3837,19 +3837,6 @@ int devlink_resources_validate(struct devlink *devlink, return err; } -static void devlink_param_notify(struct devlink *devlink, - unsigned int port_index, - struct devlink_param_item *param_item, - enum devlink_command cmd); - -struct devlink_info_req { - struct sk_buff *msg; - void (*version_cb)(const char *version_name, - enum devlink_info_version_type version_type, - void *version_cb_priv); - void *version_cb_priv; -}; - static const struct devlink_param devlink_param_generic[] = { { .id = DEVLINK_PARAM_GENERIC_ID_INT_ERR_RESET, diff --git a/net/ieee802154/header_ops.c b/net/ieee802154/header_ops.c index af337cf62764..35d384dfe29d 100644 --- a/net/ieee802154/header_ops.c +++ b/net/ieee802154/header_ops.c @@ -120,6 +120,30 @@ ieee802154_hdr_push(struct sk_buff *skb, struct ieee802154_hdr *hdr) } EXPORT_SYMBOL_GPL(ieee802154_hdr_push); +int ieee802154_beacon_push(struct sk_buff *skb, + struct ieee802154_beacon_frame *beacon) +{ + struct ieee802154_beacon_hdr *mac_pl = &beacon->mac_pl; + struct ieee802154_hdr *mhr = &beacon->mhr; + int ret; + + skb_reserve(skb, sizeof(*mhr)); + ret = ieee802154_hdr_push(skb, mhr); + if (ret < 0) + return ret; + + skb_reset_mac_header(skb); + skb->mac_len = ret; + + skb_put_data(skb, mac_pl, sizeof(*mac_pl)); + + if (mac_pl->pend_short_addr_count || mac_pl->pend_ext_addr_count) + return -EOPNOTSUPP; + + return 0; +} +EXPORT_SYMBOL_GPL(ieee802154_beacon_push); + static int ieee802154_hdr_get_addr(const u8 *buf, int mode, bool omit_pan, struct ieee802154_addr *addr) diff --git a/net/ieee802154/nl802154.c b/net/ieee802154/nl802154.c index 248ad5e46969..2215f576ee37 100644 --- a/net/ieee802154/nl802154.c +++ b/net/ieee802154/nl802154.c @@ -187,8 +187,8 @@ static const struct nla_policy nl802154_policy[NL802154_ATTR_MAX+1] = { [NL802154_ATTR_WPAN_DEV] = { .type = NLA_U64 }, - [NL802154_ATTR_PAGE] = { .type = NLA_U8, }, - [NL802154_ATTR_CHANNEL] = { .type = NLA_U8, }, + [NL802154_ATTR_PAGE] = NLA_POLICY_MAX(NLA_U8, IEEE802154_MAX_PAGE), + [NL802154_ATTR_CHANNEL] = NLA_POLICY_MAX(NLA_U8, IEEE802154_MAX_CHANNEL), [NL802154_ATTR_TX_POWER] = { .type = NLA_S32, }, @@ -221,6 +221,20 @@ static const struct nla_policy nl802154_policy[NL802154_ATTR_MAX+1] = { [NL802154_ATTR_COORDINATOR] = { .type = NLA_NESTED }, + [NL802154_ATTR_SCAN_TYPE] = + NLA_POLICY_RANGE(NLA_U8, NL802154_SCAN_ED, NL802154_SCAN_RIT_PASSIVE), + [NL802154_ATTR_SCAN_CHANNELS] = + NLA_POLICY_MASK(NLA_U32, GENMASK(IEEE802154_MAX_CHANNEL, 0)), + [NL802154_ATTR_SCAN_PREAMBLE_CODES] = { .type = NLA_REJECT }, + [NL802154_ATTR_SCAN_MEAN_PRF] = { .type = NLA_REJECT }, + [NL802154_ATTR_SCAN_DURATION] = + NLA_POLICY_MAX(NLA_U8, IEEE802154_MAX_SCAN_DURATION), + [NL802154_ATTR_SCAN_DONE_REASON] = + NLA_POLICY_RANGE(NLA_U8, NL802154_SCAN_DONE_REASON_FINISHED, + NL802154_SCAN_DONE_REASON_ABORTED), + [NL802154_ATTR_BEACON_INTERVAL] = + NLA_POLICY_MAX(NLA_U8, IEEE802154_MAX_SCAN_DURATION), + #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL [NL802154_ATTR_SEC_ENABLED] = { .type = NLA_U8, }, [NL802154_ATTR_SEC_OUT_LEVEL] = { .type = NLA_U32, }, @@ -969,8 +983,7 @@ static int nl802154_set_channel(struct sk_buff *skb, struct genl_info *info) channel = nla_get_u8(info->attrs[NL802154_ATTR_CHANNEL]); /* check 802.15.4 constraints */ - if (page > IEEE802154_MAX_PAGE || channel > IEEE802154_MAX_CHANNEL || - !(rdev->wpan_phy.supported.channels[page] & BIT(channel))) + if (!ieee802154_chan_is_valid(&rdev->wpan_phy, page, channel)) return -EINVAL; return rdev_set_channel(rdev, page, channel); @@ -1384,6 +1397,236 @@ int nl802154_scan_event(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, } EXPORT_SYMBOL_GPL(nl802154_scan_event); +static int nl802154_trigger_scan(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg802154_registered_device *rdev = info->user_ptr[0]; + struct net_device *dev = info->user_ptr[1]; + struct wpan_dev *wpan_dev = dev->ieee802154_ptr; + struct wpan_phy *wpan_phy = &rdev->wpan_phy; + struct cfg802154_scan_request *request; + u8 type; + int err; + + if (wpan_dev->iftype == NL802154_IFTYPE_MONITOR) { + NL_SET_ERR_MSG(info->extack, "Monitors are not allowed to perform scans"); + return -EOPNOTSUPP; + } + + if (!nla_get_u8(info->attrs[NL802154_ATTR_SCAN_TYPE])) { + NL_SET_ERR_MSG(info->extack, "Malformed request, missing scan type"); + return -EINVAL; + } + + request = kzalloc(sizeof(*request), GFP_KERNEL); + if (!request) + return -ENOMEM; + + request->wpan_dev = wpan_dev; + request->wpan_phy = wpan_phy; + + type = nla_get_u8(info->attrs[NL802154_ATTR_SCAN_TYPE]); + switch (type) { + case NL802154_SCAN_PASSIVE: + request->type = type; + break; + default: + NL_SET_ERR_MSG_FMT(info->extack, "Unsupported scan type: %d", type); + err = -EINVAL; + goto free_request; + } + + /* Use current page by default */ + if (info->attrs[NL802154_ATTR_PAGE]) + request->page = nla_get_u8(info->attrs[NL802154_ATTR_PAGE]); + else + request->page = wpan_phy->current_page; + + /* Scan all supported channels by default */ + if (info->attrs[NL802154_ATTR_SCAN_CHANNELS]) + request->channels = nla_get_u32(info->attrs[NL802154_ATTR_SCAN_CHANNELS]); + else + request->channels = wpan_phy->supported.channels[request->page]; + + /* Use maximum duration order by default */ + if (info->attrs[NL802154_ATTR_SCAN_DURATION]) + request->duration = nla_get_u8(info->attrs[NL802154_ATTR_SCAN_DURATION]); + else + request->duration = IEEE802154_MAX_SCAN_DURATION; + + err = rdev_trigger_scan(rdev, request); + if (err) { + pr_err("Failure starting scanning (%d)\n", err); + goto free_request; + } + + return 0; + +free_request: + kfree(request); + + return err; +} + +static int nl802154_prep_scan_msg(struct sk_buff *msg, + struct cfg802154_registered_device *rdev, + struct wpan_dev *wpan_dev, u32 portid, + u32 seq, int flags, u8 cmd, u8 arg) +{ + void *hdr; + + hdr = nl802154hdr_put(msg, portid, seq, flags, cmd); + if (!hdr) + return -ENOBUFS; + + if (nla_put_u32(msg, NL802154_ATTR_WPAN_PHY, rdev->wpan_phy_idx)) + goto nla_put_failure; + + if (wpan_dev->netdev && + nla_put_u32(msg, NL802154_ATTR_IFINDEX, wpan_dev->netdev->ifindex)) + goto nla_put_failure; + + if (nla_put_u64_64bit(msg, NL802154_ATTR_WPAN_DEV, + wpan_dev_id(wpan_dev), NL802154_ATTR_PAD)) + goto nla_put_failure; + + if (cmd == NL802154_CMD_SCAN_DONE && + nla_put_u8(msg, NL802154_ATTR_SCAN_DONE_REASON, arg)) + goto nla_put_failure; + + genlmsg_end(msg, hdr); + + return 0; + +nla_put_failure: + genlmsg_cancel(msg, hdr); + + return -EMSGSIZE; +} + +static int nl802154_send_scan_msg(struct cfg802154_registered_device *rdev, + struct wpan_dev *wpan_dev, u8 cmd, u8 arg) +{ + struct sk_buff *msg; + int ret; + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + ret = nl802154_prep_scan_msg(msg, rdev, wpan_dev, 0, 0, 0, cmd, arg); + if (ret < 0) { + nlmsg_free(msg); + return ret; + } + + return genlmsg_multicast_netns(&nl802154_fam, + wpan_phy_net(&rdev->wpan_phy), msg, 0, + NL802154_MCGRP_SCAN, GFP_KERNEL); +} + +int nl802154_scan_started(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev) +{ + struct cfg802154_registered_device *rdev = wpan_phy_to_rdev(wpan_phy); + int err; + + /* Ignore errors when there are no listeners */ + err = nl802154_send_scan_msg(rdev, wpan_dev, NL802154_CMD_TRIGGER_SCAN, 0); + if (err == -ESRCH) + err = 0; + + return err; +} +EXPORT_SYMBOL_GPL(nl802154_scan_started); + +int nl802154_scan_done(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, + enum nl802154_scan_done_reasons reason) +{ + struct cfg802154_registered_device *rdev = wpan_phy_to_rdev(wpan_phy); + int err; + + /* Ignore errors when there are no listeners */ + err = nl802154_send_scan_msg(rdev, wpan_dev, NL802154_CMD_SCAN_DONE, reason); + if (err == -ESRCH) + err = 0; + + return err; +} +EXPORT_SYMBOL_GPL(nl802154_scan_done); + +static int nl802154_abort_scan(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg802154_registered_device *rdev = info->user_ptr[0]; + struct net_device *dev = info->user_ptr[1]; + struct wpan_dev *wpan_dev = dev->ieee802154_ptr; + + /* Resources are released in the notification helper above */ + return rdev_abort_scan(rdev, wpan_dev); +} + +static int +nl802154_send_beacons(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg802154_registered_device *rdev = info->user_ptr[0]; + struct net_device *dev = info->user_ptr[1]; + struct wpan_dev *wpan_dev = dev->ieee802154_ptr; + struct wpan_phy *wpan_phy = &rdev->wpan_phy; + struct cfg802154_beacon_request *request; + int err; + + if (wpan_dev->iftype != NL802154_IFTYPE_COORD) { + NL_SET_ERR_MSG(info->extack, "Only coordinators can send beacons"); + return -EOPNOTSUPP; + } + + if (wpan_dev->pan_id == cpu_to_le16(IEEE802154_PANID_BROADCAST)) { + NL_SET_ERR_MSG(info->extack, "Device is not part of any PAN"); + return -EPERM; + } + + request = kzalloc(sizeof(*request), GFP_KERNEL); + if (!request) + return -ENOMEM; + + request->wpan_dev = wpan_dev; + request->wpan_phy = wpan_phy; + + /* Use maximum duration order by default */ + if (info->attrs[NL802154_ATTR_BEACON_INTERVAL]) + request->interval = nla_get_u8(info->attrs[NL802154_ATTR_BEACON_INTERVAL]); + else + request->interval = IEEE802154_MAX_SCAN_DURATION; + + err = rdev_send_beacons(rdev, request); + if (err) { + pr_err("Failure starting sending beacons (%d)\n", err); + goto free_request; + } + + return 0; + +free_request: + kfree(request); + + return err; +} + +void nl802154_beaconing_done(struct wpan_dev *wpan_dev) +{ + /* NOP */ +} +EXPORT_SYMBOL_GPL(nl802154_beaconing_done); + +static int +nl802154_stop_beacons(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg802154_registered_device *rdev = info->user_ptr[0]; + struct net_device *dev = info->user_ptr[1]; + struct wpan_dev *wpan_dev = dev->ieee802154_ptr; + + /* Resources are released in the notification helper above */ + return rdev_stop_beacons(rdev, wpan_dev); +} + #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL static const struct nla_policy nl802154_dev_addr_policy[NL802154_DEV_ADDR_ATTR_MAX + 1] = { [NL802154_DEV_ADDR_ATTR_PAN_ID] = { .type = NLA_U16 }, @@ -2474,6 +2717,38 @@ static const struct genl_ops nl802154_ops[] = { .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, + { + .cmd = NL802154_CMD_TRIGGER_SCAN, + .doit = nl802154_trigger_scan, + .flags = GENL_ADMIN_PERM, + .internal_flags = NL802154_FLAG_NEED_NETDEV | + NL802154_FLAG_CHECK_NETDEV_UP | + NL802154_FLAG_NEED_RTNL, + }, + { + .cmd = NL802154_CMD_ABORT_SCAN, + .doit = nl802154_abort_scan, + .flags = GENL_ADMIN_PERM, + .internal_flags = NL802154_FLAG_NEED_NETDEV | + NL802154_FLAG_CHECK_NETDEV_UP | + NL802154_FLAG_NEED_RTNL, + }, + { + .cmd = NL802154_CMD_SEND_BEACONS, + .doit = nl802154_send_beacons, + .flags = GENL_ADMIN_PERM, + .internal_flags = NL802154_FLAG_NEED_NETDEV | + NL802154_FLAG_CHECK_NETDEV_UP | + NL802154_FLAG_NEED_RTNL, + }, + { + .cmd = NL802154_CMD_STOP_BEACONS, + .doit = nl802154_stop_beacons, + .flags = GENL_ADMIN_PERM, + .internal_flags = NL802154_FLAG_NEED_NETDEV | + NL802154_FLAG_CHECK_NETDEV_UP | + NL802154_FLAG_NEED_RTNL, + }, #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL { .cmd = NL802154_CMD_SET_SEC_PARAMS, diff --git a/net/ieee802154/nl802154.h b/net/ieee802154/nl802154.h index 89b805500032..d69d950f9a6a 100644 --- a/net/ieee802154/nl802154.h +++ b/net/ieee802154/nl802154.h @@ -6,5 +6,9 @@ int nl802154_init(void); void nl802154_exit(void); int nl802154_scan_event(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, struct ieee802154_coord_desc *desc); +int nl802154_scan_started(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev); +int nl802154_scan_done(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev, + enum nl802154_scan_done_reasons reason); +void nl802154_beaconing_done(struct wpan_dev *wpan_dev); #endif /* __IEEE802154_NL802154_H */ diff --git a/net/ieee802154/rdev-ops.h b/net/ieee802154/rdev-ops.h index 598f5af49775..5eaae15c610e 100644 --- a/net/ieee802154/rdev-ops.h +++ b/net/ieee802154/rdev-ops.h @@ -209,6 +209,62 @@ rdev_set_ackreq_default(struct cfg802154_registered_device *rdev, return ret; } +static inline int rdev_trigger_scan(struct cfg802154_registered_device *rdev, + struct cfg802154_scan_request *request) +{ + int ret; + + if (!rdev->ops->trigger_scan) + return -EOPNOTSUPP; + + trace_802154_rdev_trigger_scan(&rdev->wpan_phy, request); + ret = rdev->ops->trigger_scan(&rdev->wpan_phy, request); + trace_802154_rdev_return_int(&rdev->wpan_phy, ret); + return ret; +} + +static inline int rdev_abort_scan(struct cfg802154_registered_device *rdev, + struct wpan_dev *wpan_dev) +{ + int ret; + + if (!rdev->ops->abort_scan) + return -EOPNOTSUPP; + + trace_802154_rdev_abort_scan(&rdev->wpan_phy, wpan_dev); + ret = rdev->ops->abort_scan(&rdev->wpan_phy, wpan_dev); + trace_802154_rdev_return_int(&rdev->wpan_phy, ret); + return ret; +} + +static inline int rdev_send_beacons(struct cfg802154_registered_device *rdev, + struct cfg802154_beacon_request *request) +{ + int ret; + + if (!rdev->ops->send_beacons) + return -EOPNOTSUPP; + + trace_802154_rdev_send_beacons(&rdev->wpan_phy, request); + ret = rdev->ops->send_beacons(&rdev->wpan_phy, request); + trace_802154_rdev_return_int(&rdev->wpan_phy, ret); + return ret; +} + +static inline int rdev_stop_beacons(struct cfg802154_registered_device *rdev, + struct wpan_dev *wpan_dev) +{ + int ret; + + if (!rdev->ops->stop_beacons) + return -EOPNOTSUPP; + + trace_802154_rdev_stop_beacons(&rdev->wpan_phy, wpan_dev); + ret = rdev->ops->stop_beacons(&rdev->wpan_phy, wpan_dev); + trace_802154_rdev_return_int(&rdev->wpan_phy, ret); + return ret; +} + #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL /* TODO this is already a nl802154, so move into ieee802154 */ static inline void diff --git a/net/ieee802154/trace.h b/net/ieee802154/trace.h index 19c2e5d60e76..e5d8439b9e45 100644 --- a/net/ieee802154/trace.h +++ b/net/ieee802154/trace.h @@ -295,6 +295,67 @@ TRACE_EVENT(802154_rdev_set_ackreq_default, WPAN_DEV_PR_ARG, BOOL_TO_STR(__entry->ackreq)) ); +TRACE_EVENT(802154_rdev_trigger_scan, + TP_PROTO(struct wpan_phy *wpan_phy, + struct cfg802154_scan_request *request), + TP_ARGS(wpan_phy, request), + TP_STRUCT__entry( + WPAN_PHY_ENTRY + __field(u8, page) + __field(u32, channels) + __field(u8, duration) + ), + TP_fast_assign( + WPAN_PHY_ASSIGN; + __entry->page = request->page; + __entry->channels = request->channels; + __entry->duration = request->duration; + ), + TP_printk(WPAN_PHY_PR_FMT ", scan, page: %d, channels: %x, duration %d", + WPAN_PHY_PR_ARG, __entry->page, __entry->channels, __entry->duration) +); + +TRACE_EVENT(802154_rdev_send_beacons, + TP_PROTO(struct wpan_phy *wpan_phy, + struct cfg802154_beacon_request *request), + TP_ARGS(wpan_phy, request), + TP_STRUCT__entry( + WPAN_PHY_ENTRY + __field(u8, interval) + ), + TP_fast_assign( + WPAN_PHY_ASSIGN; + __entry->interval = request->interval; + ), + TP_printk(WPAN_PHY_PR_FMT ", sending beacons (interval order: %d)", + WPAN_PHY_PR_ARG, __entry->interval) +); + +DECLARE_EVENT_CLASS(802154_wdev_template, + TP_PROTO(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev), + TP_ARGS(wpan_phy, wpan_dev), + TP_STRUCT__entry( + WPAN_PHY_ENTRY + WPAN_DEV_ENTRY + ), + TP_fast_assign( + WPAN_PHY_ASSIGN; + WPAN_DEV_ASSIGN; + ), + TP_printk(WPAN_PHY_PR_FMT ", " WPAN_DEV_PR_FMT, + WPAN_PHY_PR_ARG, WPAN_DEV_PR_ARG) +); + +DEFINE_EVENT(802154_wdev_template, 802154_rdev_abort_scan, + TP_PROTO(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev), + TP_ARGS(wpan_phy, wpan_dev) +); + +DEFINE_EVENT(802154_wdev_template, 802154_rdev_stop_beacons, + TP_PROTO(struct wpan_phy *wpan_phy, struct wpan_dev *wpan_dev), + TP_ARGS(wpan_phy, wpan_dev) +); + TRACE_EVENT(802154_rdev_return_int, TP_PROTO(struct wpan_phy *wpan_phy, int ret), TP_ARGS(wpan_phy, ret), diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index eedcf4146d29..65ad4251f6fd 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -1122,8 +1122,7 @@ static void inet_clone_ulp(const struct request_sock *req, struct sock *newsk, if (!icsk->icsk_ulp_ops) return; - if (icsk->icsk_ulp_ops->clone) - icsk->icsk_ulp_ops->clone(req, newsk, priority); + icsk->icsk_ulp_ops->clone(req, newsk, priority); } /** diff --git a/net/ipv4/netfilter/nf_reject_ipv4.c b/net/ipv4/netfilter/nf_reject_ipv4.c index d640adcaf1b1..f33aeab9424f 100644 --- a/net/ipv4/netfilter/nf_reject_ipv4.c +++ b/net/ipv4/netfilter/nf_reject_ipv4.c @@ -280,6 +280,7 @@ void nf_send_reset(struct net *net, struct sock *sk, struct sk_buff *oldskb, goto free_nskb; nf_ct_attach(nskb, oldskb); + nf_ct_set_closing(skb_nfct(oldskb)); #if IS_ENABLED(CONFIG_BRIDGE_NETFILTER) /* If we use ip_local_out for bridged traffic, the MAC source on diff --git a/net/ipv6/icmp.c b/net/ipv6/icmp.c index f32bc98155bf..1f53f2a74480 100644 --- a/net/ipv6/icmp.c +++ b/net/ipv6/icmp.c @@ -705,7 +705,7 @@ int ip6_err_gen_icmpv6_unreach(struct sk_buff *skb, int nhs, int type, } EXPORT_SYMBOL(ip6_err_gen_icmpv6_unreach); -static void icmpv6_echo_reply(struct sk_buff *skb) +static enum skb_drop_reason icmpv6_echo_reply(struct sk_buff *skb) { struct net *net = dev_net(skb->dev); struct sock *sk; @@ -719,18 +719,19 @@ static void icmpv6_echo_reply(struct sk_buff *skb) struct dst_entry *dst; struct ipcm6_cookie ipc6; u32 mark = IP6_REPLY_MARK(net, skb->mark); + SKB_DR(reason); bool acast; u8 type; if (ipv6_addr_is_multicast(&ipv6_hdr(skb)->daddr) && net->ipv6.sysctl.icmpv6_echo_ignore_multicast) - return; + return reason; saddr = &ipv6_hdr(skb)->daddr; acast = ipv6_anycast_destination(skb_dst(skb), saddr); if (acast && net->ipv6.sysctl.icmpv6_echo_ignore_anycast) - return; + return reason; if (!ipv6_unicast_destination(skb) && !(net->ipv6.sysctl.anycast_src_echo_reply && acast)) @@ -804,6 +805,7 @@ static void icmpv6_echo_reply(struct sk_buff *skb) } else { icmpv6_push_pending_frames(sk, &fl6, &tmp_hdr, skb->len + sizeof(struct icmp6hdr)); + reason = SKB_CONSUMED; } out_dst_release: dst_release(dst); @@ -811,6 +813,7 @@ out: icmpv6_xmit_unlock(sk); out_bh_enable: local_bh_enable(); + return reason; } enum skb_drop_reason icmpv6_notify(struct sk_buff *skb, u8 type, @@ -929,12 +932,12 @@ static int icmpv6_rcv(struct sk_buff *skb) switch (type) { case ICMPV6_ECHO_REQUEST: if (!net->ipv6.sysctl.icmpv6_echo_ignore_all) - icmpv6_echo_reply(skb); + reason = icmpv6_echo_reply(skb); break; case ICMPV6_EXT_ECHO_REQUEST: if (!net->ipv6.sysctl.icmpv6_echo_ignore_all && READ_ONCE(net->ipv4.sysctl_icmp_echo_enable_probe)) - icmpv6_echo_reply(skb); + reason = icmpv6_echo_reply(skb); break; case ICMPV6_ECHO_REPLY: diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c index 9548b5a44714..c4be62c99f73 100644 --- a/net/ipv6/ndisc.c +++ b/net/ipv6/ndisc.c @@ -783,7 +783,7 @@ void ndisc_update(const struct net_device *dev, struct neighbour *neigh, ndisc_ops_update(dev, neigh, flags, icmp6_type, ndopts); } -static void ndisc_recv_ns(struct sk_buff *skb) +static enum skb_drop_reason ndisc_recv_ns(struct sk_buff *skb) { struct nd_msg *msg = (struct nd_msg *)skb_transport_header(skb); const struct in6_addr *saddr = &ipv6_hdr(skb)->saddr; @@ -797,18 +797,17 @@ static void ndisc_recv_ns(struct sk_buff *skb) struct inet6_dev *idev = NULL; struct neighbour *neigh; int dad = ipv6_addr_any(saddr); - bool inc; int is_router = -1; + SKB_DR(reason); u64 nonce = 0; + bool inc; - if (skb->len < sizeof(struct nd_msg)) { - ND_PRINTK(2, warn, "NS: packet too short\n"); - return; - } + if (skb->len < sizeof(struct nd_msg)) + return SKB_DROP_REASON_PKT_TOO_SMALL; if (ipv6_addr_is_multicast(&msg->target)) { ND_PRINTK(2, warn, "NS: multicast target address\n"); - return; + return reason; } /* @@ -817,20 +816,18 @@ static void ndisc_recv_ns(struct sk_buff *skb) */ if (dad && !ipv6_addr_is_solict_mult(daddr)) { ND_PRINTK(2, warn, "NS: bad DAD packet (wrong destination)\n"); - return; + return reason; } - if (!ndisc_parse_options(dev, msg->opt, ndoptlen, &ndopts)) { - ND_PRINTK(2, warn, "NS: invalid ND options\n"); - return; - } + if (!ndisc_parse_options(dev, msg->opt, ndoptlen, &ndopts)) + return SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS; if (ndopts.nd_opts_src_lladdr) { lladdr = ndisc_opt_addr_data(ndopts.nd_opts_src_lladdr, dev); if (!lladdr) { ND_PRINTK(2, warn, "NS: invalid link-layer address length\n"); - return; + return reason; } /* RFC2461 7.1.1: @@ -841,7 +838,7 @@ static void ndisc_recv_ns(struct sk_buff *skb) if (dad) { ND_PRINTK(2, warn, "NS: bad DAD packet (link-layer address option)\n"); - return; + return reason; } } if (ndopts.nd_opts_nonce && ndopts.nd_opts_nonce->nd_opt_len == 1) @@ -869,7 +866,7 @@ have_ifp: * so fail our DAD process */ addrconf_dad_failure(skb, ifp); - return; + return reason; } else { /* * This is not a dad solicitation. @@ -901,7 +898,7 @@ have_ifp: idev = in6_dev_get(dev); if (!idev) { /* XXX: count this drop? */ - return; + return reason; } if (ipv6_chk_acast_addr(net, dev, &msg->target) || @@ -924,8 +921,10 @@ have_ifp: pneigh_enqueue(&nd_tbl, idev->nd_parms, n); goto out; } - } else + } else { + SKB_DR_SET(reason, IPV6_NDISC_NS_OTHERHOST); goto out; + } } if (is_router < 0) @@ -958,6 +957,7 @@ have_ifp: true, (ifp != NULL && inc), inc); if (neigh) neigh_release(neigh); + reason = SKB_CONSUMED; } out: @@ -965,6 +965,7 @@ out: in6_ifa_put(ifp); else in6_dev_put(idev); + return reason; } static int accept_untracked_na(struct net_device *dev, struct in6_addr *saddr) @@ -986,7 +987,7 @@ static int accept_untracked_na(struct net_device *dev, struct in6_addr *saddr) } } -static void ndisc_recv_na(struct sk_buff *skb) +static enum skb_drop_reason ndisc_recv_na(struct sk_buff *skb) { struct nd_msg *msg = (struct nd_msg *)skb_transport_header(skb); struct in6_addr *saddr = &ipv6_hdr(skb)->saddr; @@ -999,22 +1000,21 @@ static void ndisc_recv_na(struct sk_buff *skb) struct inet6_dev *idev = __in6_dev_get(dev); struct inet6_ifaddr *ifp; struct neighbour *neigh; + SKB_DR(reason); u8 new_state; - if (skb->len < sizeof(struct nd_msg)) { - ND_PRINTK(2, warn, "NA: packet too short\n"); - return; - } + if (skb->len < sizeof(struct nd_msg)) + return SKB_DROP_REASON_PKT_TOO_SMALL; if (ipv6_addr_is_multicast(&msg->target)) { ND_PRINTK(2, warn, "NA: target address is multicast\n"); - return; + return reason; } if (ipv6_addr_is_multicast(daddr) && msg->icmph.icmp6_solicited) { ND_PRINTK(2, warn, "NA: solicited NA is multicasted\n"); - return; + return reason; } /* For some 802.11 wireless deployments (and possibly other networks), @@ -1024,18 +1024,17 @@ static void ndisc_recv_na(struct sk_buff *skb) */ if (!msg->icmph.icmp6_solicited && idev && idev->cnf.drop_unsolicited_na) - return; + return reason; + + if (!ndisc_parse_options(dev, msg->opt, ndoptlen, &ndopts)) + return SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS; - if (!ndisc_parse_options(dev, msg->opt, ndoptlen, &ndopts)) { - ND_PRINTK(2, warn, "NS: invalid ND option\n"); - return; - } if (ndopts.nd_opts_tgt_lladdr) { lladdr = ndisc_opt_addr_data(ndopts.nd_opts_tgt_lladdr, dev); if (!lladdr) { ND_PRINTK(2, warn, "NA: invalid link-layer address length\n"); - return; + return reason; } } ifp = ipv6_get_ifaddr(dev_net(dev), &msg->target, dev, 1); @@ -1043,7 +1042,7 @@ static void ndisc_recv_na(struct sk_buff *skb) if (skb->pkt_type != PACKET_LOOPBACK && (ifp->flags & IFA_F_TENTATIVE)) { addrconf_dad_failure(skb, ifp); - return; + return reason; } /* What should we make now? The advertisement is invalid, but ndisc specs say nothing @@ -1059,7 +1058,7 @@ static void ndisc_recv_na(struct sk_buff *skb) "NA: %pM advertised our address %pI6c on %s!\n", eth_hdr(skb)->h_source, &ifp->addr, ifp->idev->dev->name); in6_ifa_put(ifp); - return; + return reason; } neigh = neigh_lookup(&nd_tbl, &msg->target, dev); @@ -1120,13 +1119,14 @@ static void ndisc_recv_na(struct sk_buff *skb) */ rt6_clean_tohost(dev_net(dev), saddr); } - + reason = SKB_CONSUMED; out: neigh_release(neigh); } + return reason; } -static void ndisc_recv_rs(struct sk_buff *skb) +static enum skb_drop_reason ndisc_recv_rs(struct sk_buff *skb) { struct rs_msg *rs_msg = (struct rs_msg *)skb_transport_header(skb); unsigned long ndoptlen = skb->len - sizeof(*rs_msg); @@ -1135,14 +1135,15 @@ static void ndisc_recv_rs(struct sk_buff *skb) const struct in6_addr *saddr = &ipv6_hdr(skb)->saddr; struct ndisc_options ndopts; u8 *lladdr = NULL; + SKB_DR(reason); if (skb->len < sizeof(*rs_msg)) - return; + return SKB_DROP_REASON_PKT_TOO_SMALL; idev = __in6_dev_get(skb->dev); if (!idev) { ND_PRINTK(1, err, "RS: can't find in6 device\n"); - return; + return reason; } /* Don't accept RS if we're not in router mode */ @@ -1157,10 +1158,8 @@ static void ndisc_recv_rs(struct sk_buff *skb) goto out; /* Parse ND options */ - if (!ndisc_parse_options(skb->dev, rs_msg->opt, ndoptlen, &ndopts)) { - ND_PRINTK(2, notice, "NS: invalid ND option, ignored\n"); - goto out; - } + if (!ndisc_parse_options(skb->dev, rs_msg->opt, ndoptlen, &ndopts)) + return SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS; if (ndopts.nd_opts_src_lladdr) { lladdr = ndisc_opt_addr_data(ndopts.nd_opts_src_lladdr, @@ -1177,9 +1176,10 @@ static void ndisc_recv_rs(struct sk_buff *skb) NEIGH_UPDATE_F_OVERRIDE_ISROUTER, NDISC_ROUTER_SOLICITATION, &ndopts); neigh_release(neigh); + reason = SKB_CONSUMED; } out: - return; + return reason; } static void ndisc_ra_useropt(struct sk_buff *ra, struct nd_opt_hdr *opt) @@ -1228,20 +1228,21 @@ errout: rtnl_set_sk_err(net, RTNLGRP_ND_USEROPT, err); } -static void ndisc_router_discovery(struct sk_buff *skb) +static enum skb_drop_reason ndisc_router_discovery(struct sk_buff *skb) { struct ra_msg *ra_msg = (struct ra_msg *)skb_transport_header(skb); + bool send_ifinfo_notify = false; struct neighbour *neigh = NULL; - struct inet6_dev *in6_dev; + struct ndisc_options ndopts; struct fib6_info *rt = NULL; + struct inet6_dev *in6_dev; u32 defrtr_usr_metric; + unsigned int pref = 0; + __u32 old_if_flags; struct net *net; + SKB_DR(reason); int lifetime; - struct ndisc_options ndopts; int optlen; - unsigned int pref = 0; - __u32 old_if_flags; - bool send_ifinfo_notify = false; __u8 *opt = (__u8 *)(ra_msg + 1); @@ -1253,17 +1254,15 @@ static void ndisc_router_discovery(struct sk_buff *skb) __func__, skb->dev->name); if (!(ipv6_addr_type(&ipv6_hdr(skb)->saddr) & IPV6_ADDR_LINKLOCAL)) { ND_PRINTK(2, warn, "RA: source address is not link-local\n"); - return; - } - if (optlen < 0) { - ND_PRINTK(2, warn, "RA: packet too short\n"); - return; + return reason; } + if (optlen < 0) + return SKB_DROP_REASON_PKT_TOO_SMALL; #ifdef CONFIG_IPV6_NDISC_NODETYPE if (skb->ndisc_nodetype == NDISC_NODETYPE_HOST) { ND_PRINTK(2, warn, "RA: from host or unauthorized router\n"); - return; + return reason; } #endif @@ -1275,13 +1274,11 @@ static void ndisc_router_discovery(struct sk_buff *skb) if (!in6_dev) { ND_PRINTK(0, err, "RA: can't find inet6 device for %s\n", skb->dev->name); - return; + return reason; } - if (!ndisc_parse_options(skb->dev, opt, optlen, &ndopts)) { - ND_PRINTK(2, warn, "RA: invalid ND options\n"); - return; - } + if (!ndisc_parse_options(skb->dev, opt, optlen, &ndopts)) + return SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS; if (!ipv6_accept_ra(in6_dev)) { ND_PRINTK(2, info, @@ -1362,7 +1359,7 @@ static void ndisc_router_discovery(struct sk_buff *skb) "RA: %s got default router without neighbour\n", __func__); fib6_info_release(rt); - return; + return reason; } } /* Set default route metric as specified by user */ @@ -1387,7 +1384,7 @@ static void ndisc_router_discovery(struct sk_buff *skb) ND_PRINTK(0, err, "RA: %s failed to add default route\n", __func__); - return; + return reason; } neigh = ip6_neigh_lookup(&rt->fib6_nh->fib_nh_gw6, @@ -1398,7 +1395,7 @@ static void ndisc_router_discovery(struct sk_buff *skb) "RA: %s got default router without neighbour\n", __func__); fib6_info_release(rt); - return; + return reason; } neigh->flags |= NTF_ROUTER; } else if (rt && IPV6_EXTRACT_PREF(rt->fib6_flags) != pref) { @@ -1485,6 +1482,7 @@ skip_linkparms: NEIGH_UPDATE_F_OVERRIDE_ISROUTER| NEIGH_UPDATE_F_ISROUTER, NDISC_ROUTER_ADVERTISEMENT, &ndopts); + reason = SKB_CONSUMED; } if (!ipv6_accept_ra(in6_dev)) { @@ -1595,15 +1593,17 @@ out: fib6_info_release(rt); if (neigh) neigh_release(neigh); + return reason; } -static void ndisc_redirect_rcv(struct sk_buff *skb) +static enum skb_drop_reason ndisc_redirect_rcv(struct sk_buff *skb) { - u8 *hdr; - struct ndisc_options ndopts; struct rd_msg *msg = (struct rd_msg *)skb_transport_header(skb); u32 ndoptlen = skb_tail_pointer(skb) - (skb_transport_header(skb) + offsetof(struct rd_msg, opt)); + struct ndisc_options ndopts; + SKB_DR(reason); + u8 *hdr; #ifdef CONFIG_IPV6_NDISC_NODETYPE switch (skb->ndisc_nodetype) { @@ -1611,31 +1611,31 @@ static void ndisc_redirect_rcv(struct sk_buff *skb) case NDISC_NODETYPE_NODEFAULT: ND_PRINTK(2, warn, "Redirect: from host or unauthorized router\n"); - return; + return reason; } #endif if (!(ipv6_addr_type(&ipv6_hdr(skb)->saddr) & IPV6_ADDR_LINKLOCAL)) { ND_PRINTK(2, warn, "Redirect: source address is not link-local\n"); - return; + return reason; } if (!ndisc_parse_options(skb->dev, msg->opt, ndoptlen, &ndopts)) - return; + return SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS; if (!ndopts.nd_opts_rh) { ip6_redirect_no_header(skb, dev_net(skb->dev), skb->dev->ifindex); - return; + return reason; } hdr = (u8 *)ndopts.nd_opts_rh; hdr += 8; if (!pskb_pull(skb, hdr - skb_transport_header(skb))) - return; + return SKB_DROP_REASON_PKT_TOO_SMALL; - icmpv6_notify(skb, NDISC_REDIRECT, 0, 0); + return icmpv6_notify(skb, NDISC_REDIRECT, 0, 0); } static void ndisc_fill_redirect_hdr_option(struct sk_buff *skb, @@ -1781,8 +1781,9 @@ release: static void pndisc_redo(struct sk_buff *skb) { - ndisc_recv_ns(skb); - kfree_skb(skb); + enum skb_drop_reason reason = ndisc_recv_ns(skb); + + kfree_skb_reason(skb, reason); } static int ndisc_is_multicast(const void *pkey) @@ -1834,23 +1835,23 @@ enum skb_drop_reason ndisc_rcv(struct sk_buff *skb) switch (msg->icmph.icmp6_type) { case NDISC_NEIGHBOUR_SOLICITATION: memset(NEIGH_CB(skb), 0, sizeof(struct neighbour_cb)); - ndisc_recv_ns(skb); + reason = ndisc_recv_ns(skb); break; case NDISC_NEIGHBOUR_ADVERTISEMENT: - ndisc_recv_na(skb); + reason = ndisc_recv_na(skb); break; case NDISC_ROUTER_SOLICITATION: - ndisc_recv_rs(skb); + reason = ndisc_recv_rs(skb); break; case NDISC_ROUTER_ADVERTISEMENT: - ndisc_router_discovery(skb); + reason = ndisc_router_discovery(skb); break; case NDISC_REDIRECT: - ndisc_redirect_rcv(skb); + reason = ndisc_redirect_rcv(skb); break; } diff --git a/net/ipv6/netfilter/nf_reject_ipv6.c b/net/ipv6/netfilter/nf_reject_ipv6.c index f61d4f18e1cf..58ccdb08c0fd 100644 --- a/net/ipv6/netfilter/nf_reject_ipv6.c +++ b/net/ipv6/netfilter/nf_reject_ipv6.c @@ -345,6 +345,7 @@ void nf_send_reset6(struct net *net, struct sock *sk, struct sk_buff *oldskb, nf_reject_ip6_tcphdr_put(nskb, oldskb, otcph, otcplen); nf_ct_attach(nskb, oldskb); + nf_ct_set_closing(skb_nfct(oldskb)); #if IS_ENABLED(CONFIG_BRIDGE_NETFILTER) /* If we use ip6_local_out for bridged traffic, the MAC source on diff --git a/net/mac802154/Makefile b/net/mac802154/Makefile index 4059295fdbf8..43d1347b37ee 100644 --- a/net/mac802154/Makefile +++ b/net/mac802154/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_MAC802154) += mac802154.o mac802154-objs := main.o rx.o tx.o mac_cmd.o mib.o \ - iface.o llsec.o util.o cfg.o trace.o + iface.o llsec.o util.o cfg.o scan.o trace.o CFLAGS_trace.o := -I$(src) diff --git a/net/mac802154/cfg.c b/net/mac802154/cfg.c index dc2d918fac68..5c3cb019f751 100644 --- a/net/mac802154/cfg.c +++ b/net/mac802154/cfg.c @@ -114,11 +114,15 @@ ieee802154_set_channel(struct wpan_phy *wpan_phy, u8 page, u8 channel) wpan_phy->current_channel == channel) return 0; + /* Refuse to change channels during scanning or beaconing */ + if (mac802154_is_scanning(local) || mac802154_is_beaconing(local)) + return -EBUSY; + ret = drv_set_channel(local, page, channel); if (!ret) { wpan_phy->current_page = page; wpan_phy->current_channel = channel; - ieee802154_configure_durations(wpan_phy); + ieee802154_configure_durations(wpan_phy, page, channel); } return ret; @@ -261,6 +265,56 @@ ieee802154_set_ackreq_default(struct wpan_phy *wpan_phy, return 0; } +static int mac802154_trigger_scan(struct wpan_phy *wpan_phy, + struct cfg802154_scan_request *request) +{ + struct ieee802154_sub_if_data *sdata; + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(request->wpan_dev); + + ASSERT_RTNL(); + + return mac802154_trigger_scan_locked(sdata, request); +} + +static int mac802154_abort_scan(struct wpan_phy *wpan_phy, + struct wpan_dev *wpan_dev) +{ + struct ieee802154_local *local = wpan_phy_priv(wpan_phy); + struct ieee802154_sub_if_data *sdata; + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(wpan_dev); + + ASSERT_RTNL(); + + return mac802154_abort_scan_locked(local, sdata); +} + +static int mac802154_send_beacons(struct wpan_phy *wpan_phy, + struct cfg802154_beacon_request *request) +{ + struct ieee802154_sub_if_data *sdata; + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(request->wpan_dev); + + ASSERT_RTNL(); + + return mac802154_send_beacons_locked(sdata, request); +} + +static int mac802154_stop_beacons(struct wpan_phy *wpan_phy, + struct wpan_dev *wpan_dev) +{ + struct ieee802154_local *local = wpan_phy_priv(wpan_phy); + struct ieee802154_sub_if_data *sdata; + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(wpan_dev); + + ASSERT_RTNL(); + + return mac802154_stop_beacons_locked(local, sdata); +} + #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL static void ieee802154_get_llsec_table(struct wpan_phy *wpan_phy, @@ -468,6 +522,10 @@ const struct cfg802154_ops mac802154_config_ops = { .set_max_frame_retries = ieee802154_set_max_frame_retries, .set_lbt_mode = ieee802154_set_lbt_mode, .set_ackreq_default = ieee802154_set_ackreq_default, + .trigger_scan = mac802154_trigger_scan, + .abort_scan = mac802154_abort_scan, + .send_beacons = mac802154_send_beacons, + .stop_beacons = mac802154_stop_beacons, #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL .get_llsec_table = ieee802154_get_llsec_table, .lock_llsec_table = ieee802154_lock_llsec_table, diff --git a/net/mac802154/ieee802154_i.h b/net/mac802154/ieee802154_i.h index 509e0172fe82..63bab99ed368 100644 --- a/net/mac802154/ieee802154_i.h +++ b/net/mac802154/ieee802154_i.h @@ -21,6 +21,11 @@ #include "llsec.h" +enum ieee802154_ongoing { + IEEE802154_IS_SCANNING = BIT(0), + IEEE802154_IS_BEACONING = BIT(1), +}; + /* mac802154 device private data */ struct ieee802154_local { struct ieee802154_hw hw; @@ -43,15 +48,32 @@ struct ieee802154_local { struct list_head interfaces; struct mutex iflist_mtx; - /* This one is used for scanning and other jobs not to be interfered - * with serial driver. - */ + /* Data related workqueue */ struct workqueue_struct *workqueue; + /* MAC commands related workqueue */ + struct workqueue_struct *mac_wq; struct hrtimer ifs_timer; + /* Scanning */ + u8 scan_page; + u8 scan_channel; + struct cfg802154_scan_request __rcu *scan_req; + struct delayed_work scan_work; + + /* Beaconing */ + unsigned int beacon_interval; + struct ieee802154_beacon_frame beacon; + struct cfg802154_beacon_request __rcu *beacon_req; + struct delayed_work beacon_work; + + /* Asynchronous tasks */ + struct list_head rx_beacon_list; + struct work_struct rx_beacon_work; + bool started; bool suspended; + unsigned long ongoing; struct tasklet_struct tasklet; struct sk_buff_head skb_queue; @@ -141,10 +163,16 @@ int ieee802154_mlme_op_pre(struct ieee802154_local *local); int ieee802154_mlme_tx(struct ieee802154_local *local, struct ieee802154_sub_if_data *sdata, struct sk_buff *skb); +int ieee802154_mlme_tx_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + struct sk_buff *skb); void ieee802154_mlme_op_post(struct ieee802154_local *local); int ieee802154_mlme_tx_one(struct ieee802154_local *local, struct ieee802154_sub_if_data *sdata, struct sk_buff *skb); +int ieee802154_mlme_tx_one_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + struct sk_buff *skb); netdev_tx_t ieee802154_monitor_start_xmit(struct sk_buff *skb, struct net_device *dev); netdev_tx_t @@ -220,6 +248,33 @@ void mac802154_unlock_table(struct net_device *dev); int mac802154_wpan_update_llsec(struct net_device *dev); +/* PAN management handling */ +void mac802154_scan_worker(struct work_struct *work); +int mac802154_trigger_scan_locked(struct ieee802154_sub_if_data *sdata, + struct cfg802154_scan_request *request); +int mac802154_abort_scan_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata); +int mac802154_process_beacon(struct ieee802154_local *local, + struct sk_buff *skb, + u8 page, u8 channel); +void mac802154_rx_beacon_worker(struct work_struct *work); + +static inline bool mac802154_is_scanning(struct ieee802154_local *local) +{ + return test_bit(IEEE802154_IS_SCANNING, &local->ongoing); +} + +void mac802154_beacon_worker(struct work_struct *work); +int mac802154_send_beacons_locked(struct ieee802154_sub_if_data *sdata, + struct cfg802154_beacon_request *request); +int mac802154_stop_beacons_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata); + +static inline bool mac802154_is_beaconing(struct ieee802154_local *local) +{ + return test_bit(IEEE802154_IS_BEACONING, &local->ongoing); +} + /* interface handling */ int ieee802154_iface_init(void); void ieee802154_iface_exit(void); diff --git a/net/mac802154/iface.c b/net/mac802154/iface.c index ac0b28025fb0..c0e2da5072be 100644 --- a/net/mac802154/iface.c +++ b/net/mac802154/iface.c @@ -302,6 +302,12 @@ static int mac802154_slave_close(struct net_device *dev) ASSERT_RTNL(); + if (mac802154_is_scanning(local)) + mac802154_abort_scan_locked(local, sdata); + + if (mac802154_is_beaconing(local)) + mac802154_stop_beacons_locked(local, sdata); + netif_stop_queue(dev); local->open_count--; diff --git a/net/mac802154/llsec.c b/net/mac802154/llsec.c index 55550ead2ced..8d2eabc71bbe 100644 --- a/net/mac802154/llsec.c +++ b/net/mac802154/llsec.c @@ -707,7 +707,10 @@ int mac802154_llsec_encrypt(struct mac802154_llsec *sec, struct sk_buff *skb) hlen = ieee802154_hdr_pull(skb, &hdr); - if (hlen < 0 || hdr.fc.type != IEEE802154_FC_TYPE_DATA) + /* TODO: control frames security support */ + if (hlen < 0 || + (hdr.fc.type != IEEE802154_FC_TYPE_DATA && + hdr.fc.type != IEEE802154_FC_TYPE_BEACON)) return -EINVAL; if (!hdr.fc.security_enabled || diff --git a/net/mac802154/main.c b/net/mac802154/main.c index 3ed31daf7b9c..ee23e234b998 100644 --- a/net/mac802154/main.c +++ b/net/mac802154/main.c @@ -89,6 +89,7 @@ ieee802154_alloc_hw(size_t priv_data_len, const struct ieee802154_ops *ops) local->ops = ops; INIT_LIST_HEAD(&local->interfaces); + INIT_LIST_HEAD(&local->rx_beacon_list); mutex_init(&local->iflist_mtx); tasklet_setup(&local->tasklet, ieee802154_tasklet_handler); @@ -96,6 +97,9 @@ ieee802154_alloc_hw(size_t priv_data_len, const struct ieee802154_ops *ops) skb_queue_head_init(&local->skb_queue); INIT_WORK(&local->sync_tx_work, ieee802154_xmit_sync_worker); + INIT_DELAYED_WORK(&local->scan_work, mac802154_scan_worker); + INIT_WORK(&local->rx_beacon_work, mac802154_rx_beacon_worker); + INIT_DELAYED_WORK(&local->beacon_work, mac802154_beacon_worker); /* init supported flags with 802.15.4 default ranges */ phy->supported.max_minbe = 8; @@ -113,32 +117,33 @@ ieee802154_alloc_hw(size_t priv_data_len, const struct ieee802154_ops *ops) } EXPORT_SYMBOL(ieee802154_alloc_hw); -void ieee802154_configure_durations(struct wpan_phy *phy) +void ieee802154_configure_durations(struct wpan_phy *phy, + unsigned int page, unsigned int channel) { u32 duration = 0; - switch (phy->current_page) { + switch (page) { case 0: - if (BIT(phy->current_channel) & 0x1) + if (BIT(channel) & 0x1) /* 868 MHz BPSK 802.15.4-2003: 20 ksym/s */ duration = 50 * NSEC_PER_USEC; - else if (BIT(phy->current_channel) & 0x7FE) + else if (BIT(channel) & 0x7FE) /* 915 MHz BPSK 802.15.4-2003: 40 ksym/s */ duration = 25 * NSEC_PER_USEC; - else if (BIT(phy->current_channel) & 0x7FFF800) + else if (BIT(channel) & 0x7FFF800) /* 2400 MHz O-QPSK 802.15.4-2006: 62.5 ksym/s */ duration = 16 * NSEC_PER_USEC; break; case 2: - if (BIT(phy->current_channel) & 0x1) + if (BIT(channel) & 0x1) /* 868 MHz O-QPSK 802.15.4-2006: 25 ksym/s */ duration = 40 * NSEC_PER_USEC; - else if (BIT(phy->current_channel) & 0x7FE) + else if (BIT(channel) & 0x7FE) /* 915 MHz O-QPSK 802.15.4-2006: 62.5 ksym/s */ duration = 16 * NSEC_PER_USEC; break; case 3: - if (BIT(phy->current_channel) & 0x3FFF) + if (BIT(channel) & 0x3FFF) /* 2.4 GHz CSS 802.15.4a-2007: 1/6 Msym/s */ duration = 6 * NSEC_PER_USEC; break; @@ -184,6 +189,7 @@ static void ieee802154_setup_wpan_phy_pib(struct wpan_phy *wpan_phy) int ieee802154_register_hw(struct ieee802154_hw *hw) { struct ieee802154_local *local = hw_to_local(hw); + char mac_wq_name[IFNAMSIZ + 10] = {}; struct net_device *dev; int rc = -ENOSYS; @@ -194,6 +200,13 @@ int ieee802154_register_hw(struct ieee802154_hw *hw) goto out; } + snprintf(mac_wq_name, IFNAMSIZ + 10, "%s-mac-cmds", wpan_phy_name(local->phy)); + local->mac_wq = create_singlethread_workqueue(mac_wq_name); + if (!local->mac_wq) { + rc = -ENOMEM; + goto out_wq; + } + hrtimer_init(&local->ifs_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); local->ifs_timer.function = ieee802154_xmit_ifs_timer; @@ -201,7 +214,8 @@ int ieee802154_register_hw(struct ieee802154_hw *hw) ieee802154_setup_wpan_phy_pib(local->phy); - ieee802154_configure_durations(local->phy); + ieee802154_configure_durations(local->phy, local->phy->current_page, + local->phy->current_channel); if (!(hw->flags & IEEE802154_HW_CSMA_PARAMS)) { local->phy->supported.min_csma_backoffs = 4; @@ -222,7 +236,7 @@ int ieee802154_register_hw(struct ieee802154_hw *hw) rc = wpan_phy_register(local->phy); if (rc < 0) - goto out_wq; + goto out_mac_wq; rtnl_lock(); @@ -241,6 +255,8 @@ int ieee802154_register_hw(struct ieee802154_hw *hw) out_phy: wpan_phy_unregister(local->phy); +out_mac_wq: + destroy_workqueue(local->mac_wq); out_wq: destroy_workqueue(local->workqueue); out: @@ -261,6 +277,7 @@ void ieee802154_unregister_hw(struct ieee802154_hw *hw) rtnl_unlock(); + destroy_workqueue(local->mac_wq); destroy_workqueue(local->workqueue); wpan_phy_unregister(local->phy); } diff --git a/net/mac802154/rx.c b/net/mac802154/rx.c index 97bb4401dd3e..da0628ee3c89 100644 --- a/net/mac802154/rx.c +++ b/net/mac802154/rx.c @@ -29,12 +29,31 @@ static int ieee802154_deliver_skb(struct sk_buff *skb) return netif_receive_skb(skb); } +void mac802154_rx_beacon_worker(struct work_struct *work) +{ + struct ieee802154_local *local = + container_of(work, struct ieee802154_local, rx_beacon_work); + struct cfg802154_mac_pkt *mac_pkt; + + mac_pkt = list_first_entry_or_null(&local->rx_beacon_list, + struct cfg802154_mac_pkt, node); + if (!mac_pkt) + return; + + mac802154_process_beacon(local, mac_pkt->skb, mac_pkt->page, mac_pkt->channel); + + list_del(&mac_pkt->node); + kfree_skb(mac_pkt->skb); + kfree(mac_pkt); +} + static int ieee802154_subif_frame(struct ieee802154_sub_if_data *sdata, struct sk_buff *skb, const struct ieee802154_hdr *hdr) { - struct wpan_dev *wpan_dev = &sdata->wpan_dev; struct wpan_phy *wpan_phy = sdata->local->hw.phy; + struct wpan_dev *wpan_dev = &sdata->wpan_dev; + struct cfg802154_mac_pkt *mac_pkt; __le16 span, sshort; int rc; @@ -106,6 +125,21 @@ ieee802154_subif_frame(struct ieee802154_sub_if_data *sdata, switch (mac_cb(skb)->type) { case IEEE802154_FC_TYPE_BEACON: + dev_dbg(&sdata->dev->dev, "BEACON received\n"); + if (!mac802154_is_scanning(sdata->local)) + goto fail; + + mac_pkt = kzalloc(sizeof(*mac_pkt), GFP_ATOMIC); + if (!mac_pkt) + goto fail; + + mac_pkt->skb = skb_get(skb); + mac_pkt->sdata = sdata; + mac_pkt->page = sdata->local->scan_page; + mac_pkt->channel = sdata->local->scan_channel; + list_add_tail(&mac_pkt->node, &sdata->local->rx_beacon_list); + queue_work(sdata->local->mac_wq, &sdata->local->rx_beacon_work); + return NET_RX_SUCCESS; case IEEE802154_FC_TYPE_ACK: case IEEE802154_FC_TYPE_MAC_CMD: goto fail; diff --git a/net/mac802154/scan.c b/net/mac802154/scan.c new file mode 100644 index 000000000000..9b0933a185eb --- /dev/null +++ b/net/mac802154/scan.c @@ -0,0 +1,456 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * IEEE 802.15.4 scanning management + * + * Copyright (C) 2021 Qorvo US, Inc + * Authors: + * - David Girault <david.girault@qorvo.com> + * - Miquel Raynal <miquel.raynal@bootlin.com> + */ + +#include <linux/module.h> +#include <linux/rtnetlink.h> +#include <net/mac802154.h> + +#include "ieee802154_i.h" +#include "driver-ops.h" +#include "../ieee802154/nl802154.h" + +#define IEEE802154_BEACON_MHR_SZ 13 +#define IEEE802154_BEACON_PL_SZ 4 +#define IEEE802154_BEACON_SKB_SZ (IEEE802154_BEACON_MHR_SZ + \ + IEEE802154_BEACON_PL_SZ) + +/* mac802154_scan_cleanup_locked() must be called upon scan completion or abort. + * - Completions are asynchronous, not locked by the rtnl and decided by the + * scan worker. + * - Aborts are decided by userspace, and locked by the rtnl. + * + * Concurrent modifications to the PHY, the interfaces or the hardware is in + * general prevented by the rtnl. So in most cases we don't need additional + * protection. + * + * However, the scan worker get's triggered without anybody noticing and thus we + * must ensure the presence of the devices as well as data consistency: + * - The sub-interface and device driver module get both their reference + * counters incremented whenever we start a scan, so they cannot disappear + * during operation. + * - Data consistency is achieved by the use of rcu protected pointers. + */ +static int mac802154_scan_cleanup_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + bool aborted) +{ + struct wpan_dev *wpan_dev = &sdata->wpan_dev; + struct wpan_phy *wpan_phy = local->phy; + struct cfg802154_scan_request *request; + u8 arg; + + /* Prevent any further use of the scan request */ + clear_bit(IEEE802154_IS_SCANNING, &local->ongoing); + cancel_delayed_work(&local->scan_work); + request = rcu_replace_pointer(local->scan_req, NULL, 1); + if (!request) + return 0; + kfree_rcu(request); + + /* Advertize first, while we know the devices cannot be removed */ + if (aborted) + arg = NL802154_SCAN_DONE_REASON_ABORTED; + else + arg = NL802154_SCAN_DONE_REASON_FINISHED; + nl802154_scan_done(wpan_phy, wpan_dev, arg); + + /* Cleanup software stack */ + ieee802154_mlme_op_post(local); + + /* Set the hardware back in its original state */ + drv_set_channel(local, wpan_phy->current_page, + wpan_phy->current_channel); + ieee802154_configure_durations(wpan_phy, wpan_phy->current_page, + wpan_phy->current_channel); + drv_stop(local); + synchronize_net(); + sdata->required_filtering = sdata->iface_default_filtering; + drv_start(local, sdata->required_filtering, &local->addr_filt); + + return 0; +} + +int mac802154_abort_scan_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata) +{ + ASSERT_RTNL(); + + if (!mac802154_is_scanning(local)) + return -ESRCH; + + return mac802154_scan_cleanup_locked(local, sdata, true); +} + +static unsigned int mac802154_scan_get_channel_time(u8 duration_order, + u8 symbol_duration) +{ + u64 base_super_frame_duration = (u64)symbol_duration * + IEEE802154_SUPERFRAME_PERIOD * IEEE802154_SLOT_PERIOD; + + return usecs_to_jiffies(base_super_frame_duration * + (BIT(duration_order) + 1)); +} + +static void mac802154_flush_queued_beacons(struct ieee802154_local *local) +{ + struct cfg802154_mac_pkt *mac_pkt, *tmp; + + list_for_each_entry_safe(mac_pkt, tmp, &local->rx_beacon_list, node) { + list_del(&mac_pkt->node); + kfree_skb(mac_pkt->skb); + kfree(mac_pkt); + } +} + +static void +mac802154_scan_get_next_channel(struct ieee802154_local *local, + struct cfg802154_scan_request *scan_req, + u8 *channel) +{ + (*channel)++; + *channel = find_next_bit((const unsigned long *)&scan_req->channels, + IEEE802154_MAX_CHANNEL + 1, + *channel); +} + +static int mac802154_scan_find_next_chan(struct ieee802154_local *local, + struct cfg802154_scan_request *scan_req, + u8 page, u8 *channel) +{ + mac802154_scan_get_next_channel(local, scan_req, channel); + if (*channel > IEEE802154_MAX_CHANNEL) + return -EINVAL; + + return 0; +} + +void mac802154_scan_worker(struct work_struct *work) +{ + struct ieee802154_local *local = + container_of(work, struct ieee802154_local, scan_work.work); + struct cfg802154_scan_request *scan_req; + struct ieee802154_sub_if_data *sdata; + unsigned int scan_duration = 0; + struct wpan_phy *wpan_phy; + u8 scan_req_duration; + u8 page, channel; + int ret; + + /* Ensure the device receiver is turned off when changing channels + * because there is no atomic way to change the channel and know on + * which one a beacon might have been received. + */ + drv_stop(local); + synchronize_net(); + mac802154_flush_queued_beacons(local); + + rcu_read_lock(); + scan_req = rcu_dereference(local->scan_req); + if (unlikely(!scan_req)) { + rcu_read_unlock(); + return; + } + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(scan_req->wpan_dev); + + /* Wait an arbitrary amount of time in case we cannot use the device */ + if (local->suspended || !ieee802154_sdata_running(sdata)) { + rcu_read_unlock(); + queue_delayed_work(local->mac_wq, &local->scan_work, + msecs_to_jiffies(1000)); + return; + } + + wpan_phy = scan_req->wpan_phy; + scan_req_duration = scan_req->duration; + + /* Look for the next valid chan */ + page = local->scan_page; + channel = local->scan_channel; + do { + ret = mac802154_scan_find_next_chan(local, scan_req, page, &channel); + if (ret) { + rcu_read_unlock(); + goto end_scan; + } + } while (!ieee802154_chan_is_valid(scan_req->wpan_phy, page, channel)); + + rcu_read_unlock(); + + /* Bypass the stack on purpose when changing the channel */ + rtnl_lock(); + ret = drv_set_channel(local, page, channel); + rtnl_unlock(); + if (ret) { + dev_err(&sdata->dev->dev, + "Channel change failure during scan, aborting (%d)\n", ret); + goto end_scan; + } + + local->scan_page = page; + local->scan_channel = channel; + + rtnl_lock(); + ret = drv_start(local, IEEE802154_FILTERING_3_SCAN, &local->addr_filt); + rtnl_unlock(); + if (ret) { + dev_err(&sdata->dev->dev, + "Restarting failure after channel change, aborting (%d)\n", ret); + goto end_scan; + } + + ieee802154_configure_durations(wpan_phy, page, channel); + scan_duration = mac802154_scan_get_channel_time(scan_req_duration, + wpan_phy->symbol_duration); + dev_dbg(&sdata->dev->dev, + "Scan page %u channel %u for %ums\n", + page, channel, jiffies_to_msecs(scan_duration)); + queue_delayed_work(local->mac_wq, &local->scan_work, scan_duration); + return; + +end_scan: + rtnl_lock(); + mac802154_scan_cleanup_locked(local, sdata, false); + rtnl_unlock(); +} + +int mac802154_trigger_scan_locked(struct ieee802154_sub_if_data *sdata, + struct cfg802154_scan_request *request) +{ + struct ieee802154_local *local = sdata->local; + + ASSERT_RTNL(); + + if (mac802154_is_scanning(local)) + return -EBUSY; + + /* TODO: support other scanning type */ + if (request->type != NL802154_SCAN_PASSIVE) + return -EOPNOTSUPP; + + /* Store scanning parameters */ + rcu_assign_pointer(local->scan_req, request); + + /* Software scanning requires to set promiscuous mode, so we need to + * pause the Tx queue during the entire operation. + */ + ieee802154_mlme_op_pre(local); + + sdata->required_filtering = IEEE802154_FILTERING_3_SCAN; + local->scan_page = request->page; + local->scan_channel = -1; + set_bit(IEEE802154_IS_SCANNING, &local->ongoing); + + nl802154_scan_started(request->wpan_phy, request->wpan_dev); + + queue_delayed_work(local->mac_wq, &local->scan_work, 0); + + return 0; +} + +int mac802154_process_beacon(struct ieee802154_local *local, + struct sk_buff *skb, + u8 page, u8 channel) +{ + struct ieee802154_beacon_hdr *bh = (void *)skb->data; + struct ieee802154_addr *src = &mac_cb(skb)->source; + struct cfg802154_scan_request *scan_req; + struct ieee802154_coord_desc desc; + + if (skb->len != sizeof(*bh)) + return -EINVAL; + + if (unlikely(src->mode == IEEE802154_ADDR_NONE)) + return -EINVAL; + + dev_dbg(&skb->dev->dev, + "BEACON received on page %u channel %u\n", + page, channel); + + memcpy(&desc.addr, src, sizeof(desc.addr)); + desc.page = page; + desc.channel = channel; + desc.link_quality = mac_cb(skb)->lqi; + desc.superframe_spec = get_unaligned_le16(skb->data); + desc.gts_permit = bh->gts_permit; + + trace_802154_scan_event(&desc); + + rcu_read_lock(); + scan_req = rcu_dereference(local->scan_req); + if (likely(scan_req)) + nl802154_scan_event(scan_req->wpan_phy, scan_req->wpan_dev, &desc); + rcu_read_unlock(); + + return 0; +} + +static int mac802154_transmit_beacon(struct ieee802154_local *local, + struct wpan_dev *wpan_dev) +{ + struct cfg802154_beacon_request *beacon_req; + struct ieee802154_sub_if_data *sdata; + struct sk_buff *skb; + int ret; + + /* Update the sequence number */ + local->beacon.mhr.seq = atomic_inc_return(&wpan_dev->bsn) & 0xFF; + + skb = alloc_skb(IEEE802154_BEACON_SKB_SZ, GFP_KERNEL); + if (!skb) + return -ENOBUFS; + + rcu_read_lock(); + beacon_req = rcu_dereference(local->beacon_req); + if (unlikely(!beacon_req)) { + rcu_read_unlock(); + kfree_skb(skb); + return -EINVAL; + } + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(beacon_req->wpan_dev); + skb->dev = sdata->dev; + + rcu_read_unlock(); + + ret = ieee802154_beacon_push(skb, &local->beacon); + if (ret) { + kfree_skb(skb); + return ret; + } + + /* Using the MLME transmission helper for sending beacons is a bit + * overkill because we do not really care about the final outcome. + * + * Even though, going through the whole net stack with a regular + * dev_queue_xmit() is not relevant either because we want beacons to be + * sent "now" rather than go through the whole net stack scheduling + * (qdisc & co). + * + * Finally, using ieee802154_subif_start_xmit() would only be an option + * if we had a generic transmit helper which would acquire the + * HARD_TX_LOCK() to prevent buffer handling conflicts with regular + * packets. + * + * So for now we keep it simple and send beacons with our MLME helper, + * even if it stops the ieee802154 queue entirely during these + * transmissions, wich anyway does not have a huge impact on the + * performances given the current design of the stack. + */ + return ieee802154_mlme_tx(local, sdata, skb); +} + +void mac802154_beacon_worker(struct work_struct *work) +{ + struct ieee802154_local *local = + container_of(work, struct ieee802154_local, beacon_work.work); + struct cfg802154_beacon_request *beacon_req; + struct ieee802154_sub_if_data *sdata; + struct wpan_dev *wpan_dev; + int ret; + + rcu_read_lock(); + beacon_req = rcu_dereference(local->beacon_req); + if (unlikely(!beacon_req)) { + rcu_read_unlock(); + return; + } + + sdata = IEEE802154_WPAN_DEV_TO_SUB_IF(beacon_req->wpan_dev); + + /* Wait an arbitrary amount of time in case we cannot use the device */ + if (local->suspended || !ieee802154_sdata_running(sdata)) { + rcu_read_unlock(); + queue_delayed_work(local->mac_wq, &local->beacon_work, + msecs_to_jiffies(1000)); + return; + } + + wpan_dev = beacon_req->wpan_dev; + + rcu_read_unlock(); + + dev_dbg(&sdata->dev->dev, "Sending beacon\n"); + ret = mac802154_transmit_beacon(local, wpan_dev); + if (ret) + dev_err(&sdata->dev->dev, + "Beacon could not be transmitted (%d)\n", ret); + + queue_delayed_work(local->mac_wq, &local->beacon_work, + local->beacon_interval); +} + +int mac802154_stop_beacons_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata) +{ + struct wpan_dev *wpan_dev = &sdata->wpan_dev; + struct cfg802154_beacon_request *request; + + ASSERT_RTNL(); + + if (!mac802154_is_beaconing(local)) + return -ESRCH; + + clear_bit(IEEE802154_IS_BEACONING, &local->ongoing); + cancel_delayed_work(&local->beacon_work); + request = rcu_replace_pointer(local->beacon_req, NULL, 1); + if (!request) + return 0; + kfree_rcu(request); + + nl802154_beaconing_done(wpan_dev); + + return 0; +} + +int mac802154_send_beacons_locked(struct ieee802154_sub_if_data *sdata, + struct cfg802154_beacon_request *request) +{ + struct ieee802154_local *local = sdata->local; + + ASSERT_RTNL(); + + if (mac802154_is_beaconing(local)) + mac802154_stop_beacons_locked(local, sdata); + + /* Store beaconing parameters */ + rcu_assign_pointer(local->beacon_req, request); + + set_bit(IEEE802154_IS_BEACONING, &local->ongoing); + + memset(&local->beacon, 0, sizeof(local->beacon)); + local->beacon.mhr.fc.type = IEEE802154_FC_TYPE_BEACON; + local->beacon.mhr.fc.security_enabled = 0; + local->beacon.mhr.fc.frame_pending = 0; + local->beacon.mhr.fc.ack_request = 0; + local->beacon.mhr.fc.intra_pan = 0; + local->beacon.mhr.fc.dest_addr_mode = IEEE802154_NO_ADDRESSING; + local->beacon.mhr.fc.version = IEEE802154_2003_STD; + local->beacon.mhr.fc.source_addr_mode = IEEE802154_EXTENDED_ADDRESSING; + atomic_set(&request->wpan_dev->bsn, -1); + local->beacon.mhr.source.mode = IEEE802154_ADDR_LONG; + local->beacon.mhr.source.pan_id = request->wpan_dev->pan_id; + local->beacon.mhr.source.extended_addr = request->wpan_dev->extended_addr; + local->beacon.mac_pl.beacon_order = request->interval; + local->beacon.mac_pl.superframe_order = request->interval; + local->beacon.mac_pl.final_cap_slot = 0xf; + local->beacon.mac_pl.battery_life_ext = 0; + /* TODO: Fill this field depending on the coordinator capacity */ + local->beacon.mac_pl.pan_coordinator = 1; + local->beacon.mac_pl.assoc_permit = 1; + + /* Start the beacon work */ + local->beacon_interval = + mac802154_scan_get_channel_time(request->interval, + request->wpan_phy->symbol_duration); + queue_delayed_work(local->mac_wq, &local->beacon_work, 0); + + return 0; +} diff --git a/net/mac802154/tx.c b/net/mac802154/tx.c index 9d8d43cf1e64..2a6f1ed763c9 100644 --- a/net/mac802154/tx.c +++ b/net/mac802154/tx.c @@ -137,34 +137,37 @@ int ieee802154_mlme_op_pre(struct ieee802154_local *local) return ieee802154_sync_and_hold_queue(local); } -int ieee802154_mlme_tx(struct ieee802154_local *local, - struct ieee802154_sub_if_data *sdata, - struct sk_buff *skb) +int ieee802154_mlme_tx_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + struct sk_buff *skb) { - int ret; - /* Avoid possible calls to ->ndo_stop() when we asynchronously perform * MLME transmissions. */ - rtnl_lock(); + ASSERT_RTNL(); /* Ensure the device was not stopped, otherwise error out */ - if (!local->open_count) { - rtnl_unlock(); + if (!local->open_count) return -ENETDOWN; - } /* Warn if the ieee802154 core thinks MLME frames can be sent while the * net interface expects this cannot happen. */ - if (WARN_ON_ONCE(!netif_running(sdata->dev))) { - rtnl_unlock(); + if (WARN_ON_ONCE(!netif_running(sdata->dev))) return -ENETDOWN; - } ieee802154_tx(local, skb); - ret = ieee802154_sync_queue(local); + return ieee802154_sync_queue(local); +} + +int ieee802154_mlme_tx(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + struct sk_buff *skb) +{ + int ret; + rtnl_lock(); + ret = ieee802154_mlme_tx_locked(local, sdata, skb); rtnl_unlock(); return ret; @@ -188,6 +191,19 @@ int ieee802154_mlme_tx_one(struct ieee802154_local *local, return ret; } +int ieee802154_mlme_tx_one_locked(struct ieee802154_local *local, + struct ieee802154_sub_if_data *sdata, + struct sk_buff *skb) +{ + int ret; + + ieee802154_mlme_op_pre(local); + ret = ieee802154_mlme_tx_locked(local, sdata, skb); + ieee802154_mlme_op_post(local); + + return ret; +} + static bool ieee802154_queue_is_stopped(struct ieee802154_local *local) { return test_bit(WPAN_PHY_FLAG_STATE_QUEUE_STOPPED, &local->phy->flags); diff --git a/net/netfilter/core.c b/net/netfilter/core.c index 5a6705a0e4ec..b2fdbbed2b4b 100644 --- a/net/netfilter/core.c +++ b/net/netfilter/core.c @@ -702,6 +702,22 @@ void nf_conntrack_destroy(struct nf_conntrack *nfct) } EXPORT_SYMBOL(nf_conntrack_destroy); +void nf_ct_set_closing(struct nf_conntrack *nfct) +{ + const struct nf_ct_hook *ct_hook; + + if (!nfct) + return; + + rcu_read_lock(); + ct_hook = rcu_dereference(nf_ct_hook); + if (ct_hook) + ct_hook->set_closing(nfct); + + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(nf_ct_set_closing); + bool nf_ct_get_tuple_skb(struct nf_conntrack_tuple *dst_tuple, const struct sk_buff *skb) { diff --git a/net/netfilter/ipvs/ip_vs_est.c b/net/netfilter/ipvs/ip_vs_est.c index ce2a1549b304..c5970ba416ae 100644 --- a/net/netfilter/ipvs/ip_vs_est.c +++ b/net/netfilter/ipvs/ip_vs_est.c @@ -549,7 +549,7 @@ void ip_vs_stop_estimator(struct netns_ipvs *ipvs, struct ip_vs_stats *stats) __set_bit(row, kd->avail); if (!kd->tick_len[row]) { RCU_INIT_POINTER(kd->ticks[row], NULL); - kfree_rcu(td); + kfree_rcu(td, rcu_head); } kd->est_count--; if (kd->est_count) { diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c index 9a830573480e..70c4f892174e 100644 --- a/net/netfilter/nf_conntrack_core.c +++ b/net/netfilter/nf_conntrack_core.c @@ -2748,11 +2748,23 @@ err_cachep: return ret; } +static void nf_conntrack_set_closing(struct nf_conntrack *nfct) +{ + struct nf_conn *ct = nf_ct_to_nf_conn(nfct); + + switch (nf_ct_protonum(ct)) { + case IPPROTO_TCP: + nf_conntrack_tcp_set_closing(ct); + break; + } +} + static const struct nf_ct_hook nf_conntrack_hook = { .update = nf_conntrack_update, .destroy = nf_ct_destroy, .get_tuple_skb = nf_conntrack_get_tuple_skb, .attach = nf_conntrack_attach, + .set_closing = nf_conntrack_set_closing, }; void nf_conntrack_init_end(void) diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c index 16ee5ebe1ce1..4018acb1d674 100644 --- a/net/netfilter/nf_conntrack_proto_tcp.c +++ b/net/netfilter/nf_conntrack_proto_tcp.c @@ -911,6 +911,41 @@ static bool tcp_can_early_drop(const struct nf_conn *ct) return false; } +void nf_conntrack_tcp_set_closing(struct nf_conn *ct) +{ + enum tcp_conntrack old_state; + const unsigned int *timeouts; + u32 timeout; + + if (!nf_ct_is_confirmed(ct)) + return; + + spin_lock_bh(&ct->lock); + old_state = ct->proto.tcp.state; + ct->proto.tcp.state = TCP_CONNTRACK_CLOSE; + + if (old_state == TCP_CONNTRACK_CLOSE || + test_bit(IPS_FIXED_TIMEOUT_BIT, &ct->status)) { + spin_unlock_bh(&ct->lock); + return; + } + + timeouts = nf_ct_timeout_lookup(ct); + if (!timeouts) { + const struct nf_tcp_net *tn; + + tn = nf_tcp_pernet(nf_ct_net(ct)); + timeouts = tn->timeouts; + } + + timeout = timeouts[TCP_CONNTRACK_CLOSE]; + WRITE_ONCE(ct->timeout, timeout + nfct_time_stamp); + + spin_unlock_bh(&ct->lock); + + nf_conntrack_event_cache(IPCT_PROTOINFO, ct); +} + static void nf_ct_tcp_state_reset(struct ip_ct_tcp_state *state) { state->td_end = 0; diff --git a/net/netfilter/nf_conntrack_proto_udp.c b/net/netfilter/nf_conntrack_proto_udp.c index 6b9206635b24..0030fbe8885c 100644 --- a/net/netfilter/nf_conntrack_proto_udp.c +++ b/net/netfilter/nf_conntrack_proto_udp.c @@ -104,7 +104,7 @@ int nf_conntrack_udp_packet(struct nf_conn *ct, /* If we've seen traffic both ways, this is some kind of UDP * stream. Set Assured. */ - if (status & IPS_SEEN_REPLY_BIT) { + if (status & IPS_SEEN_REPLY) { unsigned long extra = timeouts[UDP_CT_UNREPLIED]; bool stream = false; diff --git a/net/netfilter/nf_conntrack_standalone.c b/net/netfilter/nf_conntrack_standalone.c index 460294bd4b60..57f6724c99a7 100644 --- a/net/netfilter/nf_conntrack_standalone.c +++ b/net/netfilter/nf_conntrack_standalone.c @@ -275,7 +275,7 @@ static const char* l4proto_name(u16 proto) return "unknown"; } -static unsigned int +static void seq_print_acct(struct seq_file *s, const struct nf_conn *ct, int dir) { struct nf_conn_acct *acct; @@ -283,14 +283,12 @@ seq_print_acct(struct seq_file *s, const struct nf_conn *ct, int dir) acct = nf_conn_acct_find(ct); if (!acct) - return 0; + return; counter = acct->counter; seq_printf(s, "packets=%llu bytes=%llu ", (unsigned long long)atomic64_read(&counter[dir].packets), (unsigned long long)atomic64_read(&counter[dir].bytes)); - - return 0; } /* return 0 on success, 1 in case of error */ @@ -342,8 +340,7 @@ static int ct_seq_show(struct seq_file *s, void *v) if (seq_has_overflowed(s)) goto release; - if (seq_print_acct(s, ct, IP_CT_DIR_ORIGINAL)) - goto release; + seq_print_acct(s, ct, IP_CT_DIR_ORIGINAL); if (!(test_bit(IPS_SEEN_REPLY_BIT, &ct->status))) seq_puts(s, "[UNREPLIED] "); @@ -352,8 +349,7 @@ static int ct_seq_show(struct seq_file *s, void *v) ct_show_zone(s, ct, NF_CT_ZONE_DIR_REPL); - if (seq_print_acct(s, ct, IP_CT_DIR_REPLY)) - goto release; + seq_print_acct(s, ct, IP_CT_DIR_REPLY); if (test_bit(IPS_HW_OFFLOAD_BIT, &ct->status)) seq_puts(s, "[HW_OFFLOAD] "); diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index 974b95dece1d..d73edbd4eec4 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -3724,7 +3724,7 @@ static int nf_tables_delrule(struct sk_buff *skb, const struct nfnl_info *info, chain = nft_chain_lookup(net, table, nla[NFTA_RULE_CHAIN], genmask); if (IS_ERR(chain)) { - if (PTR_ERR(rule) == -ENOENT && + if (PTR_ERR(chain) == -ENOENT && NFNL_MSG_TYPE(info->nlh->nlmsg_type) == NFT_MSG_DESTROYRULE) return 0; @@ -7023,6 +7023,9 @@ static int nf_tables_newobj(struct sk_buff *skb, const struct nfnl_info *info, return -EOPNOTSUPP; type = __nft_obj_type_get(objtype); + if (WARN_ON_ONCE(!type)) + return -ENOENT; + nft_ctx_init(&ctx, net, skb, info->nlh, family, table, NULL, nla); return nf_tables_updobj(&ctx, type, nla[NFTA_OBJ_DATA], obj); diff --git a/net/openvswitch/flow.c b/net/openvswitch/flow.c index 416976f70322..33b21a0c0548 100644 --- a/net/openvswitch/flow.c +++ b/net/openvswitch/flow.c @@ -1041,7 +1041,8 @@ int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info, #if IS_ENABLED(CONFIG_NET_TC_SKB_EXT) if (tc_skb_ext_tc_enabled()) { tc_ext = skb_ext_find(skb, TC_SKB_EXT); - key->recirc_id = tc_ext ? tc_ext->chain : 0; + key->recirc_id = tc_ext && !tc_ext->act_miss ? + tc_ext->chain : 0; OVS_CB(skb)->mru = tc_ext ? tc_ext->mru : 0; post_ct = tc_ext ? tc_ext->post_ct : false; post_ct_snat = post_ct ? tc_ext->post_ct_snat : false; diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c index d68848fce51f..030d64f282f3 100644 --- a/net/rxrpc/input.c +++ b/net/rxrpc/input.c @@ -606,7 +606,7 @@ static void rxrpc_input_data(struct rxrpc_call *call, struct sk_buff *skb) rxrpc_proto_abort(call, sp->hdr.seq, rxrpc_badmsg_bad_jumbo); goto out_notify; } - skb = NULL; + return; out_notify: trace_rxrpc_notify_socket(call->debug_id, serial); diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c index 76eb2b9cd936..a482f88c5fc5 100644 --- a/net/rxrpc/recvmsg.c +++ b/net/rxrpc/recvmsg.c @@ -334,10 +334,23 @@ try_again: /* Find the next call and dequeue it if we're not just peeking. If we * do dequeue it, that comes with a ref that we will need to release. + * We also want to weed out calls that got requeued whilst we were + * shovelling data out. */ spin_lock(&rx->recvmsg_lock); l = rx->recvmsg_q.next; call = list_entry(l, struct rxrpc_call, recvmsg_link); + + if (!rxrpc_call_is_complete(call) && + skb_queue_empty(&call->recvmsg_queue)) { + list_del_init(&call->recvmsg_link); + spin_unlock(&rx->recvmsg_lock); + release_sock(&rx->sk); + trace_rxrpc_recvmsg(call->debug_id, rxrpc_recvmsg_unqueue, 0); + rxrpc_put_call(call, rxrpc_call_put_recvmsg); + goto try_again; + } + if (!(flags & MSG_PEEK)) list_del_init(&call->recvmsg_link); else @@ -402,7 +415,8 @@ try_again: if (rxrpc_call_has_failed(call)) goto call_failed; - rxrpc_notify_socket(call); + if (!skb_queue_empty(&call->recvmsg_queue)) + rxrpc_notify_socket(call); goto not_yet_complete; call_failed: diff --git a/net/sched/act_api.c b/net/sched/act_api.c index eda58b78da13..fce522886099 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -125,7 +125,7 @@ static void free_tcf(struct tc_action *p) free_percpu(p->cpu_bstats_hw); free_percpu(p->cpu_qstats); - tcf_set_action_cookie(&p->act_cookie, NULL); + tcf_set_action_cookie(&p->user_cookie, NULL); if (chain) tcf_chain_put_by_act(chain); @@ -268,7 +268,7 @@ static int tcf_action_offload_add_ex(struct tc_action *action, if (err) goto fl_err; - err = tc_setup_action(&fl_action->action, actions, extack); + err = tc_setup_action(&fl_action->action, actions, 0, extack); if (err) { NL_SET_ERR_MSG_MOD(extack, "Failed to setup tc actions for offload"); @@ -431,14 +431,14 @@ EXPORT_SYMBOL(tcf_idr_release); static size_t tcf_action_shared_attrs_size(const struct tc_action *act) { - struct tc_cookie *act_cookie; + struct tc_cookie *user_cookie; u32 cookie_len = 0; rcu_read_lock(); - act_cookie = rcu_dereference(act->act_cookie); + user_cookie = rcu_dereference(act->user_cookie); - if (act_cookie) - cookie_len = nla_total_size(act_cookie->len); + if (user_cookie) + cookie_len = nla_total_size(user_cookie->len); rcu_read_unlock(); return nla_total_size(0) /* action number nested */ @@ -488,7 +488,7 @@ tcf_action_dump_terse(struct sk_buff *skb, struct tc_action *a, bool from_act) goto nla_put_failure; rcu_read_lock(); - cookie = rcu_dereference(a->act_cookie); + cookie = rcu_dereference(a->user_cookie); if (cookie) { if (nla_put(skb, TCA_ACT_COOKIE, cookie->len, cookie->data)) { rcu_read_unlock(); @@ -1362,9 +1362,9 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, { bool police = flags & TCA_ACT_FLAGS_POLICE; struct nla_bitfield32 userflags = { 0, 0 }; + struct tc_cookie *user_cookie = NULL; u8 hw_stats = TCA_ACT_HW_STATS_ANY; struct nlattr *tb[TCA_ACT_MAX + 1]; - struct tc_cookie *cookie = NULL; struct tc_action *a; int err; @@ -1375,8 +1375,8 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, if (err < 0) return ERR_PTR(err); if (tb[TCA_ACT_COOKIE]) { - cookie = nla_memdup_cookie(tb); - if (!cookie) { + user_cookie = nla_memdup_cookie(tb); + if (!user_cookie) { NL_SET_ERR_MSG(extack, "No memory to generate TC cookie"); err = -ENOMEM; goto err_out; @@ -1402,7 +1402,7 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, *init_res = err; if (!police && tb[TCA_ACT_COOKIE]) - tcf_set_action_cookie(&a->act_cookie, cookie); + tcf_set_action_cookie(&a->user_cookie, user_cookie); if (!police) a->hw_stats = hw_stats; @@ -1410,9 +1410,9 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, return a; err_out: - if (cookie) { - kfree(cookie->data); - kfree(cookie); + if (user_cookie) { + kfree(user_cookie->data); + kfree(user_cookie); } return ERR_PTR(err); } diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index bfabc9c95fa9..3569e2c3660c 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -22,6 +22,7 @@ #include <linux/idr.h> #include <linux/jhash.h> #include <linux/rculist.h> +#include <linux/rhashtable.h> #include <net/net_namespace.h> #include <net/sock.h> #include <net/netlink.h> @@ -50,6 +51,109 @@ static LIST_HEAD(tcf_proto_base); /* Protects list of registered TC modules. It is pure SMP lock. */ static DEFINE_RWLOCK(cls_mod_lock); +static struct xarray tcf_exts_miss_cookies_xa; +struct tcf_exts_miss_cookie_node { + const struct tcf_chain *chain; + const struct tcf_proto *tp; + const struct tcf_exts *exts; + u32 chain_index; + u32 tp_prio; + u32 handle; + u32 miss_cookie_base; + struct rcu_head rcu; +}; + +/* Each tc action entry cookie will be comprised of 32bit miss_cookie_base + + * action index in the exts tc actions array. + */ +union tcf_exts_miss_cookie { + struct { + u32 miss_cookie_base; + u32 act_index; + }; + u64 miss_cookie; +}; + +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT) +static int +tcf_exts_miss_cookie_base_alloc(struct tcf_exts *exts, struct tcf_proto *tp, + u32 handle) +{ + struct tcf_exts_miss_cookie_node *n; + static u32 next; + int err; + + if (WARN_ON(!handle || !tp->ops->get_exts)) + return -EINVAL; + + n = kzalloc(sizeof(*n), GFP_KERNEL); + if (!n) + return -ENOMEM; + + n->chain_index = tp->chain->index; + n->chain = tp->chain; + n->tp_prio = tp->prio; + n->tp = tp; + n->exts = exts; + n->handle = handle; + + err = xa_alloc_cyclic(&tcf_exts_miss_cookies_xa, &n->miss_cookie_base, + n, xa_limit_32b, &next, GFP_KERNEL); + if (err) + goto err_xa_alloc; + + exts->miss_cookie_node = n; + return 0; + +err_xa_alloc: + kfree(n); + return err; +} + +static void tcf_exts_miss_cookie_base_destroy(struct tcf_exts *exts) +{ + struct tcf_exts_miss_cookie_node *n; + + if (!exts->miss_cookie_node) + return; + + n = exts->miss_cookie_node; + xa_erase(&tcf_exts_miss_cookies_xa, n->miss_cookie_base); + kfree_rcu(n, rcu); +} + +static struct tcf_exts_miss_cookie_node * +tcf_exts_miss_cookie_lookup(u64 miss_cookie, int *act_index) +{ + union tcf_exts_miss_cookie mc = { .miss_cookie = miss_cookie, }; + + *act_index = mc.act_index; + return xa_load(&tcf_exts_miss_cookies_xa, mc.miss_cookie_base); +} +#else /* IS_ENABLED(CONFIG_NET_TC_SKB_EXT) */ +static int +tcf_exts_miss_cookie_base_alloc(struct tcf_exts *exts, struct tcf_proto *tp, + u32 handle) +{ + return 0; +} + +static void tcf_exts_miss_cookie_base_destroy(struct tcf_exts *exts) +{ +} +#endif /* IS_ENABLED(CONFIG_NET_TC_SKB_EXT) */ + +static u64 tcf_exts_miss_cookie_get(u32 miss_cookie_base, int act_index) +{ + union tcf_exts_miss_cookie mc = { .act_index = act_index, }; + + if (!miss_cookie_base) + return 0; + + mc.miss_cookie_base = miss_cookie_base; + return mc.miss_cookie; +} + #ifdef CONFIG_NET_CLS_ACT DEFINE_STATIC_KEY_FALSE(tc_skb_ext_tc); EXPORT_SYMBOL(tc_skb_ext_tc); @@ -1549,6 +1653,8 @@ static inline int __tcf_classify(struct sk_buff *skb, const struct tcf_proto *orig_tp, struct tcf_result *res, bool compat_mode, + struct tcf_exts_miss_cookie_node *n, + int act_index, u32 *last_executed_chain) { #ifdef CONFIG_NET_CLS_ACT @@ -1560,13 +1666,36 @@ reclassify: #endif for (; tp; tp = rcu_dereference_bh(tp->next)) { __be16 protocol = skb_protocol(skb, false); - int err; + int err = 0; - if (tp->protocol != protocol && - tp->protocol != htons(ETH_P_ALL)) - continue; + if (n) { + struct tcf_exts *exts; + + if (n->tp_prio != tp->prio) + continue; + + /* We re-lookup the tp and chain based on index instead + * of having hard refs and locks to them, so do a sanity + * check if any of tp,chain,exts was replaced by the + * time we got here with a cookie from hardware. + */ + if (unlikely(n->tp != tp || n->tp->chain != n->chain || + !tp->ops->get_exts)) + return TC_ACT_SHOT; + + exts = tp->ops->get_exts(tp, n->handle); + if (unlikely(!exts || n->exts != exts)) + return TC_ACT_SHOT; - err = tc_classify(skb, tp, res); + n = NULL; + err = tcf_exts_exec_ex(skb, exts, act_index, res); + } else { + if (tp->protocol != protocol && + tp->protocol != htons(ETH_P_ALL)) + continue; + + err = tc_classify(skb, tp, res); + } #ifdef CONFIG_NET_CLS_ACT if (unlikely(err == TC_ACT_RECLASSIFY && !compat_mode)) { first_tp = orig_tp; @@ -1582,6 +1711,9 @@ reclassify: return err; } + if (unlikely(n)) + return TC_ACT_SHOT; + return TC_ACT_UNSPEC; /* signal: continue lookup */ #ifdef CONFIG_NET_CLS_ACT reset: @@ -1606,21 +1738,35 @@ int tcf_classify(struct sk_buff *skb, #if !IS_ENABLED(CONFIG_NET_TC_SKB_EXT) u32 last_executed_chain = 0; - return __tcf_classify(skb, tp, tp, res, compat_mode, + return __tcf_classify(skb, tp, tp, res, compat_mode, NULL, 0, &last_executed_chain); #else u32 last_executed_chain = tp ? tp->chain->index : 0; + struct tcf_exts_miss_cookie_node *n = NULL; const struct tcf_proto *orig_tp = tp; struct tc_skb_ext *ext; + int act_index = 0; int ret; if (block) { ext = skb_ext_find(skb, TC_SKB_EXT); - if (ext && ext->chain) { + if (ext && (ext->chain || ext->act_miss)) { struct tcf_chain *fchain; + u32 chain; + + if (ext->act_miss) { + n = tcf_exts_miss_cookie_lookup(ext->act_miss_cookie, + &act_index); + if (!n) + return TC_ACT_SHOT; - fchain = tcf_chain_lookup_rcu(block, ext->chain); + chain = n->chain_index; + } else { + chain = ext->chain; + } + + fchain = tcf_chain_lookup_rcu(block, chain); if (!fchain) return TC_ACT_SHOT; @@ -1632,7 +1778,7 @@ int tcf_classify(struct sk_buff *skb, } } - ret = __tcf_classify(skb, tp, orig_tp, res, compat_mode, + ret = __tcf_classify(skb, tp, orig_tp, res, compat_mode, n, act_index, &last_executed_chain); if (tc_skb_ext_tc_enabled()) { @@ -3056,9 +3202,48 @@ out: return skb->len; } +int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action, + int police, struct tcf_proto *tp, u32 handle, + bool use_action_miss) +{ + int err = 0; + +#ifdef CONFIG_NET_CLS_ACT + exts->type = 0; + exts->nr_actions = 0; + /* Note: we do not own yet a reference on net. + * This reference might be taken later from tcf_exts_get_net(). + */ + exts->net = net; + exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *), + GFP_KERNEL); + if (!exts->actions) + return -ENOMEM; +#endif + + exts->action = action; + exts->police = police; + + if (!use_action_miss) + return 0; + + err = tcf_exts_miss_cookie_base_alloc(exts, tp, handle); + if (err) + goto err_miss_alloc; + + return 0; + +err_miss_alloc: + tcf_exts_destroy(exts); + return err; +} +EXPORT_SYMBOL(tcf_exts_init_ex); + void tcf_exts_destroy(struct tcf_exts *exts) { #ifdef CONFIG_NET_CLS_ACT + tcf_exts_miss_cookie_base_destroy(exts); + if (exts->actions) { tcf_action_destroy(exts->actions, TCA_ACT_UNBIND); kfree(exts->actions); @@ -3490,28 +3675,28 @@ int tc_setup_cb_reoffload(struct tcf_block *block, struct tcf_proto *tp, } EXPORT_SYMBOL(tc_setup_cb_reoffload); -static int tcf_act_get_cookie(struct flow_action_entry *entry, - const struct tc_action *act) +static int tcf_act_get_user_cookie(struct flow_action_entry *entry, + const struct tc_action *act) { - struct tc_cookie *cookie; + struct tc_cookie *user_cookie; int err = 0; rcu_read_lock(); - cookie = rcu_dereference(act->act_cookie); - if (cookie) { - entry->cookie = flow_action_cookie_create(cookie->data, - cookie->len, - GFP_ATOMIC); - if (!entry->cookie) + user_cookie = rcu_dereference(act->user_cookie); + if (user_cookie) { + entry->user_cookie = flow_action_cookie_create(user_cookie->data, + user_cookie->len, + GFP_ATOMIC); + if (!entry->user_cookie) err = -ENOMEM; } rcu_read_unlock(); return err; } -static void tcf_act_put_cookie(struct flow_action_entry *entry) +static void tcf_act_put_user_cookie(struct flow_action_entry *entry) { - flow_action_cookie_destroy(entry->cookie); + flow_action_cookie_destroy(entry->user_cookie); } void tc_cleanup_offload_action(struct flow_action *flow_action) @@ -3520,7 +3705,7 @@ void tc_cleanup_offload_action(struct flow_action *flow_action) int i; flow_action_for_each(i, entry, flow_action) { - tcf_act_put_cookie(entry); + tcf_act_put_user_cookie(entry); if (entry->destructor) entry->destructor(entry->destructor_priv); } @@ -3547,6 +3732,7 @@ static int tc_setup_offload_act(struct tc_action *act, int tc_setup_action(struct flow_action *flow_action, struct tc_action *actions[], + u32 miss_cookie_base, struct netlink_ext_ack *extack) { int i, j, k, index, err = 0; @@ -3565,7 +3751,7 @@ int tc_setup_action(struct flow_action *flow_action, entry = &flow_action->entries[j]; spin_lock_bh(&act->tcfa_lock); - err = tcf_act_get_cookie(entry, act); + err = tcf_act_get_user_cookie(entry, act); if (err) goto err_out_locked; @@ -3577,7 +3763,9 @@ int tc_setup_action(struct flow_action *flow_action, for (k = 0; k < index ; k++) { entry[k].hw_stats = tc_act_hw_stats(act->hw_stats); entry[k].hw_index = act->tcfa_index; - entry[k].act_cookie = (unsigned long)act; + entry[k].cookie = (unsigned long)act; + entry[k].miss_cookie = + tcf_exts_miss_cookie_get(miss_cookie_base, i); } j += index; @@ -3600,10 +3788,15 @@ int tc_setup_offload_action(struct flow_action *flow_action, struct netlink_ext_ack *extack) { #ifdef CONFIG_NET_CLS_ACT + u32 miss_cookie_base; + if (!exts) return 0; - return tc_setup_action(flow_action, exts->actions, extack); + miss_cookie_base = exts->miss_cookie_node ? + exts->miss_cookie_node->miss_cookie_base : 0; + return tc_setup_action(flow_action, exts->actions, miss_cookie_base, + extack); #else return 0; #endif @@ -3771,6 +3964,8 @@ static int __init tc_filter_init(void) if (err) goto err_register_pernet_subsys; + xa_init_flags(&tcf_exts_miss_cookies_xa, XA_FLAGS_ALLOC1); + rtnl_register(PF_UNSPEC, RTM_NEWTFILTER, tc_new_tfilter, NULL, RTNL_FLAG_DOIT_UNLOCKED); rtnl_register(PF_UNSPEC, RTM_DELTFILTER, tc_del_tfilter, NULL, diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c index 885c95191ccf..e960a46b0520 100644 --- a/net/sched/cls_flower.c +++ b/net/sched/cls_flower.c @@ -529,6 +529,15 @@ static struct cls_fl_filter *__fl_get(struct cls_fl_head *head, u32 handle) return f; } +static struct tcf_exts *fl_get_exts(const struct tcf_proto *tp, u32 handle) +{ + struct cls_fl_head *head = rcu_dereference_bh(tp->root); + struct cls_fl_filter *f; + + f = idr_find(&head->handle_idr, handle); + return f ? &f->exts : NULL; +} + static int __fl_delete(struct tcf_proto *tp, struct cls_fl_filter *f, bool *last, bool rtnl_held, struct netlink_ext_ack *extack) @@ -2187,10 +2196,6 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, INIT_LIST_HEAD(&fnew->hw_list); refcount_set(&fnew->refcnt, 1); - err = tcf_exts_init(&fnew->exts, net, TCA_FLOWER_ACT, 0); - if (err < 0) - goto errout; - if (tb[TCA_FLOWER_FLAGS]) { fnew->flags = nla_get_u32(tb[TCA_FLOWER_FLAGS]); @@ -2200,15 +2205,46 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, } } + if (!fold) { + spin_lock(&tp->lock); + if (!handle) { + handle = 1; + err = idr_alloc_u32(&head->handle_idr, fnew, &handle, + INT_MAX, GFP_ATOMIC); + } else { + err = idr_alloc_u32(&head->handle_idr, fnew, &handle, + handle, GFP_ATOMIC); + + /* Filter with specified handle was concurrently + * inserted after initial check in cls_api. This is not + * necessarily an error if NLM_F_EXCL is not set in + * message flags. Returning EAGAIN will cause cls_api to + * try to update concurrently inserted rule. + */ + if (err == -ENOSPC) + err = -EAGAIN; + } + spin_unlock(&tp->lock); + + if (err) + goto errout; + } + fnew->handle = handle; + + err = tcf_exts_init_ex(&fnew->exts, net, TCA_FLOWER_ACT, 0, tp, handle, + !tc_skip_hw(fnew->flags)); + if (err < 0) + goto errout_idr; + err = fl_set_parms(net, tp, fnew, mask, base, tb, tca[TCA_RATE], tp->chain->tmplt_priv, flags, fnew->flags, extack); if (err) - goto errout; + goto errout_idr; err = fl_check_assign_mask(head, fnew, fold, mask); if (err) - goto errout; + goto errout_idr; err = fl_ht_insert_unique(fnew, fold, &in_ht); if (err) @@ -2274,29 +2310,9 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, refcount_dec(&fold->refcnt); __fl_put(fold); } else { - if (handle) { - /* user specifies a handle and it doesn't exist */ - err = idr_alloc_u32(&head->handle_idr, fnew, &handle, - handle, GFP_ATOMIC); - - /* Filter with specified handle was concurrently - * inserted after initial check in cls_api. This is not - * necessarily an error if NLM_F_EXCL is not set in - * message flags. Returning EAGAIN will cause cls_api to - * try to update concurrently inserted rule. - */ - if (err == -ENOSPC) - err = -EAGAIN; - } else { - handle = 1; - err = idr_alloc_u32(&head->handle_idr, fnew, &handle, - INT_MAX, GFP_ATOMIC); - } - if (err) - goto errout_hw; + idr_replace(&head->handle_idr, fnew, fnew->handle); refcount_inc(&fnew->refcnt); - fnew->handle = handle; list_add_tail_rcu(&fnew->list, &fnew->mask->filters); spin_unlock(&tp->lock); } @@ -2319,6 +2335,8 @@ errout_hw: fnew->mask->filter_ht_params); errout_mask: fl_mask_put(head, fnew->mask); +errout_idr: + idr_remove(&head->handle_idr, fnew->handle); errout: __fl_put(fnew); errout_tb: @@ -3436,6 +3454,7 @@ static struct tcf_proto_ops cls_fl_ops __read_mostly = { .tmplt_create = fl_tmplt_create, .tmplt_destroy = fl_tmplt_destroy, .tmplt_dump = fl_tmplt_dump, + .get_exts = fl_get_exts, .owner = THIS_MODULE, .flags = TCF_PROTO_OPS_DOIT_UNLOCKED, }; diff --git a/net/sched/sch_taprio.c b/net/sched/sch_taprio.c index 9781b47962bb..1f469861eae3 100644 --- a/net/sched/sch_taprio.c +++ b/net/sched/sch_taprio.c @@ -279,9 +279,17 @@ static void taprio_update_queue_max_sdu(struct taprio_sched *q, u32 max_frm_len; max_frm_len = duration_to_length(q, sched->max_open_gate_duration[tc]); - if (stab) + /* Compensate for L1 overhead from size table, + * but don't let the frame size go negative + */ + if (stab) { max_frm_len -= stab->szopts.overhead; + max_frm_len = max_t(int, max_frm_len, + dev->hard_header_len + 1); + } max_sdu_dynamic = max_frm_len - dev->hard_header_len; + if (max_sdu_dynamic > dev->max_mtu) + max_sdu_dynamic = U32_MAX; } max_sdu = min(max_sdu_dynamic, max_sdu_from_user); @@ -1833,23 +1841,6 @@ static int taprio_change(struct Qdisc *sch, struct nlattr *opt, goto free_sched; } - err = parse_taprio_schedule(q, tb, new_admin, extack); - if (err < 0) - goto free_sched; - - if (new_admin->num_entries == 0) { - NL_SET_ERR_MSG(extack, "There should be at least one entry in the schedule"); - err = -EINVAL; - goto free_sched; - } - - err = taprio_parse_clockid(sch, tb, extack); - if (err < 0) - goto free_sched; - - taprio_set_picos_per_byte(dev, q); - taprio_update_queue_max_sdu(q, new_admin, stab); - if (mqprio) { err = netdev_set_num_tc(dev, mqprio->num_tc); if (err) @@ -1867,6 +1858,23 @@ static int taprio_change(struct Qdisc *sch, struct nlattr *opt, mqprio->prio_tc_map[i]); } + err = parse_taprio_schedule(q, tb, new_admin, extack); + if (err < 0) + goto free_sched; + + if (new_admin->num_entries == 0) { + NL_SET_ERR_MSG(extack, "There should be at least one entry in the schedule"); + err = -EINVAL; + goto free_sched; + } + + err = taprio_parse_clockid(sch, tb, extack); + if (err < 0) + goto free_sched; + + taprio_set_picos_per_byte(dev, q); + taprio_update_queue_max_sdu(q, new_admin, stab); + if (FULL_OFFLOAD_IS_ENABLED(q->flags)) err = taprio_enable_offload(dev, q, new_admin, extack); else diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index a245c1b4a21b..45eef5af0a51 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -511,7 +511,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, return skb; } -static int xsk_generic_xmit(struct sock *sk) +static int __xsk_generic_xmit(struct sock *sk) { struct xdp_sock *xs = xdp_sk(sk); u32 max_batch = TX_BATCH_SIZE; @@ -594,22 +594,13 @@ out: return err; } -static int xsk_xmit(struct sock *sk) +static int xsk_generic_xmit(struct sock *sk) { - struct xdp_sock *xs = xdp_sk(sk); int ret; - if (unlikely(!(xs->dev->flags & IFF_UP))) - return -ENETDOWN; - if (unlikely(!xs->tx)) - return -ENOBUFS; - - if (xs->zc) - return xsk_wakeup(xs, XDP_WAKEUP_TX); - /* Drop the RCU lock since the SKB path might sleep. */ rcu_read_unlock(); - ret = xsk_generic_xmit(sk); + ret = __xsk_generic_xmit(sk); /* Reaquire RCU lock before going into common code. */ rcu_read_lock(); @@ -627,17 +618,31 @@ static bool xsk_no_wakeup(struct sock *sk) #endif } +static int xsk_check_common(struct xdp_sock *xs) +{ + if (unlikely(!xsk_is_bound(xs))) + return -ENXIO; + if (unlikely(!(xs->dev->flags & IFF_UP))) + return -ENETDOWN; + + return 0; +} + static int __xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len) { bool need_wait = !(m->msg_flags & MSG_DONTWAIT); struct sock *sk = sock->sk; struct xdp_sock *xs = xdp_sk(sk); struct xsk_buff_pool *pool; + int err; - if (unlikely(!xsk_is_bound(xs))) - return -ENXIO; + err = xsk_check_common(xs); + if (err) + return err; if (unlikely(need_wait)) return -EOPNOTSUPP; + if (unlikely(!xs->tx)) + return -ENOBUFS; if (sk_can_busy_loop(sk)) { if (xs->zc) @@ -649,8 +654,11 @@ static int __xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len return 0; pool = xs->pool; - if (pool->cached_need_wakeup & XDP_WAKEUP_TX) - return xsk_xmit(sk); + if (pool->cached_need_wakeup & XDP_WAKEUP_TX) { + if (xs->zc) + return xsk_wakeup(xs, XDP_WAKEUP_TX); + return xsk_generic_xmit(sk); + } return 0; } @@ -670,11 +678,11 @@ static int __xsk_recvmsg(struct socket *sock, struct msghdr *m, size_t len, int bool need_wait = !(flags & MSG_DONTWAIT); struct sock *sk = sock->sk; struct xdp_sock *xs = xdp_sk(sk); + int err; - if (unlikely(!xsk_is_bound(xs))) - return -ENXIO; - if (unlikely(!(xs->dev->flags & IFF_UP))) - return -ENETDOWN; + err = xsk_check_common(xs); + if (err) + return err; if (unlikely(!xs->rx)) return -ENOBUFS; if (unlikely(need_wait)) @@ -713,21 +721,20 @@ static __poll_t xsk_poll(struct file *file, struct socket *sock, sock_poll_wait(file, sock, wait); rcu_read_lock(); - if (unlikely(!xsk_is_bound(xs))) { - rcu_read_unlock(); - return mask; - } + if (xsk_check_common(xs)) + goto skip_tx; pool = xs->pool; if (pool->cached_need_wakeup) { if (xs->zc) xsk_wakeup(xs, pool->cached_need_wakeup); - else + else if (xs->tx) /* Poll needs to drive Tx also in copy mode */ - xsk_xmit(sk); + xsk_generic_xmit(sk); } +skip_tx: if (xs->rx && !xskq_prod_is_empty(xs->rx)) mask |= EPOLLIN | EPOLLRDNORM; if (xs->tx && xsk_tx_writeable(xs)) @@ -1294,8 +1301,6 @@ static int xsk_mmap(struct file *file, struct socket *sock, unsigned long size = vma->vm_end - vma->vm_start; struct xdp_sock *xs = xdp_sk(sock->sk); struct xsk_queue *q = NULL; - unsigned long pfn; - struct page *qpg; if (READ_ONCE(xs->state) != XSK_READY) return -EBUSY; @@ -1318,13 +1323,10 @@ static int xsk_mmap(struct file *file, struct socket *sock, /* Matches the smp_wmb() in xsk_init_queue */ smp_rmb(); - qpg = virt_to_head_page(q->ring); - if (size > page_size(qpg)) + if (size > q->ring_vmalloc_size) return -EINVAL; - pfn = virt_to_phys(q->ring) >> PAGE_SHIFT; - return remap_pfn_range(vma, vma->vm_start, pfn, - size, vma->vm_page_prot); + return remap_vmalloc_range(vma, q->ring, 0); } static int xsk_notifier(struct notifier_block *this, diff --git a/net/xdp/xsk_queue.c b/net/xdp/xsk_queue.c index 6cf9586e5027..f8905400ee07 100644 --- a/net/xdp/xsk_queue.c +++ b/net/xdp/xsk_queue.c @@ -6,6 +6,7 @@ #include <linux/log2.h> #include <linux/slab.h> #include <linux/overflow.h> +#include <linux/vmalloc.h> #include <net/xdp_sock_drv.h> #include "xsk_queue.h" @@ -23,7 +24,6 @@ static size_t xskq_get_ring_size(struct xsk_queue *q, bool umem_queue) struct xsk_queue *xskq_create(u32 nentries, bool umem_queue) { struct xsk_queue *q; - gfp_t gfp_flags; size_t size; q = kzalloc(sizeof(*q), GFP_KERNEL); @@ -33,17 +33,16 @@ struct xsk_queue *xskq_create(u32 nentries, bool umem_queue) q->nentries = nentries; q->ring_mask = nentries - 1; - gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | - __GFP_COMP | __GFP_NORETRY; size = xskq_get_ring_size(q, umem_queue); + size = PAGE_ALIGN(size); - q->ring = (struct xdp_ring *)__get_free_pages(gfp_flags, - get_order(size)); + q->ring = vmalloc_user(size); if (!q->ring) { kfree(q); return NULL; } + q->ring_vmalloc_size = size; return q; } @@ -52,6 +51,6 @@ void xskq_destroy(struct xsk_queue *q) if (!q) return; - page_frag_free(q->ring); + vfree(q->ring); kfree(q); } diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index c6fb6b763658..bfb2a7e50c26 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -45,6 +45,7 @@ struct xsk_queue { struct xdp_ring *ring; u64 invalid_descs; u64 queue_empty_descs; + size_t ring_vmalloc_size; }; /* The structure of the shared state of the rings are a simple diff --git a/samples/bpf/test_map_in_map_user.c b/samples/bpf/test_map_in_map_user.c index 9e79df4071f5..55dca43f3723 100644 --- a/samples/bpf/test_map_in_map_user.c +++ b/samples/bpf/test_map_in_map_user.c @@ -38,7 +38,7 @@ static void check_map_id(int inner_map_fd, int map_in_map_fd, uint32_t key) uint32_t info_len = sizeof(info); int ret, id; - ret = bpf_obj_get_info_by_fd(inner_map_fd, &info, &info_len); + ret = bpf_map_get_info_by_fd(inner_map_fd, &info, &info_len); assert(!ret); ret = bpf_map_lookup_elem(map_in_map_fd, &key, &id); diff --git a/samples/bpf/xdp1_user.c b/samples/bpf/xdp1_user.c index 281dc964de8d..f05e797013e9 100644 --- a/samples/bpf/xdp1_user.c +++ b/samples/bpf/xdp1_user.c @@ -153,7 +153,7 @@ int main(int argc, char **argv) return 1; } - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) { printf("can't get prog info - %s\n", strerror(errno)); return err; diff --git a/samples/bpf/xdp_adjust_tail_user.c b/samples/bpf/xdp_adjust_tail_user.c index 167646077c8f..e9426bd65420 100644 --- a/samples/bpf/xdp_adjust_tail_user.c +++ b/samples/bpf/xdp_adjust_tail_user.c @@ -184,7 +184,7 @@ int main(int argc, char **argv) return 1; } - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) { printf("can't get prog info - %s\n", strerror(errno)); return 1; diff --git a/samples/bpf/xdp_fwd_user.c b/samples/bpf/xdp_fwd_user.c index 84f57f1209ce..193b3b79b31f 100644 --- a/samples/bpf/xdp_fwd_user.c +++ b/samples/bpf/xdp_fwd_user.c @@ -76,9 +76,9 @@ static int do_detach(int ifindex, const char *ifname, const char *app_name) return prog_fd; } - err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &info_len); if (err) { - printf("ERROR: bpf_obj_get_info_by_fd failed (%s)\n", + printf("ERROR: bpf_prog_get_info_by_fd failed (%s)\n", strerror(errno)); goto close_out; } diff --git a/samples/bpf/xdp_redirect_cpu_user.c b/samples/bpf/xdp_redirect_cpu_user.c index a12381c37d2b..e1458405e2ba 100644 --- a/samples/bpf/xdp_redirect_cpu_user.c +++ b/samples/bpf/xdp_redirect_cpu_user.c @@ -494,9 +494,9 @@ int main(int argc, char **argv) goto end_cpu; } - ret = bpf_obj_get_info_by_fd(bpf_map__fd(skel->maps.cpu_map), &info, &infosz); + ret = bpf_map_get_info_by_fd(bpf_map__fd(skel->maps.cpu_map), &info, &infosz); if (ret < 0) { - fprintf(stderr, "Failed bpf_obj_get_info_by_fd for cpumap: %s\n", + fprintf(stderr, "Failed bpf_map_get_info_by_fd for cpumap: %s\n", strerror(errno)); goto end_cpu; } diff --git a/samples/bpf/xdp_rxq_info_user.c b/samples/bpf/xdp_rxq_info_user.c index 08f5331d2b00..b95e0ef61f06 100644 --- a/samples/bpf/xdp_rxq_info_user.c +++ b/samples/bpf/xdp_rxq_info_user.c @@ -602,7 +602,7 @@ int main(int argc, char **argv) return EXIT_FAIL_XDP; } - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) { printf("can't get prog info - %s\n", strerror(errno)); return err; diff --git a/samples/bpf/xdp_sample_pkts_user.c b/samples/bpf/xdp_sample_pkts_user.c index 7df7163239ac..e39d7f654f30 100644 --- a/samples/bpf/xdp_sample_pkts_user.c +++ b/samples/bpf/xdp_sample_pkts_user.c @@ -35,7 +35,7 @@ static int do_attach(int idx, int fd, const char *name) return err; } - err = bpf_obj_get_info_by_fd(fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(fd, &info, &info_len); if (err) { printf("can't get prog info - %s\n", strerror(errno)); return err; diff --git a/samples/bpf/xdp_tx_iptunnel_user.c b/samples/bpf/xdp_tx_iptunnel_user.c index 307baef6861a..7e4b2f7108a6 100644 --- a/samples/bpf/xdp_tx_iptunnel_user.c +++ b/samples/bpf/xdp_tx_iptunnel_user.c @@ -295,7 +295,7 @@ int main(int argc, char **argv) return 1; } - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) { printf("can't get prog info - %s\n", strerror(errno)); return err; diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c index 352290ba7b29..91fcb75babe3 100644 --- a/tools/bpf/bpftool/btf.c +++ b/tools/bpf/bpftool/btf.c @@ -537,7 +537,7 @@ static bool btf_is_kernel_module(__u32 btf_id) len = sizeof(btf_info); btf_info.name = ptr_to_u64(btf_name); btf_info.name_len = sizeof(btf_name); - err = bpf_obj_get_info_by_fd(btf_fd, &btf_info, &len); + err = bpf_btf_get_info_by_fd(btf_fd, &btf_info, &len); close(btf_fd); if (err) { p_err("can't get BTF (ID %u) object info: %s", btf_id, strerror(errno)); @@ -606,7 +606,7 @@ static int do_dump(int argc, char **argv) if (fd < 0) return -1; - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_prog_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get prog info: %s", strerror(errno)); goto done; @@ -789,7 +789,10 @@ build_btf_type_table(struct hashmap *tab, enum bpf_obj_type type, } memset(info, 0, *len); - err = bpf_obj_get_info_by_fd(fd, info, len); + if (type == BPF_OBJ_PROG) + err = bpf_prog_get_info_by_fd(fd, info, len); + else + err = bpf_map_get_info_by_fd(fd, info, len); close(fd); if (err) { p_err("can't get %s info: %s", names[type], @@ -931,7 +934,7 @@ show_btf(int fd, struct hashmap *btf_prog_table, int err; memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_btf_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get BTF object info: %s", strerror(errno)); return -1; @@ -943,7 +946,7 @@ show_btf(int fd, struct hashmap *btf_prog_table, info.name = ptr_to_u64(name); len = sizeof(info); - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_btf_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get BTF object info: %s", strerror(errno)); return -1; diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c index eda71fdfe95a..e7f6ec3a8f35 100644 --- a/tools/bpf/bpftool/btf_dumper.c +++ b/tools/bpf/bpftool/btf_dumper.c @@ -57,7 +57,7 @@ static int dump_prog_id_as_func_ptr(const struct btf_dumper *d, if (prog_fd < 0) goto print; - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) goto print; @@ -70,7 +70,7 @@ static int dump_prog_id_as_func_ptr(const struct btf_dumper *d, info.func_info_rec_size = finfo_rec_size; info.func_info = ptr_to_u64(&finfo); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err) goto print; diff --git a/tools/bpf/bpftool/cgroup.c b/tools/bpf/bpftool/cgroup.c index b46a998d8f8d..ac846b0805b4 100644 --- a/tools/bpf/bpftool/cgroup.c +++ b/tools/bpf/bpftool/cgroup.c @@ -82,7 +82,7 @@ static void guess_vmlinux_btf_id(__u32 attach_btf_obj_id) if (fd < 0) return; - err = bpf_obj_get_info_by_fd(fd, &btf_info, &btf_len); + err = bpf_btf_get_info_by_fd(fd, &btf_info, &btf_len); if (err) goto out; @@ -108,7 +108,7 @@ static int show_bpf_prog(int id, enum bpf_attach_type attach_type, if (prog_fd < 0) return -1; - if (bpf_obj_get_info_by_fd(prog_fd, &info, &info_len)) { + if (bpf_prog_get_info_by_fd(prog_fd, &info, &info_len)) { close(prog_fd); return -1; } diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index 620032042576..5a73ccf14332 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -353,7 +353,7 @@ void get_prog_full_name(const struct bpf_prog_info *prog_info, int prog_fd, info.func_info_rec_size = sizeof(finfo); info.func_info = ptr_to_u64(&finfo); - if (bpf_obj_get_info_by_fd(prog_fd, &info, &info_len)) + if (bpf_prog_get_info_by_fd(prog_fd, &info, &info_len)) goto copy_name; prog_btf = btf__load_from_kernel_by_id(info.btf_id); @@ -488,7 +488,7 @@ static int do_build_table_cb(const char *fpath, const struct stat *sb, goto out_close; memset(&pinned_info, 0, sizeof(pinned_info)); - if (bpf_obj_get_info_by_fd(fd, &pinned_info, &len)) + if (bpf_prog_get_info_by_fd(fd, &pinned_info, &len)) goto out_close; path = strdup(fpath); @@ -756,7 +756,7 @@ static int prog_fd_by_nametag(void *nametag, int **fds, bool tag) goto err_close_fds; } - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_prog_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get prog info (%u): %s", id, strerror(errno)); @@ -916,7 +916,7 @@ static int map_fd_by_name(char *name, int **fds) goto err_close_fds; } - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_map_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get map info (%u): %s", id, strerror(errno)); @@ -1026,7 +1026,8 @@ exit_free: return fd; } -int map_parse_fd_and_info(int *argc, char ***argv, void *info, __u32 *info_len) +int map_parse_fd_and_info(int *argc, char ***argv, struct bpf_map_info *info, + __u32 *info_len) { int err; int fd; @@ -1035,7 +1036,7 @@ int map_parse_fd_and_info(int *argc, char ***argv, void *info, __u32 *info_len) if (fd < 0) return -1; - err = bpf_obj_get_info_by_fd(fd, info, info_len); + err = bpf_map_get_info_by_fd(fd, info, info_len); if (err) { p_err("can't get map info: %s", strerror(errno)); close(fd); diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index 6f4cfe01cad4..f985b79cca27 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -145,7 +145,7 @@ static int get_prog_info(int prog_id, struct bpf_prog_info *info) return prog_fd; memset(info, 0, sizeof(*info)); - err = bpf_obj_get_info_by_fd(prog_fd, info, &len); + err = bpf_prog_get_info_by_fd(prog_fd, info, &len); if (err) p_err("can't get prog info: %s", strerror(errno)); close(prog_fd); @@ -327,7 +327,7 @@ static int do_show_link(int fd) memset(&info, 0, sizeof(info)); again: - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_link_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get link info: %s", strerror(errno)); diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h index a84224b6a604..0ef373cef4c7 100644 --- a/tools/bpf/bpftool/main.h +++ b/tools/bpf/bpftool/main.h @@ -168,7 +168,8 @@ int prog_parse_fd(int *argc, char ***argv); int prog_parse_fds(int *argc, char ***argv, int **fds); int map_parse_fd(int *argc, char ***argv); int map_parse_fds(int *argc, char ***argv, int **fds); -int map_parse_fd_and_info(int *argc, char ***argv, void *info, __u32 *info_len); +int map_parse_fd_and_info(int *argc, char ***argv, struct bpf_map_info *info, + __u32 *info_len); struct bpf_prog_linfo; #if defined(HAVE_LLVM_SUPPORT) || defined(HAVE_LIBBFD_SUPPORT) diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index 88911d3aa2d9..aaeb8939e137 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -638,7 +638,7 @@ static int do_show_subset(int argc, char **argv) if (json_output && nb_fds > 1) jsonw_start_array(json_wtr); /* root array */ for (i = 0; i < nb_fds; i++) { - err = bpf_obj_get_info_by_fd(fds[i], &info, &len); + err = bpf_map_get_info_by_fd(fds[i], &info, &len); if (err) { p_err("can't get map info: %s", strerror(errno)); @@ -708,7 +708,7 @@ static int do_show(int argc, char **argv) break; } - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_map_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get map info: %s", strerror(errno)); close(fd); @@ -764,7 +764,7 @@ static int maps_have_btf(int *fds, int nb_fds) int err, i; for (i = 0; i < nb_fds; i++) { - err = bpf_obj_get_info_by_fd(fds[i], &info, &len); + err = bpf_map_get_info_by_fd(fds[i], &info, &len); if (err) { p_err("can't get map info: %s", strerror(errno)); return -1; @@ -925,7 +925,7 @@ static int do_dump(int argc, char **argv) if (wtr && nb_fds > 1) jsonw_start_array(wtr); /* root array */ for (i = 0; i < nb_fds; i++) { - if (bpf_obj_get_info_by_fd(fds[i], &info, &len)) { + if (bpf_map_get_info_by_fd(fds[i], &info, &len)) { p_err("can't get map info: %s", strerror(errno)); break; } diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c index e87738dbffc1..afbe3ec342c8 100644 --- a/tools/bpf/bpftool/prog.c +++ b/tools/bpf/bpftool/prog.c @@ -198,7 +198,7 @@ static void show_prog_maps(int fd, __u32 num_maps) info.nr_map_ids = num_maps; info.map_ids = ptr_to_u64(map_ids); - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_prog_get_info_by_fd(fd, &info, &len); if (err || !info.nr_map_ids) return; @@ -231,7 +231,7 @@ static void *find_metadata(int prog_fd, struct bpf_map_info *map_info) memset(&prog_info, 0, sizeof(prog_info)); prog_info_len = sizeof(prog_info); - ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + ret = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); if (ret) return NULL; @@ -248,7 +248,7 @@ static void *find_metadata(int prog_fd, struct bpf_map_info *map_info) prog_info.map_ids = ptr_to_u64(map_ids); prog_info_len = sizeof(prog_info); - ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + ret = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); if (ret) goto free_map_ids; @@ -259,7 +259,7 @@ static void *find_metadata(int prog_fd, struct bpf_map_info *map_info) memset(map_info, 0, sizeof(*map_info)); map_info_len = sizeof(*map_info); - ret = bpf_obj_get_info_by_fd(map_fd, map_info, &map_info_len); + ret = bpf_map_get_info_by_fd(map_fd, map_info, &map_info_len); if (ret < 0) { close(map_fd); goto free_map_ids; @@ -580,7 +580,7 @@ static int show_prog(int fd) __u32 len = sizeof(info); int err; - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_prog_get_info_by_fd(fd, &info, &len); if (err) { p_err("can't get prog info: %s", strerror(errno)); return -1; @@ -949,7 +949,7 @@ static int do_dump(int argc, char **argv) for (i = 0; i < nb_fds; i++) { memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(fds[i], &info, &info_len); + err = bpf_prog_get_info_by_fd(fds[i], &info, &info_len); if (err) { p_err("can't get prog info: %s", strerror(errno)); break; @@ -961,7 +961,7 @@ static int do_dump(int argc, char **argv) break; } - err = bpf_obj_get_info_by_fd(fds[i], &info, &info_len); + err = bpf_prog_get_info_by_fd(fds[i], &info, &info_len); if (err) { p_err("can't get prog info: %s", strerror(errno)); break; @@ -2170,9 +2170,9 @@ static char *profile_target_name(int tgt_fd) char *name = NULL; int err; - err = bpf_obj_get_info_by_fd(tgt_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(tgt_fd, &info, &info_len); if (err) { - p_err("failed to bpf_obj_get_info_by_fd for prog FD %d", tgt_fd); + p_err("failed to get info for prog FD %d", tgt_fd); goto out; } @@ -2183,7 +2183,7 @@ static char *profile_target_name(int tgt_fd) func_info_rec_size = info.func_info_rec_size; if (info.nr_func_info == 0) { - p_err("bpf_obj_get_info_by_fd for prog FD %d found 0 func_info", tgt_fd); + p_err("found 0 func_info for prog FD %d", tgt_fd); goto out; } @@ -2192,7 +2192,7 @@ static char *profile_target_name(int tgt_fd) info.func_info_rec_size = func_info_rec_size; info.func_info = ptr_to_u64(&func_info); - err = bpf_obj_get_info_by_fd(tgt_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(tgt_fd, &info, &info_len); if (err) { p_err("failed to get func_info for prog FD %d", tgt_fd); goto out; diff --git a/tools/bpf/bpftool/struct_ops.c b/tools/bpf/bpftool/struct_ops.c index 903b80ff4e9a..b389f4830e11 100644 --- a/tools/bpf/bpftool/struct_ops.c +++ b/tools/bpf/bpftool/struct_ops.c @@ -151,7 +151,7 @@ static int get_next_struct_ops_map(const char *name, int *res_fd, return -1; } - err = bpf_obj_get_info_by_fd(fd, info, &info_len); + err = bpf_map_get_info_by_fd(fd, info, &info_len); if (err) { p_err("can't get map info: %s", strerror(errno)); close(fd); @@ -262,7 +262,7 @@ static struct res do_one_id(const char *id_str, work_func func, void *data, goto done; } - if (bpf_obj_get_info_by_fd(fd, info, &info_len)) { + if (bpf_map_get_info_by_fd(fd, info, &info_len)) { p_err("can't get map info: %s", strerror(errno)); res.nr_errs++; goto done; @@ -522,7 +522,7 @@ static int do_register(int argc, char **argv) bpf_link__disconnect(link); bpf_link__destroy(link); - if (!bpf_obj_get_info_by_fd(bpf_map__fd(map), &info, + if (!bpf_map_get_info_by_fd(bpf_map__fd(map), &info, &info_len)) p_info("Registered %s %s id %u", get_kern_struct_ops_name(&info), diff --git a/tools/include/uapi/asm/bpf_perf_event.h b/tools/include/uapi/asm/bpf_perf_event.h index d7dfeab0d71a..ff52668abf8c 100644 --- a/tools/include/uapi/asm/bpf_perf_event.h +++ b/tools/include/uapi/asm/bpf_perf_event.h @@ -6,6 +6,8 @@ #include "../../arch/s390/include/uapi/asm/bpf_perf_event.h" #elif defined(__riscv) #include "../../arch/riscv/include/uapi/asm/bpf_perf_event.h" +#elif defined(__loongarch__) +#include "../../arch/loongarch/include/uapi/asm/bpf_perf_event.h" #else #include <uapi/asm-generic/bpf_perf_event.h> #endif diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 17afd2b35ee5..62ce1f5d1b1d 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3134,6 +3134,11 @@ union bpf_attr { * **BPF_FIB_LOOKUP_OUTPUT** * Perform lookup from an egress perspective (default is * ingress). + * **BPF_FIB_LOOKUP_SKIP_NEIGH** + * Skip the neighbour table lookup. *params*->dmac + * and *params*->smac will not be set as output. A common + * use case is to call **bpf_redirect_neigh**\ () after + * doing **bpf_fib_lookup**\ (). * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -6750,6 +6755,7 @@ struct bpf_raw_tracepoint_args { enum { BPF_FIB_LOOKUP_DIRECT = (1U << 0), BPF_FIB_LOOKUP_OUTPUT = (1U << 1), + BPF_FIB_LOOKUP_SKIP_NEIGH = (1U << 2), }; enum { @@ -6917,6 +6923,17 @@ struct bpf_list_node { __u64 :64; } __attribute__((aligned(8))); +struct bpf_rb_root { + __u64 :64; + __u64 :64; +} __attribute__((aligned(8))); + +struct bpf_rb_node { + __u64 :64; + __u64 :64; + __u64 :64; +} __attribute__((aligned(8))); + struct bpf_sysctl { __u32 write; /* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 9aff98f42a3d..e750b6f5fcc3 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -1044,6 +1044,26 @@ int bpf_obj_get_info_by_fd(int bpf_fd, void *info, __u32 *info_len) return libbpf_err_errno(err); } +int bpf_prog_get_info_by_fd(int prog_fd, struct bpf_prog_info *info, __u32 *info_len) +{ + return bpf_obj_get_info_by_fd(prog_fd, info, info_len); +} + +int bpf_map_get_info_by_fd(int map_fd, struct bpf_map_info *info, __u32 *info_len) +{ + return bpf_obj_get_info_by_fd(map_fd, info, info_len); +} + +int bpf_btf_get_info_by_fd(int btf_fd, struct bpf_btf_info *info, __u32 *info_len) +{ + return bpf_obj_get_info_by_fd(btf_fd, info, info_len); +} + +int bpf_link_get_info_by_fd(int link_fd, struct bpf_link_info *info, __u32 *info_len) +{ + return bpf_obj_get_info_by_fd(link_fd, info, info_len); +} + int bpf_raw_tracepoint_open(const char *name, int prog_fd) { const size_t attr_sz = offsetofend(union bpf_attr, raw_tracepoint); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 7468978d3c27..9ed9bceb4111 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -386,6 +386,15 @@ LIBBPF_API int bpf_link_get_fd_by_id(__u32 id); LIBBPF_API int bpf_link_get_fd_by_id_opts(__u32 id, const struct bpf_get_fd_by_id_opts *opts); LIBBPF_API int bpf_obj_get_info_by_fd(int bpf_fd, void *info, __u32 *info_len); +/* Type-safe variants of bpf_obj_get_info_by_fd(). The callers still needs to + * pass info_len, which should normally be + * sizeof(struct bpf_{prog,map,btf,link}_info), in order to be compatible with + * different libbpf and kernel versions. + */ +LIBBPF_API int bpf_prog_get_info_by_fd(int prog_fd, struct bpf_prog_info *info, __u32 *info_len); +LIBBPF_API int bpf_map_get_info_by_fd(int map_fd, struct bpf_map_info *info, __u32 *info_len); +LIBBPF_API int bpf_btf_get_info_by_fd(int btf_fd, struct bpf_btf_info *info, __u32 *info_len); +LIBBPF_API int bpf_link_get_info_by_fd(int link_fd, struct bpf_link_info *info, __u32 *info_len); struct bpf_prog_query_opts { size_t sz; /* size of this struct for forward/backward compatibility */ diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index 64841117fbb2..9181d36118d2 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -1350,9 +1350,9 @@ struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) void *ptr; int err; - /* we won't know btf_size until we call bpf_obj_get_info_by_fd(). so + /* we won't know btf_size until we call bpf_btf_get_info_by_fd(). so * let's start with a sane default - 4KiB here - and resize it only if - * bpf_obj_get_info_by_fd() needs a bigger buffer. + * bpf_btf_get_info_by_fd() needs a bigger buffer. */ last_size = 4096; ptr = malloc(last_size); @@ -1362,7 +1362,7 @@ struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) memset(&btf_info, 0, sizeof(btf_info)); btf_info.btf = ptr_to_u64(ptr); btf_info.btf_size = last_size; - err = bpf_obj_get_info_by_fd(btf_fd, &btf_info, &len); + err = bpf_btf_get_info_by_fd(btf_fd, &btf_info, &len); if (!err && btf_info.btf_size > last_size) { void *temp_ptr; @@ -1380,7 +1380,7 @@ struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) btf_info.btf = ptr_to_u64(ptr); btf_info.btf_size = last_size; - err = bpf_obj_get_info_by_fd(btf_fd, &btf_info, &len); + err = bpf_btf_get_info_by_fd(btf_fd, &btf_info, &len); } if (err || btf_info.btf_size > last_size) { diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 35a698eb825d..05c4db355f28 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -4345,7 +4345,7 @@ int bpf_map__reuse_fd(struct bpf_map *map, int fd) char *new_name; memset(&info, 0, len); - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_map_get_info_by_fd(fd, &info, &len); if (err && errno == EINVAL) err = bpf_get_map_info_from_fdinfo(fd, &info); if (err) @@ -4729,7 +4729,7 @@ static int probe_module_btf(void) * kernel's module BTF support coincides with support for * name/name_len fields in struct bpf_btf_info. */ - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_btf_get_info_by_fd(fd, &info, &len); close(fd); return !err; } @@ -4892,7 +4892,7 @@ static bool map_is_reuse_compat(const struct bpf_map *map, int map_fd) int err; memset(&map_info, 0, map_info_len); - err = bpf_obj_get_info_by_fd(map_fd, &map_info, &map_info_len); + err = bpf_map_get_info_by_fd(map_fd, &map_info, &map_info_len); if (err && errno == EINVAL) err = bpf_get_map_info_from_fdinfo(map_fd, &map_info); if (err) { @@ -5437,7 +5437,7 @@ static int load_module_btfs(struct bpf_object *obj) info.name = ptr_to_u64(name); info.name_len = sizeof(name); - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_btf_get_info_by_fd(fd, &info, &len); if (err) { err = -errno; pr_warn("failed to get BTF object #%d info: %d\n", id, err); @@ -9030,9 +9030,9 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd) int err; memset(&info, 0, info_len); - err = bpf_obj_get_info_by_fd(attach_prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(attach_prog_fd, &info, &info_len); if (err) { - pr_warn("failed bpf_obj_get_info_by_fd for FD %d: %d\n", + pr_warn("failed bpf_prog_get_info_by_fd for FD %d: %d\n", attach_prog_fd, err); return err; } @@ -11741,7 +11741,7 @@ static struct perf_buffer *__perf_buffer__new(int map_fd, size_t page_cnt, /* best-effort sanity checks */ memset(&map, 0, sizeof(map)); map_info_len = sizeof(map); - err = bpf_obj_get_info_by_fd(map_fd, &map, &map_info_len); + err = bpf_map_get_info_by_fd(map_fd, &map, &map_info_len); if (err) { err = -errno; /* if BPF_OBJ_GET_INFO_BY_FD is supported, will return diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 11c36a3c1a9f..50dde1f6521e 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -384,4 +384,9 @@ LIBBPF_1.1.0 { } LIBBPF_1.0.0; LIBBPF_1.2.0 { + global: + bpf_btf_get_info_by_fd; + bpf_link_get_info_by_fd; + bpf_map_get_info_by_fd; + bpf_prog_get_info_by_fd; } LIBBPF_1.1.0; diff --git a/tools/lib/bpf/netlink.c b/tools/lib/bpf/netlink.c index cb082a04ffa8..1653e7a8b0a1 100644 --- a/tools/lib/bpf/netlink.c +++ b/tools/lib/bpf/netlink.c @@ -689,7 +689,7 @@ static int tc_add_fd_and_name(struct libbpf_nla_req *req, int fd) int len, ret; memset(&info, 0, info_len); - ret = bpf_obj_get_info_by_fd(fd, &info, &info_len); + ret = bpf_prog_get_info_by_fd(fd, &info, &info_len); if (ret < 0) return ret; diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c index 47855af25f3b..02199364db13 100644 --- a/tools/lib/bpf/ringbuf.c +++ b/tools/lib/bpf/ringbuf.c @@ -83,7 +83,7 @@ int ring_buffer__add(struct ring_buffer *rb, int map_fd, memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(map_fd, &info, &len); + err = bpf_map_get_info_by_fd(map_fd, &info, &len); if (err) { err = -errno; pr_warn("ringbuf: failed to get map info for fd=%d: %d\n", @@ -359,7 +359,7 @@ static int user_ringbuf_map(struct user_ring_buffer *rb, int map_fd) memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(map_fd, &info, &len); + err = bpf_map_get_info_by_fd(map_fd, &info, &len); if (err) { err = -errno; pr_warn("user ringbuf: failed to get map info for fd=%d: %d\n", map_fd, err); diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index c4b5c44cdee2..b677dcd0b77a 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -152,14 +152,13 @@ endif # NOTE: Semicolon at the end is critical to override lib.mk's default static # rule for binaries. $(notdir $(TEST_GEN_PROGS) \ - $(TEST_PROGS) \ - $(TEST_PROGS_EXTENDED) \ $(TEST_GEN_PROGS_EXTENDED) \ $(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ; # sort removes libbpf duplicates when not cross-building -MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \ - $(HOST_BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/resolve_btfids \ +MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \ + $(BUILD_DIR)/bpftool $(HOST_BUILD_DIR)/bpftool \ + $(HOST_BUILD_DIR)/resolve_btfids \ $(RUNQSLOWER_OUTPUT) $(INCLUDE_DIR)) $(MAKE_DIRS): $(call msg,MKDIR,,$@) @@ -209,6 +208,14 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(wildcard bpf_testmod/Makefile bpf_tes $(Q)cp bpf_testmod/bpf_testmod.ko $@ DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool +ifneq ($(CROSS_COMPILE),) +CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool +TRUNNER_BPFTOOL := $(CROSS_BPFTOOL) +USE_BOOTSTRAP := "" +else +TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL) +USE_BOOTSTRAP := "bootstrap/" +endif $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT) $(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \ @@ -220,7 +227,7 @@ $(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT) EXTRA_LDFLAGS='$(SAN_LDFLAGS)' && \ cp $(RUNQSLOWER_OUTPUT)runqslower $@ -TEST_GEN_PROGS_EXTENDED += $(DEFAULT_BPFTOOL) +TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL) $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED): $(BPFOBJ) @@ -258,6 +265,18 @@ $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ LIBBPF_DESTDIR=$(HOST_SCRATCH_DIR)/ \ prefix= DESTDIR=$(HOST_SCRATCH_DIR)/ install-bin +ifneq ($(CROSS_COMPILE),) +$(CROSS_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ + $(BPFOBJ) | $(BUILD_DIR)/bpftool + $(Q)$(MAKE) $(submake_extras) -C $(BPFTOOLDIR) \ + ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) \ + EXTRA_CFLAGS='-g -O0' \ + OUTPUT=$(BUILD_DIR)/bpftool/ \ + LIBBPF_OUTPUT=$(BUILD_DIR)/libbpf/ \ + LIBBPF_DESTDIR=$(SCRATCH_DIR)/ \ + prefix= DESTDIR=$(SCRATCH_DIR)/ install-bin +endif + all: docs docs: @@ -523,11 +542,12 @@ endif $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS) \ $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ) \ $(RESOLVE_BTFIDS) \ + $(TRUNNER_BPFTOOL) \ | $(TRUNNER_BINARY)-extras $$(call msg,BINARY,,$$@) $(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) -o $$@ $(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@ - $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/bootstrap/bpftool \ + $(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)bpftool \ $(OUTPUT)/$(if $2,$2/)bpftool endef @@ -618,6 +638,7 @@ $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h $(OUTPUT)/bench_local_storage_rcu_tasks_trace.o: $(OUTPUT)/local_storage_rcu_tasks_trace_bench.skel.h +$(OUTPUT)/bench_bpf_hashmap_lookup.o: $(OUTPUT)/bpf_hashmap_lookup.skel.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm $(OUTPUT)/bench: $(OUTPUT)/bench.o \ @@ -632,7 +653,9 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_strncmp.o \ $(OUTPUT)/bench_bpf_hashmap_full_update.o \ $(OUTPUT)/bench_local_storage.o \ - $(OUTPUT)/bench_local_storage_rcu_tasks_trace.o + $(OUTPUT)/bench_local_storage_rcu_tasks_trace.o \ + $(OUTPUT)/bench_bpf_hashmap_lookup.o \ + # $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@ diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index c1f20a147462..0b2a53bb8460 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -16,6 +16,7 @@ struct env env = { .warmup_sec = 1, .duration_sec = 5, .affinity = false, + .quiet = false, .consumer_cnt = 1, .producer_cnt = 1, }; @@ -262,6 +263,7 @@ static const struct argp_option opts[] = { { "consumers", 'c', "NUM", 0, "Number of consumer threads"}, { "verbose", 'v', NULL, 0, "Verbose debug output"}, { "affinity", 'a', NULL, 0, "Set consumer/producer thread affinity"}, + { "quiet", 'q', NULL, 0, "Be more quiet"}, { "prod-affinity", ARG_PROD_AFFINITY_SET, "CPUSET", 0, "Set of CPUs for producer threads; implies --affinity"}, { "cons-affinity", ARG_CONS_AFFINITY_SET, "CPUSET", 0, @@ -275,6 +277,7 @@ extern struct argp bench_bpf_loop_argp; extern struct argp bench_local_storage_argp; extern struct argp bench_local_storage_rcu_tasks_trace_argp; extern struct argp bench_strncmp_argp; +extern struct argp bench_hashmap_lookup_argp; static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 }, @@ -284,13 +287,15 @@ static const struct argp_child bench_parsers[] = { { &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 }, { &bench_local_storage_rcu_tasks_trace_argp, 0, "local_storage RCU Tasks Trace slowdown benchmark", 0 }, + { &bench_hashmap_lookup_argp, 0, "Hashmap lookup benchmark", 0 }, {}, }; +/* Make pos_args global, so that we can run argp_parse twice, if necessary */ +static int pos_args; + static error_t parse_arg(int key, char *arg, struct argp_state *state) { - static int pos_args; - switch (key) { case 'v': env.verbose = true; @@ -329,6 +334,9 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) case 'a': env.affinity = true; break; + case 'q': + env.quiet = true; + break; case ARG_PROD_AFFINITY_SET: env.affinity = true; if (parse_num_list(arg, &env.prod_cpus.cpus, @@ -359,7 +367,7 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) return 0; } -static void parse_cmdline_args(int argc, char **argv) +static void parse_cmdline_args_init(int argc, char **argv) { static const struct argp argp = { .options = opts, @@ -369,9 +377,25 @@ static void parse_cmdline_args(int argc, char **argv) }; if (argp_parse(&argp, argc, argv, 0, NULL, NULL)) exit(1); - if (!env.list && !env.bench_name) { - argp_help(&argp, stderr, ARGP_HELP_DOC, "bench"); - exit(1); +} + +static void parse_cmdline_args_final(int argc, char **argv) +{ + struct argp_child bench_parsers[2] = {}; + const struct argp argp = { + .options = opts, + .parser = parse_arg, + .doc = argp_program_doc, + .children = bench_parsers, + }; + + /* Parse arguments the second time with the correct set of parsers */ + if (bench->argp) { + bench_parsers[0].argp = bench->argp; + bench_parsers[0].header = bench->name; + pos_args = 0; + if (argp_parse(&argp, argc, argv, 0, NULL, NULL)) + exit(1); } } @@ -490,6 +514,7 @@ extern const struct bench bench_local_storage_cache_seq_get; extern const struct bench bench_local_storage_cache_interleaved_get; extern const struct bench bench_local_storage_cache_hashmap_control; extern const struct bench bench_local_storage_tasks_trace; +extern const struct bench bench_bpf_hashmap_lookup; static const struct bench *benchs[] = { &bench_count_global, @@ -529,17 +554,17 @@ static const struct bench *benchs[] = { &bench_local_storage_cache_interleaved_get, &bench_local_storage_cache_hashmap_control, &bench_local_storage_tasks_trace, + &bench_bpf_hashmap_lookup, }; -static void setup_benchmark() +static void find_benchmark(void) { - int i, err; + int i; if (!env.bench_name) { fprintf(stderr, "benchmark name is not specified\n"); exit(1); } - for (i = 0; i < ARRAY_SIZE(benchs); i++) { if (strcmp(benchs[i]->name, env.bench_name) == 0) { bench = benchs[i]; @@ -550,8 +575,14 @@ static void setup_benchmark() fprintf(stderr, "benchmark '%s' not found\n", env.bench_name); exit(1); } +} - printf("Setting up benchmark '%s'...\n", bench->name); +static void setup_benchmark(void) +{ + int i, err; + + if (!env.quiet) + printf("Setting up benchmark '%s'...\n", bench->name); state.producers = calloc(env.producer_cnt, sizeof(*state.producers)); state.consumers = calloc(env.consumer_cnt, sizeof(*state.consumers)); @@ -597,7 +628,8 @@ static void setup_benchmark() next_cpu(&env.prod_cpus)); } - printf("Benchmark '%s' started.\n", bench->name); + if (!env.quiet) + printf("Benchmark '%s' started.\n", bench->name); } static pthread_mutex_t bench_done_mtx = PTHREAD_MUTEX_INITIALIZER; @@ -621,7 +653,7 @@ static void collect_measurements(long delta_ns) { int main(int argc, char **argv) { - parse_cmdline_args(argc, argv); + parse_cmdline_args_init(argc, argv); if (env.list) { int i; @@ -633,6 +665,9 @@ int main(int argc, char **argv) return 0; } + find_benchmark(); + parse_cmdline_args_final(argc, argv); + setup_benchmark(); setup_timer(); diff --git a/tools/testing/selftests/bpf/bench.h b/tools/testing/selftests/bpf/bench.h index d748255877e2..402729c6a3ac 100644 --- a/tools/testing/selftests/bpf/bench.h +++ b/tools/testing/selftests/bpf/bench.h @@ -24,6 +24,7 @@ struct env { bool verbose; bool list; bool affinity; + bool quiet; int consumer_cnt; int producer_cnt; struct cpu_set prod_cpus; @@ -47,6 +48,7 @@ struct bench_res { struct bench { const char *name; + const struct argp *argp; void (*validate)(void); void (*setup)(void); void *(*producer_thread)(void *ctx); diff --git a/tools/testing/selftests/bpf/benchs/bench_bloom_filter_map.c b/tools/testing/selftests/bpf/benchs/bench_bloom_filter_map.c index 5bcb8a8cdeb2..7c8ccc108313 100644 --- a/tools/testing/selftests/bpf/benchs/bench_bloom_filter_map.c +++ b/tools/testing/selftests/bpf/benchs/bench_bloom_filter_map.c @@ -428,6 +428,7 @@ static void *consumer(void *input) const struct bench bench_bloom_lookup = { .name = "bloom-lookup", + .argp = &bench_bloom_map_argp, .validate = validate, .setup = bloom_lookup_setup, .producer_thread = producer, @@ -439,6 +440,7 @@ const struct bench bench_bloom_lookup = { const struct bench bench_bloom_update = { .name = "bloom-update", + .argp = &bench_bloom_map_argp, .validate = validate, .setup = bloom_update_setup, .producer_thread = producer, @@ -450,6 +452,7 @@ const struct bench bench_bloom_update = { const struct bench bench_bloom_false_positive = { .name = "bloom-false-positive", + .argp = &bench_bloom_map_argp, .validate = validate, .setup = false_positive_setup, .producer_thread = producer, @@ -461,6 +464,7 @@ const struct bench bench_bloom_false_positive = { const struct bench bench_hashmap_without_bloom = { .name = "hashmap-without-bloom", + .argp = &bench_bloom_map_argp, .validate = validate, .setup = hashmap_no_bloom_setup, .producer_thread = producer, @@ -472,6 +476,7 @@ const struct bench bench_hashmap_without_bloom = { const struct bench bench_hashmap_with_bloom = { .name = "hashmap-with-bloom", + .argp = &bench_bloom_map_argp, .validate = validate, .setup = hashmap_with_bloom_setup, .producer_thread = producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c b/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c index cec51e0ff4b8..75abe8137b6c 100644 --- a/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c +++ b/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_full_update.c @@ -1,7 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2022 Bytedance */ -#include <argp.h> #include "bench.h" #include "bpf_hashmap_full_update_bench.skel.h" #include "bpf_util.h" @@ -68,7 +67,7 @@ static void setup(void) bpf_map_update_elem(map_fd, &i, &i, BPF_ANY); } -void hashmap_report_final(struct bench_res res[], int res_cnt) +static void hashmap_report_final(struct bench_res res[], int res_cnt) { unsigned int nr_cpus = bpf_num_possible_cpus(); int i; @@ -85,7 +84,7 @@ void hashmap_report_final(struct bench_res res[], int res_cnt) } const struct bench bench_bpf_hashmap_full_update = { - .name = "bpf-hashmap-ful-update", + .name = "bpf-hashmap-full-update", .validate = validate, .setup = setup, .producer_thread = producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c b/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c new file mode 100644 index 000000000000..8dbb02f75cff --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/bench_bpf_hashmap_lookup.c @@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include <sys/random.h> +#include <argp.h> +#include "bench.h" +#include "bpf_hashmap_lookup.skel.h" +#include "bpf_util.h" + +/* BPF triggering benchmarks */ +static struct ctx { + struct bpf_hashmap_lookup *skel; +} ctx; + +/* only available to kernel, so define it here */ +#define BPF_MAX_LOOPS (1<<23) + +#define MAX_KEY_SIZE 1024 /* the size of the key map */ + +static struct { + __u32 key_size; + __u32 map_flags; + __u32 max_entries; + __u32 nr_entries; + __u32 nr_loops; +} args = { + .key_size = 4, + .map_flags = 0, + .max_entries = 1000, + .nr_entries = 500, + .nr_loops = 1000000, +}; + +enum { + ARG_KEY_SIZE = 8001, + ARG_MAP_FLAGS, + ARG_MAX_ENTRIES, + ARG_NR_ENTRIES, + ARG_NR_LOOPS, +}; + +static const struct argp_option opts[] = { + { "key_size", ARG_KEY_SIZE, "KEY_SIZE", 0, + "The hashmap key size (max 1024)"}, + { "map_flags", ARG_MAP_FLAGS, "MAP_FLAGS", 0, + "The hashmap flags passed to BPF_MAP_CREATE"}, + { "max_entries", ARG_MAX_ENTRIES, "MAX_ENTRIES", 0, + "The hashmap max entries"}, + { "nr_entries", ARG_NR_ENTRIES, "NR_ENTRIES", 0, + "The number of entries to insert/lookup"}, + { "nr_loops", ARG_NR_LOOPS, "NR_LOOPS", 0, + "The number of loops for the benchmark"}, + {}, +}; + +static error_t parse_arg(int key, char *arg, struct argp_state *state) +{ + long ret; + + switch (key) { + case ARG_KEY_SIZE: + ret = strtol(arg, NULL, 10); + if (ret < 1 || ret > MAX_KEY_SIZE) { + fprintf(stderr, "invalid key_size"); + argp_usage(state); + } + args.key_size = ret; + break; + case ARG_MAP_FLAGS: + ret = strtol(arg, NULL, 0); + if (ret < 0 || ret > UINT_MAX) { + fprintf(stderr, "invalid map_flags"); + argp_usage(state); + } + args.map_flags = ret; + break; + case ARG_MAX_ENTRIES: + ret = strtol(arg, NULL, 10); + if (ret < 1 || ret > UINT_MAX) { + fprintf(stderr, "invalid max_entries"); + argp_usage(state); + } + args.max_entries = ret; + break; + case ARG_NR_ENTRIES: + ret = strtol(arg, NULL, 10); + if (ret < 1 || ret > UINT_MAX) { + fprintf(stderr, "invalid nr_entries"); + argp_usage(state); + } + args.nr_entries = ret; + break; + case ARG_NR_LOOPS: + ret = strtol(arg, NULL, 10); + if (ret < 1 || ret > BPF_MAX_LOOPS) { + fprintf(stderr, "invalid nr_loops: %ld (min=1 max=%u)\n", + ret, BPF_MAX_LOOPS); + argp_usage(state); + } + args.nr_loops = ret; + break; + default: + return ARGP_ERR_UNKNOWN; + } + + return 0; +} + +const struct argp bench_hashmap_lookup_argp = { + .options = opts, + .parser = parse_arg, +}; + +static void validate(void) +{ + if (env.consumer_cnt != 1) { + fprintf(stderr, "benchmark doesn't support multi-consumer!\n"); + exit(1); + } + + if (args.nr_entries > args.max_entries) { + fprintf(stderr, "args.nr_entries is too big! (max %u, got %u)\n", + args.max_entries, args.nr_entries); + exit(1); + } +} + +static void *producer(void *input) +{ + while (true) { + /* trigger the bpf program */ + syscall(__NR_getpgid); + } + return NULL; +} + +static void *consumer(void *input) +{ + return NULL; +} + +static void measure(struct bench_res *res) +{ +} + +static inline void patch_key(u32 i, u32 *key) +{ +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ + *key = i + 1; +#else + *key = __builtin_bswap32(i + 1); +#endif + /* the rest of key is random */ +} + +static void setup(void) +{ + struct bpf_link *link; + int map_fd; + int ret; + int i; + + setup_libbpf(); + + ctx.skel = bpf_hashmap_lookup__open(); + if (!ctx.skel) { + fprintf(stderr, "failed to open skeleton\n"); + exit(1); + } + + bpf_map__set_max_entries(ctx.skel->maps.hash_map_bench, args.max_entries); + bpf_map__set_key_size(ctx.skel->maps.hash_map_bench, args.key_size); + bpf_map__set_value_size(ctx.skel->maps.hash_map_bench, 8); + bpf_map__set_map_flags(ctx.skel->maps.hash_map_bench, args.map_flags); + + ctx.skel->bss->nr_entries = args.nr_entries; + ctx.skel->bss->nr_loops = args.nr_loops / args.nr_entries; + + if (args.key_size > 4) { + for (i = 1; i < args.key_size/4; i++) + ctx.skel->bss->key[i] = 2654435761 * i; + } + + ret = bpf_hashmap_lookup__load(ctx.skel); + if (ret) { + bpf_hashmap_lookup__destroy(ctx.skel); + fprintf(stderr, "failed to load map: %s", strerror(-ret)); + exit(1); + } + + /* fill in the hash_map */ + map_fd = bpf_map__fd(ctx.skel->maps.hash_map_bench); + for (u64 i = 0; i < args.nr_entries; i++) { + patch_key(i, ctx.skel->bss->key); + bpf_map_update_elem(map_fd, ctx.skel->bss->key, &i, BPF_ANY); + } + + link = bpf_program__attach(ctx.skel->progs.benchmark); + if (!link) { + fprintf(stderr, "failed to attach program!\n"); + exit(1); + } +} + +static inline double events_from_time(u64 time) +{ + if (time) + return args.nr_loops * 1000000000llu / time / 1000000.0L; + + return 0; +} + +static int compute_events(u64 *times, double *events_mean, double *events_stddev, u64 *mean_time) +{ + int i, n = 0; + + *events_mean = 0; + *events_stddev = 0; + *mean_time = 0; + + for (i = 0; i < 32; i++) { + if (!times[i]) + break; + *mean_time += times[i]; + *events_mean += events_from_time(times[i]); + n += 1; + } + if (!n) + return 0; + + *mean_time /= n; + *events_mean /= n; + + if (n > 1) { + for (i = 0; i < n; i++) { + double events_i = *events_mean - events_from_time(times[i]); + *events_stddev += events_i * events_i / (n - 1); + } + *events_stddev = sqrt(*events_stddev); + } + + return n; +} + +static void hashmap_report_final(struct bench_res res[], int res_cnt) +{ + unsigned int nr_cpus = bpf_num_possible_cpus(); + double events_mean, events_stddev; + u64 mean_time; + int i, n; + + for (i = 0; i < nr_cpus; i++) { + n = compute_events(ctx.skel->bss->percpu_times[i], &events_mean, + &events_stddev, &mean_time); + if (n == 0) + continue; + + if (env.quiet) { + /* we expect only one cpu to be present */ + if (env.affinity) + printf("%.3lf\n", events_mean); + else + printf("cpu%02d %.3lf\n", i, events_mean); + } else { + printf("cpu%02d: lookup %.3lfM ± %.3lfM events/sec" + " (approximated from %d samples of ~%lums)\n", + i, events_mean, 2*events_stddev, + n, mean_time / 1000000); + } + } +} + +const struct bench bench_bpf_hashmap_lookup = { + .name = "bpf-hashmap-lookup", + .argp = &bench_hashmap_lookup_argp, + .validate = validate, + .setup = setup, + .producer_thread = producer, + .consumer_thread = consumer, + .measure = measure, + .report_progress = NULL, + .report_final = hashmap_report_final, +}; diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_loop.c b/tools/testing/selftests/bpf/benchs/bench_bpf_loop.c index d0a6572bfab6..d8a0394e10b1 100644 --- a/tools/testing/selftests/bpf/benchs/bench_bpf_loop.c +++ b/tools/testing/selftests/bpf/benchs/bench_bpf_loop.c @@ -95,6 +95,7 @@ static void setup(void) const struct bench bench_bpf_loop = { .name = "bpf-loop", + .argp = &bench_bpf_loop_argp, .validate = validate, .setup = setup, .producer_thread = producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_local_storage.c b/tools/testing/selftests/bpf/benchs/bench_local_storage.c index 5a378c84e81f..d4b2817306d4 100644 --- a/tools/testing/selftests/bpf/benchs/bench_local_storage.c +++ b/tools/testing/selftests/bpf/benchs/bench_local_storage.c @@ -255,6 +255,7 @@ static void *producer(void *input) */ const struct bench bench_local_storage_cache_seq_get = { .name = "local-storage-cache-seq-get", + .argp = &bench_local_storage_argp, .validate = validate, .setup = local_storage_cache_get_setup, .producer_thread = producer, @@ -266,6 +267,7 @@ const struct bench bench_local_storage_cache_seq_get = { const struct bench bench_local_storage_cache_interleaved_get = { .name = "local-storage-cache-int-get", + .argp = &bench_local_storage_argp, .validate = validate, .setup = local_storage_cache_get_interleaved_setup, .producer_thread = producer, @@ -277,6 +279,7 @@ const struct bench bench_local_storage_cache_interleaved_get = { const struct bench bench_local_storage_cache_hashmap_control = { .name = "local-storage-cache-hashmap-control", + .argp = &bench_local_storage_argp, .validate = validate, .setup = hashmap_setup, .producer_thread = producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_local_storage_rcu_tasks_trace.c b/tools/testing/selftests/bpf/benchs/bench_local_storage_rcu_tasks_trace.c index 43f109d93130..d5eb5587f2aa 100644 --- a/tools/testing/selftests/bpf/benchs/bench_local_storage_rcu_tasks_trace.c +++ b/tools/testing/selftests/bpf/benchs/bench_local_storage_rcu_tasks_trace.c @@ -12,17 +12,14 @@ static struct { __u32 nr_procs; __u32 kthread_pid; - bool quiet; } args = { .nr_procs = 1000, .kthread_pid = 0, - .quiet = false, }; enum { ARG_NR_PROCS = 7000, ARG_KTHREAD_PID = 7001, - ARG_QUIET = 7002, }; static const struct argp_option opts[] = { @@ -30,8 +27,6 @@ static const struct argp_option opts[] = { "Set number of user processes to spin up"}, { "kthread_pid", ARG_KTHREAD_PID, "PID", 0, "Pid of rcu_tasks_trace kthread for ticks tracking"}, - { "quiet", ARG_QUIET, "{0,1}", 0, - "If true, don't report progress"}, {}, }; @@ -56,14 +51,6 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) } args.kthread_pid = ret; break; - case ARG_QUIET: - ret = strtol(arg, NULL, 10); - if (ret < 0 || ret > 1) { - fprintf(stderr, "invalid quiet %ld\n", ret); - argp_usage(state); - } - args.quiet = ret; - break; break; default: return ARGP_ERR_UNKNOWN; @@ -230,7 +217,7 @@ static void report_progress(int iter, struct bench_res *res, long delta_ns) exit(1); } - if (args.quiet) + if (env.quiet) return; printf("Iter %d\t avg tasks_trace grace period latency\t%lf ns\n", @@ -271,6 +258,7 @@ static void report_final(struct bench_res res[], int res_cnt) */ const struct bench bench_local_storage_tasks_trace = { .name = "local-storage-tasks-trace", + .argp = &bench_local_storage_rcu_tasks_trace_argp, .validate = validate, .setup = local_storage_tasks_trace_setup, .producer_thread = producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_ringbufs.c b/tools/testing/selftests/bpf/benchs/bench_ringbufs.c index c2554f9695ff..fc91fdac4faa 100644 --- a/tools/testing/selftests/bpf/benchs/bench_ringbufs.c +++ b/tools/testing/selftests/bpf/benchs/bench_ringbufs.c @@ -518,6 +518,7 @@ static void *perfbuf_custom_consumer(void *input) const struct bench bench_rb_libbpf = { .name = "rb-libbpf", + .argp = &bench_ringbufs_argp, .validate = bufs_validate, .setup = ringbuf_libbpf_setup, .producer_thread = bufs_sample_producer, @@ -529,6 +530,7 @@ const struct bench bench_rb_libbpf = { const struct bench bench_rb_custom = { .name = "rb-custom", + .argp = &bench_ringbufs_argp, .validate = bufs_validate, .setup = ringbuf_custom_setup, .producer_thread = bufs_sample_producer, @@ -540,6 +542,7 @@ const struct bench bench_rb_custom = { const struct bench bench_pb_libbpf = { .name = "pb-libbpf", + .argp = &bench_ringbufs_argp, .validate = bufs_validate, .setup = perfbuf_libbpf_setup, .producer_thread = bufs_sample_producer, @@ -551,6 +554,7 @@ const struct bench bench_pb_libbpf = { const struct bench bench_pb_custom = { .name = "pb-custom", + .argp = &bench_ringbufs_argp, .validate = bufs_validate, .setup = perfbuf_libbpf_setup, .producer_thread = bufs_sample_producer, diff --git a/tools/testing/selftests/bpf/benchs/bench_strncmp.c b/tools/testing/selftests/bpf/benchs/bench_strncmp.c index 494b591c0289..d3fad2ba6916 100644 --- a/tools/testing/selftests/bpf/benchs/bench_strncmp.c +++ b/tools/testing/selftests/bpf/benchs/bench_strncmp.c @@ -140,6 +140,7 @@ static void strncmp_measure(struct bench_res *res) const struct bench bench_strncmp_no_helper = { .name = "strncmp-no-helper", + .argp = &bench_strncmp_argp, .validate = strncmp_validate, .setup = strncmp_no_helper_setup, .producer_thread = strncmp_producer, @@ -151,6 +152,7 @@ const struct bench bench_strncmp_no_helper = { const struct bench bench_strncmp_helper = { .name = "strncmp-helper", + .argp = &bench_strncmp_argp, .validate = strncmp_validate, .setup = strncmp_helper_setup, .producer_thread = strncmp_producer, diff --git a/tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh b/tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh index 1e2de838f9fa..cd2efd3fdef3 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_bpf_hashmap_full_update.sh @@ -6,6 +6,6 @@ source ./benchs/run_common.sh set -eufo pipefail nr_threads=`expr $(cat /proc/cpuinfo | grep "processor"| wc -l) - 1` -summary=$($RUN_BENCH -p $nr_threads bpf-hashmap-ful-update) +summary=$($RUN_BENCH -p $nr_threads bpf-hashmap-full-update) printf "$summary" printf "\n" diff --git a/tools/testing/selftests/bpf/benchs/run_bench_local_storage_rcu_tasks_trace.sh b/tools/testing/selftests/bpf/benchs/run_bench_local_storage_rcu_tasks_trace.sh index 5dac1f02892c..3e8a969f2096 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_local_storage_rcu_tasks_trace.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_local_storage_rcu_tasks_trace.sh @@ -8,4 +8,4 @@ if [ -z $kthread_pid ]; then exit 1 fi -./bench --nr_procs 15000 --kthread_pid $kthread_pid -d 600 --quiet 1 local-storage-tasks-trace +./bench --nr_procs 15000 --kthread_pid $kthread_pid -d 600 --quiet local-storage-tasks-trace diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 424f7bbbfe9b..dbd2c729781a 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -65,4 +65,28 @@ extern struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ks */ extern struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym; +/* Description + * Remove 'node' from rbtree with root 'root' + * Returns + * Pointer to the removed node, or NULL if 'root' didn't contain 'node' + */ +extern struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, + struct bpf_rb_node *node) __ksym; + +/* Description + * Add 'node' to rbtree with root 'root' using comparator 'less' + * Returns + * Nothing + */ +extern void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) __ksym; + +/* Description + * Return the first (leftmost) node in input tree + * Returns + * Pointer to the node, which is _not_ removed from the tree. If the tree + * contains no nodes, returns NULL. + */ +extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym; + #endif diff --git a/tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c b/tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c index f472d28ad11a..16f1671e4bde 100644 --- a/tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c +++ b/tools/testing/selftests/bpf/map_tests/map_in_map_batch_ops.c @@ -18,7 +18,7 @@ static __u32 get_map_id_from_fd(int map_fd) uint32_t info_len = sizeof(map_info); int ret; - ret = bpf_obj_get_info_by_fd(map_fd, &map_info, &info_len); + ret = bpf_map_get_info_by_fd(map_fd, &map_info, &info_len); CHECK(ret < 0, "Finding map info failed", "error:%s\n", strerror(errno)); diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_iter.c b/tools/testing/selftests/bpf/prog_tests/bpf_iter.c index 3af6450763e9..1f02168103dd 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_iter.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_iter.c @@ -195,8 +195,8 @@ static void check_bpf_link_info(const struct bpf_program *prog) return; info_len = sizeof(info); - err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &info, &info_len); - ASSERT_OK(err, "bpf_obj_get_info_by_fd"); + err = bpf_link_get_info_by_fd(bpf_link__fd(link), &info, &info_len); + ASSERT_OK(err, "bpf_link_get_info_by_fd"); ASSERT_EQ(info.iter.task.tid, getpid(), "check_task_tid"); bpf_link__destroy(link); @@ -684,13 +684,13 @@ static void test_overflow(bool test_e2big_overflow, bool ret1) /* setup filtering map_id in bpf program */ map_info_len = sizeof(map_info); - err = bpf_obj_get_info_by_fd(map1_fd, &map_info, &map_info_len); + err = bpf_map_get_info_by_fd(map1_fd, &map_info, &map_info_len); if (CHECK(err, "get_map_info", "get map info failed: %s\n", strerror(errno))) goto free_map2; skel->bss->map1_id = map_info.id; - err = bpf_obj_get_info_by_fd(map2_fd, &map_info, &map_info_len); + err = bpf_map_get_info_by_fd(map2_fd, &map_info, &map_info_len); if (CHECK(err, "get_map_info", "get map info failed: %s\n", strerror(errno))) goto free_map2; diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c b/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c index e1c1e521cca2..675b90b15280 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c @@ -44,7 +44,7 @@ void serial_test_bpf_obj_id(void) CHECK(err >= 0 || errno != ENOENT, "get-fd-by-notexist-link-id", "err %d errno %d\n", err, errno); - /* Check bpf_obj_get_info_by_fd() */ + /* Check bpf_map_get_info_by_fd() */ bzero(zeros, sizeof(zeros)); for (i = 0; i < nr_iters; i++) { now = time(NULL); @@ -79,7 +79,7 @@ void serial_test_bpf_obj_id(void) /* Check getting map info */ info_len = sizeof(struct bpf_map_info) * 2; bzero(&map_infos[i], info_len); - err = bpf_obj_get_info_by_fd(map_fds[i], &map_infos[i], + err = bpf_map_get_info_by_fd(map_fds[i], &map_infos[i], &info_len); if (CHECK(err || map_infos[i].type != BPF_MAP_TYPE_ARRAY || @@ -118,8 +118,8 @@ void serial_test_bpf_obj_id(void) err = clock_gettime(CLOCK_BOOTTIME, &boot_time_ts); if (CHECK_FAIL(err)) goto done; - err = bpf_obj_get_info_by_fd(prog_fds[i], &prog_infos[i], - &info_len); + err = bpf_prog_get_info_by_fd(prog_fds[i], &prog_infos[i], + &info_len); load_time = (real_time_ts.tv_sec - boot_time_ts.tv_sec) + (prog_infos[i].load_time / nsec_per_sec); if (CHECK(err || @@ -161,8 +161,8 @@ void serial_test_bpf_obj_id(void) bzero(&link_infos[i], info_len); link_infos[i].raw_tracepoint.tp_name = ptr_to_u64(&tp_name); link_infos[i].raw_tracepoint.tp_name_len = sizeof(tp_name); - err = bpf_obj_get_info_by_fd(bpf_link__fd(links[i]), - &link_infos[i], &info_len); + err = bpf_link_get_info_by_fd(bpf_link__fd(links[i]), + &link_infos[i], &info_len); if (CHECK(err || link_infos[i].type != BPF_LINK_TYPE_RAW_TRACEPOINT || link_infos[i].prog_id != prog_infos[i].id || @@ -217,7 +217,7 @@ void serial_test_bpf_obj_id(void) * prog_info.map_ids = NULL */ prog_info.nr_map_ids = 1; - err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &info_len); if (CHECK(!err || errno != EFAULT, "get-prog-fd-bad-nr-map-ids", "err %d errno %d(%d)", err, errno, EFAULT)) @@ -228,7 +228,7 @@ void serial_test_bpf_obj_id(void) saved_map_id = *(int *)((long)prog_infos[i].map_ids); prog_info.map_ids = prog_infos[i].map_ids; prog_info.nr_map_ids = 2; - err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &info_len); prog_infos[i].jited_prog_insns = 0; prog_infos[i].xlated_prog_insns = 0; CHECK(err || info_len != sizeof(struct bpf_prog_info) || @@ -277,7 +277,7 @@ void serial_test_bpf_obj_id(void) if (CHECK_FAIL(err)) goto done; - err = bpf_obj_get_info_by_fd(map_fd, &map_info, &info_len); + err = bpf_map_get_info_by_fd(map_fd, &map_info, &info_len); CHECK(err || info_len != sizeof(struct bpf_map_info) || memcmp(&map_info, &map_infos[i], info_len) || array_value != array_magic_value, @@ -322,7 +322,7 @@ void serial_test_bpf_obj_id(void) nr_id_found++; - err = bpf_obj_get_info_by_fd(link_fd, &link_info, &info_len); + err = bpf_link_get_info_by_fd(link_fd, &link_info, &info_len); cmp_res = memcmp(&link_info, &link_infos[i], offsetof(struct bpf_link_info, raw_tracepoint)); CHECK(err || info_len != sizeof(link_info) || cmp_res, diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c index de1b5b9eb93a..cbb600be943d 100644 --- a/tools/testing/selftests/bpf/prog_tests/btf.c +++ b/tools/testing/selftests/bpf/prog_tests/btf.c @@ -4422,7 +4422,7 @@ static int test_big_btf_info(unsigned int test_num) info->btf = ptr_to_u64(user_btf); info->btf_size = raw_btf_size; - err = bpf_obj_get_info_by_fd(btf_fd, info, &info_len); + err = bpf_btf_get_info_by_fd(btf_fd, info, &info_len); if (CHECK(!err, "!err")) { err = -1; goto done; @@ -4435,7 +4435,7 @@ static int test_big_btf_info(unsigned int test_num) * to userspace. */ info_garbage.garbage = 0; - err = bpf_obj_get_info_by_fd(btf_fd, info, &info_len); + err = bpf_btf_get_info_by_fd(btf_fd, info, &info_len); if (CHECK(err || info_len != sizeof(*info), "err:%d errno:%d info_len:%u sizeof(*info):%zu", err, errno, info_len, sizeof(*info))) { @@ -4499,7 +4499,7 @@ static int test_btf_id(unsigned int test_num) /* Test BPF_OBJ_GET_INFO_BY_ID on btf_id */ info_len = sizeof(info[0]); - err = bpf_obj_get_info_by_fd(btf_fd[0], &info[0], &info_len); + err = bpf_btf_get_info_by_fd(btf_fd[0], &info[0], &info_len); if (CHECK(err, "errno:%d", errno)) { err = -1; goto done; @@ -4512,7 +4512,7 @@ static int test_btf_id(unsigned int test_num) } ret = 0; - err = bpf_obj_get_info_by_fd(btf_fd[1], &info[1], &info_len); + err = bpf_btf_get_info_by_fd(btf_fd[1], &info[1], &info_len); if (CHECK(err || info[0].id != info[1].id || info[0].btf_size != info[1].btf_size || (ret = memcmp(user_btf[0], user_btf[1], info[0].btf_size)), @@ -4535,7 +4535,7 @@ static int test_btf_id(unsigned int test_num) } info_len = sizeof(map_info); - err = bpf_obj_get_info_by_fd(map_fd, &map_info, &info_len); + err = bpf_map_get_info_by_fd(map_fd, &map_info, &info_len); if (CHECK(err || map_info.btf_id != info[0].id || map_info.btf_key_type_id != 1 || map_info.btf_value_type_id != 2, "err:%d errno:%d info.id:%u btf_id:%u btf_key_type_id:%u btf_value_type_id:%u", @@ -4638,7 +4638,7 @@ static void do_test_get_info(unsigned int test_num) info.btf_size = user_btf_size; ret = 0; - err = bpf_obj_get_info_by_fd(btf_fd, &info, &info_len); + err = bpf_btf_get_info_by_fd(btf_fd, &info, &info_len); if (CHECK(err || !info.id || info_len != sizeof(info) || info.btf_size != raw_btf_size || (ret = memcmp(raw_btf, user_btf, expected_nbytes)), @@ -4755,7 +4755,7 @@ static void do_test_file(unsigned int test_num) /* get necessary program info */ info_len = sizeof(struct bpf_prog_info); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err < 0, "invalid get info (1st) errno:%d", errno)) { fprintf(stderr, "%s\n", btf_log_buf); @@ -4787,7 +4787,7 @@ static void do_test_file(unsigned int test_num) info.func_info_rec_size = rec_size; info.func_info = ptr_to_u64(func_info); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err < 0, "invalid get info (2nd) errno:%d", errno)) { fprintf(stderr, "%s\n", btf_log_buf); @@ -6405,7 +6405,7 @@ static int test_get_finfo(const struct prog_info_raw_test *test, /* get necessary lens */ info_len = sizeof(struct bpf_prog_info); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err < 0, "invalid get info (1st) errno:%d", errno)) { fprintf(stderr, "%s\n", btf_log_buf); return -1; @@ -6435,7 +6435,7 @@ static int test_get_finfo(const struct prog_info_raw_test *test, info.nr_func_info = nr_func_info; info.func_info_rec_size = rec_size; info.func_info = ptr_to_u64(func_info); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err < 0, "invalid get info (2nd) errno:%d", errno)) { fprintf(stderr, "%s\n", btf_log_buf); err = -1; @@ -6499,7 +6499,7 @@ static int test_get_linfo(const struct prog_info_raw_test *test, nr_jited_func_lens = nr_jited_ksyms; info_len = sizeof(struct bpf_prog_info); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err < 0, "err:%d errno:%d", err, errno)) { err = -1; goto done; @@ -6573,7 +6573,7 @@ static int test_get_linfo(const struct prog_info_raw_test *test, info.jited_func_lens = ptr_to_u64(jited_func_lens); } - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); /* * Only recheck the info.*line_info* fields. diff --git a/tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c b/tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c index eb90a6b8850d..a8b53b8736f0 100644 --- a/tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c +++ b/tools/testing/selftests/bpf/prog_tests/btf_map_in_map.c @@ -14,7 +14,7 @@ static __u32 bpf_map_id(struct bpf_map *map) int err; memset(&info, 0, info_len); - err = bpf_obj_get_info_by_fd(bpf_map__fd(map), &info, &info_len); + err = bpf_map_get_info_by_fd(bpf_map__fd(map), &info, &info_len); if (err) return 0; return info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/cgrp_kfunc.c b/tools/testing/selftests/bpf/prog_tests/cgrp_kfunc.c index f3bb0e16e088..b3f7985c8504 100644 --- a/tools/testing/selftests/bpf/prog_tests/cgrp_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/cgrp_kfunc.c @@ -8,9 +8,6 @@ #include "cgrp_kfunc_failure.skel.h" #include "cgrp_kfunc_success.skel.h" -static size_t log_buf_sz = 1 << 20; /* 1 MB */ -static char obj_log_buf[1048576]; - static struct cgrp_kfunc_success *open_load_cgrp_kfunc_skel(void) { struct cgrp_kfunc_success *skel; @@ -89,65 +86,6 @@ static const char * const success_tests[] = { "test_cgrp_get_ancestors", }; -static struct { - const char *prog_name; - const char *expected_err_msg; -} failure_tests[] = { - {"cgrp_kfunc_acquire_untrusted", "Possibly NULL pointer passed to trusted arg0"}, - {"cgrp_kfunc_acquire_fp", "arg#0 pointer type STRUCT cgroup must point"}, - {"cgrp_kfunc_acquire_unsafe_kretprobe", "reg type unsupported for arg#0 function"}, - {"cgrp_kfunc_acquire_trusted_walked", "R1 must be referenced or trusted"}, - {"cgrp_kfunc_acquire_null", "Possibly NULL pointer passed to trusted arg0"}, - {"cgrp_kfunc_acquire_unreleased", "Unreleased reference"}, - {"cgrp_kfunc_get_non_kptr_param", "arg#0 expected pointer to map value"}, - {"cgrp_kfunc_get_non_kptr_acquired", "arg#0 expected pointer to map value"}, - {"cgrp_kfunc_get_null", "arg#0 expected pointer to map value"}, - {"cgrp_kfunc_xchg_unreleased", "Unreleased reference"}, - {"cgrp_kfunc_get_unreleased", "Unreleased reference"}, - {"cgrp_kfunc_release_untrusted", "arg#0 is untrusted_ptr_or_null_ expected ptr_ or socket"}, - {"cgrp_kfunc_release_fp", "arg#0 pointer type STRUCT cgroup must point"}, - {"cgrp_kfunc_release_null", "arg#0 is ptr_or_null_ expected ptr_ or socket"}, - {"cgrp_kfunc_release_unacquired", "release kernel function bpf_cgroup_release expects"}, -}; - -static void verify_fail(const char *prog_name, const char *expected_err_msg) -{ - LIBBPF_OPTS(bpf_object_open_opts, opts); - struct cgrp_kfunc_failure *skel; - int err, i; - - opts.kernel_log_buf = obj_log_buf; - opts.kernel_log_size = log_buf_sz; - opts.kernel_log_level = 1; - - skel = cgrp_kfunc_failure__open_opts(&opts); - if (!ASSERT_OK_PTR(skel, "cgrp_kfunc_failure__open_opts")) - goto cleanup; - - for (i = 0; i < ARRAY_SIZE(failure_tests); i++) { - struct bpf_program *prog; - const char *curr_name = failure_tests[i].prog_name; - - prog = bpf_object__find_program_by_name(skel->obj, curr_name); - if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name")) - goto cleanup; - - bpf_program__set_autoload(prog, !strcmp(curr_name, prog_name)); - } - - err = cgrp_kfunc_failure__load(skel); - if (!ASSERT_ERR(err, "unexpected load success")) - goto cleanup; - - if (!ASSERT_OK_PTR(strstr(obj_log_buf, expected_err_msg), "expected_err_msg")) { - fprintf(stderr, "Expected err_msg: %s\n", expected_err_msg); - fprintf(stderr, "Verifier output: %s\n", obj_log_buf); - } - -cleanup: - cgrp_kfunc_failure__destroy(skel); -} - void test_cgrp_kfunc(void) { int i, err; @@ -163,12 +101,7 @@ void test_cgrp_kfunc(void) run_success_test(success_tests[i]); } - for (i = 0; i < ARRAY_SIZE(failure_tests); i++) { - if (!test__start_subtest(failure_tests[i].prog_name)) - continue; - - verify_fail(failure_tests[i].prog_name, failure_tests[i].expected_err_msg); - } + RUN_TESTS(cgrp_kfunc_failure); cleanup: cleanup_cgroup_environment(); diff --git a/tools/testing/selftests/bpf/prog_tests/check_mtu.c b/tools/testing/selftests/bpf/prog_tests/check_mtu.c index 12f4395f18b3..5338d2ea0460 100644 --- a/tools/testing/selftests/bpf/prog_tests/check_mtu.c +++ b/tools/testing/selftests/bpf/prog_tests/check_mtu.c @@ -59,7 +59,7 @@ static void test_check_mtu_xdp_attach(void) memset(&link_info, 0, sizeof(link_info)); fd = bpf_link__fd(link); - err = bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len); + err = bpf_link_get_info_by_fd(fd, &link_info, &link_info_len); if (CHECK(err, "link_info", "failed: %d\n", err)) goto out; diff --git a/tools/testing/selftests/bpf/prog_tests/dynptr.c b/tools/testing/selftests/bpf/prog_tests/dynptr.c index 7faaf6d9e0d4..b99264ec0d9c 100644 --- a/tools/testing/selftests/bpf/prog_tests/dynptr.c +++ b/tools/testing/selftests/bpf/prog_tests/dynptr.c @@ -5,14 +5,10 @@ #include "dynptr_fail.skel.h" #include "dynptr_success.skel.h" -static struct { - const char *prog_name; - const char *expected_err_msg; -} dynptr_tests[] = { - /* success cases */ - {"test_read_write", NULL}, - {"test_data_slice", NULL}, - {"test_ringbuf", NULL}, +static const char * const success_tests[] = { + "test_read_write", + "test_data_slice", + "test_ringbuf", }; static void verify_success(const char *prog_name) @@ -53,11 +49,11 @@ void test_dynptr(void) { int i; - for (i = 0; i < ARRAY_SIZE(dynptr_tests); i++) { - if (!test__start_subtest(dynptr_tests[i].prog_name)) + for (i = 0; i < ARRAY_SIZE(success_tests); i++) { + if (!test__start_subtest(success_tests[i])) continue; - verify_success(dynptr_tests[i].prog_name); + verify_success(success_tests[i]); } RUN_TESTS(dynptr_fail); diff --git a/tools/testing/selftests/bpf/prog_tests/enable_stats.c b/tools/testing/selftests/bpf/prog_tests/enable_stats.c index 2cb2085917e7..75f85d0fe74a 100644 --- a/tools/testing/selftests/bpf/prog_tests/enable_stats.c +++ b/tools/testing/selftests/bpf/prog_tests/enable_stats.c @@ -28,7 +28,7 @@ void test_enable_stats(void) prog_fd = bpf_program__fd(skel->progs.test_enable_stats); memset(&info, 0, info_len); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err, "get_prog_info", "failed to get bpf_prog_info for fd %d\n", prog_fd)) goto cleanup; diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 20f5fa0fcec9..8ec73fdfcdab 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -79,7 +79,7 @@ static void test_fexit_bpf2bpf_common(const char *obj_file, return; info_len = sizeof(prog_info); - err = bpf_obj_get_info_by_fd(tgt_fd, &prog_info, &info_len); + err = bpf_prog_get_info_by_fd(tgt_fd, &prog_info, &info_len); if (!ASSERT_OK(err, "tgt_fd_get_info")) goto close_prog; @@ -136,8 +136,8 @@ static void test_fexit_bpf2bpf_common(const char *obj_file, info_len = sizeof(link_info); memset(&link_info, 0, sizeof(link_info)); - err = bpf_obj_get_info_by_fd(bpf_link__fd(link[i]), - &link_info, &info_len); + err = bpf_link_get_info_by_fd(bpf_link__fd(link[i]), + &link_info, &info_len); ASSERT_OK(err, "link_fd_get_info"); ASSERT_EQ(link_info.tracing.attach_type, bpf_program__expected_attach_type(prog[i]), @@ -417,7 +417,7 @@ static int find_prog_btf_id(const char *name, __u32 attach_prog_fd) struct btf *btf; int ret; - ret = bpf_obj_get_info_by_fd(attach_prog_fd, &info, &info_len); + ret = bpf_prog_get_info_by_fd(attach_prog_fd, &info, &info_len); if (ret) return ret; @@ -483,12 +483,12 @@ static void test_fentry_to_cgroup_bpf(void) if (!ASSERT_GE(fentry_fd, 0, "load_fentry")) goto cleanup; - /* Make sure bpf_obj_get_info_by_fd works correctly when attaching + /* Make sure bpf_prog_get_info_by_fd works correctly when attaching * to another BPF program. */ - ASSERT_OK(bpf_obj_get_info_by_fd(fentry_fd, &info, &info_len), - "bpf_obj_get_info_by_fd"); + ASSERT_OK(bpf_prog_get_info_by_fd(fentry_fd, &info, &info_len), + "bpf_prog_get_info_by_fd"); ASSERT_EQ(info.btf_id, 0, "info.btf_id"); ASSERT_EQ(info.attach_btf_id, btf_id, "info.attach_btf_id"); diff --git a/tools/testing/selftests/bpf/prog_tests/fib_lookup.c b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c new file mode 100644 index 000000000000..61ccddccf485 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include <sys/types.h> +#include <net/if.h> + +#include "test_progs.h" +#include "network_helpers.h" +#include "fib_lookup.skel.h" + +#define SYS(fmt, ...) \ + ({ \ + char cmd[1024]; \ + snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \ + if (!ASSERT_OK(system(cmd), cmd)) \ + goto fail; \ + }) + +#define NS_TEST "fib_lookup_ns" +#define IPV6_IFACE_ADDR "face::face" +#define IPV6_NUD_FAILED_ADDR "face::1" +#define IPV6_NUD_STALE_ADDR "face::2" +#define IPV4_IFACE_ADDR "10.0.0.254" +#define IPV4_NUD_FAILED_ADDR "10.0.0.1" +#define IPV4_NUD_STALE_ADDR "10.0.0.2" +#define DMAC "11:11:11:11:11:11" +#define DMAC_INIT { 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, } + +struct fib_lookup_test { + const char *desc; + const char *daddr; + int expected_ret; + int lookup_flags; + __u8 dmac[6]; +}; + +static const struct fib_lookup_test tests[] = { + { .desc = "IPv6 failed neigh", + .daddr = IPV6_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_NO_NEIGH, }, + { .desc = "IPv6 stale neigh", + .daddr = IPV6_NUD_STALE_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .dmac = DMAC_INIT, }, + { .desc = "IPv6 skip neigh", + .daddr = IPV6_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, }, + { .desc = "IPv4 failed neigh", + .daddr = IPV4_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_NO_NEIGH, }, + { .desc = "IPv4 stale neigh", + .daddr = IPV4_NUD_STALE_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .dmac = DMAC_INIT, }, + { .desc = "IPv4 skip neigh", + .daddr = IPV4_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, }, +}; + +static int ifindex; + +static int setup_netns(void) +{ + int err; + + SYS("ip link add veth1 type veth peer name veth2"); + SYS("ip link set dev veth1 up"); + + SYS("ip addr add %s/64 dev veth1 nodad", IPV6_IFACE_ADDR); + SYS("ip neigh add %s dev veth1 nud failed", IPV6_NUD_FAILED_ADDR); + SYS("ip neigh add %s dev veth1 lladdr %s nud stale", IPV6_NUD_STALE_ADDR, DMAC); + + SYS("ip addr add %s/24 dev veth1 nodad", IPV4_IFACE_ADDR); + SYS("ip neigh add %s dev veth1 nud failed", IPV4_NUD_FAILED_ADDR); + SYS("ip neigh add %s dev veth1 lladdr %s nud stale", IPV4_NUD_STALE_ADDR, DMAC); + + err = write_sysctl("/proc/sys/net/ipv4/conf/veth1/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.conf.veth1.forwarding)")) + goto fail; + + err = write_sysctl("/proc/sys/net/ipv6/conf/veth1/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv6.conf.veth1.forwarding)")) + goto fail; + + return 0; +fail: + return -1; +} + +static int set_lookup_params(struct bpf_fib_lookup *params, const char *daddr) +{ + int ret; + + memset(params, 0, sizeof(*params)); + + params->l4_protocol = IPPROTO_TCP; + params->ifindex = ifindex; + + if (inet_pton(AF_INET6, daddr, params->ipv6_dst) == 1) { + params->family = AF_INET6; + ret = inet_pton(AF_INET6, IPV6_IFACE_ADDR, params->ipv6_src); + if (!ASSERT_EQ(ret, 1, "inet_pton(IPV6_IFACE_ADDR)")) + return -1; + return 0; + } + + ret = inet_pton(AF_INET, daddr, ¶ms->ipv4_dst); + if (!ASSERT_EQ(ret, 1, "convert IP[46] address")) + return -1; + params->family = AF_INET; + ret = inet_pton(AF_INET, IPV4_IFACE_ADDR, ¶ms->ipv4_src); + if (!ASSERT_EQ(ret, 1, "inet_pton(IPV4_IFACE_ADDR)")) + return -1; + + return 0; +} + +static void mac_str(char *b, const __u8 *mac) +{ + sprintf(b, "%02X:%02X:%02X:%02X:%02X:%02X", + mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]); +} + +void test_fib_lookup(void) +{ + struct bpf_fib_lookup *fib_params; + struct nstoken *nstoken = NULL; + struct __sk_buff skb = { }; + struct fib_lookup *skel; + int prog_fd, err, ret, i; + + /* The test does not use the skb->data, so + * use pkt_v6 for both v6 and v4 test. + */ + LIBBPF_OPTS(bpf_test_run_opts, run_opts, + .data_in = &pkt_v6, + .data_size_in = sizeof(pkt_v6), + .ctx_in = &skb, + .ctx_size_in = sizeof(skb), + ); + + skel = fib_lookup__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel open_and_load")) + return; + prog_fd = bpf_program__fd(skel->progs.fib_lookup); + + SYS("ip netns add %s", NS_TEST); + + nstoken = open_netns(NS_TEST); + if (!ASSERT_OK_PTR(nstoken, "open_netns")) + goto fail; + + if (setup_netns()) + goto fail; + + ifindex = if_nametoindex("veth1"); + skb.ifindex = ifindex; + fib_params = &skel->bss->fib_params; + + for (i = 0; i < ARRAY_SIZE(tests); i++) { + printf("Testing %s\n", tests[i].desc); + + if (set_lookup_params(fib_params, tests[i].daddr)) + continue; + skel->bss->fib_lookup_ret = -1; + skel->bss->lookup_flags = BPF_FIB_LOOKUP_OUTPUT | + tests[i].lookup_flags; + + err = bpf_prog_test_run_opts(prog_fd, &run_opts); + if (!ASSERT_OK(err, "bpf_prog_test_run_opts")) + continue; + + ASSERT_EQ(tests[i].expected_ret, skel->bss->fib_lookup_ret, + "fib_lookup_ret"); + + ret = memcmp(tests[i].dmac, fib_params->dmac, sizeof(tests[i].dmac)); + if (!ASSERT_EQ(ret, 0, "dmac not match")) { + char expected[18], actual[18]; + + mac_str(expected, tests[i].dmac); + mac_str(actual, fib_params->dmac); + printf("dmac expected %s actual %s\n", expected, actual); + } + } + +fail: + if (nstoken) + close_netns(nstoken); + system("ip netns del " NS_TEST " &> /dev/null"); + fib_lookup__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/prog_tests/flow_dissector_reattach.c b/tools/testing/selftests/bpf/prog_tests/flow_dissector_reattach.c index 7c79462d2702..9333f7346d15 100644 --- a/tools/testing/selftests/bpf/prog_tests/flow_dissector_reattach.c +++ b/tools/testing/selftests/bpf/prog_tests/flow_dissector_reattach.c @@ -60,9 +60,9 @@ static __u32 query_prog_id(int prog) __u32 info_len = sizeof(info); int err; - err = bpf_obj_get_info_by_fd(prog, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog, &info, &info_len); if (CHECK_FAIL(err || info_len != sizeof(info))) { - perror("bpf_obj_get_info_by_fd"); + perror("bpf_prog_get_info_by_fd"); return 0; } @@ -497,7 +497,7 @@ static void test_link_get_info(int netns, int prog1, int prog2) } info_len = sizeof(info); - err = bpf_obj_get_info_by_fd(link, &info, &info_len); + err = bpf_link_get_info_by_fd(link, &info, &info_len); if (CHECK_FAIL(err)) { perror("bpf_obj_get_info"); goto out_unlink; @@ -521,7 +521,7 @@ static void test_link_get_info(int netns, int prog1, int prog2) link_id = info.id; info_len = sizeof(info); - err = bpf_obj_get_info_by_fd(link, &info, &info_len); + err = bpf_link_get_info_by_fd(link, &info, &info_len); if (CHECK_FAIL(err)) { perror("bpf_obj_get_info"); goto out_unlink; @@ -546,7 +546,7 @@ static void test_link_get_info(int netns, int prog1, int prog2) netns = -1; info_len = sizeof(info); - err = bpf_obj_get_info_by_fd(link, &info, &info_len); + err = bpf_link_get_info_by_fd(link, &info, &info_len); if (CHECK_FAIL(err)) { perror("bpf_obj_get_info"); goto out_unlink; diff --git a/tools/testing/selftests/bpf/prog_tests/htab_reuse.c b/tools/testing/selftests/bpf/prog_tests/htab_reuse.c new file mode 100644 index 000000000000..a742dd994d60 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/htab_reuse.c @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (C) 2023. Huawei Technologies Co., Ltd */ +#define _GNU_SOURCE +#include <sched.h> +#include <stdbool.h> +#include <test_progs.h> +#include "htab_reuse.skel.h" + +struct htab_op_ctx { + int fd; + int loop; + bool stop; +}; + +struct htab_val { + unsigned int lock; + unsigned int data; +}; + +static void *htab_lookup_fn(void *arg) +{ + struct htab_op_ctx *ctx = arg; + int i = 0; + + while (i++ < ctx->loop && !ctx->stop) { + struct htab_val value; + unsigned int key; + + /* Use BPF_F_LOCK to use spin-lock in map value. */ + key = 7; + bpf_map_lookup_elem_flags(ctx->fd, &key, &value, BPF_F_LOCK); + } + + return NULL; +} + +static void *htab_update_fn(void *arg) +{ + struct htab_op_ctx *ctx = arg; + int i = 0; + + while (i++ < ctx->loop && !ctx->stop) { + struct htab_val value; + unsigned int key; + + key = 7; + value.lock = 0; + value.data = key; + bpf_map_update_elem(ctx->fd, &key, &value, BPF_F_LOCK); + bpf_map_delete_elem(ctx->fd, &key); + + key = 24; + value.lock = 0; + value.data = key; + bpf_map_update_elem(ctx->fd, &key, &value, BPF_F_LOCK); + bpf_map_delete_elem(ctx->fd, &key); + } + + return NULL; +} + +void test_htab_reuse(void) +{ + unsigned int i, wr_nr = 1, rd_nr = 4; + pthread_t tids[wr_nr + rd_nr]; + struct htab_reuse *skel; + struct htab_op_ctx ctx; + int err; + + skel = htab_reuse__open_and_load(); + if (!ASSERT_OK_PTR(skel, "htab_reuse__open_and_load")) + return; + + ctx.fd = bpf_map__fd(skel->maps.htab); + ctx.loop = 500; + ctx.stop = false; + + memset(tids, 0, sizeof(tids)); + for (i = 0; i < wr_nr; i++) { + err = pthread_create(&tids[i], NULL, htab_update_fn, &ctx); + if (!ASSERT_OK(err, "pthread_create")) { + ctx.stop = true; + goto reap; + } + } + for (i = 0; i < rd_nr; i++) { + err = pthread_create(&tids[i + wr_nr], NULL, htab_lookup_fn, &ctx); + if (!ASSERT_OK(err, "pthread_create")) { + ctx.stop = true; + goto reap; + } + } + +reap: + for (i = 0; i < wr_nr + rd_nr; i++) { + if (!tids[i]) + continue; + pthread_join(tids[i], NULL); + } + htab_reuse__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c b/tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c index 72800b1e8395..8cd298b78e44 100644 --- a/tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c +++ b/tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c @@ -10,17 +10,11 @@ #include <test_progs.h> #include "test_kfunc_dynptr_param.skel.h" -static size_t log_buf_sz = 1048576; /* 1 MB */ -static char obj_log_buf[1048576]; - static struct { const char *prog_name; - const char *expected_verifier_err_msg; int expected_runtime_err; } kfunc_dynptr_tests[] = { - {"not_valid_dynptr", "cannot pass in dynptr at an offset=-8", 0}, - {"not_ptr_to_stack", "arg#0 expected pointer to stack or dynptr_ptr", 0}, - {"dynptr_data_null", NULL, -EBADMSG}, + {"dynptr_data_null", -EBADMSG}, }; static bool kfunc_not_supported; @@ -38,29 +32,15 @@ static int libbpf_print_cb(enum libbpf_print_level level, const char *fmt, return 0; } -static void verify_fail(const char *prog_name, const char *expected_err_msg) +static bool has_pkcs7_kfunc_support(void) { struct test_kfunc_dynptr_param *skel; - LIBBPF_OPTS(bpf_object_open_opts, opts); libbpf_print_fn_t old_print_cb; - struct bpf_program *prog; int err; - opts.kernel_log_buf = obj_log_buf; - opts.kernel_log_size = log_buf_sz; - opts.kernel_log_level = 1; - - skel = test_kfunc_dynptr_param__open_opts(&opts); - if (!ASSERT_OK_PTR(skel, "test_kfunc_dynptr_param__open_opts")) - goto cleanup; - - prog = bpf_object__find_program_by_name(skel->obj, prog_name); - if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name")) - goto cleanup; - - bpf_program__set_autoload(prog, true); - - bpf_map__set_max_entries(skel->maps.ringbuf, getpagesize()); + skel = test_kfunc_dynptr_param__open(); + if (!ASSERT_OK_PTR(skel, "test_kfunc_dynptr_param__open")) + return false; kfunc_not_supported = false; @@ -72,26 +52,18 @@ static void verify_fail(const char *prog_name, const char *expected_err_msg) fprintf(stderr, "%s:SKIP:bpf_verify_pkcs7_signature() kfunc not supported\n", __func__); - test__skip(); - goto cleanup; - } - - if (!ASSERT_ERR(err, "unexpected load success")) - goto cleanup; - - if (!ASSERT_OK_PTR(strstr(obj_log_buf, expected_err_msg), "expected_err_msg")) { - fprintf(stderr, "Expected err_msg: %s\n", expected_err_msg); - fprintf(stderr, "Verifier output: %s\n", obj_log_buf); + test_kfunc_dynptr_param__destroy(skel); + return false; } -cleanup: test_kfunc_dynptr_param__destroy(skel); + + return true; } static void verify_success(const char *prog_name, int expected_runtime_err) { struct test_kfunc_dynptr_param *skel; - libbpf_print_fn_t old_print_cb; struct bpf_program *prog; struct bpf_link *link; __u32 next_id; @@ -103,21 +75,7 @@ static void verify_success(const char *prog_name, int expected_runtime_err) skel->bss->pid = getpid(); - bpf_map__set_max_entries(skel->maps.ringbuf, getpagesize()); - - kfunc_not_supported = false; - - old_print_cb = libbpf_set_print(libbpf_print_cb); err = test_kfunc_dynptr_param__load(skel); - libbpf_set_print(old_print_cb); - - if (err < 0 && kfunc_not_supported) { - fprintf(stderr, - "%s:SKIP:bpf_verify_pkcs7_signature() kfunc not supported\n", - __func__); - test__skip(); - goto cleanup; - } if (!ASSERT_OK(err, "test_kfunc_dynptr_param__load")) goto cleanup; @@ -147,15 +105,15 @@ void test_kfunc_dynptr_param(void) { int i; + if (!has_pkcs7_kfunc_support()) + return; + for (i = 0; i < ARRAY_SIZE(kfunc_dynptr_tests); i++) { if (!test__start_subtest(kfunc_dynptr_tests[i].prog_name)) continue; - if (kfunc_dynptr_tests[i].expected_verifier_err_msg) - verify_fail(kfunc_dynptr_tests[i].prog_name, - kfunc_dynptr_tests[i].expected_verifier_err_msg); - else - verify_success(kfunc_dynptr_tests[i].prog_name, - kfunc_dynptr_tests[i].expected_runtime_err); + verify_success(kfunc_dynptr_tests[i].prog_name, + kfunc_dynptr_tests[i].expected_runtime_err); } + RUN_TESTS(test_kfunc_dynptr_param); } diff --git a/tools/testing/selftests/bpf/prog_tests/libbpf_get_fd_by_id_opts.c b/tools/testing/selftests/bpf/prog_tests/libbpf_get_fd_by_id_opts.c index 25e5dfa9c315..a3f238f51d05 100644 --- a/tools/testing/selftests/bpf/prog_tests/libbpf_get_fd_by_id_opts.c +++ b/tools/testing/selftests/bpf/prog_tests/libbpf_get_fd_by_id_opts.c @@ -29,9 +29,9 @@ void test_libbpf_get_fd_by_id_opts(void) if (!ASSERT_OK(ret, "test_libbpf_get_fd_by_id_opts__attach")) goto close_prog; - ret = bpf_obj_get_info_by_fd(bpf_map__fd(skel->maps.data_input), + ret = bpf_map_get_info_by_fd(bpf_map__fd(skel->maps.data_input), &info_m, &len); - if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd")) + if (!ASSERT_OK(ret, "bpf_map_get_info_by_fd")) goto close_prog; fd = bpf_map_get_fd_by_id(info_m.id); diff --git a/tools/testing/selftests/bpf/prog_tests/linked_list.c b/tools/testing/selftests/bpf/prog_tests/linked_list.c index 9a7d4c47af63..0ed8132ce1c3 100644 --- a/tools/testing/selftests/bpf/prog_tests/linked_list.c +++ b/tools/testing/selftests/bpf/prog_tests/linked_list.c @@ -58,12 +58,12 @@ static struct { TEST(inner_map, pop_front) TEST(inner_map, pop_back) #undef TEST - { "map_compat_kprobe", "tracing progs cannot use bpf_list_head yet" }, - { "map_compat_kretprobe", "tracing progs cannot use bpf_list_head yet" }, - { "map_compat_tp", "tracing progs cannot use bpf_list_head yet" }, - { "map_compat_perf", "tracing progs cannot use bpf_list_head yet" }, - { "map_compat_raw_tp", "tracing progs cannot use bpf_list_head yet" }, - { "map_compat_raw_tp_w", "tracing progs cannot use bpf_list_head yet" }, + { "map_compat_kprobe", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, + { "map_compat_kretprobe", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, + { "map_compat_tp", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, + { "map_compat_perf", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, + { "map_compat_raw_tp", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, + { "map_compat_raw_tp_w", "tracing progs cannot use bpf_{list_head,rb_root} yet" }, { "obj_type_id_oor", "local type ID argument must be in range [0, U32_MAX]" }, { "obj_new_no_composite", "bpf_obj_new type ID argument must be of a struct" }, { "obj_new_no_struct", "bpf_obj_new type ID argument must be of a struct" }, @@ -78,8 +78,6 @@ static struct { { "direct_write_head", "direct access to bpf_list_head is disallowed" }, { "direct_read_node", "direct access to bpf_list_node is disallowed" }, { "direct_write_node", "direct access to bpf_list_node is disallowed" }, - { "write_after_push_front", "only read is supported" }, - { "write_after_push_back", "only read is supported" }, { "use_after_unlock_push_front", "invalid mem access 'scalar'" }, { "use_after_unlock_push_back", "invalid mem access 'scalar'" }, { "double_push_front", "arg#1 expected pointer to allocated object" }, @@ -717,6 +715,43 @@ static void test_btf(void) btf__free(btf); break; } + + while (test__start_subtest("btf: list_node and rb_node in same struct")) { + btf = init_btf(); + if (!ASSERT_OK_PTR(btf, "init_btf")) + break; + + id = btf__add_struct(btf, "bpf_rb_node", 24); + if (!ASSERT_EQ(id, 5, "btf__add_struct bpf_rb_node")) + break; + id = btf__add_struct(btf, "bar", 40); + if (!ASSERT_EQ(id, 6, "btf__add_struct bar")) + break; + err = btf__add_field(btf, "a", LIST_NODE, 0, 0); + if (!ASSERT_OK(err, "btf__add_field bar::a")) + break; + err = btf__add_field(btf, "c", 5, 128, 0); + if (!ASSERT_OK(err, "btf__add_field bar::c")) + break; + + id = btf__add_struct(btf, "foo", 20); + if (!ASSERT_EQ(id, 7, "btf__add_struct foo")) + break; + err = btf__add_field(btf, "a", LIST_HEAD, 0, 0); + if (!ASSERT_OK(err, "btf__add_field foo::a")) + break; + err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0); + if (!ASSERT_OK(err, "btf__add_field foo::b")) + break; + id = btf__add_decl_tag(btf, "contains:bar:a", 7, 0); + if (!ASSERT_EQ(id, 8, "btf__add_decl_tag contains:bar:a")) + break; + + err = btf__load_into_kernel(btf); + ASSERT_EQ(err, -EINVAL, "check btf"); + btf__free(btf); + break; + } } void test_linked_list(void) diff --git a/tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c b/tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c index f117bfef68a1..130a3b21e467 100644 --- a/tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c +++ b/tools/testing/selftests/bpf/prog_tests/lsm_cgroup.c @@ -47,7 +47,8 @@ static __u32 query_prog_cnt(int cgroup_fd, const char *attach_func) fd = bpf_prog_get_fd_by_id(p.prog_ids[i]); ASSERT_GE(fd, 0, "prog_get_fd_by_id"); - ASSERT_OK(bpf_obj_get_info_by_fd(fd, &info, &info_len), "prog_info_by_fd"); + ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &info_len), + "prog_info_by_fd"); close(fd); if (info.attach_btf_id == diff --git a/tools/testing/selftests/bpf/prog_tests/metadata.c b/tools/testing/selftests/bpf/prog_tests/metadata.c index 2c53eade88e3..8b67dfc10f5c 100644 --- a/tools/testing/selftests/bpf/prog_tests/metadata.c +++ b/tools/testing/selftests/bpf/prog_tests/metadata.c @@ -16,7 +16,7 @@ static int duration; static int prog_holds_map(int prog_fd, int map_fd) { struct bpf_prog_info prog_info = {}; - struct bpf_prog_info map_info = {}; + struct bpf_map_info map_info = {}; __u32 prog_info_len; __u32 map_info_len; __u32 *map_ids; @@ -25,12 +25,12 @@ static int prog_holds_map(int prog_fd, int map_fd) int i; map_info_len = sizeof(map_info); - ret = bpf_obj_get_info_by_fd(map_fd, &map_info, &map_info_len); + ret = bpf_map_get_info_by_fd(map_fd, &map_info, &map_info_len); if (ret) return -errno; prog_info_len = sizeof(prog_info); - ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + ret = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); if (ret) return -errno; @@ -44,7 +44,7 @@ static int prog_holds_map(int prog_fd, int map_fd) prog_info.map_ids = ptr_to_u64(map_ids); prog_info_len = sizeof(prog_info); - ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + ret = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); if (ret) { ret = -errno; goto free_map_ids; diff --git a/tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c b/tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c index eb2feaac81fe..653b0a20fab9 100644 --- a/tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +++ b/tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c @@ -488,7 +488,7 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_servers; } - /* Tie requests to the first four listners */ + /* Tie requests to the first four listeners */ err = start_clients(test_case); if (!ASSERT_OK(err, "start_clients")) goto close_clients; diff --git a/tools/testing/selftests/bpf/prog_tests/mmap.c b/tools/testing/selftests/bpf/prog_tests/mmap.c index 37b002ca1167..a271d5a0f7ab 100644 --- a/tools/testing/selftests/bpf/prog_tests/mmap.c +++ b/tools/testing/selftests/bpf/prog_tests/mmap.c @@ -64,7 +64,7 @@ void test_mmap(void) /* get map's ID */ memset(&map_info, 0, map_info_sz); - err = bpf_obj_get_info_by_fd(data_map_fd, &map_info, &map_info_sz); + err = bpf_map_get_info_by_fd(data_map_fd, &map_info, &map_info_sz); if (CHECK(err, "map_get_info", "failed %d\n", errno)) goto cleanup; data_map_id = map_info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/perf_link.c b/tools/testing/selftests/bpf/prog_tests/perf_link.c index 224eba6fef2e..3a25f1c743a1 100644 --- a/tools/testing/selftests/bpf/prog_tests/perf_link.c +++ b/tools/testing/selftests/bpf/prog_tests/perf_link.c @@ -54,7 +54,7 @@ void serial_test_perf_link(void) goto cleanup; memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(link_fd, &info, &info_len); + err = bpf_link_get_info_by_fd(link_fd, &info, &info_len); if (!ASSERT_OK(err, "link_get_info")) goto cleanup; diff --git a/tools/testing/selftests/bpf/prog_tests/pinning.c b/tools/testing/selftests/bpf/prog_tests/pinning.c index d95cee5867b7..c799a3c5ad1f 100644 --- a/tools/testing/selftests/bpf/prog_tests/pinning.c +++ b/tools/testing/selftests/bpf/prog_tests/pinning.c @@ -18,7 +18,7 @@ __u32 get_map_id(struct bpf_object *obj, const char *name) if (CHECK(!map, "find map", "NULL map")) return 0; - err = bpf_obj_get_info_by_fd(bpf_map__fd(map), + err = bpf_map_get_info_by_fd(bpf_map__fd(map), &map_info, &map_info_len); CHECK(err, "get map info", "err %d errno %d", err, errno); return map_info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/prog_run_opts.c b/tools/testing/selftests/bpf/prog_tests/prog_run_opts.c index 1ccd2bdf8fa8..01f1d1b6715a 100644 --- a/tools/testing/selftests/bpf/prog_tests/prog_run_opts.c +++ b/tools/testing/selftests/bpf/prog_tests/prog_run_opts.c @@ -12,7 +12,7 @@ static void check_run_cnt(int prog_fd, __u64 run_cnt) __u32 info_len = sizeof(info); int err; - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (CHECK(err, "get_prog_info", "failed to get bpf_prog_info for fd %d\n", prog_fd)) return; diff --git a/tools/testing/selftests/bpf/prog_tests/rbtree.c b/tools/testing/selftests/bpf/prog_tests/rbtree.c new file mode 100644 index 000000000000..156fa95c42f6 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/rbtree.c @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> + +#include "rbtree.skel.h" +#include "rbtree_fail.skel.h" +#include "rbtree_btf_fail__wrong_node_type.skel.h" +#include "rbtree_btf_fail__add_wrong_type.skel.h" + +static void test_rbtree_add_nodes(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct rbtree *skel; + int ret; + + skel = rbtree__open_and_load(); + if (!ASSERT_OK_PTR(skel, "rbtree__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.rbtree_add_nodes), &opts); + ASSERT_OK(ret, "rbtree_add_nodes run"); + ASSERT_OK(opts.retval, "rbtree_add_nodes retval"); + ASSERT_EQ(skel->data->less_callback_ran, 1, "rbtree_add_nodes less_callback_ran"); + + rbtree__destroy(skel); +} + +static void test_rbtree_add_and_remove(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct rbtree *skel; + int ret; + + skel = rbtree__open_and_load(); + if (!ASSERT_OK_PTR(skel, "rbtree__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.rbtree_add_and_remove), &opts); + ASSERT_OK(ret, "rbtree_add_and_remove"); + ASSERT_OK(opts.retval, "rbtree_add_and_remove retval"); + ASSERT_EQ(skel->data->removed_key, 5, "rbtree_add_and_remove first removed key"); + + rbtree__destroy(skel); +} + +static void test_rbtree_first_and_remove(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct rbtree *skel; + int ret; + + skel = rbtree__open_and_load(); + if (!ASSERT_OK_PTR(skel, "rbtree__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.rbtree_first_and_remove), &opts); + ASSERT_OK(ret, "rbtree_first_and_remove"); + ASSERT_OK(opts.retval, "rbtree_first_and_remove retval"); + ASSERT_EQ(skel->data->first_data[0], 2, "rbtree_first_and_remove first rbtree_first()"); + ASSERT_EQ(skel->data->removed_key, 1, "rbtree_first_and_remove first removed key"); + ASSERT_EQ(skel->data->first_data[1], 4, "rbtree_first_and_remove second rbtree_first()"); + + rbtree__destroy(skel); +} + +void test_rbtree_success(void) +{ + if (test__start_subtest("rbtree_add_nodes")) + test_rbtree_add_nodes(); + if (test__start_subtest("rbtree_add_and_remove")) + test_rbtree_add_and_remove(); + if (test__start_subtest("rbtree_first_and_remove")) + test_rbtree_first_and_remove(); +} + +#define BTF_FAIL_TEST(suffix) \ +void test_rbtree_btf_fail__##suffix(void) \ +{ \ + struct rbtree_btf_fail__##suffix *skel; \ + \ + skel = rbtree_btf_fail__##suffix##__open_and_load(); \ + if (!ASSERT_ERR_PTR(skel, \ + "rbtree_btf_fail__" #suffix "__open_and_load unexpected success")) \ + rbtree_btf_fail__##suffix##__destroy(skel); \ +} + +#define RUN_BTF_FAIL_TEST(suffix) \ + if (test__start_subtest("rbtree_btf_fail__" #suffix)) \ + test_rbtree_btf_fail__##suffix(); + +BTF_FAIL_TEST(wrong_node_type); +BTF_FAIL_TEST(add_wrong_type); + +void test_rbtree_btf_fail(void) +{ + RUN_BTF_FAIL_TEST(wrong_node_type); + RUN_BTF_FAIL_TEST(add_wrong_type); +} + +void test_rbtree_fail(void) +{ + RUN_TESTS(rbtree_fail); +} diff --git a/tools/testing/selftests/bpf/prog_tests/recursion.c b/tools/testing/selftests/bpf/prog_tests/recursion.c index f3af2627b599..23552d3e3365 100644 --- a/tools/testing/selftests/bpf/prog_tests/recursion.c +++ b/tools/testing/selftests/bpf/prog_tests/recursion.c @@ -31,8 +31,8 @@ void test_recursion(void) bpf_map_delete_elem(bpf_map__fd(skel->maps.hash2), &key); ASSERT_EQ(skel->bss->pass2, 2, "pass2 == 2"); - err = bpf_obj_get_info_by_fd(bpf_program__fd(skel->progs.on_delete), - &prog_info, &prog_info_len); + err = bpf_prog_get_info_by_fd(bpf_program__fd(skel->progs.on_delete), + &prog_info, &prog_info_len); if (!ASSERT_OK(err, "get_prog_info")) goto out; ASSERT_EQ(prog_info.recursion_misses, 2, "recursion_misses"); diff --git a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c index 0aa088900699..0ce25a967481 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +++ b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c @@ -299,9 +299,9 @@ static __u32 query_prog_id(int prog_fd) __u32 info_len = sizeof(info); int err; - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd") || - !ASSERT_EQ(info_len, sizeof(info), "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd") || + !ASSERT_EQ(info_len, sizeof(info), "bpf_prog_get_info_by_fd")) return 0; return info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/task_local_storage.c b/tools/testing/selftests/bpf/prog_tests/task_local_storage.c index a176bd75a748..ea8537c54413 100644 --- a/tools/testing/selftests/bpf/prog_tests/task_local_storage.c +++ b/tools/testing/selftests/bpf/prog_tests/task_local_storage.c @@ -119,19 +119,19 @@ static void test_recursion(void) prog_fd = bpf_program__fd(skel->progs.on_lookup); memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); ASSERT_OK(err, "get prog info"); ASSERT_GT(info.recursion_misses, 0, "on_lookup prog recursion"); prog_fd = bpf_program__fd(skel->progs.on_update); memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); ASSERT_OK(err, "get prog info"); ASSERT_EQ(info.recursion_misses, 0, "on_update prog recursion"); prog_fd = bpf_program__fd(skel->progs.on_enter); memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); ASSERT_OK(err, "get prog info"); ASSERT_EQ(info.recursion_misses, 0, "on_enter prog recursion"); @@ -221,7 +221,7 @@ static void test_nodeadlock(void) info_len = sizeof(info); prog_fd = bpf_program__fd(skel->progs.socket_post_create); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); ASSERT_OK(err, "get prog info"); ASSERT_EQ(info.recursion_misses, 0, "prog recursion"); diff --git a/tools/testing/selftests/bpf/prog_tests/tc_bpf.c b/tools/testing/selftests/bpf/prog_tests/tc_bpf.c index 4a505a5adf4d..e873766276d1 100644 --- a/tools/testing/selftests/bpf/prog_tests/tc_bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/tc_bpf.c @@ -29,8 +29,8 @@ static int test_tc_bpf_basic(const struct bpf_tc_hook *hook, int fd) __u32 info_len = sizeof(info); int ret; - ret = bpf_obj_get_info_by_fd(fd, &info, &info_len); - if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd")) + ret = bpf_prog_get_info_by_fd(fd, &info, &info_len); + if (!ASSERT_OK(ret, "bpf_prog_get_info_by_fd")) return ret; ret = bpf_tc_attach(hook, &opts); diff --git a/tools/testing/selftests/bpf/prog_tests/test_global_funcs.c b/tools/testing/selftests/bpf/prog_tests/test_global_funcs.c index 7295cc60f724..e0879df38639 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_global_funcs.c +++ b/tools/testing/selftests/bpf/prog_tests/test_global_funcs.c @@ -1,104 +1,43 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2020 Facebook */ #include <test_progs.h> - -const char *err_str; -bool found; - -static int libbpf_debug_print(enum libbpf_print_level level, - const char *format, va_list args) -{ - char *log_buf; - - if (level != LIBBPF_WARN || - strcmp(format, "libbpf: \n%s\n")) { - vprintf(format, args); - return 0; - } - - log_buf = va_arg(args, char *); - if (!log_buf) - goto out; - if (err_str && strstr(log_buf, err_str) == 0) - found = true; -out: - printf(format, log_buf); - return 0; -} - -extern int extra_prog_load_log_flags; - -static int check_load(const char *file) -{ - struct bpf_object *obj = NULL; - struct bpf_program *prog; - int err; - - found = false; - - obj = bpf_object__open_file(file, NULL); - err = libbpf_get_error(obj); - if (err) - return err; - - prog = bpf_object__next_program(obj, NULL); - if (!prog) { - err = -ENOENT; - goto err_out; - } - - bpf_program__set_flags(prog, BPF_F_TEST_RND_HI32); - bpf_program__set_log_level(prog, extra_prog_load_log_flags); - - err = bpf_object__load(obj); - -err_out: - bpf_object__close(obj); - return err; -} - -struct test_def { - const char *file; - const char *err_str; -}; +#include "test_global_func1.skel.h" +#include "test_global_func2.skel.h" +#include "test_global_func3.skel.h" +#include "test_global_func4.skel.h" +#include "test_global_func5.skel.h" +#include "test_global_func6.skel.h" +#include "test_global_func7.skel.h" +#include "test_global_func8.skel.h" +#include "test_global_func9.skel.h" +#include "test_global_func10.skel.h" +#include "test_global_func11.skel.h" +#include "test_global_func12.skel.h" +#include "test_global_func13.skel.h" +#include "test_global_func14.skel.h" +#include "test_global_func15.skel.h" +#include "test_global_func16.skel.h" +#include "test_global_func17.skel.h" +#include "test_global_func_ctx_args.skel.h" void test_test_global_funcs(void) { - struct test_def tests[] = { - { "test_global_func1.bpf.o", "combined stack size of 4 calls is 544" }, - { "test_global_func2.bpf.o" }, - { "test_global_func3.bpf.o", "the call stack of 8 frames" }, - { "test_global_func4.bpf.o" }, - { "test_global_func5.bpf.o", "expected pointer to ctx, but got PTR" }, - { "test_global_func6.bpf.o", "modified ctx ptr R2" }, - { "test_global_func7.bpf.o", "foo() doesn't return scalar" }, - { "test_global_func8.bpf.o" }, - { "test_global_func9.bpf.o" }, - { "test_global_func10.bpf.o", "invalid indirect read from stack" }, - { "test_global_func11.bpf.o", "Caller passes invalid args into func#1" }, - { "test_global_func12.bpf.o", "invalid mem access 'mem_or_null'" }, - { "test_global_func13.bpf.o", "Caller passes invalid args into func#1" }, - { "test_global_func14.bpf.o", "reference type('FWD S') size cannot be determined" }, - { "test_global_func15.bpf.o", "At program exit the register R0 has value" }, - { "test_global_func16.bpf.o", "invalid indirect read from stack" }, - { "test_global_func17.bpf.o", "Caller passes invalid args into func#1" }, - }; - libbpf_print_fn_t old_print_fn = NULL; - int err, i, duration = 0; - - old_print_fn = libbpf_set_print(libbpf_debug_print); - - for (i = 0; i < ARRAY_SIZE(tests); i++) { - const struct test_def *test = &tests[i]; - - if (!test__start_subtest(test->file)) - continue; - - err_str = test->err_str; - err = check_load(test->file); - CHECK_FAIL(!!err ^ !!err_str); - if (err_str) - CHECK(found, "", "expected string '%s'", err_str); - } - libbpf_set_print(old_print_fn); + RUN_TESTS(test_global_func1); + RUN_TESTS(test_global_func2); + RUN_TESTS(test_global_func3); + RUN_TESTS(test_global_func4); + RUN_TESTS(test_global_func5); + RUN_TESTS(test_global_func6); + RUN_TESTS(test_global_func7); + RUN_TESTS(test_global_func8); + RUN_TESTS(test_global_func9); + RUN_TESTS(test_global_func10); + RUN_TESTS(test_global_func11); + RUN_TESTS(test_global_func12); + RUN_TESTS(test_global_func13); + RUN_TESTS(test_global_func14); + RUN_TESTS(test_global_func15); + RUN_TESTS(test_global_func16); + RUN_TESTS(test_global_func17); + RUN_TESTS(test_global_func_ctx_args); } diff --git a/tools/testing/selftests/bpf/prog_tests/tp_attach_query.c b/tools/testing/selftests/bpf/prog_tests/tp_attach_query.c index a479080533db..770fcc3bb1ba 100644 --- a/tools/testing/selftests/bpf/prog_tests/tp_attach_query.c +++ b/tools/testing/selftests/bpf/prog_tests/tp_attach_query.c @@ -45,8 +45,9 @@ void serial_test_tp_attach_query(void) prog_info.xlated_prog_len = 0; prog_info.nr_map_ids = 0; info_len = sizeof(prog_info); - err = bpf_obj_get_info_by_fd(prog_fd[i], &prog_info, &info_len); - if (CHECK(err, "bpf_obj_get_info_by_fd", "err %d errno %d\n", + err = bpf_prog_get_info_by_fd(prog_fd[i], &prog_info, + &info_len); + if (CHECK(err, "bpf_prog_get_info_by_fd", "err %d errno %d\n", err, errno)) goto cleanup1; saved_prog_ids[i] = prog_info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/trampoline_count.c b/tools/testing/selftests/bpf/prog_tests/trampoline_count.c index 8fd4c0d78089..e91d0d1769f1 100644 --- a/tools/testing/selftests/bpf/prog_tests/trampoline_count.c +++ b/tools/testing/selftests/bpf/prog_tests/trampoline_count.c @@ -79,7 +79,7 @@ void serial_test_trampoline_count(void) if (!ASSERT_EQ(link, NULL, "ptr_is_null")) goto cleanup; - /* and finaly execute the probe */ + /* and finally execute the probe */ prog_fd = bpf_program__fd(prog); if (!ASSERT_GE(prog_fd, 0, "bpf_program__fd")) goto cleanup; diff --git a/tools/testing/selftests/bpf/prog_tests/unpriv_bpf_disabled.c b/tools/testing/selftests/bpf/prog_tests/unpriv_bpf_disabled.c index 1ed3cc2092db..8383a99f610f 100644 --- a/tools/testing/selftests/bpf/prog_tests/unpriv_bpf_disabled.c +++ b/tools/testing/selftests/bpf/prog_tests/unpriv_bpf_disabled.c @@ -179,7 +179,7 @@ static void test_unpriv_bpf_disabled_negative(struct test_unpriv_bpf_disabled *s ASSERT_EQ(bpf_prog_get_next_id(prog_id, &next), -EPERM, "prog_get_next_id_fails"); ASSERT_EQ(bpf_prog_get_next_id(0, &next), -EPERM, "prog_get_next_id_fails"); - if (ASSERT_OK(bpf_obj_get_info_by_fd(map_fds[0], &map_info, &map_info_len), + if (ASSERT_OK(bpf_map_get_info_by_fd(map_fds[0], &map_info, &map_info_len), "obj_get_info_by_fd")) { ASSERT_EQ(bpf_map_get_fd_by_id(map_info.id), -EPERM, "map_get_fd_by_id_fails"); ASSERT_EQ(bpf_map_get_next_id(map_info.id, &next), -EPERM, @@ -187,8 +187,8 @@ static void test_unpriv_bpf_disabled_negative(struct test_unpriv_bpf_disabled *s } ASSERT_EQ(bpf_map_get_next_id(0, &next), -EPERM, "map_get_next_id_fails"); - if (ASSERT_OK(bpf_obj_get_info_by_fd(bpf_link__fd(skel->links.sys_nanosleep_enter), - &link_info, &link_info_len), + if (ASSERT_OK(bpf_link_get_info_by_fd(bpf_link__fd(skel->links.sys_nanosleep_enter), + &link_info, &link_info_len), "obj_get_info_by_fd")) { ASSERT_EQ(bpf_link_get_fd_by_id(link_info.id), -EPERM, "link_get_fd_by_id_fails"); ASSERT_EQ(bpf_link_get_next_id(link_info.id, &next), -EPERM, @@ -269,7 +269,7 @@ void test_unpriv_bpf_disabled(void) } prog_fd = bpf_program__fd(skel->progs.sys_nanosleep_enter); - ASSERT_OK(bpf_obj_get_info_by_fd(prog_fd, &prog_info, &prog_info_len), + ASSERT_OK(bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len), "obj_get_info_by_fd"); prog_id = prog_info.id; ASSERT_GT(prog_id, 0, "valid_prog_id"); diff --git a/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c b/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c index dae68de285b9..3a13e102c149 100644 --- a/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c +++ b/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c @@ -19,8 +19,6 @@ #include "../progs/test_user_ringbuf.h" -static size_t log_buf_sz = 1 << 20; /* 1 MB */ -static char obj_log_buf[1048576]; static const long c_sample_size = sizeof(struct sample) + BPF_RINGBUF_HDR_SZ; static const long c_ringbuf_size = 1 << 12; /* 1 small page */ static const long c_max_entries = c_ringbuf_size / c_sample_size; @@ -663,23 +661,6 @@ cleanup: user_ringbuf_success__destroy(skel); } -static struct { - const char *prog_name; - const char *expected_err_msg; -} failure_tests[] = { - /* failure cases */ - {"user_ringbuf_callback_bad_access1", "negative offset dynptr_ptr ptr"}, - {"user_ringbuf_callback_bad_access2", "dereference of modified dynptr_ptr ptr"}, - {"user_ringbuf_callback_write_forbidden", "invalid mem access 'dynptr_ptr'"}, - {"user_ringbuf_callback_null_context_write", "invalid mem access 'scalar'"}, - {"user_ringbuf_callback_null_context_read", "invalid mem access 'scalar'"}, - {"user_ringbuf_callback_discard_dynptr", "cannot release unowned const bpf_dynptr"}, - {"user_ringbuf_callback_submit_dynptr", "cannot release unowned const bpf_dynptr"}, - {"user_ringbuf_callback_invalid_return", "At callback return the register R0 has value"}, - {"user_ringbuf_callback_reinit_dynptr_mem", "Dynptr has to be an uninitialized dynptr"}, - {"user_ringbuf_callback_reinit_dynptr_ringbuf", "Dynptr has to be an uninitialized dynptr"}, -}; - #define SUCCESS_TEST(_func) { _func, #_func } static struct { @@ -700,42 +681,6 @@ static struct { SUCCESS_TEST(test_user_ringbuf_blocking_reserve), }; -static void verify_fail(const char *prog_name, const char *expected_err_msg) -{ - LIBBPF_OPTS(bpf_object_open_opts, opts); - struct bpf_program *prog; - struct user_ringbuf_fail *skel; - int err; - - opts.kernel_log_buf = obj_log_buf; - opts.kernel_log_size = log_buf_sz; - opts.kernel_log_level = 1; - - skel = user_ringbuf_fail__open_opts(&opts); - if (!ASSERT_OK_PTR(skel, "dynptr_fail__open_opts")) - goto cleanup; - - prog = bpf_object__find_program_by_name(skel->obj, prog_name); - if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name")) - goto cleanup; - - bpf_program__set_autoload(prog, true); - - bpf_map__set_max_entries(skel->maps.user_ringbuf, getpagesize()); - - err = user_ringbuf_fail__load(skel); - if (!ASSERT_ERR(err, "unexpected load success")) - goto cleanup; - - if (!ASSERT_OK_PTR(strstr(obj_log_buf, expected_err_msg), "expected_err_msg")) { - fprintf(stderr, "Expected err_msg: %s\n", expected_err_msg); - fprintf(stderr, "Verifier output: %s\n", obj_log_buf); - } - -cleanup: - user_ringbuf_fail__destroy(skel); -} - void test_user_ringbuf(void) { int i; @@ -747,10 +692,5 @@ void test_user_ringbuf(void) success_tests[i].test_callback(); } - for (i = 0; i < ARRAY_SIZE(failure_tests); i++) { - if (!test__start_subtest(failure_tests[i].prog_name)) - continue; - - verify_fail(failure_tests[i].prog_name, failure_tests[i].expected_err_msg); - } + RUN_TESTS(user_ringbuf_fail); } diff --git a/tools/testing/selftests/bpf/prog_tests/verif_stats.c b/tools/testing/selftests/bpf/prog_tests/verif_stats.c index a47e7c0e1ffd..af4b95f57ac1 100644 --- a/tools/testing/selftests/bpf/prog_tests/verif_stats.c +++ b/tools/testing/selftests/bpf/prog_tests/verif_stats.c @@ -16,8 +16,9 @@ void test_verif_stats(void) if (!ASSERT_OK_PTR(skel, "trace_vprintk__open_and_load")) goto cleanup; - err = bpf_obj_get_info_by_fd(skel->progs.sys_enter.prog_fd, &info, &len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(skel->progs.sys_enter.prog_fd, + &info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) goto cleanup; if (!ASSERT_GT(info.verified_insns, 0, "verified_insns")) diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_attach.c b/tools/testing/selftests/bpf/prog_tests/xdp_attach.c index 062fbc8c8e5e..d4cd9f873c14 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_attach.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_attach.c @@ -18,7 +18,7 @@ void serial_test_xdp_attach(void) err = bpf_prog_test_load(file, BPF_PROG_TYPE_XDP, &obj1, &fd1); if (CHECK_FAIL(err)) return; - err = bpf_obj_get_info_by_fd(fd1, &info, &len); + err = bpf_prog_get_info_by_fd(fd1, &info, &len); if (CHECK_FAIL(err)) goto out_1; id1 = info.id; @@ -28,7 +28,7 @@ void serial_test_xdp_attach(void) goto out_1; memset(&info, 0, sizeof(info)); - err = bpf_obj_get_info_by_fd(fd2, &info, &len); + err = bpf_prog_get_info_by_fd(fd2, &info, &len); if (CHECK_FAIL(err)) goto out_2; id2 = info.id; diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c b/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c index f775a1613833..481626a875d1 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c @@ -33,8 +33,8 @@ static void test_xdp_with_cpumap_helpers(void) prog_fd = bpf_program__fd(skel->progs.xdp_dummy_cm); map_fd = bpf_map__fd(skel->maps.cpu_map); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(prog_fd, &info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) goto out_close; val.bpf_prog.fd = prog_fd; @@ -85,8 +85,8 @@ static void test_xdp_with_cpumap_frags_helpers(void) frags_prog_fd = bpf_program__fd(skel->progs.xdp_dummy_cm_frags); map_fd = bpf_map__fd(skel->maps.cpu_map); - err = bpf_obj_get_info_by_fd(frags_prog_fd, &info, &len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(frags_prog_fd, &info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) goto out_close; val.bpf_prog.fd = frags_prog_fd; diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c b/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c index ead40016c324..ce6812558287 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c @@ -35,8 +35,8 @@ static void test_xdp_with_devmap_helpers(void) dm_fd = bpf_program__fd(skel->progs.xdp_dummy_dm); map_fd = bpf_map__fd(skel->maps.dm_ports); - err = bpf_obj_get_info_by_fd(dm_fd, &info, &len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(dm_fd, &info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) goto out_close; val.bpf_prog.fd = dm_fd; @@ -98,8 +98,8 @@ static void test_xdp_with_devmap_frags_helpers(void) dm_fd_frags = bpf_program__fd(skel->progs.xdp_dummy_dm_frags); map_fd = bpf_map__fd(skel->maps.dm_ports); - err = bpf_obj_get_info_by_fd(dm_fd_frags, &info, &len); - if (!ASSERT_OK(err, "bpf_obj_get_info_by_fd")) + err = bpf_prog_get_info_by_fd(dm_fd_frags, &info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) goto out_close; val.bpf_prog.fd = dm_fd_frags; diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_info.c b/tools/testing/selftests/bpf/prog_tests/xdp_info.c index 286c21ecdc65..1dbddcab87a8 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_info.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_info.c @@ -34,7 +34,7 @@ void serial_test_xdp_info(void) if (CHECK_FAIL(err)) return; - err = bpf_obj_get_info_by_fd(prog_fd, &info, &len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &len); if (CHECK(err, "get_prog_info", "errno=%d\n", errno)) goto out_close; diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_link.c b/tools/testing/selftests/bpf/prog_tests/xdp_link.c index 3e9d5c5521f0..e7e9f3c22edf 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_link.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_link.c @@ -29,13 +29,13 @@ void serial_test_xdp_link(void) prog_fd2 = bpf_program__fd(skel2->progs.xdp_handler); memset(&prog_info, 0, sizeof(prog_info)); - err = bpf_obj_get_info_by_fd(prog_fd1, &prog_info, &prog_info_len); + err = bpf_prog_get_info_by_fd(prog_fd1, &prog_info, &prog_info_len); if (!ASSERT_OK(err, "fd_info1")) goto cleanup; id1 = prog_info.id; memset(&prog_info, 0, sizeof(prog_info)); - err = bpf_obj_get_info_by_fd(prog_fd2, &prog_info, &prog_info_len); + err = bpf_prog_get_info_by_fd(prog_fd2, &prog_info, &prog_info_len); if (!ASSERT_OK(err, "fd_info2")) goto cleanup; id2 = prog_info.id; @@ -119,7 +119,8 @@ void serial_test_xdp_link(void) goto cleanup; memset(&link_info, 0, sizeof(link_info)); - err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &link_info, &link_info_len); + err = bpf_link_get_info_by_fd(bpf_link__fd(link), + &link_info, &link_info_len); if (!ASSERT_OK(err, "link_info")) goto cleanup; @@ -137,7 +138,8 @@ void serial_test_xdp_link(void) goto cleanup; memset(&link_info, 0, sizeof(link_info)); - err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &link_info, &link_info_len); + err = bpf_link_get_info_by_fd(bpf_link__fd(link), + &link_info, &link_info_len); ASSERT_OK(err, "link_info"); ASSERT_EQ(link_info.prog_id, id1, "link_prog_id"); diff --git a/tools/testing/selftests/bpf/progs/bpf_hashmap_lookup.c b/tools/testing/selftests/bpf/progs/bpf_hashmap_lookup.c new file mode 100644 index 000000000000..1eb74ddca414 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bpf_hashmap_lookup.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Isovalent */ + +#include "vmlinux.h" + +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); +} hash_map_bench SEC(".maps"); + +/* The number of slots to store times */ +#define NR_SLOTS 32 +#define NR_CPUS 256 +#define CPU_MASK (NR_CPUS-1) + +/* Configured by userspace */ +u64 nr_entries; +u64 nr_loops; +u32 __attribute__((__aligned__(8))) key[NR_CPUS]; + +/* Filled by us */ +u64 __attribute__((__aligned__(256))) percpu_times_index[NR_CPUS]; +u64 __attribute__((__aligned__(256))) percpu_times[NR_CPUS][NR_SLOTS]; + +static inline void patch_key(u32 i) +{ +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ + key[0] = i + 1; +#else + key[0] = __builtin_bswap32(i + 1); +#endif + /* the rest of key is random and is configured by userspace */ +} + +static int lookup_callback(__u32 index, u32 *unused) +{ + patch_key(index); + return bpf_map_lookup_elem(&hash_map_bench, key) ? 0 : 1; +} + +static int loop_lookup_callback(__u32 index, u32 *unused) +{ + return bpf_loop(nr_entries, lookup_callback, NULL, 0) ? 0 : 1; +} + +SEC("fentry/" SYS_PREFIX "sys_getpgid") +int benchmark(void *ctx) +{ + u32 cpu = bpf_get_smp_processor_id(); + u32 times_index; + u64 start_time; + + times_index = percpu_times_index[cpu & CPU_MASK] % NR_SLOTS; + start_time = bpf_ktime_get_ns(); + bpf_loop(nr_loops, loop_lookup_callback, NULL, 0); + percpu_times[cpu & CPU_MASK][times_index] = bpf_ktime_get_ns() - start_time; + percpu_times_index[cpu & CPU_MASK] += 1; + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/btf_dump_test_case_syntax.c b/tools/testing/selftests/bpf/progs/btf_dump_test_case_syntax.c index 26fffb02ed10..ad21ee8c7e23 100644 --- a/tools/testing/selftests/bpf/progs/btf_dump_test_case_syntax.c +++ b/tools/testing/selftests/bpf/progs/btf_dump_test_case_syntax.c @@ -84,7 +84,7 @@ typedef void (*printf_fn_t)(const char *, ...); * typedef int (*fn_t)(int); * typedef char * const * (*fn_ptr2_t)(s_t, fn_t); * - * - `fn_complext_t`: pointer to a function returning struct and accepting + * - `fn_complex_t`: pointer to a function returning struct and accepting * union and struct. All structs and enum are anonymous and defined inline. * * - `signal_t: pointer to a function accepting a pointer to a function as an diff --git a/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c b/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c index a1369b5ebcf8..4ad7fe24966d 100644 --- a/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c +++ b/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c @@ -5,6 +5,7 @@ #include <bpf/bpf_tracing.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" #include "cgrp_kfunc_common.h" char _license[] SEC("license") = "GPL"; @@ -28,6 +29,7 @@ static struct __cgrps_kfunc_map_value *insert_lookup_cgrp(struct cgroup *cgrp) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("Possibly NULL pointer passed to trusted arg0") int BPF_PROG(cgrp_kfunc_acquire_untrusted, struct cgroup *cgrp, const char *path) { struct cgroup *acquired; @@ -45,6 +47,7 @@ int BPF_PROG(cgrp_kfunc_acquire_untrusted, struct cgroup *cgrp, const char *path } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 pointer type STRUCT cgroup must point") int BPF_PROG(cgrp_kfunc_acquire_fp, struct cgroup *cgrp, const char *path) { struct cgroup *acquired, *stack_cgrp = (struct cgroup *)&path; @@ -57,6 +60,7 @@ int BPF_PROG(cgrp_kfunc_acquire_fp, struct cgroup *cgrp, const char *path) } SEC("kretprobe/cgroup_destroy_locked") +__failure __msg("reg type unsupported for arg#0 function") int BPF_PROG(cgrp_kfunc_acquire_unsafe_kretprobe, struct cgroup *cgrp) { struct cgroup *acquired; @@ -69,6 +73,7 @@ int BPF_PROG(cgrp_kfunc_acquire_unsafe_kretprobe, struct cgroup *cgrp) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("cgrp_kfunc_acquire_trusted_walked") int BPF_PROG(cgrp_kfunc_acquire_trusted_walked, struct cgroup *cgrp, const char *path) { struct cgroup *acquired; @@ -80,8 +85,8 @@ int BPF_PROG(cgrp_kfunc_acquire_trusted_walked, struct cgroup *cgrp, const char return 0; } - SEC("tp_btf/cgroup_mkdir") +__failure __msg("Possibly NULL pointer passed to trusted arg0") int BPF_PROG(cgrp_kfunc_acquire_null, struct cgroup *cgrp, const char *path) { struct cgroup *acquired; @@ -96,6 +101,7 @@ int BPF_PROG(cgrp_kfunc_acquire_null, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("Unreleased reference") int BPF_PROG(cgrp_kfunc_acquire_unreleased, struct cgroup *cgrp, const char *path) { struct cgroup *acquired; @@ -108,6 +114,7 @@ int BPF_PROG(cgrp_kfunc_acquire_unreleased, struct cgroup *cgrp, const char *pat } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 expected pointer to map value") int BPF_PROG(cgrp_kfunc_get_non_kptr_param, struct cgroup *cgrp, const char *path) { struct cgroup *kptr; @@ -123,6 +130,7 @@ int BPF_PROG(cgrp_kfunc_get_non_kptr_param, struct cgroup *cgrp, const char *pat } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 expected pointer to map value") int BPF_PROG(cgrp_kfunc_get_non_kptr_acquired, struct cgroup *cgrp, const char *path) { struct cgroup *kptr, *acquired; @@ -141,6 +149,7 @@ int BPF_PROG(cgrp_kfunc_get_non_kptr_acquired, struct cgroup *cgrp, const char * } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 expected pointer to map value") int BPF_PROG(cgrp_kfunc_get_null, struct cgroup *cgrp, const char *path) { struct cgroup *kptr; @@ -156,6 +165,7 @@ int BPF_PROG(cgrp_kfunc_get_null, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("Unreleased reference") int BPF_PROG(cgrp_kfunc_xchg_unreleased, struct cgroup *cgrp, const char *path) { struct cgroup *kptr; @@ -175,6 +185,7 @@ int BPF_PROG(cgrp_kfunc_xchg_unreleased, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("Unreleased reference") int BPF_PROG(cgrp_kfunc_get_unreleased, struct cgroup *cgrp, const char *path) { struct cgroup *kptr; @@ -194,6 +205,7 @@ int BPF_PROG(cgrp_kfunc_get_unreleased, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 is untrusted_ptr_or_null_ expected ptr_ or socket") int BPF_PROG(cgrp_kfunc_release_untrusted, struct cgroup *cgrp, const char *path) { struct __cgrps_kfunc_map_value *v; @@ -209,6 +221,7 @@ int BPF_PROG(cgrp_kfunc_release_untrusted, struct cgroup *cgrp, const char *path } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 pointer type STRUCT cgroup must point") int BPF_PROG(cgrp_kfunc_release_fp, struct cgroup *cgrp, const char *path) { struct cgroup *acquired = (struct cgroup *)&path; @@ -220,6 +233,7 @@ int BPF_PROG(cgrp_kfunc_release_fp, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("arg#0 is ptr_or_null_ expected ptr_ or socket") int BPF_PROG(cgrp_kfunc_release_null, struct cgroup *cgrp, const char *path) { struct __cgrps_kfunc_map_value local, *v; @@ -251,6 +265,7 @@ int BPF_PROG(cgrp_kfunc_release_null, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") +__failure __msg("release kernel function bpf_cgroup_release expects") int BPF_PROG(cgrp_kfunc_release_unacquired, struct cgroup *cgrp, const char *path) { /* Cannot release trusted cgroup pointer which was not acquired. */ diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c index 5950ad6ec2e6..aa5b69354b91 100644 --- a/tools/testing/selftests/bpf/progs/dynptr_fail.c +++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c @@ -630,7 +630,7 @@ static int release_twice_callback_fn(__u32 index, void *data) } /* Test that releasing a dynptr twice, where one of the releases happens - * within a calback function, fails + * within a callback function, fails */ SEC("?raw_tp") __failure __msg("arg 1 is an unacquired reference") diff --git a/tools/testing/selftests/bpf/progs/fib_lookup.c b/tools/testing/selftests/bpf/progs/fib_lookup.c new file mode 100644 index 000000000000..c4514dd58c62 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/fib_lookup.c @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include <linux/types.h> +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_tracing_net.h" + +struct bpf_fib_lookup fib_params = {}; +int fib_lookup_ret = 0; +int lookup_flags = 0; + +SEC("tc") +int fib_lookup(struct __sk_buff *skb) +{ + fib_lookup_ret = bpf_fib_lookup(skb, &fib_params, sizeof(fib_params), + lookup_flags); + + return TC_ACT_SHOT; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/htab_reuse.c b/tools/testing/selftests/bpf/progs/htab_reuse.c new file mode 100644 index 000000000000..7f7368cb3095 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/htab_reuse.c @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (C) 2023. Huawei Technologies Co., Ltd */ +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +char _license[] SEC("license") = "GPL"; + +struct htab_val { + struct bpf_spin_lock lock; + unsigned int data; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 64); + __type(key, unsigned int); + __type(value, struct htab_val); + __uint(map_flags, BPF_F_NO_PREALLOC); +} htab SEC(".maps"); diff --git a/tools/testing/selftests/bpf/progs/linked_list.c b/tools/testing/selftests/bpf/progs/linked_list.c index 4ad88da5cda2..4fa4a9b01bde 100644 --- a/tools/testing/selftests/bpf/progs/linked_list.c +++ b/tools/testing/selftests/bpf/progs/linked_list.c @@ -260,7 +260,7 @@ int test_list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head { int ret; - ret = list_push_pop_multiple(lock ,head, false); + ret = list_push_pop_multiple(lock, head, false); if (ret) return ret; return list_push_pop_multiple(lock, head, true); diff --git a/tools/testing/selftests/bpf/progs/linked_list_fail.c b/tools/testing/selftests/bpf/progs/linked_list_fail.c index 1d9017240e19..69cdc07cba13 100644 --- a/tools/testing/selftests/bpf/progs/linked_list_fail.c +++ b/tools/testing/selftests/bpf/progs/linked_list_fail.c @@ -54,28 +54,44 @@ return 0; \ } -CHECK(kptr, push_front, &f->head); -CHECK(kptr, push_back, &f->head); CHECK(kptr, pop_front, &f->head); CHECK(kptr, pop_back, &f->head); -CHECK(global, push_front, &ghead); -CHECK(global, push_back, &ghead); CHECK(global, pop_front, &ghead); CHECK(global, pop_back, &ghead); -CHECK(map, push_front, &v->head); -CHECK(map, push_back, &v->head); CHECK(map, pop_front, &v->head); CHECK(map, pop_back, &v->head); -CHECK(inner_map, push_front, &iv->head); -CHECK(inner_map, push_back, &iv->head); CHECK(inner_map, pop_front, &iv->head); CHECK(inner_map, pop_back, &iv->head); #undef CHECK +#define CHECK(test, op, hexpr, nexpr) \ + SEC("?tc") \ + int test##_missing_lock_##op(void *ctx) \ + { \ + INIT; \ + void (*p)(void *, void *) = (void *)&bpf_list_##op; \ + p(hexpr, nexpr); \ + return 0; \ + } + +CHECK(kptr, push_front, &f->head, b); +CHECK(kptr, push_back, &f->head, b); + +CHECK(global, push_front, &ghead, f); +CHECK(global, push_back, &ghead, f); + +CHECK(map, push_front, &v->head, f); +CHECK(map, push_back, &v->head, f); + +CHECK(inner_map, push_front, &iv->head, f); +CHECK(inner_map, push_back, &iv->head, f); + +#undef CHECK + #define CHECK(test, op, lexpr, hexpr) \ SEC("?tc") \ int test##_incorrect_lock_##op(void *ctx) \ @@ -108,13 +124,49 @@ CHECK(inner_map, pop_back, &iv->head); CHECK(inner_map_global, op, &iv->lock, &ghead); \ CHECK(inner_map_map, op, &iv->lock, &v->head); -CHECK_OP(push_front); -CHECK_OP(push_back); CHECK_OP(pop_front); CHECK_OP(pop_back); #undef CHECK #undef CHECK_OP + +#define CHECK(test, op, lexpr, hexpr, nexpr) \ + SEC("?tc") \ + int test##_incorrect_lock_##op(void *ctx) \ + { \ + INIT; \ + void (*p)(void *, void*) = (void *)&bpf_list_##op; \ + bpf_spin_lock(lexpr); \ + p(hexpr, nexpr); \ + return 0; \ + } + +#define CHECK_OP(op) \ + CHECK(kptr_kptr, op, &f1->lock, &f2->head, b); \ + CHECK(kptr_global, op, &f1->lock, &ghead, f); \ + CHECK(kptr_map, op, &f1->lock, &v->head, f); \ + CHECK(kptr_inner_map, op, &f1->lock, &iv->head, f); \ + \ + CHECK(global_global, op, &glock2, &ghead, f); \ + CHECK(global_kptr, op, &glock, &f1->head, b); \ + CHECK(global_map, op, &glock, &v->head, f); \ + CHECK(global_inner_map, op, &glock, &iv->head, f); \ + \ + CHECK(map_map, op, &v->lock, &v2->head, f); \ + CHECK(map_kptr, op, &v->lock, &f2->head, b); \ + CHECK(map_global, op, &v->lock, &ghead, f); \ + CHECK(map_inner_map, op, &v->lock, &iv->head, f); \ + \ + CHECK(inner_map_inner_map, op, &iv->lock, &iv2->head, f); \ + CHECK(inner_map_kptr, op, &iv->lock, &f2->head, b); \ + CHECK(inner_map_global, op, &iv->lock, &ghead, f); \ + CHECK(inner_map_map, op, &iv->lock, &v->head, f); + +CHECK_OP(push_front); +CHECK_OP(push_back); + +#undef CHECK +#undef CHECK_OP #undef INIT SEC("?kprobe/xyz") @@ -304,34 +356,6 @@ int direct_write_node(void *ctx) } static __always_inline -int write_after_op(void (*push_op)(void *head, void *node)) -{ - struct foo *f; - - f = bpf_obj_new(typeof(*f)); - if (!f) - return 0; - bpf_spin_lock(&glock); - push_op(&ghead, &f->node); - f->data = 42; - bpf_spin_unlock(&glock); - - return 0; -} - -SEC("?tc") -int write_after_push_front(void *ctx) -{ - return write_after_op((void *)bpf_list_push_front); -} - -SEC("?tc") -int write_after_push_back(void *ctx) -{ - return write_after_op((void *)bpf_list_push_back); -} - -static __always_inline int use_after_unlock(void (*op)(void *head, void *node)) { struct foo *f; diff --git a/tools/testing/selftests/bpf/progs/map_kptr.c b/tools/testing/selftests/bpf/progs/map_kptr.c index eb8217803493..228ec45365a8 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr.c +++ b/tools/testing/selftests/bpf/progs/map_kptr.c @@ -62,21 +62,23 @@ extern struct prog_test_ref_kfunc * bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **p, int a, int b) __ksym; extern void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; +#define WRITE_ONCE(x, val) ((*(volatile typeof(x) *) &(x)) = (val)) + static void test_kptr_unref(struct map_value *v) { struct prog_test_ref_kfunc *p; p = v->unref_ptr; /* store untrusted_ptr_or_null_ */ - v->unref_ptr = p; + WRITE_ONCE(v->unref_ptr, p); if (!p) return; if (p->a + p->b > 100) return; /* store untrusted_ptr_ */ - v->unref_ptr = p; + WRITE_ONCE(v->unref_ptr, p); /* store NULL */ - v->unref_ptr = NULL; + WRITE_ONCE(v->unref_ptr, NULL); } static void test_kptr_ref(struct map_value *v) @@ -85,7 +87,7 @@ static void test_kptr_ref(struct map_value *v) p = v->ref_ptr; /* store ptr_or_null_ */ - v->unref_ptr = p; + WRITE_ONCE(v->unref_ptr, p); if (!p) return; if (p->a + p->b > 100) @@ -99,7 +101,7 @@ static void test_kptr_ref(struct map_value *v) return; } /* store ptr_ */ - v->unref_ptr = p; + WRITE_ONCE(v->unref_ptr, p); bpf_kfunc_call_test_release(p); p = bpf_kfunc_call_test_acquire(&(unsigned long){0}); diff --git a/tools/testing/selftests/bpf/progs/rbtree.c b/tools/testing/selftests/bpf/progs/rbtree.c new file mode 100644 index 000000000000..e5db1a4287e5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rbtree.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_experimental.h" + +struct node_data { + long key; + long data; + struct bpf_rb_node node; +}; + +long less_callback_ran = -1; +long removed_key = -1; +long first_data[2] = {-1, -1}; + +#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock glock; +private(A) struct bpf_rb_root groot __contains(node_data, node); + +static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + less_callback_ran = 1; + + return node_a->key < node_b->key; +} + +static long __add_three(struct bpf_rb_root *root, struct bpf_spin_lock *lock) +{ + struct node_data *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + n->key = 5; + + m = bpf_obj_new(typeof(*m)); + if (!m) { + bpf_obj_drop(n); + return 2; + } + m->key = 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_rbtree_add(&groot, &m->node, less); + bpf_spin_unlock(&glock); + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 3; + n->key = 3; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + return 0; +} + +SEC("tc") +long rbtree_add_nodes(void *ctx) +{ + return __add_three(&groot, &glock); +} + +SEC("tc") +long rbtree_add_and_remove(void *ctx) +{ + struct bpf_rb_node *res = NULL; + struct node_data *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + goto err_out; + n->key = 5; + + m = bpf_obj_new(typeof(*m)); + if (!m) + goto err_out; + m->key = 3; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_rbtree_add(&groot, &m->node, less); + res = bpf_rbtree_remove(&groot, &n->node); + bpf_spin_unlock(&glock); + + n = container_of(res, struct node_data, node); + removed_key = n->key; + + bpf_obj_drop(n); + + return 0; +err_out: + if (n) + bpf_obj_drop(n); + if (m) + bpf_obj_drop(m); + return 1; +} + +SEC("tc") +long rbtree_first_and_remove(void *ctx) +{ + struct bpf_rb_node *res = NULL; + struct node_data *n, *m, *o; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + n->key = 3; + n->data = 4; + + m = bpf_obj_new(typeof(*m)); + if (!m) + goto err_out; + m->key = 5; + m->data = 6; + + o = bpf_obj_new(typeof(*o)); + if (!o) + goto err_out; + o->key = 1; + o->data = 2; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_rbtree_add(&groot, &m->node, less); + bpf_rbtree_add(&groot, &o->node, less); + + res = bpf_rbtree_first(&groot); + if (!res) { + bpf_spin_unlock(&glock); + return 2; + } + + o = container_of(res, struct node_data, node); + first_data[0] = o->data; + + res = bpf_rbtree_remove(&groot, &o->node); + bpf_spin_unlock(&glock); + + o = container_of(res, struct node_data, node); + removed_key = o->key; + + bpf_obj_drop(o); + + bpf_spin_lock(&glock); + res = bpf_rbtree_first(&groot); + if (!res) { + bpf_spin_unlock(&glock); + return 3; + } + + o = container_of(res, struct node_data, node); + first_data[1] = o->data; + bpf_spin_unlock(&glock); + + return 0; +err_out: + if (n) + bpf_obj_drop(n); + if (m) + bpf_obj_drop(m); + return 1; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/rbtree_btf_fail__add_wrong_type.c b/tools/testing/selftests/bpf/progs/rbtree_btf_fail__add_wrong_type.c new file mode 100644 index 000000000000..60079b202c07 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rbtree_btf_fail__add_wrong_type.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_experimental.h" + +struct node_data { + int key; + int data; + struct bpf_rb_node node; +}; + +struct node_data2 { + int key; + struct bpf_rb_node node; + int data; +}; + +static bool less2(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data2 *node_a; + struct node_data2 *node_b; + + node_a = container_of(a, struct node_data2, node); + node_b = container_of(b, struct node_data2, node); + + return node_a->key < node_b->key; +} + +#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock glock; +private(A) struct bpf_rb_root groot __contains(node_data, node); + +SEC("tc") +long rbtree_api_add__add_wrong_type(void *ctx) +{ + struct node_data2 *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less2); + bpf_spin_unlock(&glock); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/rbtree_btf_fail__wrong_node_type.c b/tools/testing/selftests/bpf/progs/rbtree_btf_fail__wrong_node_type.c new file mode 100644 index 000000000000..340f97da1084 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rbtree_btf_fail__wrong_node_type.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_experimental.h" + +/* BTF load should fail as bpf_rb_root __contains this type and points to + * 'node', but 'node' is not a bpf_rb_node + */ +struct node_data { + int key; + int data; + struct bpf_list_node node; +}; + +static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + + return node_a->key < node_b->key; +} + +#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock glock; +private(A) struct bpf_rb_root groot __contains(node_data, node); + +SEC("tc") +long rbtree_api_add__wrong_node_type(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_first(&groot); + bpf_spin_unlock(&glock); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/rbtree_fail.c b/tools/testing/selftests/bpf/progs/rbtree_fail.c new file mode 100644 index 000000000000..bf3cba115897 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rbtree_fail.c @@ -0,0 +1,322 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_experimental.h" +#include "bpf_misc.h" + +struct node_data { + long key; + long data; + struct bpf_rb_node node; +}; + +#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock glock; +private(A) struct bpf_rb_root groot __contains(node_data, node); +private(A) struct bpf_rb_root groot2 __contains(node_data, node); + +static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + + return node_a->key < node_b->key; +} + +SEC("?tc") +__failure __msg("bpf_spin_lock at off=16 must be held for bpf_rb_root") +long rbtree_api_nolock_add(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_rbtree_add(&groot, &n->node, less); + return 0; +} + +SEC("?tc") +__failure __msg("bpf_spin_lock at off=16 must be held for bpf_rb_root") +long rbtree_api_nolock_remove(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + + bpf_rbtree_remove(&groot, &n->node); + return 0; +} + +SEC("?tc") +__failure __msg("bpf_spin_lock at off=16 must be held for bpf_rb_root") +long rbtree_api_nolock_first(void *ctx) +{ + bpf_rbtree_first(&groot); + return 0; +} + +SEC("?tc") +__failure __msg("rbtree_remove node input must be non-owning ref") +long rbtree_api_remove_unadded_node(void *ctx) +{ + struct node_data *n, *m; + struct bpf_rb_node *res; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + m = bpf_obj_new(typeof(*m)); + if (!m) { + bpf_obj_drop(n); + return 1; + } + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + + /* This remove should pass verifier */ + res = bpf_rbtree_remove(&groot, &n->node); + n = container_of(res, struct node_data, node); + + /* This remove shouldn't, m isn't in an rbtree */ + res = bpf_rbtree_remove(&groot, &m->node); + m = container_of(res, struct node_data, node); + bpf_spin_unlock(&glock); + + if (n) + bpf_obj_drop(n); + if (m) + bpf_obj_drop(m); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference id=2 alloc_insn=11") +long rbtree_api_remove_no_drop(void *ctx) +{ + struct bpf_rb_node *res; + struct node_data *n; + + bpf_spin_lock(&glock); + res = bpf_rbtree_first(&groot); + if (!res) + goto unlock_err; + + res = bpf_rbtree_remove(&groot, res); + + n = container_of(res, struct node_data, node); + bpf_spin_unlock(&glock); + + /* bpf_obj_drop(n) is missing here */ + return 0; + +unlock_err: + bpf_spin_unlock(&glock); + return 1; +} + +SEC("?tc") +__failure __msg("arg#1 expected pointer to allocated object") +long rbtree_api_add_to_multiple_trees(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + + /* This add should fail since n already in groot's tree */ + bpf_rbtree_add(&groot2, &n->node, less); + bpf_spin_unlock(&glock); + return 0; +} + +SEC("?tc") +__failure __msg("rbtree_remove node input must be non-owning ref") +long rbtree_api_add_release_unlock_escape(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + + bpf_spin_lock(&glock); + /* After add() in previous critical section, n should be + * release_on_unlock and released after previous spin_unlock, + * so should not be possible to use it here + */ + bpf_rbtree_remove(&groot, &n->node); + bpf_spin_unlock(&glock); + return 0; +} + +SEC("?tc") +__failure __msg("rbtree_remove node input must be non-owning ref") +long rbtree_api_release_aliasing(void *ctx) +{ + struct node_data *n, *m, *o; + struct bpf_rb_node *res; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + + bpf_spin_lock(&glock); + + /* m and o point to the same node, + * but verifier doesn't know this + */ + res = bpf_rbtree_first(&groot); + if (!res) + return 1; + o = container_of(res, struct node_data, node); + + res = bpf_rbtree_first(&groot); + if (!res) + return 1; + m = container_of(res, struct node_data, node); + + bpf_rbtree_remove(&groot, &m->node); + /* This second remove shouldn't be possible. Retval of previous + * remove returns owning reference to m, which is the same + * node o's non-owning ref is pointing at + * + * In order to preserve property + * * owning ref must not be in rbtree + * * non-owning ref must be in rbtree + * + * o's ref must be invalidated after previous remove. Otherwise + * we'd have non-owning ref to node that isn't in rbtree, and + * verifier wouldn't be able to use type system to prevent remove + * of ref that already isn't in any tree. Would have to do runtime + * checks in that case. + */ + bpf_rbtree_remove(&groot, &o->node); + + bpf_spin_unlock(&glock); + return 0; +} + +SEC("?tc") +__failure __msg("rbtree_remove node input must be non-owning ref") +long rbtree_api_first_release_unlock_escape(void *ctx) +{ + struct bpf_rb_node *res; + struct node_data *n; + + bpf_spin_lock(&glock); + res = bpf_rbtree_first(&groot); + if (res) + n = container_of(res, struct node_data, node); + bpf_spin_unlock(&glock); + + bpf_spin_lock(&glock); + /* After first() in previous critical section, n should be + * release_on_unlock and released after previous spin_unlock, + * so should not be possible to use it here + */ + bpf_rbtree_remove(&groot, &n->node); + bpf_spin_unlock(&glock); + return 0; +} + +static bool less__bad_fn_call_add(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + bpf_rbtree_add(&groot, &node_a->node, less); + + return node_a->key < node_b->key; +} + +static bool less__bad_fn_call_remove(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + bpf_rbtree_remove(&groot, &node_a->node); + + return node_a->key < node_b->key; +} + +static bool less__bad_fn_call_first_unlock_after(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + bpf_rbtree_first(&groot); + bpf_spin_unlock(&glock); + + return node_a->key < node_b->key; +} + +static __always_inline +long add_with_cb(bool (cb)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, cb); + bpf_spin_unlock(&glock); + return 0; +} + +SEC("?tc") +__failure __msg("arg#1 expected pointer to allocated object") +long rbtree_api_add_bad_cb_bad_fn_call_add(void *ctx) +{ + return add_with_cb(less__bad_fn_call_add); +} + +SEC("?tc") +__failure __msg("rbtree_remove not allowed in rbtree cb") +long rbtree_api_add_bad_cb_bad_fn_call_remove(void *ctx) +{ + return add_with_cb(less__bad_fn_call_remove); +} + +SEC("?tc") +__failure __msg("can't spin_{lock,unlock} in rbtree cb") +long rbtree_api_add_bad_cb_bad_fn_call_first_unlock_after(void *ctx) +{ + return add_with_cb(less__bad_fn_call_first_unlock_after); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/strobemeta.h b/tools/testing/selftests/bpf/progs/strobemeta.h index 753718595c26..e562be6356f3 100644 --- a/tools/testing/selftests/bpf/progs/strobemeta.h +++ b/tools/testing/selftests/bpf/progs/strobemeta.h @@ -135,7 +135,7 @@ struct strobe_value_loc { * tpidr_el0 for aarch64). * TLS_IMM_EXEC: absolute address of GOT entry containing offset * from thread pointer; - * TLS_GENERAL_DYN: absolute addres of double GOT entry + * TLS_GENERAL_DYN: absolute address of double GOT entry * containing tls_index_t struct; */ int64_t offset; diff --git a/tools/testing/selftests/bpf/progs/test_cls_redirect.c b/tools/testing/selftests/bpf/progs/test_cls_redirect.c index 2833ad722cb7..66b304982245 100644 --- a/tools/testing/selftests/bpf/progs/test_cls_redirect.c +++ b/tools/testing/selftests/bpf/progs/test_cls_redirect.c @@ -600,7 +600,7 @@ static INLINING ret_t get_next_hop(buf_t *pkt, encap_headers_t *encap, return TC_ACT_SHOT; } - /* Skip the remainig next hops (may be zero). */ + /* Skip the remaining next hops (may be zero). */ return skip_next_hops(pkt, encap->unigue.hop_count - encap->unigue.next_hop - 1); } @@ -610,8 +610,8 @@ static INLINING ret_t get_next_hop(buf_t *pkt, encap_headers_t *encap, * * fill_tuple(&t, foo, sizeof(struct iphdr), 123, 321) * - * clang will substitue a costant for sizeof, which allows the verifier - * to track it's value. Based on this, it can figure out the constant + * clang will substitute a constant for sizeof, which allows the verifier + * to track its value. Based on this, it can figure out the constant * return value, and calling code works while still being "generic" to * IPv4 and IPv6. */ diff --git a/tools/testing/selftests/bpf/progs/test_global_func1.c b/tools/testing/selftests/bpf/progs/test_global_func1.c index 7b42dad187b8..23970a20b324 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func1.c +++ b/tools/testing/selftests/bpf/progs/test_global_func1.c @@ -3,10 +3,9 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" -#ifndef MAX_STACK #define MAX_STACK (512 - 3 * 32 + 8) -#endif static __attribute__ ((noinline)) int f0(int var, struct __sk_buff *skb) @@ -39,7 +38,8 @@ int f3(int val, struct __sk_buff *skb, int var) } SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("combined stack size of 4 calls is 544") +int global_func1(struct __sk_buff *skb) { return f0(1, skb) + f1(skb) + f2(2, skb) + f3(3, skb, 4); } diff --git a/tools/testing/selftests/bpf/progs/test_global_func10.c b/tools/testing/selftests/bpf/progs/test_global_func10.c index 97b7031d0e22..98327bdbbfd2 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func10.c +++ b/tools/testing/selftests/bpf/progs/test_global_func10.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct Small { int x; @@ -21,7 +22,8 @@ __noinline int foo(const struct Big *big) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("invalid indirect read from stack") +int global_func10(struct __sk_buff *skb) { const struct Small small = {.x = skb->len }; diff --git a/tools/testing/selftests/bpf/progs/test_global_func11.c b/tools/testing/selftests/bpf/progs/test_global_func11.c index ef5277d982d9..283e036dc401 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func11.c +++ b/tools/testing/selftests/bpf/progs/test_global_func11.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct S { int x; @@ -13,7 +14,8 @@ __noinline int foo(const struct S *s) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("Caller passes invalid args into func#1") +int global_func11(struct __sk_buff *skb) { return foo((const void *)skb); } diff --git a/tools/testing/selftests/bpf/progs/test_global_func12.c b/tools/testing/selftests/bpf/progs/test_global_func12.c index 62343527cc59..7f159d83c6f6 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func12.c +++ b/tools/testing/selftests/bpf/progs/test_global_func12.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct S { int x; @@ -13,7 +14,8 @@ __noinline int foo(const struct S *s) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("invalid mem access 'mem_or_null'") +int global_func12(struct __sk_buff *skb) { const struct S s = {.x = skb->len }; diff --git a/tools/testing/selftests/bpf/progs/test_global_func13.c b/tools/testing/selftests/bpf/progs/test_global_func13.c index ff8897c1ac22..02ea80da75b5 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func13.c +++ b/tools/testing/selftests/bpf/progs/test_global_func13.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct S { int x; @@ -16,7 +17,8 @@ __noinline int foo(const struct S *s) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("Caller passes invalid args into func#1") +int global_func13(struct __sk_buff *skb) { const struct S *s = (const struct S *)(0xbedabeda); diff --git a/tools/testing/selftests/bpf/progs/test_global_func14.c b/tools/testing/selftests/bpf/progs/test_global_func14.c index 698c77199ebf..33b7d5efd7b2 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func14.c +++ b/tools/testing/selftests/bpf/progs/test_global_func14.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct S; @@ -14,7 +15,8 @@ __noinline int foo(const struct S *s) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("reference type('FWD S') size cannot be determined") +int global_func14(struct __sk_buff *skb) { return foo(NULL); diff --git a/tools/testing/selftests/bpf/progs/test_global_func15.c b/tools/testing/selftests/bpf/progs/test_global_func15.c index c19c435988d5..b512d6a6c75e 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func15.c +++ b/tools/testing/selftests/bpf/progs/test_global_func15.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __noinline int foo(unsigned int *v) { @@ -12,7 +13,8 @@ __noinline int foo(unsigned int *v) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("At program exit the register R0 has value") +int global_func15(struct __sk_buff *skb) { unsigned int v = 1; diff --git a/tools/testing/selftests/bpf/progs/test_global_func16.c b/tools/testing/selftests/bpf/progs/test_global_func16.c index 0312d1e8d8c0..e7206304632e 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func16.c +++ b/tools/testing/selftests/bpf/progs/test_global_func16.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __noinline int foo(int (*arr)[10]) { @@ -12,7 +13,8 @@ __noinline int foo(int (*arr)[10]) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__failure __msg("invalid indirect read from stack") +int global_func16(struct __sk_buff *skb) { int array[10]; diff --git a/tools/testing/selftests/bpf/progs/test_global_func17.c b/tools/testing/selftests/bpf/progs/test_global_func17.c index 2b8b9b8ba018..a32e11c7d933 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func17.c +++ b/tools/testing/selftests/bpf/progs/test_global_func17.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only #include <vmlinux.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __noinline int foo(int *p) { @@ -10,7 +11,8 @@ __noinline int foo(int *p) const volatile int i; SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("Caller passes invalid args into func#1") +int global_func17(struct __sk_buff *skb) { return foo((int *)&i); } diff --git a/tools/testing/selftests/bpf/progs/test_global_func2.c b/tools/testing/selftests/bpf/progs/test_global_func2.c index 2c18d82923a2..3dce97fb52a4 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func2.c +++ b/tools/testing/selftests/bpf/progs/test_global_func2.c @@ -1,4 +1,45 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (c) 2020 Facebook */ +#include <stddef.h> +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + #define MAX_STACK (512 - 3 * 32) -#include "test_global_func1.c" + +static __attribute__ ((noinline)) +int f0(int var, struct __sk_buff *skb) +{ + return skb->len; +} + +__attribute__ ((noinline)) +int f1(struct __sk_buff *skb) +{ + volatile char buf[MAX_STACK] = {}; + + return f0(0, skb) + skb->len; +} + +int f3(int, struct __sk_buff *skb, int); + +__attribute__ ((noinline)) +int f2(int val, struct __sk_buff *skb) +{ + return f1(skb) + f3(val, skb, 1); +} + +__attribute__ ((noinline)) +int f3(int val, struct __sk_buff *skb, int var) +{ + volatile char buf[MAX_STACK] = {}; + + return skb->ifindex * val * var; +} + +SEC("tc") +__success +int global_func2(struct __sk_buff *skb) +{ + return f0(1, skb) + f1(skb) + f2(2, skb) + f3(3, skb, 4); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c index 01bf8275dfd6..142b682d3c2f 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func3.c +++ b/tools/testing/selftests/bpf/progs/test_global_func3.c @@ -3,6 +3,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __attribute__ ((noinline)) int f1(struct __sk_buff *skb) @@ -46,20 +47,15 @@ int f7(struct __sk_buff *skb) return f6(skb); } -#ifndef NO_FN8 __attribute__ ((noinline)) int f8(struct __sk_buff *skb) { return f7(skb); } -#endif SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("the call stack of 8 frames") +int global_func3(struct __sk_buff *skb) { -#ifndef NO_FN8 return f8(skb); -#else - return f7(skb); -#endif } diff --git a/tools/testing/selftests/bpf/progs/test_global_func4.c b/tools/testing/selftests/bpf/progs/test_global_func4.c index 610f75edf276..1733d87ad3f3 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func4.c +++ b/tools/testing/selftests/bpf/progs/test_global_func4.c @@ -1,4 +1,55 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (c) 2020 Facebook */ -#define NO_FN8 -#include "test_global_func3.c" +#include <stddef.h> +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +__attribute__ ((noinline)) +int f1(struct __sk_buff *skb) +{ + return skb->len; +} + +__attribute__ ((noinline)) +int f2(int val, struct __sk_buff *skb) +{ + return f1(skb) + val; +} + +__attribute__ ((noinline)) +int f3(int val, struct __sk_buff *skb, int var) +{ + return f2(var, skb) + val; +} + +__attribute__ ((noinline)) +int f4(struct __sk_buff *skb) +{ + return f3(1, skb, 2); +} + +__attribute__ ((noinline)) +int f5(struct __sk_buff *skb) +{ + return f4(skb); +} + +__attribute__ ((noinline)) +int f6(struct __sk_buff *skb) +{ + return f5(skb); +} + +__attribute__ ((noinline)) +int f7(struct __sk_buff *skb) +{ + return f6(skb); +} + +SEC("tc") +__success +int global_func4(struct __sk_buff *skb) +{ + return f7(skb); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_func5.c b/tools/testing/selftests/bpf/progs/test_global_func5.c index 9248d03e0d06..cc55aedaf82d 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func5.c +++ b/tools/testing/selftests/bpf/progs/test_global_func5.c @@ -3,6 +3,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __attribute__ ((noinline)) int f1(struct __sk_buff *skb) @@ -25,7 +26,8 @@ int f3(int val, struct __sk_buff *skb) } SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("expected pointer to ctx, but got PTR") +int global_func5(struct __sk_buff *skb) { return f1(skb) + f2(2, skb) + f3(3, skb); } diff --git a/tools/testing/selftests/bpf/progs/test_global_func6.c b/tools/testing/selftests/bpf/progs/test_global_func6.c index af8c78bdfb25..46c38c8f2cf0 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func6.c +++ b/tools/testing/selftests/bpf/progs/test_global_func6.c @@ -3,6 +3,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __attribute__ ((noinline)) int f1(struct __sk_buff *skb) @@ -25,7 +26,8 @@ int f3(int val, struct __sk_buff *skb) } SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("modified ctx ptr R2") +int global_func6(struct __sk_buff *skb) { return f1(skb) + f2(2, skb) + f3(3, skb); } diff --git a/tools/testing/selftests/bpf/progs/test_global_func7.c b/tools/testing/selftests/bpf/progs/test_global_func7.c index 6cb8e2f5254c..f182febfde3c 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func7.c +++ b/tools/testing/selftests/bpf/progs/test_global_func7.c @@ -3,6 +3,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __attribute__ ((noinline)) void foo(struct __sk_buff *skb) @@ -11,7 +12,8 @@ void foo(struct __sk_buff *skb) } SEC("tc") -int test_cls(struct __sk_buff *skb) +__failure __msg("foo() doesn't return scalar") +int global_func7(struct __sk_buff *skb) { foo(skb); return 0; diff --git a/tools/testing/selftests/bpf/progs/test_global_func8.c b/tools/testing/selftests/bpf/progs/test_global_func8.c index d55a6544b1ab..9b9c57fa2dd3 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func8.c +++ b/tools/testing/selftests/bpf/progs/test_global_func8.c @@ -3,6 +3,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" __noinline int foo(struct __sk_buff *skb) { @@ -10,7 +11,8 @@ __noinline int foo(struct __sk_buff *skb) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__success +int global_func8(struct __sk_buff *skb) { if (!foo(skb)) return 0; diff --git a/tools/testing/selftests/bpf/progs/test_global_func9.c b/tools/testing/selftests/bpf/progs/test_global_func9.c index bd233ddede98..1f2cb0159b8d 100644 --- a/tools/testing/selftests/bpf/progs/test_global_func9.c +++ b/tools/testing/selftests/bpf/progs/test_global_func9.c @@ -2,6 +2,7 @@ #include <stddef.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "bpf_misc.h" struct S { int x; @@ -74,7 +75,8 @@ __noinline int quuz(int **p) } SEC("cgroup_skb/ingress") -int test_cls(struct __sk_buff *skb) +__success +int global_func9(struct __sk_buff *skb) { int result = 0; diff --git a/tools/testing/selftests/bpf/progs/test_global_func_ctx_args.c b/tools/testing/selftests/bpf/progs/test_global_func_ctx_args.c new file mode 100644 index 000000000000..7faa8eef0598 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_func_ctx_args.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_core_read.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +static long stack[256]; + +/* + * KPROBE contexts + */ + +__weak int kprobe_typedef_ctx_subprog(bpf_user_pt_regs_t *ctx) +{ + return bpf_get_stack(ctx, &stack, sizeof(stack), 0); +} + +SEC("?kprobe") +__success +int kprobe_typedef_ctx(void *ctx) +{ + return kprobe_typedef_ctx_subprog(ctx); +} + +#define pt_regs_struct_t typeof(*(__PT_REGS_CAST((struct pt_regs *)NULL))) + +__weak int kprobe_struct_ctx_subprog(pt_regs_struct_t *ctx) +{ + return bpf_get_stack((void *)ctx, &stack, sizeof(stack), 0); +} + +SEC("?kprobe") +__success +int kprobe_resolved_ctx(void *ctx) +{ + return kprobe_struct_ctx_subprog(ctx); +} + +/* this is current hack to make this work on old kernels */ +struct bpf_user_pt_regs_t {}; + +__weak int kprobe_workaround_ctx_subprog(struct bpf_user_pt_regs_t *ctx) +{ + return bpf_get_stack(ctx, &stack, sizeof(stack), 0); +} + +SEC("?kprobe") +__success +int kprobe_workaround_ctx(void *ctx) +{ + return kprobe_workaround_ctx_subprog(ctx); +} + +/* + * RAW_TRACEPOINT contexts + */ + +__weak int raw_tp_ctx_subprog(struct bpf_raw_tracepoint_args *ctx) +{ + return bpf_get_stack(ctx, &stack, sizeof(stack), 0); +} + +SEC("?raw_tp") +__success +int raw_tp_ctx(void *ctx) +{ + return raw_tp_ctx_subprog(ctx); +} + +/* + * RAW_TRACEPOINT_WRITABLE contexts + */ + +__weak int raw_tp_writable_ctx_subprog(struct bpf_raw_tracepoint_args *ctx) +{ + return bpf_get_stack(ctx, &stack, sizeof(stack), 0); +} + +SEC("?raw_tp") +__success +int raw_tp_writable_ctx(void *ctx) +{ + return raw_tp_writable_ctx_subprog(ctx); +} + +/* + * PERF_EVENT contexts + */ + +__weak int perf_event_ctx_subprog(struct bpf_perf_event_data *ctx) +{ + return bpf_get_stack(ctx, &stack, sizeof(stack), 0); +} + +SEC("?perf_event") +__success +int perf_event_ctx(void *ctx) +{ + return perf_event_ctx_subprog(ctx); +} diff --git a/tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c b/tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c index f4a8250329b2..2fbef3cc7ad8 100644 --- a/tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c +++ b/tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c @@ -10,6 +10,7 @@ #include <errno.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +#include "bpf_misc.h" extern struct bpf_key *bpf_lookup_system_key(__u64 id) __ksym; extern void bpf_key_put(struct bpf_key *key) __ksym; @@ -19,6 +20,7 @@ extern int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_ptr, struct { __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 4096); } ringbuf SEC(".maps"); struct { @@ -33,6 +35,7 @@ int err, pid; char _license[] SEC("license") = "GPL"; SEC("?lsm.s/bpf") +__failure __msg("cannot pass in dynptr at an offset=-8") int BPF_PROG(not_valid_dynptr, int cmd, union bpf_attr *attr, unsigned int size) { unsigned long val; @@ -42,6 +45,7 @@ int BPF_PROG(not_valid_dynptr, int cmd, union bpf_attr *attr, unsigned int size) } SEC("?lsm.s/bpf") +__failure __msg("arg#0 expected pointer to stack or dynptr_ptr") int BPF_PROG(not_ptr_to_stack, int cmd, union bpf_attr *attr, unsigned int size) { unsigned long val; diff --git a/tools/testing/selftests/bpf/progs/test_subprogs.c b/tools/testing/selftests/bpf/progs/test_subprogs.c index f8e9256cf18d..a8d602d7c88a 100644 --- a/tools/testing/selftests/bpf/progs/test_subprogs.c +++ b/tools/testing/selftests/bpf/progs/test_subprogs.c @@ -47,7 +47,7 @@ static __noinline int sub5(int v) return sub1(v) - 1; /* compensates sub1()'s + 1 */ } -/* unfortunately verifier rejects `struct task_struct *t` as an unkown pointer +/* unfortunately verifier rejects `struct task_struct *t` as an unknown pointer * type, so we need to accept pointer as integer and then cast it inside the * function */ diff --git a/tools/testing/selftests/bpf/progs/test_xdp_vlan.c b/tools/testing/selftests/bpf/progs/test_xdp_vlan.c index cdf3c48d6cbb..4ddcb6dfe500 100644 --- a/tools/testing/selftests/bpf/progs/test_xdp_vlan.c +++ b/tools/testing/selftests/bpf/progs/test_xdp_vlan.c @@ -98,7 +98,7 @@ bool parse_eth_frame(struct ethhdr *eth, void *data_end, struct parse_pkt *pkt) return true; } -/* Hint, VLANs are choosen to hit network-byte-order issues */ +/* Hint, VLANs are chosen to hit network-byte-order issues */ #define TESTVLAN 4011 /* 0xFAB */ // #define TO_VLAN 4000 /* 0xFA0 (hint 0xOA0 = 160) */ diff --git a/tools/testing/selftests/bpf/progs/user_ringbuf_fail.c b/tools/testing/selftests/bpf/progs/user_ringbuf_fail.c index f3201dc69a60..03ee946c6bf7 100644 --- a/tools/testing/selftests/bpf/progs/user_ringbuf_fail.c +++ b/tools/testing/selftests/bpf/progs/user_ringbuf_fail.c @@ -16,6 +16,7 @@ struct sample { struct { __uint(type, BPF_MAP_TYPE_USER_RINGBUF); + __uint(max_entries, 4096); } user_ringbuf SEC(".maps"); struct { @@ -39,7 +40,8 @@ bad_access1(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to read before the pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("negative offset dynptr_ptr ptr") int user_ringbuf_callback_bad_access1(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, bad_access1, NULL, 0); @@ -61,7 +63,8 @@ bad_access2(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to read past the end of the pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("dereference of modified dynptr_ptr ptr") int user_ringbuf_callback_bad_access2(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, bad_access2, NULL, 0); @@ -80,7 +83,8 @@ write_forbidden(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to write to that pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("invalid mem access 'dynptr_ptr'") int user_ringbuf_callback_write_forbidden(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, write_forbidden, NULL, 0); @@ -99,7 +103,8 @@ null_context_write(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to write to that pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") int user_ringbuf_callback_null_context_write(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, null_context_write, NULL, 0); @@ -120,7 +125,8 @@ null_context_read(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to write to that pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") int user_ringbuf_callback_null_context_read(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, null_context_read, NULL, 0); @@ -139,7 +145,8 @@ try_discard_dynptr(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to read past the end of the pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("cannot release unowned const bpf_dynptr") int user_ringbuf_callback_discard_dynptr(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, try_discard_dynptr, NULL, 0); @@ -158,7 +165,8 @@ try_submit_dynptr(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to read past the end of the pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("cannot release unowned const bpf_dynptr") int user_ringbuf_callback_submit_dynptr(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, try_submit_dynptr, NULL, 0); @@ -175,7 +183,8 @@ invalid_drain_callback_return(struct bpf_dynptr *dynptr, void *context) /* A callback that accesses a dynptr in a bpf_user_ringbuf_drain callback should * not be able to write to that pointer. */ -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("At callback return the register R0 has value") int user_ringbuf_callback_invalid_return(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, invalid_drain_callback_return, NULL, 0); @@ -197,14 +206,16 @@ try_reinit_dynptr_ringbuf(struct bpf_dynptr *dynptr, void *context) return 0; } -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("Dynptr has to be an uninitialized dynptr") int user_ringbuf_callback_reinit_dynptr_mem(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, try_reinit_dynptr_mem, NULL, 0); return 0; } -SEC("?raw_tp/") +SEC("?raw_tp") +__failure __msg("Dynptr has to be an uninitialized dynptr") int user_ringbuf_callback_reinit_dynptr_ringbuf(void *ctx) { bpf_user_ringbuf_drain(&user_ringbuf, try_reinit_dynptr_ringbuf, NULL, 0); diff --git a/tools/testing/selftests/bpf/test_cpp.cpp b/tools/testing/selftests/bpf/test_cpp.cpp index 0bd9990e83fa..f4936834f76f 100644 --- a/tools/testing/selftests/bpf/test_cpp.cpp +++ b/tools/testing/selftests/bpf/test_cpp.cpp @@ -91,7 +91,7 @@ static void try_skeleton_template() skel.detach(); - /* destructor will destory underlying skeleton */ + /* destructor will destroy underlying skeleton */ } int main(int argc, char *argv[]) diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c index b73152822aa2..7fc00e423e4d 100644 --- a/tools/testing/selftests/bpf/test_maps.c +++ b/tools/testing/selftests/bpf/test_maps.c @@ -1275,7 +1275,7 @@ static void test_map_in_map(void) goto out_map_in_map; } - err = bpf_obj_get_info_by_fd(fd, &info, &len); + err = bpf_map_get_info_by_fd(fd, &info, &len); if (err) { printf("Failed to get map info by fd %d: %d", fd, errno); diff --git a/tools/testing/selftests/bpf/test_skb_cgroup_id_user.c b/tools/testing/selftests/bpf/test_skb_cgroup_id_user.c index 3256de30f563..ed518d075d1d 100644 --- a/tools/testing/selftests/bpf/test_skb_cgroup_id_user.c +++ b/tools/testing/selftests/bpf/test_skb_cgroup_id_user.c @@ -93,7 +93,7 @@ int get_map_fd_by_prog_id(int prog_id) info.nr_map_ids = 1; info.map_ids = (__u64) (unsigned long) map_ids; - if (bpf_obj_get_info_by_fd(prog_fd, &info, &info_len)) { + if (bpf_prog_get_info_by_fd(prog_fd, &info, &info_len)) { log_err("Failed to get info by prog fd %d", prog_fd); goto err; } diff --git a/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c b/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c index 5c8ef062f760..32df93747095 100644 --- a/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c +++ b/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c @@ -96,7 +96,7 @@ static int get_map_fd_by_prog_id(int prog_id, bool *xdp) info.nr_map_ids = 1; info.map_ids = (__u64)(unsigned long)map_ids; - if (bpf_obj_get_info_by_fd(prog_fd, &info, &info_len)) { + if (bpf_prog_get_info_by_fd(prog_fd, &info, &info_len)) { log_err("Failed to get info by prog fd %d", prog_fd); goto err; } diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 887c49dc5abd..8b9949bb833d 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -1239,8 +1239,8 @@ static int get_xlated_program(int fd_prog, struct bpf_insn **buf, int *cnt) __u32 xlated_prog_len; __u32 buf_element_size = sizeof(struct bpf_insn); - if (bpf_obj_get_info_by_fd(fd_prog, &info, &info_len)) { - perror("bpf_obj_get_info_by_fd failed"); + if (bpf_prog_get_info_by_fd(fd_prog, &info, &info_len)) { + perror("bpf_prog_get_info_by_fd failed"); return -1; } @@ -1261,8 +1261,8 @@ static int get_xlated_program(int fd_prog, struct bpf_insn **buf, int *cnt) bzero(&info, sizeof(info)); info.xlated_prog_len = xlated_prog_len; info.xlated_prog_insns = (__u64)(unsigned long)*buf; - if (bpf_obj_get_info_by_fd(fd_prog, &info, &info_len)) { - perror("second bpf_obj_get_info_by_fd failed"); + if (bpf_prog_get_info_by_fd(fd_prog, &info, &info_len)) { + perror("second bpf_prog_get_info_by_fd failed"); goto out_free_buf; } diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c index 9695318e8132..6c44153755e6 100644 --- a/tools/testing/selftests/bpf/testing_helpers.c +++ b/tools/testing/selftests/bpf/testing_helpers.c @@ -164,7 +164,7 @@ __u32 link_info_prog_id(const struct bpf_link *link, struct bpf_link_info *info) int err; memset(info, 0, sizeof(*info)); - err = bpf_obj_get_info_by_fd(bpf_link__fd(link), info, &info_len); + err = bpf_link_get_info_by_fd(bpf_link__fd(link), info, &info_len); if (err) { printf("failed to get link info: %d\n", -errno); return 0; diff --git a/tools/testing/selftests/bpf/verifier/bounds_mix_sign_unsign.c b/tools/testing/selftests/bpf/verifier/bounds_mix_sign_unsign.c index c2aa6f26738b..bf82b923c5fe 100644 --- a/tools/testing/selftests/bpf/verifier/bounds_mix_sign_unsign.c +++ b/tools/testing/selftests/bpf/verifier/bounds_mix_sign_unsign.c @@ -1,13 +1,14 @@ { "bounds checks mixing signed and unsigned, positive bounds", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, 2), BPF_JMP_REG(BPF_JGE, BPF_REG_2, BPF_REG_1, 3), @@ -17,20 +18,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 3), @@ -40,20 +42,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 2", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 5), @@ -65,20 +68,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 3", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 4), @@ -89,20 +93,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 4", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, 1), BPF_ALU64_REG(BPF_AND, BPF_REG_1, BPF_REG_2), @@ -112,19 +117,20 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .result = ACCEPT, }, { "bounds checks mixing signed and unsigned, variant 5", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 5), @@ -135,17 +141,20 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 6", .insns = { + BPF_MOV64_REG(BPF_REG_9, BPF_REG_1), + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_9), BPF_MOV64_IMM(BPF_REG_2, 0), BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -512), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_6, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_4, BPF_REG_6, 5), @@ -163,13 +172,14 @@ { "bounds checks mixing signed and unsigned, variant 7", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, 1024 * 1024 * 1024), BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 3), @@ -179,19 +189,20 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .result = ACCEPT, }, { "bounds checks mixing signed and unsigned, variant 8", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGT, BPF_REG_2, BPF_REG_1, 2), @@ -203,20 +214,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 9", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 10), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_LD_IMM64(BPF_REG_2, -9223372036854775808ULL), BPF_JMP_REG(BPF_JGT, BPF_REG_2, BPF_REG_1, 2), @@ -228,19 +240,20 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .result = ACCEPT, }, { "bounds checks mixing signed and unsigned, variant 10", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, 0), BPF_JMP_REG(BPF_JGT, BPF_REG_2, BPF_REG_1, 2), @@ -252,20 +265,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 11", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_JMP_REG(BPF_JGE, BPF_REG_2, BPF_REG_1, 2), @@ -278,20 +292,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 12", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -6), BPF_JMP_REG(BPF_JGE, BPF_REG_2, BPF_REG_1, 2), @@ -303,20 +318,21 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 13", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, 2), BPF_JMP_REG(BPF_JGE, BPF_REG_2, BPF_REG_1, 2), @@ -331,7 +347,7 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, @@ -340,13 +356,14 @@ .insns = { BPF_LDX_MEM(BPF_W, BPF_REG_9, BPF_REG_1, offsetof(struct __sk_buff, mark)), + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -1), BPF_MOV64_IMM(BPF_REG_8, 2), @@ -360,20 +377,21 @@ BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, -3), BPF_JMP_IMM(BPF_JA, 0, 0, -7), }, - .fixup_map_hash_8b = { 4 }, + .fixup_map_hash_8b = { 6 }, .errstr = "unbounded min value", .result = REJECT, }, { "bounds checks mixing signed and unsigned, variant 15", .insns = { + BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -16), BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), BPF_LD_MAP_FD(BPF_REG_1, 0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, -8), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_10, -16), BPF_MOV64_IMM(BPF_REG_2, -6), BPF_JMP_REG(BPF_JGE, BPF_REG_2, BPF_REG_1, 2), @@ -387,7 +405,7 @@ BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .fixup_map_hash_8b = { 3 }, + .fixup_map_hash_8b = { 5 }, .errstr = "unbounded min value", .result = REJECT, }, diff --git a/tools/testing/selftests/bpf/verifier/bpf_st_mem.c b/tools/testing/selftests/bpf/verifier/bpf_st_mem.c new file mode 100644 index 000000000000..3af2501082b2 --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/bpf_st_mem.c @@ -0,0 +1,67 @@ +{ + "BPF_ST_MEM stack imm non-zero", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 42), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_10, -8), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, -42), + /* if value is tracked correctly R0 is zero */ + BPF_EXIT_INSN(), + }, + .result = ACCEPT, + /* Use prog type that requires return value in range [0, 1] */ + .prog_type = BPF_PROG_TYPE_SK_LOOKUP, + .expected_attach_type = BPF_SK_LOOKUP, + .runs = -1, +}, +{ + "BPF_ST_MEM stack imm zero", + .insns = { + /* mark stack 0000 0000 */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* read and sum a few bytes */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_10, -8), + BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), + BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_10, -4), + BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), + BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_10, -1), + BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), + /* if value is tracked correctly R0 is zero */ + BPF_EXIT_INSN(), + }, + .result = ACCEPT, + /* Use prog type that requires return value in range [0, 1] */ + .prog_type = BPF_PROG_TYPE_SK_LOOKUP, + .expected_attach_type = BPF_SK_LOOKUP, + .runs = -1, +}, +{ + "BPF_ST_MEM stack imm zero, variable offset", + .insns = { + /* set fp[-16], fp[-24] to zeros */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -24, 0), + /* r0 = random value in range [-32, -15] */ + BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), + BPF_JMP_IMM(BPF_JLE, BPF_REG_0, 16, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_0, 32), + /* fp[r0] = 0, make a variable offset write of zero, + * this should preserve zero marks on stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_10), + BPF_ST_MEM(BPF_B, BPF_REG_0, 0, 0), + /* r0 = fp[-20], if variable offset write was tracked correctly + * r0 would be a known zero. + */ + BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_10, -20), + /* Would fail return code verification if r0 range is not tracked correctly. */ + BPF_EXIT_INSN(), + }, + .result = ACCEPT, + /* Use prog type that requires return value in range [0, 1] */ + .prog_type = BPF_PROG_TYPE_SK_LOOKUP, + .expected_attach_type = BPF_SK_LOOKUP, + .runs = -1, +}, diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c index f961b49b8ef4..83231456d3c5 100644 --- a/tools/testing/selftests/bpf/veristat.c +++ b/tools/testing/selftests/bpf/veristat.c @@ -144,7 +144,7 @@ static struct env { struct verif_stats *prog_stats; int prog_stat_cnt; - /* baseline_stats is allocated and used only in comparsion mode */ + /* baseline_stats is allocated and used only in comparison mode */ struct verif_stats *baseline_stats; int baseline_stat_cnt; @@ -882,7 +882,7 @@ static int process_obj(const char *filename) * that BPF object file is incomplete and has to be statically * linked into a final BPF object file; instead of bailing * out, report it into stderr, mark it as skipped, and - * proceeed + * proceed */ fprintf(stderr, "Failed to open '%s': %d\n", filename, -errno); env.files_skipped++; diff --git a/tools/testing/selftests/bpf/xdp_synproxy.c b/tools/testing/selftests/bpf/xdp_synproxy.c index 6dbe0b745198..ce68c342b56f 100644 --- a/tools/testing/selftests/bpf/xdp_synproxy.c +++ b/tools/testing/selftests/bpf/xdp_synproxy.c @@ -217,9 +217,10 @@ static int syncookie_attach(const char *argv0, unsigned int ifindex, bool tc) prog_fd = bpf_program__fd(prog); - err = bpf_obj_get_info_by_fd(prog_fd, &info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len); if (err < 0) { - fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err)); + fprintf(stderr, "Error: bpf_prog_get_info_by_fd: %s\n", + strerror(-err)); goto out; } attached_tc = tc; @@ -292,9 +293,10 @@ static int syncookie_open_bpf_maps(__u32 prog_id, int *values_map_fd, int *ports }; info_len = sizeof(prog_info); - err = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &info_len); + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &info_len); if (err != 0) { - fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err)); + fprintf(stderr, "Error: bpf_prog_get_info_by_fd: %s\n", + strerror(-err)); goto out; } @@ -317,9 +319,10 @@ static int syncookie_open_bpf_maps(__u32 prog_id, int *values_map_fd, int *ports map_fd = err; info_len = sizeof(map_info); - err = bpf_obj_get_info_by_fd(map_fd, &map_info, &info_len); + err = bpf_map_get_info_by_fd(map_fd, &map_info, &info_len); if (err != 0) { - fprintf(stderr, "Error: bpf_obj_get_info_by_fd: %s\n", strerror(-err)); + fprintf(stderr, "Error: bpf_map_get_info_by_fd: %s\n", + strerror(-err)); close(map_fd); goto err_close_map_fds; } diff --git a/tools/testing/selftests/net/rps_default_mask.sh b/tools/testing/selftests/net/rps_default_mask.sh index c81c0ac7ddfe..0fd0d2db3abc 100755 --- a/tools/testing/selftests/net/rps_default_mask.sh +++ b/tools/testing/selftests/net/rps_default_mask.sh @@ -8,7 +8,9 @@ ret=0 [ $cpus -gt 2 ] || exit $ksft_skip readonly INITIAL_RPS_DEFAULT_MASK=$(cat /proc/sys/net/core/rps_default_mask) -readonly NETNS="ns-$(mktemp -u XXXXXX)" +readonly TAG="$(mktemp -u XXXXXX)" +readonly VETH="veth${TAG}" +readonly NETNS="ns-${TAG}" setup() { ip netns add "${NETNS}" @@ -21,11 +23,15 @@ cleanup() { } chk_rps() { - local rps_mask expected_rps_mask=$3 - local dev_name=$2 + local rps_mask expected_rps_mask=$4 + local dev_name=$3 + local netns=$2 + local cmd="cat" local msg=$1 - rps_mask=$(ip netns exec $NETNS cat /sys/class/net/$dev_name/queues/rx-0/rps_cpus) + [ -n "$netns" ] && cmd="ip netns exec $netns $cmd" + + rps_mask=$($cmd /sys/class/net/$dev_name/queues/rx-0/rps_cpus) printf "%-60s" "$msg" if [ $rps_mask -eq $expected_rps_mask ]; then echo "[ ok ]" @@ -39,19 +45,30 @@ trap cleanup EXIT echo 0 > /proc/sys/net/core/rps_default_mask setup -chk_rps "empty rps_default_mask" lo 0 +chk_rps "empty rps_default_mask" $NETNS lo 0 cleanup echo 1 > /proc/sys/net/core/rps_default_mask setup -chk_rps "non zero rps_default_mask" lo 1 +chk_rps "changing rps_default_mask dont affect existing devices" "" lo $INITIAL_RPS_DEFAULT_MASK echo 3 > /proc/sys/net/core/rps_default_mask -chk_rps "changing rps_default_mask dont affect existing netns" lo 1 +chk_rps "changing rps_default_mask dont affect existing netns" $NETNS lo 0 + +ip link add name $VETH type veth peer netns $NETNS name $VETH +ip link set dev $VETH up +ip -n $NETNS link set dev $VETH up +chk_rps "changing rps_default_mask affect newly created devices" "" $VETH 3 +chk_rps "changing rps_default_mask don't affect newly child netns[II]" $NETNS $VETH 0 +ip netns del $NETNS + +setup +chk_rps "rps_default_mask is 0 by default in child netns" "$NETNS" lo 0 + +ip netns exec $NETNS sysctl -qw net.core.rps_default_mask=1 +ip link add name $VETH type veth peer netns $NETNS name $VETH +chk_rps "changing rps_default_mask in child ns don't affect the main one" "" lo $INITIAL_RPS_DEFAULT_MASK +chk_rps "changing rps_default_mask in child ns affects new childns devices" $NETNS $VETH 1 +chk_rps "changing rps_default_mask in child ns don't affect existing devices" $NETNS lo 0 -ip -n $NETNS link add type veth -ip -n $NETNS link set dev veth0 up -ip -n $NETNS link set dev veth1 up -chk_rps "changing rps_default_mask affect newly created devices" veth0 3 -chk_rps "changing rps_default_mask affect newly created devices[II]" veth1 3 exit $ret |