From f796feabb9f5b1e5c48780a7a0023ab4b82336dd Mon Sep 17 00:00:00 2001 From: Paolo Abeni Date: Tue, 20 Feb 2024 12:00:01 +0100 Subject: udp: add local "peek offset enabled" flag We want to re-organize the struct sock layout. The sk_peek_off field location is problematic, as most protocols want it in the RX read area, while UDP wants it on a cacheline different from sk_receive_queue. Create a local (inside udp_sock) copy of the 'peek offset is enabled' flag and place it inside the same cacheline of reader_queue. Check such flag before reading sk_peek_off. This will save potential false sharing and cache misses in the fast-path. Tested under UDP flood with small packets. The struct sock layout update causes a 4% performance drop, and this patch restores completely the original tput. Signed-off-by: Paolo Abeni Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/r/67ab679c15fbf49fa05b3ffe05d91c47ab84f147.1708426665.git.pabeni@redhat.com Signed-off-by: Jakub Kicinski --- net/ipv4/udp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'net/ipv4/udp.c') diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index f631b0a21af4..38cce7cc51f6 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1589,7 +1589,7 @@ int udp_init_sock(struct sock *sk) void skb_consume_udp(struct sock *sk, struct sk_buff *skb, int len) { - if (unlikely(READ_ONCE(sk->sk_peek_off) >= 0)) { + if (unlikely(READ_ONCE(udp_sk(sk)->peeking_with_offset))) { bool slow = lock_sock_fast(sk); sk_peek_offset_bwd(sk, len); -- cgit