summaryrefslogtreecommitdiff
path: root/kernel/rcu/sync.c
diff options
context:
space:
mode:
authorPeter Zijlstra <peterz@infradead.org>2016-07-14 20:08:46 +0200
committerIngo Molnar <mingo@kernel.org>2016-08-10 14:34:01 +0200
commit80127a39681bd68c959f0953f84a830cbd7c3b1c (patch)
tree223bcc2a5cbec5c0873f8fae85a98797f94e6c56 /kernel/rcu/sync.c
parent08be8f63c40c030b5cf95b4368e314e563a86301 (diff)
locking/percpu-rwsem: Optimize readers and reduce global impact
Currently the percpu-rwsem switches to (global) atomic ops while a writer is waiting; which could be quite a while and slows down releasing the readers. This patch cures this problem by ordering the reader-state vs reader-count (see the comments in __percpu_down_read() and percpu_down_write()). This changes a global atomic op into a full memory barrier, which doesn't have the global cacheline contention. This also enables using the percpu-rwsem with rcu_sync disabled in order to bias the implementation differently, reducing the writer latency by adding some cost to readers. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Oleg Nesterov <oleg@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Paul McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: linux-kernel@vger.kernel.org [ Fixed modular build. ] Signed-off-by: Ingo Molnar <mingo@kernel.org>
Diffstat (limited to 'kernel/rcu/sync.c')
-rw-r--r--kernel/rcu/sync.c2
1 files changed, 2 insertions, 0 deletions
diff --git a/kernel/rcu/sync.c b/kernel/rcu/sync.c
index be922c9f3d37..198473d90f81 100644
--- a/kernel/rcu/sync.c
+++ b/kernel/rcu/sync.c
@@ -68,6 +68,8 @@ void rcu_sync_lockdep_assert(struct rcu_sync *rsp)
RCU_LOCKDEP_WARN(!gp_ops[rsp->gp_type].held(),
"suspicious rcu_sync_is_idle() usage");
}
+
+EXPORT_SYMBOL_GPL(rcu_sync_lockdep_assert);
#endif
/**