From 1c6c1ca318585f1096d4d04bc722297c85e9fb8a Mon Sep 17 00:00:00 2001 From: Peter Zijlstra Date: Thu, 13 Jun 2019 15:43:19 +0200 Subject: mips/atomic: Fix loongson_llsc_mb() wreckage The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. Clarify the comment by explaining that these issue cause a coherence fail. For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asm() implementation does too. We cannot rely on the barriers from cmpxchg() because cmpxchg_local() is implemented with the same macro, and branch prediction and speculation are, too, CPU local. Fixes: e02e07e3127d ("MIPS: Loongson: Introduce and use loongson_llsc_mb()") Cc: Huacai Chen Cc: Huang Pei Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Paul Burton --- arch/mips/include/asm/barrier.h | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) (limited to 'arch/mips/include/asm/barrier.h') diff --git a/arch/mips/include/asm/barrier.h b/arch/mips/include/asm/barrier.h index b865e317a14f..f9a6da96aae1 100644 --- a/arch/mips/include/asm/barrier.h +++ b/arch/mips/include/asm/barrier.h @@ -238,36 +238,40 @@ /* * Some Loongson 3 CPUs have a bug wherein execution of a memory access (load, - * store or pref) in between an ll & sc can cause the sc instruction to + * store or prefetch) in between an LL & SC can cause the SC instruction to * erroneously succeed, breaking atomicity. Whilst it's unusual to write code * containing such sequences, this bug bites harder than we might otherwise * expect due to reordering & speculation: * - * 1) A memory access appearing prior to the ll in program order may actually - * be executed after the ll - this is the reordering case. + * 1) A memory access appearing prior to the LL in program order may actually + * be executed after the LL - this is the reordering case. * - * In order to avoid this we need to place a memory barrier (ie. a sync - * instruction) prior to every ll instruction, in between it & any earlier - * memory access instructions. Many of these cases are already covered by - * smp_mb__before_llsc() but for the remaining cases, typically ones in - * which multiple CPUs may operate on a memory location but ordering is not - * usually guaranteed, we use loongson_llsc_mb() below. + * In order to avoid this we need to place a memory barrier (ie. a SYNC + * instruction) prior to every LL instruction, in between it and any earlier + * memory access instructions. * * This reordering case is fixed by 3A R2 CPUs, ie. 3A2000 models and later. * - * 2) If a conditional branch exists between an ll & sc with a target outside - * of the ll-sc loop, for example an exit upon value mismatch in cmpxchg() + * 2) If a conditional branch exists between an LL & SC with a target outside + * of the LL-SC loop, for example an exit upon value mismatch in cmpxchg() * or similar, then misprediction of the branch may allow speculative - * execution of memory accesses from outside of the ll-sc loop. + * execution of memory accesses from outside of the LL-SC loop. * - * In order to avoid this we need a memory barrier (ie. a sync instruction) + * In order to avoid this we need a memory barrier (ie. a SYNC instruction) * at each affected branch target, for which we also use loongson_llsc_mb() * defined below. * * This case affects all current Loongson 3 CPUs. + * + * The above described cases cause an error in the cache coherence protocol; + * such that the Invalidate of a competing LL-SC goes 'missing' and SC + * erroneously observes its core still has Exclusive state and lets the SC + * proceed. + * + * Therefore the error only occurs on SMP systems. */ #ifdef CONFIG_CPU_LOONGSON3_WORKAROUNDS /* Loongson-3's LLSC workaround */ -#define loongson_llsc_mb() __asm__ __volatile__(__WEAK_LLSC_MB : : :"memory") +#define loongson_llsc_mb() __asm__ __volatile__("sync" : : :"memory") #else #define loongson_llsc_mb() do { } while (0) #endif -- cgit