diff options
| author | Keith Busch <kbusch@kernel.org> | 2020-05-08 13:04:06 -0700 |
|---|---|---|
| committer | Christoph Hellwig <hch@lst.de> | 2020-05-12 18:02:24 +0200 |
| commit | b69e2ef24b7b4867f80f47e2781e95d0bacd15cb (patch) | |
| tree | 849b398f5ec603aa97bc6a1f6e387bfa90a7ba82 | |
| parent | 59c7c3caaaf8750df4ec3255082f15eb4e371514 (diff) | |
nvme-pci: dma read memory barrier for completions
Control dependencies do not guarantee load order across the condition,
allowing a CPU to predict and speculate memory reads.
Commit 324b494c2862 inlined verifying a new completion with its
handling. At least one architecture was observed to access the contents
out of order, resulting in the driver using stale data for the
completion.
Add a dma read barrier before reading the completion queue entry and
after the condition its contents depend on to ensure the read order is
determinsitic.
Reported-by: John Garry <john.garry@huawei.com>
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Tested-by: John Garry <john.garry@huawei.com>
Acked-by: Will Deacon <will@kernel.org>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Christoph Hellwig <hch@lst.de>
| -rw-r--r-- | drivers/nvme/host/pci.c | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index e13c370de830..3726dc780d15 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq) while (nvme_cqe_pending(nvmeq)) { found++; + /* + * load-load control dependency between phase and the rest of + * the cqe requires a full read memory barrier + */ + dma_rmb(); nvme_handle_cqe(nvmeq, nvmeq->cq_head); nvme_update_cq_head(nvmeq); } |