From f95ccffc715bf0fc6792fda52d24e0a92ad955e7 Mon Sep 17 00:00:00 2001 From: Jack Morgenstein Date: Thu, 26 Jul 2018 10:08:37 +0300 Subject: IB/mlx4: Use 4K pages for kernel QP's WQE buffer In the current implementation, the driver tries to allocate contiguous memory, and if it fails, it falls back to 4K fragmented allocation. Once the memory is fragmented, the first allocation might take a lot of time, and even fail, which can cause connection failures. This patch changes the logic to always allocate with 4K granularity, since it's more robust and more likely to succeed. This patch was tested with Lustre and no performance degradation was observed. Note: This commit eliminates the "shrinking WQE" feature. This feature depended on using vmap to create a virtually contiguous send WQ. vmap use was abandoned due to problems with several processors (see the commit cited below). As a result, shrinking WQE was available only with physically contiguous send WQs. Allocating such send WQs caused the problems described above. Therefore, as a side effect of eliminating the use of large physically contiguous send WQs, the shrinking WQE feature became unavailable. Warning example: worker/20:1: page allocation failure: order:8, mode:0x80d0 CPU: 20 PID: 513 Comm: kworker/20:1 Tainted: G OE ------------ Workqueue: ib_cm cm_work_handler [ib_cm] Call Trace: [] dump_stack+0x19/0x1b [] warn_alloc_failed+0x110/0x180 [] __alloc_pages_nodemask+0x9b4/0xba0 [] alloc_pages_current+0x98/0x110 [] __get_free_pages+0xe/0x50 [] swiotlb_alloc_coherent+0x5e/0x150 [] x86_swiotlb_alloc_coherent+0x41/0x50 [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [] create_qp_common+0x536/0x1000 [mlx4_ib] [] ? dma_pool_free+0xa7/0xd0 [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [] ib_create_qp+0x7a/0x2f0 [ib_core] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [] kiblnd_create_conn+0xbf9/0x1950 [ko2iblnd] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] Fixes: 73898db04301 ("net/mlx4: Avoid wrong virtual mappings") Signed-off-by: Jack Morgenstein Signed-off-by: Leon Romanovsky Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx4/mlx4_ib.h | 1 - 1 file changed, 1 deletion(-) (limited to 'drivers/infiniband/hw/mlx4/mlx4_ib.h') diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index e817a2f55546..e10dccc7958f 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -322,7 +322,6 @@ struct mlx4_ib_qp { u32 doorbell_qpn; __be32 sq_signal_bits; unsigned sq_next_wqe; - int sq_max_wqes_per_wr; int sq_spare_wqes; struct mlx4_ib_wq sq; -- cgit