From a1bd3baeb2f18b2b3d0f98ce5fdaa725149b950b Mon Sep 17 00:00:00 2001 From: Allen Hubbe Date: Thu, 9 Apr 2015 10:33:20 -0400 Subject: NTB: Add NTB hardware abstraction layer Abstract the NTB device behind a programming interface, so that it can support different hardware and client drivers. Signed-off-by: Allen Hubbe Signed-off-by: Jon Mason --- Documentation/ntb.txt | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 Documentation/ntb.txt (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt new file mode 100644 index 000000000000..9d46dc9712a8 --- /dev/null +++ b/Documentation/ntb.txt @@ -0,0 +1,32 @@ +# NTB Drivers + +NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects +the separate memory systems of two computers to the same PCI-Express fabric. +Existing NTB hardware supports a common feature set, including scratchpad +registers, doorbell registers, and memory translation windows. Scratchpad +registers are read-and-writable registers that are accessible from either side +of the device, so that peers can exchange a small amount of information at a +fixed address. Doorbell registers provide a way for peers to send interrupt +events. Memory windows allow translated read and write access to the peer +memory. + +## NTB Core Driver (ntb) + +The NTB core driver defines an api wrapping the common feature set, and allows +clients interested in NTB features to discover NTB the devices supported by +hardware drivers. The term "client" is used here to mean an upper layer +component making use of the NTB api. The term "driver," or "hardware driver," +is used here to mean a driver for a specific vendor and model of NTB hardware. + +## NTB Client Drivers + +NTB client drivers should register with the NTB core driver. After +registering, the client probe and remove functions will be called appropriately +as ntb hardware, or hardware drivers, are inserted and removed. The +registration uses the Linux Device framework, so it should feel familiar to +anyone who has written a pci driver. + +## NTB Hardware Drivers + +NTB hardware drivers should register devices with the NTB core driver. After +registering, clients probe and remove functions will be called. -- cgit From e26a5843f7f5014ae4460030ca4de029a3ac35d3 Mon Sep 17 00:00:00 2001 From: Allen Hubbe Date: Thu, 9 Apr 2015 10:33:20 -0400 Subject: NTB: Split ntb_hw_intel and ntb_transport drivers Change ntb_hw_intel to use the new NTB hardware abstraction layer. Split ntb_transport into its own driver. Change it to use the new NTB hardware abstraction layer. Signed-off-by: Allen Hubbe Signed-off-by: Jon Mason --- Documentation/ntb.txt | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index 9d46dc9712a8..725ba1e6c127 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -26,7 +26,33 @@ as ntb hardware, or hardware drivers, are inserted and removed. The registration uses the Linux Device framework, so it should feel familiar to anyone who has written a pci driver. +### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev) + +The primary client for NTB is the Transport client, used in tandem with NTB +Netdev. These drivers function together to create a logical link to the peer, +across the ntb, to exchange packets of network data. The Transport client +establishes a logical link to the peer, and creates queue pairs to exchange +messages and data. The NTB Netdev then creates an ethernet device using a +Transport queue pair. Network data is copied between socket buffers and the +Transport queue pair buffer. The Transport client may be used for other things +besides Netdev, however no other applications have yet been written. + ## NTB Hardware Drivers NTB hardware drivers should register devices with the NTB core driver. After registering, clients probe and remove functions will be called. + +### NTB Intel Hardware Driver (ntb\_hw\_intel) + +The Intel hardware driver supports NTB on Xeon and Atom CPUs. + +Module Parameters: + +* b2b\_mw\_idx - If the peer ntb is to be accessed via a memory window, then use + this memory window to access the peer ntb. A value of zero or positive + starts from the first mw idx, and a negative value starts from the last + mw idx. Both sides MUST set the same value here! The default value is + `-1`. +* b2b\_mw\_share - If the peer ntb is to be accessed via a memory window, and if + the memory window is large enough, still allow the client to use the + second half of the memory window for address translation to the peer. -- cgit From 42fefc86a680f975e2816cd86c4ba50069766ef6 Mon Sep 17 00:00:00 2001 From: Allen Hubbe Date: Mon, 11 May 2015 05:45:30 -0400 Subject: NTB: Add parameters for Intel SNB B2B addresses Add module parameters for the addresses to be used in B2B topology. Signed-off-by: Allen Hubbe Signed-off-by: Jon Mason --- Documentation/ntb.txt | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index 725ba1e6c127..00a3f92832a5 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -56,3 +56,13 @@ Module Parameters: * b2b\_mw\_share - If the peer ntb is to be accessed via a memory window, and if the memory window is large enough, still allow the client to use the second half of the memory window for address translation to the peer. +* snb\_b2b\_usd\_bar2\_addr64 - If using B2B topology on Xeon hardware, use this + 64 bit address on the bus between the NTB devices for the window at + BAR2, on the upstream side of the link. +* snb\_b2b\_usd\_bar4\_addr64 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_usd\_bar4\_addr32 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_usd\_bar5\_addr32 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_dsd\_bar2\_addr64 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_dsd\_bar4\_addr64 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_dsd\_bar4\_addr32 - See *snb\_b2b\_bar2\_addr64*. +* snb\_b2b\_dsd\_bar5\_addr32 - See *snb\_b2b\_bar2\_addr64*. -- cgit From 963de4739fb4f8fa8d5cd87969109a7d3282ed13 Mon Sep 17 00:00:00 2001 From: Allen Hubbe Date: Wed, 15 Apr 2015 11:12:41 -0400 Subject: NTB: Add ping pong test client This is a simple ping pong driver that exercises the scratch pads and doorbells of the ntb hardware. This driver may be used to test that your ntb hardware and drivers are functioning at a basic level. Signed-off-by: Allen Hubbe Signed-off-by: Jon Mason --- Documentation/ntb.txt | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index 00a3f92832a5..e5d1591478ca 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -37,6 +37,33 @@ Transport queue pair. Network data is copied between socket buffers and the Transport queue pair buffer. The Transport client may be used for other things besides Netdev, however no other applications have yet been written. +### NTB Ping Pong Test Client (ntb\_pingpong) + +The Ping Pong test client serves as a demonstration to exercise the doorbell +and scratchpad registers of NTB hardware, and as an example simple NTB client. +Ping Pong enables the link when started, waits for the NTB link to come up, and +then proceeds to read and write the doorbell scratchpad registers of the NTB. +The peers interrupt each other using a bit mask of doorbell bits, which is +shifted by one in each round, to test the behavior of multiple doorbell bits +and interrupt vectors. The Ping Pong driver also reads the first local +scratchpad, and writes the value plus one to the first peer scratchpad, each +round before writing the peer doorbell register. + +Module Parameters: + +* unsafe - Some hardware has known issues with scratchpad and doorbell + registers. By default, Ping Pong will not attempt to exercise such + hardware. You may override this behavior at your own risk by setting + unsafe=1. +* delay\_ms - Specify the delay between receiving a doorbell + interrupt event and setting the peer doorbell register for the next + round. +* init\_db - Specify the doorbell bits to start new series of rounds. A new + series begins once all the doorbell bits have been shifted out of + range. +* dyndbg - It is suggested to specify dyndbg=+p when loading this module, and + then to observe debugging output on the console. + ## NTB Hardware Drivers NTB hardware drivers should register devices with the NTB core driver. After -- cgit From 578b881ba9c4b253482903bf8fae438318f5629b Mon Sep 17 00:00:00 2001 From: Allen Hubbe Date: Thu, 21 May 2015 02:51:39 -0400 Subject: NTB: Add tool test client This is a simple debugging driver that enables the doorbell and scratch pad registers to be read and written from the debugfs. This tool enables more complicated debugging to be scripted from user space. This driver may be used to test that your ntb hardware and drivers are functioning at a basic level. Signed-off-by: Allen Hubbe Signed-off-by: Jon Mason --- Documentation/ntb.txt | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index e5d1591478ca..b48249a7b607 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -64,6 +64,38 @@ Module Parameters: * dyndbg - It is suggested to specify dyndbg=+p when loading this module, and then to observe debugging output on the console. +### NTB Tool Test Client (ntb\_tool) + +The Tool test client serves for debugging, primarily, ntb hardware and drivers. +The Tool provides access through debugfs for reading, setting, and clearing the +NTB doorbell, and reading and writing scratchpads. + +The Tool does not currently have any module parameters. + +Debugfs Files: + +* *debugfs*/ntb\_tool/*hw*/ - A directory in debugfs will be created for each + NTB device probed by the tool. This directory is shortened to *hw* + below. +* *hw*/db - This file is used to read, set, and clear the local doorbell. Not + all operations may be supported by all hardware. To read the doorbell, + read the file. To set the doorbell, write `s` followed by the bits to + set (eg: `echo 's 0x0101' > db`). To clear the doorbell, write `c` + followed by the bits to clear. +* *hw*/mask - This file is used to read, set, and clear the local doorbell mask. + See *db* for details. +* *hw*/peer\_db - This file is used to read, set, and clear the peer doorbell. + See *db* for details. +* *hw*/peer\_mask - This file is used to read, set, and clear the peer doorbell + mask. See *db* for details. +* *hw*/spad - This file is used to read and write local scratchpads. To read + the values of all scratchpads, read the file. To write values, write a + series of pairs of scratchpad number and value + (eg: `echo '4 0x123 7 0xabc' > spad` + # to set scratchpads `4` and `7` to `0x123` and `0xabc`, respectively). +* *hw*/peer\_spad - This file is used to read and write peer scratchpads. See + *spad* for details. + ## NTB Hardware Drivers NTB hardware drivers should register devices with the NTB core driver. After -- cgit From 2f887b9a44015a8146d52e40bef9e2b7bc6cd275 Mon Sep 17 00:00:00 2001 From: Dave Jiang Date: Wed, 20 May 2015 12:55:47 -0400 Subject: NTB: Rename Intel code names to platform names Instead of using the platform code names, use the correct platform names to identify the respective Intel NTB hardware. Signed-off-by: Dave Jiang Signed-off-by: Jon Mason --- Documentation/ntb.txt | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) (limited to 'Documentation') diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt index b48249a7b607..1d9bbabb6c79 100644 --- a/Documentation/ntb.txt +++ b/Documentation/ntb.txt @@ -115,13 +115,13 @@ Module Parameters: * b2b\_mw\_share - If the peer ntb is to be accessed via a memory window, and if the memory window is large enough, still allow the client to use the second half of the memory window for address translation to the peer. -* snb\_b2b\_usd\_bar2\_addr64 - If using B2B topology on Xeon hardware, use this - 64 bit address on the bus between the NTB devices for the window at - BAR2, on the upstream side of the link. -* snb\_b2b\_usd\_bar4\_addr64 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_usd\_bar4\_addr32 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_usd\_bar5\_addr32 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_dsd\_bar2\_addr64 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_dsd\_bar4\_addr64 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_dsd\_bar4\_addr32 - See *snb\_b2b\_bar2\_addr64*. -* snb\_b2b\_dsd\_bar5\_addr32 - See *snb\_b2b\_bar2\_addr64*. +* xeon\_b2b\_usd\_bar2\_addr64 - If using B2B topology on Xeon hardware, use + this 64 bit address on the bus between the NTB devices for the window + at BAR2, on the upstream side of the link. +* xeon\_b2b\_usd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_usd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_usd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_dsd\_bar2\_addr64 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_dsd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_dsd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*. +* xeon\_b2b\_dsd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*. -- cgit