summaryrefslogtreecommitdiff
path: root/Documentation
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2017-07-14 13:31:52 -0700
committerLinus Torvalds <torvalds@linux-foundation.org>2017-07-14 13:31:52 -0700
commitccd5d1b91f22351b55feb6fdee504cb84d97752f (patch)
treec85966f14a3c1efbb2379a3697b00d43937e20b2 /Documentation
parent4d25ec19669292a65a32498eabdabdd32b1a8747 (diff)
parent854b1dd9c39d8c8c8647a44de47ef18506ae11f9 (diff)
Merge tag 'ntb-4.13' of git://github.com/jonmason/ntb
Pull NTB updates from Jon Mason: "The major change in the series is a rework of the NTB infrastructure to all for IDT hardware to be supported (and resulting fallout from that). There are also a few clean-ups, etc. New IDT NTB driver and changes to the NTB infrastructure to allow for this different kind of NTB HW, some style fixes (per Greg KH recommendation), and some ntb_test tweaks" * tag 'ntb-4.13' of git://github.com/jonmason/ntb: ntb_netdev: set the net_device's parent ntb: Add error path/handling to Debug FS entry creation ntb: Add more debugfs support for ntb_perf testing options ntb: Remove debug-fs variables from the context structure ntb: Add a module option to control affinity of DMA channels NTB: Add IDT 89HPESxNTx PCIe-switches support ntb_hw_intel: Style fixes: open code macros that just obfuscate code ntb_hw_amd: Style fixes: open code macros that just obfuscate code NTB: Add ntb.h comments NTB: Add PCIe Gen4 link speed NTB: Add new Memory Windows API documentation NTB: Add Messaging NTB API NTB: Alter Scratchpads API to support multi-ports devices NTB: Alter MW API to support multi-ports devices NTB: Alter link-state API to support multi-port devices NTB: Add indexed ports NTB API NTB: Make link-state API being declared first NTB: ntb_test: add parameter for doorbell bitmask NTB: ntb_test: modprobe on remote host
Diffstat (limited to 'Documentation')
-rw-r--r--Documentation/ntb.txt99
1 files changed, 91 insertions, 8 deletions
diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt
index 1d9bbabb6c79..a5af4f0159f3 100644
--- a/Documentation/ntb.txt
+++ b/Documentation/ntb.txt
@@ -1,14 +1,16 @@
# NTB Drivers
NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects
-the separate memory systems of two computers to the same PCI-Express fabric.
-Existing NTB hardware supports a common feature set, including scratchpad
-registers, doorbell registers, and memory translation windows. Scratchpad
-registers are read-and-writable registers that are accessible from either side
-of the device, so that peers can exchange a small amount of information at a
-fixed address. Doorbell registers provide a way for peers to send interrupt
-events. Memory windows allow translated read and write access to the peer
-memory.
+the separate memory systems of two or more computers to the same PCI-Express
+fabric. Existing NTB hardware supports a common feature set: doorbell
+registers and memory translation windows, as well as non common features like
+scratchpad and message registers. Scratchpad registers are read-and-writable
+registers that are accessible from either side of the device, so that peers can
+exchange a small amount of information at a fixed address. Message registers can
+be utilized for the same purpose. Additionally they are provided with with
+special status bits to make sure the information isn't rewritten by another
+peer. Doorbell registers provide a way for peers to send interrupt events.
+Memory windows allow translated read and write access to the peer memory.
## NTB Core Driver (ntb)
@@ -26,6 +28,87 @@ as ntb hardware, or hardware drivers, are inserted and removed. The
registration uses the Linux Device framework, so it should feel familiar to
anyone who has written a pci driver.
+### NTB Typical client driver implementation
+
+Primary purpose of NTB is to share some peace of memory between at least two
+systems. So the NTB device features like Scratchpad/Message registers are
+mainly used to perform the proper memory window initialization. Typically
+there are two types of memory window interfaces supported by the NTB API:
+inbound translation configured on the local ntb port and outbound translation
+configured by the peer, on the peer ntb port. The first type is
+depicted on the next figure
+
+Inbound translation:
+ Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
+ ____________
+ | dma-mapped |-ntb_mw_set_trans(addr) |
+ | memory | _v____________ | ______________
+ | (addr) |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO
+ |------------| |--------------| | |--------------|
+
+So typical scenario of the first type memory window initialization looks:
+1) allocate a memory region, 2) put translated address to NTB config,
+3) somehow notify a peer device of performed initialization, 4) peer device
+maps corresponding outbound memory window so to have access to the shared
+memory region.
+
+The second type of interface, that implies the shared windows being
+initialized by a peer device, is depicted on the figure:
+
+Outbound translation:
+ Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
+ ____________ ______________
+ | dma-mapped | | | MW base addr |<== memory-mapped IO
+ | memory | | |--------------|
+ | (addr) |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr)
+ |------------| | |--------------|
+
+Typical scenario of the second type interface initialization would be:
+1) allocate a memory region, 2) somehow deliver a translated address to a peer
+device, 3) peer puts the translated address to NTB config, 4) peer device maps
+outbound memory window so to have access to the shared memory region.
+
+As one can see the described scenarios can be combined in one portable
+algorithm.
+ Local device:
+ 1) Allocate memory for a shared window
+ 2) Initialize memory window by translated address of the allocated region
+ (it may fail if local memory window initialization is unsupported)
+ 3) Send the translated address and memory window index to a peer device
+ Peer device:
+ 1) Initialize memory window with retrieved address of the allocated
+ by another device memory region (it may fail if peer memory window
+ initialization is unsupported)
+ 2) Map outbound memory window
+
+In accordance with this scenario, the NTB Memory Window API can be used as
+follows:
+ Local device:
+ 1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can
+ be allocated for memory windows between local device and peer device
+ of port with specified index.
+ 2) ntb_get_align(pidx, midx) - retrieve parameters restricting the
+ shared memory region alignment and size. Then memory can be properly
+ allocated.
+ 3) Allocate physically contiguous memory region in compliance with
+ restrictions retrieved in 2).
+ 4) ntb_mw_set_trans(pidx, midx) - try to set translation address of
+ the memory window with specified index for the defined peer device
+ (it may fail if local translated address setting is not supported)
+ 5) Send translated base address (usually together with memory window
+ number) to the peer device using, for instance, scratchpad or message
+ registers.
+ Peer device:
+ 1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other
+ device (related to pidx) translated address for specified memory
+ window. It may fail if retrieved address, for instance, exceeds
+ maximum possible address or isn't properly aligned.
+ 2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory
+ window so to have an access to the shared memory.
+
+Also it is worth to note, that method ntb_mw_count(pidx) should return the
+same value as ntb_peer_mw_count() on the peer with port index - pidx.
+
### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev)
The primary client for NTB is the Transport client, used in tandem with NTB