nm_minidriver.h File Reference

Definition of the driver interface. More...

#include <nm_public.h>
#include <Padico/Puk.h>
#include <sys/uio.h>

Include dependency graph for nm_minidriver.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	nm_drv_profile_s
	Performance information for driver. More...
struct	nm_minidriver_capabilities_s
	Static driver capabilities. More...
struct	nm_minidriver_hints_s
	some hints to help drivers tune themselves More...
struct	nm_minidriver_properties_s
struct	nm_minidriver_iface_s
	Interface driver for the 'NewMad_minidriver' component interface. More...
struct	nm_prefetch_key_s
	key to match prefetch entries with actual send_iov_post More...

Functions
	PUK_IFACE_TYPE (NewMad_minidriver, struct nm_minidriver_iface_s)
static int	nm_prefetch_key_eq (const struct nm_prefetch_key_s *p_key1, const struct nm_prefetch_key_s *p_key2)
static uint32_t	nm_prefetch_key_hash (const struct nm_prefetch_key_s *p_key)

Detailed Description

Definition of the driver interface.

A driver with a context corresponds to a NIC. The same driver may be used in multiple contexts in case of multi-rail. An instance is created for each gate.

A driver must define a subset of functions of struct nm_minidriver_iface_s. ** Init **

• getprops: the function is used to get properties of the driver. It is mandatory that all drivers define this function. In the properties, the field 'hints' contains hints given by upper layers for the driver; all other fields are initialized to 0 and are supposed to be filled-in by the driver. This function is always called first, before init. Additionnal hints are available as context attributes: "session_size", "rank", "wide_url_support".
• init: the function is used to initialize the driver in the given context. It is mandatory that all drivers define this function. It returns an url that the launcher will provide to other nodes. The url is per-context. If a driver needs per instance urls, it is its responsibility to manage per-instance urls through the unique per-context url. In case wide_url_support is granted by the launcher, it may initialize a vector of urls, packed as a single wide per-context url. It must still support cases where non-scalable launcher (e.g. PMI2) does not support wide urls.
• close: the function closes a driver in the given context. It is always called last, after all instances have been disconnected. This function may be left NULL.

** Connection establishment ** A driver must define at least one method for connection establishment among synchronous or asynchronous.

• For synchronous connection, connect() will be called on all nodes in sequence for each gate; it is assumed to establish the connection towards the node of the given url. This function is expected to be called on both nodes at the same time and is allowed to block. The remote node is known only through its url.
• For asynchronous connection, connect_async() is called on all nodes for each gate, then connect_wait() is called for each gate. Multiple connection establishments may run at the same time. Function connect_async() may not block. A driver supporting fully asynchronous connection establishment is allowed to only define connect_async() and leave connect_wait() to NULL. If both methods arre available, a launcher that supports asynchronous connection will prefer asynchronous for better scalability. Function disconnect() is optionnal and may be left NULL. It is guaranteed that no communication will take place after disconnect.

** Sending ** 3 methods are available to initiate a send operation: buffer-based, iov-based, data-based. 2 methods are available to manage completion: polling and blocking wait. A driver must provide at least one method to initiate a send, must provide polling, and may optionnaly provide blocking wait.

• buffer-based: the user calls send_buf_get() to get a buffer, fills it, then calls send_buf_post() to send it. The driver has thus the opportunity to allocate buffers directly in registered memory. This method is only possible for drivers for small packets.
• iov-base: the user calls send_iov_post() whith an iovec that describes the packet to send. If the driver does not support iovecs (capability supports_iovec=0), then send_iov_post() will always be called with n=1.
• data-based: the user calls send_data_post() with an nm_data_s that describes the data layout. The implementor should use the method that allows the lowest number of memory copies: buffer-based for small packets on network with memory registration or shared memory, data-based for large packets that need a memory copy, iov-based (with iovec support, as much as possible) otherwise. For completion, support of send_poll() is mandatory; support of send_wait() is optionnal. Even when using blocking wait, send_poll() will be called at least once before. It must be noted that these functions may be called from different threads, but not concurrently.

** Receiving ** 3 methods are available to initiate a recv operation: buffer-based, iov-base, data-based. 2 methods are available to manage completion: polling and blocking wait. A driver must provide at least one method to initiate a recv, must provide polling, and may optionnaly provide blocking wait. For scalability, before posting a recv request on a given instance, it is possible to ask globally (context-wide) for which instance has data available for receive. This may be done in a non-blocking way with recv_probe_any() or blocking with recv_wait_any(). After one if these functions returns, it is guaranteed that posting a receive on the returned instance will succeed immediately. This mechanism is reserved to drivers for small packet, without rendez-vous.

• buffer-based: the user calls recv_buf_poll() to get a buffer with incoming data, then calls recv_buf_release() to give it back to the driver.
• iov-based: the user calls recv_iov_post(), then polls its completion with recv_poll_one() or waits with recv_wait_one(). If the driver does not support iovecs (capability supports_iovec=0), then send_iov_post() will always be called with n=1. Please note that capability supports_iovec is global, for both sending and receiving.
• data-based: the user calls recv_data_post(), then polls its completion with recv_poll_one() or waits with recv_wait_one(). For completion, support of recv_poll_one() is mandatory for iov-based and data-based methods; blocking wait is optionnal. Upon termination, the user may want to cancel pending requests:
• cancel_recv() cancels a posted recv_iov_post() or recv_data_post()
• cancel_recv_any() cancels a recv_wait_any() that may be running at the same time in another thread.

** Prefetching **

To optimize transfer on networks that require memory registration, it is possible to allow speculative registration on the sender or receiver side while a rendez-vous is still in progress. The user calls send_iov_prefetch() when it is likely that the message will be accepted on the given network, before having received the RTR, and recv_iov_prefetch() when it is likely that the message will be arriving through the given network, before having received rendez-vous request. In case the data arrives through another network or with a different layout (multiple chunks), functions send_iov_unfetch() and recv_iov_unfetch() are called to cancel prefetch. The driver may match calls to send_iov_prefetch() with the corresponding send_iov_post() through the iovec.

** Rendez-vous data ** The driver may want to piggy-back some data with the RTR messages. In this case, the user calls get_rdv_data() on the receiver side after the receive has been posted, transports this data allongside with the RTR, and gives it to the driver on the sender side by calling set_rdv_data(). This may be usefull to driver implementors to transfer memory registration information or addresses for RDMA.

Definition in file nm_minidriver.h.

Function Documentation

nm_prefetch_key_eq()

static int nm_prefetch_key_eq ( const struct nm_prefetch_key_s *  p_key1, const struct nm_prefetch_key_s *  p_key2  )

inlinestatic

Definition at line 302 of file nm_minidriver.h.

References nm_prefetch_key_s::n, and nm_prefetch_key_s::v.

nm_prefetch_key_hash()

static uint32_t nm_prefetch_key_hash ( const struct nm_prefetch_key_s *  p_key )

inlinestatic

Definition at line 309 of file nm_minidriver.h.

References nm_prefetch_key_s::v.

PUK_IFACE_TYPE()

PUK_IFACE_TYPE	(	NewMad_minidriver	,
		struct nm_minidriver_iface_s
	)