NewMadeleine README

This document describes nmad installation and configuration.

for any question, send mail to: Alexandre.Denis@inria.fr.

Prerequisites

The following development tools are required to compile NewMadeleine:

• GNU C Compiler gcc (version 6.0 and higher) or compatible (icc, clang).
• GNU make (version 3.81 and higher).
• autoconf (v 2.60 or later)
• pkg-config
• hwloc
• libexpat XML parser (set $EXPAT_ROOT if it cannot be found by pkg-config)
• libucontext on systems where the libc does not support ucontext (e.g. musl libc). Not needed on systems with glibc.
• ibverbs/OFED for InfiniBand networks support (set $IBHOME if not installed in /usr) (optional)
• PSM for Intel InfiniPath networks (optional)
• PSM2 for Intel OmniPath networks (set PSM2_DIR if not installed in system directories) (optional)
• Portals4 for Atos BXI networks (set PORTALS4_DIR if not installed in system directories) (optional)
• libfabric (OFI) for Cray Slingshot networks (optional)
• libpmi2 for PMI2 (slurm) integration and/or libpmix for PMIx (set $PMIX_ROOT if not installed in /usr)
• other PM2 modules:
  • Puk: mandatory
  • PadicoTM: recommended, used as launcher
  • pioman: optional, for progression
  • these modules are built automatically when using the automated build system.

Download

NewMadeleine may be downloaded either as a tarball from https://pm2.gitlabpages.inria.fr/releases/.

or use the git master from https://gitlab.inria.fr/pm2/pm2.

Installation

Installation of NewMadeleine is a standard sequence of ./configure ; make ; make install. However, to build and install all modules required by NewMadeleine, we propose a script to build the modules with the right order and parameters.

Automated build (recommended)

To build all modules required by nmad, we recommend to use the build script located in pm2/scripts/pm2-build-packages using a given or a custom configuration file, e.g.:

 % cd pm2/scripts
 % ./pm2-build-packages ./madmpi.conf --prefix=$HOME/soft/x86_64

For a standard multi-threaded build, it is advised to use madmpi.conf. For a non-threaded (no progression!), configuration from madmpi-mini.conf will lead to a slightly more efficient library.

Manual build (not recommended, advanced users only)

Module nmad requires other pm2 modules: Puk, PadicoTM, PukABI (optionnal), pioman (optionnal).

For each module:

./autogen.sh
mkdir build ; cd build
../configure [your options here]
make
make install

Usefull configure flags (see ./configure –help)

--enable-sampling       Enable network sampling
--enable-mpi            Enable builtin MPI implementation MadMPI
--with-pioman           use pioman I/O manager [default=no]
--with-ibverbs          use Infiniband ibverbs [default=check]
--with-psm              use Intel Performance Scaled Messasging (PSM) [default=check]
--with-psm2             use Intel Performance Scaled Messasging 2 (PSM2) [default=check]

Building for developpers

The pm2-build-packages script builds everything in $prefix/build/. It is possible to re-build selectively a module without re-building the full pm2 stack, e.g.:

% cd $prefix/build/nmad
% make -j install

The following build targets are useful:

• install: build and install. Parallel build with -j is supported.
• config: re-run config.status (re-generate all autogenerate files, when a *.in file was modified).
• help: display targets

By default, build is pretty-printed. To disable this behavior and display the full commands, set V=1:

% make install V=1

Installation through spack

NewMadeleine is included in the spack package manager. Installing NewMadeleine git master from spack is as easy as:

% spack install nmad@master

Notable options to consider are:

• pioman to enable or disable threading support. Default is enabled.
• pukabi to enable or disable Puk ABI interceptor, used to invalidate Infiniband registration cache. Default is disabled (and thus ibverbs uses IODP).
• ibverbs, psm, psm2, ofi, ucx to enable these drivers.
• pmix, craypmi to enable the given version of PMI to interface with slurm. Default is pmix enabled.

Building application code

For an MPI applicatiion using MadMPI, use the standard mpicc, mpif77 and mpif90 compiler frontends to build and link.

To build an application using native NewMadeleine interface, get the required flags through pkg-config. For CFLAGS:

% pkg-config --cflags nmad

For libraries:

% pkg-config --libs nmad

In a Makefile, you will typically need:

CFLAGS += $(shell pkg-config --cflags)
LIBS += $(shell pkg-config --libs)

Launcher

For MadMPI use the standard mpirun as launcher. Please see mpirun --help for up-to-date documentation. Please note that MadMPI mpirun is a frontend to padico-launch so it accepts all options described below. For example:

% mpirun -n 2 -nodelist henri0,henri1 nm_mpi_cpi

When their is a risk of using the mpirun command from another MPI library, it is possible to ensure that we are using the mpirun from MadMPI by calling mpirun.madmpi.

For native NewMadeleine applications, it is possible to use mpirun as well as padico-launch as a launcher for nmad. It accepts parameters similar to mpirun. Please see padico-launch --help for up-to-date documentation. For example:

% padico-launch -n 2 -nodelist jack0,jack1 nm_bench_sendrecv

starts program nm_bench_sendrecv on hosts jack0 and jack1, using auto-detected network.

Environment variables may be set using -D parameters, e.g.:

% padico-launch -c -p -n 2 -nodelist jack0,jack1 -DNMAD_DRIVER=ib nm_bench_sendrecv

starts program nm_bench_sendrecv on hosts jack0 and jack1, over Infiniband, using one console per process.

On clusters using slurm, mpirun and padico-launch will start processes using srun. It is possible for the user to directly use srun without using mpirun at all. Make sure to enable pmi2 or pmix, e.g.:

% srun -N 2 --exclusive --mpi=pmix nm_bench_sendrecv

For better scalability, all jobs may be started in MPI mode using padico-launch -mpi.

Debug

gdb

To launch each process in a gdb debugger, use argument -d in association with -c to get one console per node, i.e.:

% padico-launch -c -d -n 2 nm_sr_hello

valgrind

To launch each process in valgrind memcheck tool, use argument --padico-valgrind, in association with -c, i.e.:

% padico-launch -c --padico-valgrind -n 2 nm_sr_hello

AddressSanitizer (ASAN)

AddressSanitizer is supported in the configure and makefiles. Its support is driven by the configure flag --enable-asan. All modules must be configured with the flag. To do so, it may given as an additional flag to the build command:

 % ./pm2-build-packages ./madmpi.conf --prefix=$HOME/soft/x86_64 -- --enable-asan

Then start the binaries as usual. padico-launch adds defaults flags when ASAN support is built-in. LeakSanitizer is activated by default.

See https://github.com/google/sanitizers/wiki/addresssanitizer for more info about using and configuring ASAN.

Detect invalid data change

NewMadeleine comes with a tool to detect if user data is modified while a non-blocking send is manipulating the data, which leads to data corruption. To detect such a bug in application code, please set the environment variable NMAD_ISEND_CHECK to a non-null value, i.e.:

% padico-launch -n 2 -DNMAD_ISEND_CHECK=1 nm_sr_hello

It checks whether user buffer has been modified between nm_sr_isend and nm_sr_swait, or between MPI_Isend and MPI_Test or MPI_Wait. This feature is only available when nmad is built in debug mode.

Verbosity

By default, NewMadeleine is quiet and outputs only warnings and fatal errors. To display info about the init (network detection, addresses, drivers used), it is advised to use verbose mode with the -v parameter:

% padico-launch -v -n 2 nm_sr_hello

Verbose mode is the default when NewMadeleine is built in debug mode. It is possible to switch to quiet mode with parameter -q. Verbosity may be controlled in a fine way. See traces.md in Puk for documentation, available online: https://pm2.gitlabpages.inria.fr/pm2/Puk/doc/PukTracePolicy.html

Per-node logging

To help debug code on a large number of nodes, standard output and stderr may be captured and sent to disk, with one file per node using the --log parameter, e.g.:

% padico-launch --log=${HOME}/log-$$ -n 2 nm_sr_hello

will send output to files created in directory ${HOME}/log-$$. Directory is created if needed. File names contain: the username, the session uuid, the node rank, the hostname, and the node uuid, to avoid collisions and to allow easy browsing.

Deadlocks

To help debug deadlocks in communications, NewMadeleine is able to detect stalled packets using the environment variable NMAD_PWSEND_TIMEOUT, i.e. :

% padico-launch -n 2 -DNMAD_PWSEND_TIMEOUT=1 nm_sr_hello

It checks whether a packet wrapper takes more than 30 seconds to be sent on any track, or to be received only on large track. Since it relies on timers from profiling, it requires NewMadeleine to be built with profiling (--enable-profile at configure).

In addition, when built with PadicoTM (the default), this flag enables a watchdog to check how often the optimizing strategy is called.

Memory

To help diagnose OOM errors, a memory monitor is available to display the allocated memory. It may be enabled by using the MemMonitor PadicoTM module. It is loaded by using -iload-MemMonitor init flag, e.g.:

% padico-launch -n 2 -iload-MemMonitor -DPADICO_MEM_MONITOR_PERIOD=5 nm_sr_hello

By default, it displays periodically the memory usage of the whole process, as given by getrusage. The MaxRSS (maximum resident set size) is displayed in kilobytes. The period may be tuned through the optionnal environment variable PADICO_MEM_MONITOR_PERIOD (in seconds); the default is 3 seconds.

In addition, when Puk profiling is enabled (--enable-profile for module Puk), it displays a detailed view of the allocation by module:

• puk_profile_total_bytes is the current total amount of memory allocated through Puk wrappers, in bytes.
• puk_profile_malloc is the number of times malloc was called through the wrappers.
• puk_profile_free is the nomber of times free was called.
• the line starting with puk_mods gives the amount of memory allocated by each loaded module. Some modules may display a negative amount in case a block was allocated in a module then freed by another module.

The memory accounted here is memory allocated through Puk wrappers, which is the memory used by Puk+pioman+PadicoTM+NewMadeleine. Memory allocated by the application is not counted nor seen by this tool, except in maxrssKB which measures the complete Unix process.

To extract time series for each module and to draw graphs, use the script padico-mem-monitor-extract.

Other fields returned by getrusage() may be displayed with parameter PADICO_MEM_MONITOR_RUSAGE. Note however that on Linux systems, most fields are unused/unimplemented by the kernel.

PAJE traces

NewMadeleine may generate a trace of its internal state in the PAJE format. To do so, you may use the --enable-trace configuration flag. This option requires the external library GTG.

When compiled with traces, NewMadeleine will automatically generate a PAJE trace file in the current directory at the end of the execution. A single file is generated for all nodes.

The content of traces may be controlled by the NMAD_TRACE environment variable. It must contain a coma-separated list of the following items:

- core    trace state of nmad core (beware: huge traces)
- driver  trace state of packet-wrappers
- pack    trace state of pack/unpack requests
- link    generate arrows for messages
- all     all of the above
- none    no trace
- ^core   remove core state
- ^driver remove pw state
- ^pack   remove request state
- ^link   remove arrows

Operands are evaluated in order when adding/removing filters. The default when the variable is not set by user is NMAD_TRACE=all,^core.

Profiling

To enable profiling counters in NewMadeleine, please give --enable-profile to the configure.

Then it is possible to control which counters are displayed using the PUK_PROFILE environment variable which gives a filter to be matched against the profiling varibale name. By default, nothing is displayed. Use ‘PUK_PROFILE=’*'to display all variables, PUK_PROFILE=nm_drv.*to only display variables from nmad drivers. Multiple patterns may be given separated by a semicolon, e.g.PUK_PROFILE=nm_drv.*;*.max_memto display all variables from nm_drvnamespace and all variablesmax_mem` from all modules

Set PUK_DISPLAY_PROFILE=yes to display the description of all profiling variables.

Memory profiling is available only if Puk, in addition to nmad, was configured with the --enable-profile option.

Profiling is usually displayed only at process termination. It is possible to display it periodically. To do so, use the -iload-ProfMonitor init flag, e.g.:

% padico-launch -n 2 -iload-ProfMonitor -DPUK_PROFILE='*' nm_sr_hello

The period may be tuned using the optionnal environment variable PADICO_PROF_MONITOR_PERIOD, in seconds. The default is 10 seconds.

Advanced Tuning

Parameters

NewMadeleine is tuned through parameters that can be set through environment variables or programmatically (see Puk-opt.h). For convenience, environment variables may be set on the command line using the following syntax:

% padico-launch -DVAR=value

to set a value to environment variable VAR.

Parameters are typed (string, int, unsigned, bool). Valid values for boolean variables are: 0/1, y/n, yes/no, true/false, on/off, enabled/disabled.

To display the list of all parameters and their value, give the parameter -DPUK_DISPLAY_ENV=yes

Strategy

The strategy used by nmad is selected using the following rules:

1. if the environment variable NMAD_STRATEGY is set, it is used whatever the other configuration parameters are.
2. if the variable is not set, strategy aggreg is used by default. Please note that while used with StarPU, StarPU automatically forces strategy prio.

Valid strategies are: default, aggreg, split_balance, prio.

The following are deprecated/unmatained: split_all, qos

The default choice should fit most cases.

Strategy prio limits the total number of simultaneous outgoing packets. This number may be tuned using environment variable NMAD_PRIO_MAX_PW. The default value is -1 for an adaptive value depanding on the number of nodes.

Drivers

The drivers used by nmad are selected automatically according to the detected hardware. It is expected to make the best decision in most cases. It is using the following rules:

1. if the environment variable NMAD_DRIVER is set, it is used by default. It may contain the name of a single driver for single rail, or a list of multiple drivers separated by + for multirail, e.g. NMAD_DRIVER=mx+ibverbs

   The following driver names are recognized:

   • ibverbs for default InfiniBand drivers
   • ibrcache, iblr2, ibsrq or ibbuf to force the InfiniBand protocol
   • tcp or tcp_epoll for TCP sockets using epoll; tcp_posix for TCP with Posix poll
   • psm for Infinipath
   • psm2 for Omni-Path
   • bxi or portals4 for Portals4 network (tested only with Atos BXI)
   • ucx for UCX library. This is the only way to use ucx, since it is never used automatically.
   • ofi or ofi_rdm for libfabric in RDM mode; ofi_msg for msg mode.
   • shm for shared memory on the same node
   • cma for shared memory with cross-memory attach
   • local for Unix domain sockets (basic driver, for debug)
   • self for intra-process loopback is always added by default by nmad and does not need to be given by the enduser.
   • other drivers (mx, sisci, cci, dcfa, qsnet) are deprecated.

   It should be noted that this method uses a single driver on all links, and thus cannot distinguish intra-node and inter-node communications. The next method based on PADICO_NETSELECTOR_LIST should be prefered.

   Environment variables may not be propagated to nodes, depanding on the used launcher. It is safer to explicitely let the launcher set the variable on each node:

   % padico-launch -n 2 -DNMAD_DRIVER=ibverbs nm_sr_hello

2. if nmad is launched with mpirun, srun, or padico-launch, then NetSelector is used to select the driver according to the available hardware. Customization is possible through the use of the PADICO_NETSELECTOR_LIST environment variable. It contains a comma-separated list of allowed or forbiden (using a caret ^) drivers. It is interpreted in order: to decide whether to enable or disable a driver, the first matching rule is applied. The available drivers are the following:

   • self is used for intra-process
   • cma for intra-node inter-process, using Cross Memory Attachment. It may not be available on all machines and is not auto-detected.
   • shm for intra-node inter-process
   • local for intra-node Unix domain sockets (basic driver, for debug)
   • psm2 for OmniPath, if present (auto-detected)
   • psm for InfiniPath, if present (auto-detected)
   • ibverbs for InfiniBand is present (auto-detected when nodes are on the same IB subnet). Thus it is important to configure subnet GID prefix and not keep the factory default GID prefix in opensm. Note that nmad actually uses separate drivers for small and large packets; this is normally hidden to the user with most drivers. For InfiniBand, ibverbs uses a heuristic for the right combination, but the drivers for small and large packets may be configured separately.
   • ibsrq for InfiniBand small packets, using Shared Receive Queues. This is the most scalable mechanism for large number of nodes. This is used by default for 16+ processes.
   • ibbuf for InfiniBand small packets, using buffer-based RDMA. It gets a very low latency for small numbers of nodes, but does not scale well.
   • ibrcache for InfiniBand large packets, using registration cache. This is used by default for 100G+ IB boards.
   • iblr2 for InfiniBand large packets, using pipelined copy. It gets a good bandwidth on slow IB boards and a fast CPU. This is used by default for < 100G IB boards.
   • ibsendrecv for InfiniBand large packets, using send/recv. It leverages a registration cache but is able to send non-contiguous data, contrary to ibrcache.
   • portals4 for BXI networks using Portals4 library. Other networks than BXI relying on Portals4 interface will not be detected.
   • cxi for Cray Slingshot 11 using OFI RDM. Other networks than Cray Slingshot will not use OFI by default.
   • tcp_epoll for TCP/IP with Linux epoll interface
   • tcp for TCP/IP sockets
   • control as last resort, routed messages over control channel are used if no direct connection is possible.
   • multirail enables the use of multi-rail, i.e. aggregate multiple networks. See Multirail and multi-NIC policy for more information. It is disabled by default. This is a special rule that recursively invokes the same algorithm to decide what the rails are.
   • defaults is a shortcut for the default value. Its value is: self,shm,psm2,psm,ibverbs,portals4,cxi,tcp_epoll,tcp,control

   When establishing connexions, drivers are always considered in the order of the above list. The order of drivers in PADICO_NETSELECTOR_LIST is only used to decide whether to enable or disable a driver, not an order of precedence for drivers.

   For example, to use the defaults but with CMA for intra-node, use: PADICO_NETSELECTOR_LIST=cma,defaults

   To disable the use of PSM2, use: PADICO_NETSELECTOR_LIST=^psm2,defaults

   To force SRQ and rcache on InfiniBand: PADICO_NETSELECTOR_LIST=ibsrq,ibrcache,defaults

   To only consider self, shared-memory and psm2, without considering other networks: PADICO_NETSELECTOR_LIST=self,shm,psm2

   Note that not all drivers are able to send data to self, thus a configuration without driver self may lead to unspecified behavior.

3. if nmad is launched through the cmdline launcher, then a -R <string> parameter is taken as a railstring, with the same syntax as NMAD_DRIVER. Please note that cmdline launcher is only for debug purpose and manages only 2 nodes.
4. if another custom launcher is used, it may set a selector using the 'sesion' interface.
5. in any other case, self is used for intra-process; tcp for inter-process.

NMAD_DISPLAY_DRIVERS=1 displays strategy and drivers used by each process.

For most users, auto-detection should do the right thing and endusers are not expected to manually select a driver. Users who want to customize the drivers are encouraged to use PADICO_NETSELECTOR_LIST.

GPU Support (CUDA, HIP)

NewMadeleine and MadMPI are GPU aware. It may be given CUDA or HIP pointers.

GPU support is disabled by default. It must be:

• enabled explicitely at build time with --with-cuda or --with-hip, e.g.: % ./pm2-build-packages ./madmpi.conf --prefix=$HOME/soft/x86_64 -- --with-cuda
• enabled explicitely at runtime with -DNMAD_GPU_SUPPORT=1

The library is fully GPU aware, meaning that it is able to access data in GPU memory. In most cases, data is copied transparently into host memory before sending it to the network. We get good results with drivers shm and iblr2 which are fully GPU aware.

If available, we may use GPU Direct, a feature to move data directly from GPU memory to the network interface, without going through host memory. It is currently available only on the following networks:

• ofi (libfabric): the feature is automatically detected. It has been tested successfuly on Cray Slingshot network
• psm2: GPU Direct is enabled only if the user manually sets PSM2_CUDA=1 and PSM2_GPUDIRECT=1 as required by the PSM2 library.

Other networks are not supported yet for GPU Direct. Moreover, GDRCopy is not supported yet.

Thread safety and communication progression

Thread safety and progression are closely related. Several levels of support are available:

1. no thread safety, no progression: when building without pioman, communication progresses only in nmad or MPI calls; thread safety is not available. Latency is slightly lower than with pioman. Maximum thread level support is MPI_THREAD_SERIALIZED.

   This mode is obtained when building with profile madmpi-mini.conf (or nmad-pioman when building with spack).

2. thread safety, no progression: when building with pioman, thread safety is available by default and MPI_THREAD_MULTIPLE is supported.

   However, even whith pioman, progression is disabled by default.

   This mode is obtained when building with profile madmpi.conf (or nmad+pioman when building with spack). This is the default.

   By default non-blocking request submission does not make communication progress, for scalability reasons. Set NMAD_AUTO_FLUSH=1 to flush outgoing packets after every posted send. It ensures data is sent earlier; in return, it increses contention between threads and prevent the aggreg strategy to actually aggregate messages.

3. thread safety, progression: with pioman, progression may be enabled manually by the user. Please see instructions in the README from pioman for more details. Available policies are:

   – default progression with PIOM_ENABLE_PROGRESSION=1 (timer-based and low-priority thread for opportunistic scheduling). Please note that pioman threads may collide with application threads.

   – dedicated core: set PIOM_DEDICATED=1 to enable dedicated thread, PIOM_DEDICATED_LEVEL=core and PIOM_DEDICATED_DISTRIB=1 to get the dedicated thread on the first core.

   – dedicated core controlled at runtime: set PIOM_DEDICATED_WAIT=1 to indicated that binding will be set at runtime through piom_ltask_set_bound_thread_indexes() or piom_ltask_set_bound_thread_os_indexes(). This is the default when used with StarPU.

   – set PIOM_VERBOSE=1 to let pioman display what it is doing.

   Set these variables using -D parameters as described in section Launcher.

Shm tuning

By defdault, intra-node shared memory communication uses a pipelined copy. The more efficient "Cross Memory Attachment" (CMA) method may be used on systems that support it using environment variable NMAD_SHM_CMA=1 or by setting PADICO_NETSELECTOR_LIST=cma,defaults.

Binding

NUIOA (Non-Uniform I/O Access) may be taken into account by NewMadeleine. To do so, use NMAD_NUIOA_ENABLE=1 to automatically bind threads to the NUMA node where the network board is attached.

This is disabled by default.

Tuning for sparse data handling

When sending sparse data (a message not contiguous in memory), different methods are used depending on the size and layout of data and the capabilities of the underlying driver.

• small packets are constructed from a header and small data chunks. By default, memory copy is used to assemble the packet. When the underlying driver supports iovec, data chunks larger than a given threshold are sent using iovec instead of memory copy.
• for large data sent with a rendez-vous, when the message is not contiguous in memory and the driver is not able to send it in a single packet using iovec or native nm_data iterators, then the methods depends on data density. Messages with an average block size lower than the iov threshold, and messages that the driver cannot send as packets larger than the large threshold, are considered low density. The following algorithm is used:
  • messages considered as high-density are sent as multiple chunks, using multiple rendez-vous. Chunks are as large as allowed by the capabilities of the driver with iovecs.
  • messages with low density are sent through an intermediate memory copy in a contiguous buffer. This copy may be pipelined using a fixed buffer size, to reduce pressure on memory.

These mechanisms may be tuned using the following parameters:

• NMAD_DATA_LARGE_THRESHOLD is the minimum size to consider chunks as large, sent through rendez-vous. Chunks smaller than that will be sent without rendez-vous, through copy. The default value is 4096.
• NMAD_DATA_IOV_THRESHOLD is the threshold to switch between copy and iovec. The default value is 512 bytes.
• NMAD_DATA_COPY_BLOCKSIZE is the block size used to perform the copy, when an intermediate copy is used for large messages. The default value is 4194304 (4MB). A value of 0 disabled blocking; it may be usefull for deep datatypes where sub-blocking cost may be higher than the gain brought by sub-blocking.

Infiniband tuning

NewMadeleine drivers are actually pairs of drivers: a driver is used for small messages and another for large messages sent with rendez-vous. On InfiniBand, multiple options are available for each driver. For small messages, drivers are:

• bycopy: this driver copies data into a registered memory zone, then send it through RDMA write. It is basic and should be considered as legacy.
• bybuf: the driver provides nmad with pre-registered buffers, so that to construct packets in an already registered memory zone. This is the default driver for a number of nodes up to 16.
• srq: this driver uses ibverbs Shared Receive Queues (SRQ) for better scalability. It is the default for more than 16 nodes.

Large drivers for InfiniBand are as follows:

• rcache: uses RDMA write with a registration cache. See below for details about the cache policy. It is able to send only data contiguous in memory. However, nmad core may fragment data in multiple contiguous chunks if needed. This is the default driver for 100G+ InfiniBand cards.
• lr2: leverages a pipelined copy to bring data into pre-registered memory. It is slower and consumes more CPU than ibrcache but does not rely on tracking of memory mappings, which may solve comptbility issues. It is able to send efficiently any datatype; it shines especially with sparse data. This is the default driver for InfiniBand cards below 100G.
• sendrecv: uses InfiniBand send/recv instead of RDMA. Performance is a little slower than rcache, but it allows to send sparse data comprised of up to 16 chunks. This is still experimental and never used by default.

When forcing a network through NMAD_DRIVER, the name actually encompasses a pair of drivers for small+large. The pairs are as follows:

• iblr2 = bybuf + lr2
• ibsrq = srq + lr2
• ibsrqrcache = srq + rcache
• ibcopy = bycopy + lr2
• ibrcache = bybuf + rcache
• ibcopyrcache = bycopy + rcache
• ibsendrecv = bybuf + sendrecv

This method based on NMAD_DRIVER should be considered obsolete; users are advised to use the method based on PADICO_NETSELECTOR_LIST instead which is more flexible.

Infiniband may be tuned at run time through environment variables:

• NMAD_IBVERBS_RCACHE=1 forces the use of the registration cache in all cases. Without this option, the cache is used only for 100+GB/s IB network. Setting this option is the same as setting PADICO_NETSELECTOR_LIST=ibrcache,defaults.
• NMAD_RCACHE_POLICY is used to give a list of policies that are tried (and ignored if detected as unsupported by software or hardware) for drivers using a registration cache (rcache and sendrecv). Various policies are available to manage the cache and cache invalidation when its memory mappings change:
  • pukabi uses a cache of registered segments; cache invalidation, tracking memory mapping, is performed using PukABI to intercept symbols of memory mangement functions through the use of LD_PRELOAD. It depends on the availability of the PukABI module. This method should be prefered, and thus it is recommended to include module PukABI when using InfiniBand networks by using a flavor such as pukabi+madmpi.conf or pukabi+madmpi-debug.conf.
  • iodp registers the full process memory for Implicit On-Demand Paging (IODP), with prefetch. This mode is available on ConnectX-5+ hardware. It is safe to use, compatible with any memory allocator, and gets fair performance (although a little slower than pukabi).
  • odp registers memory blocks with explicit On-Demand Paging (ODP), with page fault prefetch, without cache. Performance is poor.
  • nocache implements no cache at all; memory is registered and deregistered for each packet. Performance is poor. It is the fallback when nothing else is available.
  • mallopt uses calls to mallopt() to make malloc/free never unmap memory and disables some advanced malloc features, so as to implement rcache without invalidation at all. It gets good network performance but impacts significantly the malloc performance. In addition, it does not work if application directly uses mmap(). Use at your own risks.
  • hooks implements a cache and uses __free_hook() from libs to invalidate cache entries. It is not thread safe and should be considered as deprecated.
  • userfaultfd implements a cache of registered segments, with invalidation based on a tracking of memory mappings through the userfaultfd Linux kernel API. This policy is still considered experimental.

The default value is: NMAD_RCACHE_POLICY=pukabi,iodp,nocache

• NMAD_IBVERBS_SRQ=1 forces the use of Shared Requests Queues for scalability in number of nodes. The latency penalty is low. It is used by default for > 16 nodes. Setting this option is the same as setting PADICO_NETSELECTOR_LIST=ibsrq,defaults.
• NMAD_IBVERBS_CHECKSUM=1 enables the checksum computation on the fly in the driver.
• NMAD_IBVERBS_ALIGN=<n> sets alignment of every packets sent through InfiniBand to <n>, using padding. Default is 64 bytes.
• NMAD_IBVERBS_MEMALIGN=<n> enforces alignment of buffers used internally in all InfiniBand drivers. Default is 4096 bytes.
• NMAD_IBVERBS_COMP_CHANNEL=1 enables the completion channel (to use blocking syscalls) with driver srq.
• NMAD_RCACHE_CHECKSUM=1 enables checksums for the rcache driver, for debugging purpose only.
• NMAD_IBVERBS_LAZY_CONNECT=0 disables lazy connections in all InfiniBand drivers. Lazy connections are enabled by default.

To use a specific IB device or port, they mey be specified in the driver string through driver attributes. The supported attributes for IB drivers ar ibv_device and ibv_port, e.g.:

% padico-launch -n 2 -DNMAD_DRIVER=ibverbs:ibv_device=mlx5_0:ibv_port=1 nm_sr_hello

Either ibv_device, ibv_port, or both, may be given.

When opensm is used as subnet manager, subnet GID must be customized with a value unique to the given subnet, so as nmad is able to automatically detect IB connectivity. As root:

• create the default opensm config file:

  % opensm -o -c /var/cache/opensm/opensm.opts
  

• in the above file, customize the line with subnet_prefix to some other value than the factory default 0xfe80000000000000. Set the same subnet GID on all nodes of the subnet.
• restart opensm:

  % /etc/init.d/infiniband restart
  

PSM2 tuning

By default, nmad sets HFI_NO_CPUAFFINITY=1 if no value was set by the user, to ensure that PSM2 does not mangle with thread binding as set by mpirun. To disable this feature, the user can set HFI_NO_CPUAFFINITY=0 explicitely.

When multiple Omni-Path ports are present, nmad uses by default psm2 automatic port selection. A specific port may be selected by using the port attribute, e.g.:

% padico-launch -n 2 -DNMAD_DRIVER=psm2:port=2 nm_sr_hello

to select the second port. Ports are numbered from 1. port=0 enables the automatic port selection.

Multirail and multi-NIC policy

When nodes have multiple network cards, multiple policies are possible to share the cards between processes.

• when multiple processes run on such nodes, we usually want to have each process use one card. This behavior may be controlled through the PADICO_MULTINIC_POLICY parameter; it accepts the following values:

  • auto: do not give any hint, let the driver choose by itself.
  • first: always use the first card
  • last: always use the last card
  • roundrobin: use a round-robin policy, i.e. the first process uses the first card, the second process uses the second card, etc., and wrap-around if we have more processes than network cards.

  The default is PADICO_MULTINIC_POLICY=roundrobin

• when we have a single process on a node with multiple cards, we usually want to use multirail, i.e. split messages to utilize all cards at once. Currently, our multirail component is able to aggregate only 2 NICs at the same time.

  Multirail must be enabled manually so that it would be considered by the netselector. To do so, add the method multirail to PADICO_NETSELECTOR_LIST as defined in Drivers, e.g. -DPADICO_NETSELECTOR_LIST=multirail,defaults

  Multirail is attempted only for a selection of networks, as controlled by the parameter PADICO_NETSELECTOR_MULTIRAIL_NETWORKS giving a list of patterns to match against network names. By default, its value is Infiniband-*,PSM2-*,BXI,CXI-*. Run padico-topo-print on the nodes to discover the exact name under which the networks are recognized.

Launcher advanced tuning

The appropriate launcher to use is usually selected automatically. For testing and debugging, it may be forced using environment variable NMAD_LAUNCHER. Valid values are:

• madico: use PadicoTM as launcher. Launch nodes through ssh.
• pmi2: use slurm PMI2
• pmix: use slurm PMIx
• single: single node
• cmdline: processes are launched by user, connection information is given on command-line. This launcher is able to launch only 2 processes per job.

The default is pmix if PMIx is detected in the job, pmi2 if a slurm job is detected with pmi2 enabled, madico if we detect job was launched with padico-launch and neither PMI2 nor PMIx are available, single if nothing else is available. cmdline is never selected by default and should be used only for debug.

For PMIx and PMI2 support to be available, PMIX / PMI2 must have been detected at config time.

In case PMIx is not installed in system directories, a full path may be given to --with-pmix=/full/path to configure, or it may be set globally through the PMIX_ROOT environment variable.

Moreover, on some systems, PMIX is not enabled by default in every job, and the user may need to give --mpi=pmix to salloc or sbatch.

Multicast tuning

The default routing tree for the multicast interface is a binomial one. You can change it with the environment variable NMAD_MCAST_TREE set to binary, 3ary, 4ary, 8ary, binomial, 3nomial, 4nomial, 8nomial, flat, chain, bitree, ladder or simply default. The default choice will use 4nomial for messages < 32kB then binomial for larger messages.

When the bitree routing tree is selected, the multicast tree will be split in two trees. You have to provide the characteristics of the bitrees with environment variables:

• the type of the first tree (containing recipients with the higher priorities if priorities are used) with the environment variable NMAD_MCAST_BITREE_FIRST. It can take the same values than NMAD_MCAST_TREE (except bitree).
• similarly, the type of the second tree (containing remaining recipients) with NMAD_MCAST_BITREE_SECOND.
• the number of recipients in the first tree has to be set with the environment variable NMAD_MCAST_BITREE_THRESOLD. If the number of recipients in the whole multicast is lower than this value, only the first tree type will be used.

If you are not using bitrees, you can use the delegate option, which has to be enbled with the environment variable NMAD_MCAST_DELEGATE=1. With this option, the root node of a multicast sends data to the first recipient node and lets this recipient manage the rest of the multicast (perform a binomial tree if this kind of tree is selected, etc).

By default, broadcasting trees are reordered to take into account message priorities. You can change the behavior of the reordering with the environment variable NMAD_MCAST_REORDER_TREE:

• default: default behavior: reorder according to message priorities (most prioritary recipient is on top of the tree)
• reverse: the tree is reordered by respecting the opposite order of priorities (least prioritary recipient is on top of the tree; mostly for experimental purpose)
• default: the tree is reordered in a random fashion (mostly for experimental purpose)
• none: the tree is not reordered

Simulation with simgrid

Simulation may be performed by compiling NewMadeleine with support for simgrid. To do so, the requirements are:

• simgrid installed (tested with simgrid >= 3.31)
• support for dladdr() in the libc (GNU extension, glibc only)
• objdump
• Nix patchelf (>= 0.18) NewMadeleine must be configured with --with-simgrid.

Compilation of user code is done as usual, with mpicc for MPI code or by using pkg-config for native NewMadeleine code. Note that build with -fPIC and linking with -shared will be forced, so as to generate a dynamically loadable object instead of a plain binary. This should be transparent for configure/makefiles as long as they do not try to start the binary (which is actually a dynamic object).

Then launching must be done with nm_simgrid_run instead of padico-launch or mpirun. See nm_simgrid_run -h for help on accepted parameters.

NewMadeleine will automatically perform global symbols privatization, and supports dynamic linking. Dynamic libraries that needs to be privatized must be declared with -lib to nm_simgrid_run. To do so, libraries are automatically duplicated thus enough disk space must be available in <prefix>/var/tmp/. To start a large number of simulated nodes (several hundreds), it may be needed to increase /proc/sys/vm/max_map_count.

Documentation

To generate doxygen documentation:

% cd $prefix/build/nmad
% make docs

It is available online at https://pm2.gitlabpages.inria.fr/pm2/nmad/doc/.