Documentation for NewMadeleine-2025-03-18+b4ed7d07041aecd589e8d5bb9d281654cd4543a7 (build runner-7cspzrfxd-project-25853-concurrent-0 Thu Nov 13 12:04:24 UTC 2025)

Introduction

NewMadeleine is a complete redesign and rewrite of the communication library Madeleine. The new architecture aims at enabling the use of a much wider range of communication flow optimization techniques. It is entirely modular: The request scheduler itself is interchangeable, allowing experimentations with multiple approaches or on multiple issues with regard to processing communication flows. We have implemented an optimizing scheduler called SchedOpt. SchedOpt targets applications with irregular, multi-flow communication schemes such as found in the increasingly common application conglomerates made of multiple programming environments and coupled pieces of code, for instance. SchedOpt itself is easily extensible through the concepts of optimization strategies (what to optimize for, what the optimization goal is) expressed in terms of tactics (how to optimize to reach the optimization goal). Tactics themselves are made of basic communication flows operations such as packet merging or reordering.

More information on NewMadeleine can be found at http://pm2.gitlabpages.inria.fr/newmadeleine/.

Getting the source code

NewMadeleine files are hosted by the PM2 project on the Inria Gitlab at https:/gitlab.inria.fr/pm2/pm2/

The source code is managed by a GIT server hosted by the Inria Gitlab. Anonymous GIT access is available, so that you don't need to become a project member to access the repository with the latest source. Alternatively, you can download a released PM2 tarball.

Installation and configuration

For up to date installation details and various configuration parameters see NewMadeleine README

Alternatively, NewMadeleine may be installed through spack or Guix package.

User APIs

Several APIs are provided to NewMadeleine users:

the MadMPI: MPI Interface known as MadMPI, which is a regular MPI implementation with full MPI-1 compliance, and parts of MPI-2, MPI-3, and MPI-4 additions (multi-threading, RMA, non-blocking collectives, sessions, partitioned communications, MPI Tool, large counts). Notable missing features are: graph topologies, neighborhood collectives, dynamic processes

or the native NewMadeleine APIs:

the Public definitions. containing definitions common to all NewMadeleine interfaces.
the Internal data representation: data iterators
the Session interface to open/close sessions.
the Launcher interface to initialize NewMadeleine and get informations on nodes.
the sendrecv interface as general purpose messaging interface based on send/receive, using sessions.
the RPC interface for asynchronous Remote Procedure Calls (RPC).
the Collective interface for collective operations.
the Pack interface for incremental packing/unpacking of messages.
the One-sided interface. for one-sided operations (RMA)
the Multicast interface for dynamic multicasts.

users may use MPI and native interfaces at the same time. However, they must be initialized separately (MPI_Init for MPI, and Launcher interface for native interfaces).

Examples

See README in examples/ directory.

The most basic example to consider first is examples/sendrecv/nm_sr_hello2.c

Hacking in NewMadeleine

Prerequisites

NewMadeleine relies on other modules:

Puk for data structures (vectors, lists, hashtables, interval trees, priority lists), component model, dynamic module loading, display
- see Puk documentation
pioman for progression
- see pioman documentation
PukABI for symbol interception to install memory hooks for InfiniBand (optional)
- see PukABI documentation
PadicoTM as a launcher to deploy processes, establish a control channel, detect networks, select drivers, and exchange addresses.
- see PadicoTM documentation

Building

See Building for developpers for more details on build useful for developpers, so as not to rebuild everything everytime.

To debug and for tests, use gdb (see gdb), use valgrind (see valgrind), use ASAN (see AddressSanitizer (ASAN)). See Debug for other tools.

To check for available makefile rules, use the target help:

% cd $prefix/build/nmad
% make help

Tests

NewMadeleine comes with a large collection of tests. They can be started using the tests make target for the full test suite, or the quicktests target for quick tests:

% cd $prefix/build/nmad
% make tests

By default, tests are run on local host using tcp driver. The following variables are available for tuning:

NMAD_HOSTS: comma separated list of hosts where to run tests
NMAD_DRIVER: driver to use (see Drivers)
NMAD_EXTRA_ARGS: extra arguments given to launcher

Example:

% make -C $prefix/build/nmad tests NMAD_DRIVER=ibverbs NMAD_HOSTS=henri0,henri1

Tests are run at the following places:

quicktests on Gitlab CI
nightly tests on daltons
weekly tests on plafrim

Benchmarks are performed automatically:

nitghtly on daltons
weekly on plafrim

Benchmark and tests results are available here: https://pm2.gitlabpages.inria.fr/benchmark/nmad/

Internal interfaces

The internal interfaces not designed for end-users:

the nmad core interface for direct access to nmad packet scheduler. This is the interface upon all other interfaces rely. This is a private interface and may change at any time.
the Connector interface used internally to exchange urls. It is deprecated.
the MadMPI Private Interface used internally in MadMPI.
the nmad internal interfaces. used internally by the nmad scheduler
the Minidriver interface exposed by drivers
the interface exposed by strategies: struct nm_strategy_iface_s
the interface exposed by launchers: struct nm_launcher_driver_s

Coding rules

When coding in NewMadeleine, most coding rules from PadicoTM apply.

In short, to ensure consistency and performance:

language is C99
visibility: external symbols have a cost when loading. Use static visibility as much as possible. When inter-file, intra-lib visibility is wanted, use __PUK_SYM_INTERNAL.
const: to let the optimizer do its job, use const whenever possible.
code is documented using doxygen. We use the JavaDoc syntax:
/** @brief This is a doxygen example. */
- documentation is built when flag --enable-docs is given for configure.
- documentation is built by Gitlab CI and published automatically at: https://pm2.gitlabpages.inria.fr/pm2/nmad/doc/
naming things:
- lower case with underscores ("snake_case") is used for symbols and files.
- functions starts with nm_, then add the name of the submodule (e.g. collectives start with nm_coll_)
- typedefs end with _t, struct types end with _s, enum types end with _e, variable which are pointers start with p_.
- attributes and environment variables start with NMAD_.
indentation:
- indentation uses spaces (no tabs), 2 spaces, space is trimmed at the end of lines.
- for emacs users, file .dir-locals.el at the root of the project enforces this style.
- open/close braces are on their own line
- pointers use no space: int*p_foo = NULL;
- no space between if, for, while, function name, etc., and the opening parenthesis.

Git

Subscribe to the pm2-commits list: https://sympa.inria.fr/sympa/info/pm2-commits
Do not create merge commits. Use pull.ff=only in git config and rebase manually if needed
When working with emacs, use whitespace-mode to make sure commits are not polluted with spurious spaces.
Make small commits, with a clear target, that will allow cherry-picking if needed. Avoid large cleanup commits that fix things from multiple previous commits at once; cleanup commits must have the same span as the commit they fix, and should be squashed if possible.
Rebase your branch often so that it does not diverge too far, making it impossible to merge.
Make short lived branches, with a clear focus. Make the code review easy.
Host your branch in the pm2 project to make sure your branch survives your account when you leave.

Continuous integration

For each git push on gitlab, continous integration (CI) performs automated tests driven by .gitlab-ci.yml. The CI build and run quick tests on a selection a config flavors, and on a selection of Linux distributions in docker containers.
dockerfiles are located in admin/docker/. These dockerfiles are designed for CI and tests only. They are not designed to be used by end-users.
To debug an issue that happens only on some distribution, it is conveninent to run the same docker images locally, outside of gitlab CI. To do so, a script is provided: admin/docker/run.sh. See ./run.sh --help for an up-to-date documentation.

Example of usage:
cd admin/docker/

./run.sh alpine madmpi-mini-debug /bin/bash

This builds an image based on Alpine Linux, builds pm2 with madmpi-mini-debug.conf flavor, and gives a shell in the container.