2. Installing MPICH-Madeleine

2.1 Getting the Source Code

MPICH-Madeleine files are hosted by the project MPICH-Madeleine on the InriaGforge at https://gforge.inria.fr/projects/mpich-mad/.

2.1.1 Getting the Latest Stable Release

You need to download the package for MPICH-Madeleine as well as the one for PM2. The latest release for these two packages, pm2 and mpich-mad, is available from the main page of the project MPICH-Madeleine at https://gforge.inria.fr/projects/mpich-mad/ under the tab Latest File Releases.

Note that the latest release for the package pm2 is also available from the main page of the project PM2 at https://gforge.inria.fr/projects/pm2/.

2.1.2 Getting the Latest Version

The source code for PM2 and MPICH-Madeleine is managed by a Subversion server hosted by the InriaGforge. To get the source code, you need:

To install the client side of the software Subversion if it is not already available on your system. The software can be obtained from http://subversion.tigris.org/.
To become a member of both projects mpich-mad and pm2. For this, you first need to get an account to the gForge server. You can then become a member of the projects by contacting one of the project administrators. The list of the project administrators is available from the main project page.

You can also choose to check out both project's SVN repository through anonymous access. In that case, you do not need to become a member, and you will have limited access to the repository.

More information on how to get a gForge account, to become a member of the specified projects, or on any other related task can be obtained from the InriaGforge at https://gforge.inria.fr/. The most important thing is to upload your public SSH key on the gForge server (see the FAQ at http://siteadmin.gforge.inria.fr/FAQ.html#Q6 for instructions).

The commands for getting the source code are given in the following sections.

2.2 Required development tools

The following development tools are required to compile MPICH-Madeleine:

GNU C Compiler gcc (version 3.2 and higher).
GNU make (version 3.81 and higher).
GNU Bison (http://www.gnu.org/software/bison/) and Flex (http://www.gnu.org/software/flex/).

2.3 Prerequisites

If you plan to use Myrinet and SCI, you must get the appropriate drivers. Madeleine III supports Myrinet networks through the GM and MX drivers and SCI networks through the SISCI driver. The default installation directories for these drivers are:

/opt/gm for Myrinet/GM;
/opt/mx for Myrinet/MX;
/opt/DIS for SCI/SISCI.

In case, your installation differs from these directories, you will need to set the following environment variables to point to your own installation directories:

% export MX_DIR=/softs/mx
% export GM_DIR=/softs/gm
% export SISCI_PATH=/opt/DIS

The second step is then to create symbolic links from the sub-directory .mpich-mad in your home directory pointing to the drivers's distributions:

A link to the GM distribution (including header files, libraries, ...) called gm, such as:
```
% mkdir $HOME/.mpich-mad
% cd $HOME/.mpich-mad
% ln -s /opt/gm gm
```
AND/OR a link to the MX distribution (including header files, libraries, ...) called mx, such as:
```
% mkdir $HOME/.mpich-mad
% cd $HOME/.mpich-mad
% ln -s /opt/mx mx
```

AND/OR a link to the SISCI distribution called DIS, such as:

% mkdir $HOME/.mpich-mad
% cd $HOME/.mpich-mad
% ln -s /opt/DIS DIS

The name of these links are important, so please respect them.

2.4 Installing PM2

Here are the required steps to install PM2.

Choose an installation directory for PM2, and set the environment variable PM2_ROOT to this directory.
```
% export PM2_ROOT=$HOME/soft/pm2
```
Go in the parent directory of the directory PM2_ROOT and either
1. unpack the PM2 archive;
```
% cd $HOME/soft
% tar zxf pm2-2005-03-16.tar.gz
```
  If your tar does not accept the option z, use:
```
% gunzip -c pm2-2005-03-16.tar.gz | tar xf -
```
2. or check out the latest version from the Subversion server
  - using the anonymous access.
```
% cd $HOME/soft
% svn checkout svn://scm.gforge.inria.fr/svn/pm2/trunk pm2
```
  - or using your gForge account.
```
% cd $HOME/soft
% svn checkout svn+ssh://&lt;login>@scm.gforge.inria.fr/svn/pm2/trunk pm2
```
In both cases, you should end up with a directory named pm2 as indicated by the environment variable PM2_ROOT.
You need to add to your PATH the directory pm2/bin.
```
% export PATH=${PM2_ROOT}/bin:${PATH}
```
PM2 is installed.

2.5 Installing MPICH-Madeleine

Here are the required steps to install MPICH-Madeleine.

Go in some directory of your choice, and either

unpack the MPICH-Madeleine archive;

% cd $HOME/soft
% tar zxf mpich-mad-2005-03-16.tar.gz

If your tar does not accept the option z, use:

% gunzip -c mpich-mad-2005-03-16.tar.gz | tar xf -

or check out the latest version from the Subversion server

using the anonymous access.

% cd $HOME/soft
% svn checkout svn://scm.gforge.inria.fr/svn/mpich-mad/trunk mpich-mad

or using your gForge account.

% cd $HOME/soft
% svn checkout
    svn+ssh://&lt;login>@scm.gforge.inria.fr/svn/mpich-mad/trunk
    mpich-mad

In both cases, you should end up with a directory named mpich-mad.

Choose an installation directory for MPICH-Madeleine, and set the environment variable MPICH_MAD_ROOT to this directory.
```
% export MPICH_MAD_ROOT=$HOME/soft/mpich-mad-install
```
Configure MPICH-Madeleine by issuing the following commands:
```
% cd mpich-mad
% ./configure --prefix=$MPICH_MAD_ROOT --with-arch=LINUX --with-device=ch_mad:--pm2root="$PM2_ROOT"
            
```
- PM2 is using a system of flavors to configure its different options. By default, MPICH-Madeleine is compiled using the flavor mpi-flav. If you wish to use another flavor, you can specify it as follows:
```
--with-device=ch_mad:--pm2root="$PM2_ROOT":--flavor="name of your flavor"
```
- If you want to enable SHMEM support, add the option --shmem-support.
```
--with-device=ch_mad:--pm2root="$PM2_ROOT":--shmem-support
```
  SHMEM support is dynamically deployed on SMP nodes on which several MPI processes are spawned. Therefore, you can compile MPICH-Madeleine by enabling SHMEM support even if you have a set-up with both SMP and UP nodes. It will not impact the performance.
- If you want to enable fast buffering capabilities, add the option --fast-buffer.
```
--with-device=ch_mad:--pm2root="$PM2_ROOT":--fast-buffer
```
- The default behavior is to compile all networks support (TCP, SISCI, GM; MX if available). If you DO NOT want to enable one specific network, add the following option. The network name can be any among tcp, gm, bip, mx, sisci, ....
```
--with-device=ch_mad::--pm2root="$PM2_ROOT":--mad3=no-"network name"
```
Build MPICH-Madeleine. That step might take some time.
```
% make
```
Install the MPICH-Madeleine commands.
```
% make install
```
You need to provide a network configuration file called localnet.cfg (see Section 1.1). This file should be located in the sub-directory etc of the directory you have installed MPICH-Madeleine in i.e. $MPICH_MAD_ROOT/etc. This file is mandatory but users can select another location by defining the appropriate environment variable (see Section 3.1).
Note: do NOT get rid off the PM2 source after the installation. It is still needed by users to compile and run their applications!
Users can add to their PATH the element mpich-mad-install/bin in order to easily access the commands mpicc and mpirun.
```
% export PATH=${MPICH_MAD_ROOT}/bin:${PATH}
```

2.6 Summary of Environment Variables

Here the list of environment variables you need to set (preferably in your login files such as $HOME/.bashrc) before being able to compile and execute MPICH-Madeleine applications.

## Directory where PM2 is installed
export PM2_ROOT=$HOME/soft/pm2

## Directory where MPICH-Madeleine is installed
export MPICH_MAD_ROOT=$HOME/soft/mpich-mad-install

## Access to the PM2 commands
export PATH=${PM2_ROOT}/bin:${PATH}

## Access to the MPI commands
export PATH=${MPICH_MAD_ROOT}/bin:${PATH}

2.7 Automatic Installation of MPICH-Madeleine

The tool set presented in Section 3.4.1 provides a script that allows to get the source code for PM2 and MPICH-Madeleine (or update the existing code), and to compile that code. The script has to be called as in the following example:

% installMpichMad.sh $HOME/soft $HOME/soft/mpich-mad-install

where:

$HOME/soft is the parent directory in which the source files for the softwares PM2 and MPICH-Madeleine are (or will be) installed;
$HOME/soft/mpich-mad-install is the directory in which MPICH-Madeleine is (or will be) installed.

The script will execute the following steps:

If the directory $HOME/soft/pm2 does not exist, get the latest version from the Subversion server; otherwise update the existing version.
If the directory $HOME/soft/mpich-mad does not exist, get the latest version from the Subversion server; otherwise update the existing version.
Compile MPICH-Madeleine in the directory $HOME/soft/mpich-mad-install.

To maximize the automation of the installation, the following script can also be used to automatically copy the archive of the tool set and launch the installation script of MPICH-Madeleine on a given machine.

#!/bin/bash

if [ "$4" == "" ]
then
  echo "Error. Usage: $0 &lt;mpich-mad-tools.tar.gz>
          &lt;machine to install MPICH-Madeleine on>
          &lt;PM2 & MPICH-Madeleine source directory> 
          &lt;MPICH-Madeleine installation directory> [-user &lt;inria gforge login>]"
  exit 1;
fi

DIR=`basename $1 .tar.gz`

scp $1 $2:
ssh $2 "tar zxvf $1 ; ./$DIR/installMpichMad.sh $3 $4 $5 $6 ; uname -a"

Here an example on how to call the script:

%  ./installMpichMadAndTools.sh g5k-tools.tar.gz oar.sophia.grid5000.fr ~/soft ~/soft/mpich-mad-install/

2.8 Troubleshooting

You might encounter the following errors while compiling MPICH-Madeleine.

Bison version error:

 Generating Makefile leoparse-config.mak
 building leoparse_parser.h
 source/tools/leoparse_parser.y:25: unrecognized: %error-verbose

 source/tools/leoparse_parser.y:25:    Skipping to next %
 make[1]: ***
 [/home/test/MPI/MPICH_MAD/MPICH-MAD-INST/pm2/leonie/leoparse/include/leoparse_parser.h] Error 1
 make: *** [dot_h] Error 2

This error happens because of an outdated version of the Bison software. To fix it, you can either:

Install a newest version of the Bison software ; or
Comment the %error-verbose line in the leoparse/source/tools/leoparse_parser.y file of the PM2 software package.

Another Bison error:
```
 yylloc undefined
```
This error can be resolved by installing a newest version of the Bison software or by commenting the extern keyword preceding YYLTYPE yylloc in the leoparse/source/tools/leoparse_lexer.l file of the PM2 software package.

MPICH-Madeleine Installer's, User's and Developer's Guide

Runtime

High Performance Runtime Systems for Parallel Architectures

2. Installing MPICH-Madeleine

2.1 Getting the Source Code

2.1.1 Getting the Latest Stable Release

2.1.2 Getting the Latest Version

2.2 Required development tools

2.3 Prerequisites

2.4 Installing PM2

2.5 Installing MPICH-Madeleine

2.6 Summary of Environment Variables

2.7 Automatic Installation of MPICH-Madeleine

2.8 Troubleshooting

Table of contents

1 What is MPICH-Madeleine?

2 Installing MPICH-Madeleine

3 Running MPICH-Madeleine Programs

4 Debugging MPICH-Madeleine Programs

5 Developping MPICH-Madeleine

References

A Testing the Installation of PM2