Introduction to MPI with C (2024)

I recently got to grips with parallel programming in C using MPI - below I share some my insights.

Author

Paul Norvig

Published

January 8, 2024

Introduction

I’ve been coding with MPI in C for a while now. When you’re handling data across multiple processors, getting your head around certain practices can save time and cut down on the headaches. From making sure your environment is set up correctly to correctly handling errors, below are some of tips with actual code.

Getting Started with MPI in C

As someone who has waded through the intimidating waters of parallel programming, I can say getting started with MPI in C is like learning to coordinate a symphony of computers; it’s tricky at first, but immensely powerful once you get the hang of it. MPI stands for Message Passing Interface, and it’s the go-to standard for writing programs that run on multiple nodes in a cluster.

It’s crucial to have the MPI libraries installed before you jump in. On most Unix-like systems, installing MPI is a simple affair with package managers. For instance, with apt on Ubuntu, you’d run something like:

sudo apt install mpich

Or with yum on Fedora:

sudo yum install mpich

Once MPI is installed, you’re ready to compile and run your first simple program. So, let’s start with the quintessential Hello, World in the parallel universe.

In C, your basic structure of an MPI program will always start with initializing MPI and then finalizing it before the program ends. The initialization is done by calling MPI_Init, while MPI_Finalize signifies the end of the MPI environment. Everything else happens in between these two.

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(&argc, &argv);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);

// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);

// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);

// Finalize the MPI environment.
MPI_Finalize();
}

To compile this program, you’d typically invoke the mpicc wrapper which calls the underlying C compiler with the appropriate MPI libraries linked. To run it, we use mpirun or mpiexec, followed by the -np flag that specifies the number of processes to use:

mpicc -o hello_mpi hello_mpi.c
mpirun -np 4 ./hello_mpi

This little block of code does a lot of heavy lifting. It first kicks off the MPI environment, then finds out how many processes are running (world_size), and the unique ID (world_rank) assigned to each process. Every process then prints out a hello message including its rank out of the total number of processes, and finally, the environment is shut down cleanly.

You see, it’s not that arcane. The world_rank is particularly interesting because it’s how you differentiate between processes – akin to having an orchestra where every musician knows their part based on the sheet music in front of them.

When you run the example with mpirun -np 4, you’re simulating a mini orchestra of four computers (or cores). They all run the same code but play their parts independently, based on their assigned rank. Simple, right?

And with that, you’ve laid the groundwork. You’ve now got a basic program that launches multiple processes across a computing cluster. Each process can now go off and execute tasks in parallel. While this introduction covers the initial setup, remember that MPI has a lot more depth and a suite of operations for intricate communication between processes. Keep tinkering, and you’ll find that with MPI, the computation possibilities are vast.

Basic MPI Communication Concepts

MPI, or Message Passing Interface, is, to put it plainly, the Swiss Army knife of distributed computing. As someone who dabbles in parallel programming, I’ve found that grasping the core communication concepts of MPI is pivotal.

At the heart of MPI communication is the notion of point-to-point messaging. Picture this: you have a number of processes running concurrently (imagine a team of diligent ants scurrying about their business). Now, if one ant needs to pass a crumb to another, how does it do it? This is where point-to-point comes in, using two main functions:

MPI_Send(void* data, int count, MPI_Datatype datatype, int destination, int tag, MPI_Comm communicator);

MPI_Recv(void* data, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm communicator, MPI_Status *status);

When I first encountered these, I thought of MPI_Send as handing off the crumb—specifying what to send, how much of it, to whom, and a unique tag to identify the transaction. MPI_Recv on the flip side waits with open mandibles ready to catch that crumb based on similar specifications.

Let’s code a simple message-pass:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
MPI_Init(&argc, &argv);

int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);

int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

const int PING_PONG_LIMIT = 10;
int ping_pong_count = 0;
int partner_rank = (world_rank + 1) % 2;

while (ping_pong_count < PING_PONG_LIMIT) {
if (world_rank == ping_pong_count % 2) {
ping_pong_count++;
MPI_Send(&ping_pong_count, 1, MPI_INT, partner_rank, 0, MPI_COMM_WORLD);
printf("%d sent and incremented ping_pong_count %d to %d\n", world_rank, ping_pong_count, partner_rank);
} else {
MPI_Recv(&ping_pong_count, 1, MPI_INT, partner_rank, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%d received ping_pong_count %d from %d\n", world_rank, ping_pong_count, partner_rank);
}
}

MPI_Finalize();
return 0;
}

One of the first things I discovered is that MPI allows data exchange without a direct connection between processes, thanks to the MPI_Comm that abstracts connection details.

But what if you want to shout out to all your ant buddies at once? Enter collective communication. Functions like MPI_Bcast (broadcast) or MPI_Allreduce (global reduction and distribution) come into play.

Here’s a snippet broadcasting a message to all processes:

int data;
if (world_rank == 0) {
data = 100; // Let's say this is some important value you've computed
MPI_Bcast(&data, 1, MPI_INT, 0, MPI_COMM_WORLD);
} else {
MPI_Bcast(&data, 1, MPI_INT, 0, MPI_COMM_WORLD);
printf("Process %d received data %d from root\n", world_rank, data);
}

Collective communication functions are nifty because they help you manage the complexity that can arise when handling many-to-many communication patterns.

To demystify data types: MPI has its own data types to ensure that data structures are consistently interpreted across different architectures. It’s like agreeing that a crumb is a crumb, whether it’s seen by an ant in Africa or Europe.

Now, if you’re anxious about diving into MPI’s myriad functionalities, remember that learning MPI is like learning any other language. You start small, maybe awkwardly fumbling with “Hello World”, but before long, you’re writing poetry in it.

As for resources, the MPI standard itself is quite comprehensive, and for the code examples, you have places like GitHub where countless repositories provide snippets and full-fledged MPI projects (check out mpitutorial for starters).

Mastering these communications is the linchpin to unlocking the full potential of parallel computing. With a solid understanding of MPI’s messaging essentials, you’re well on your way to orchestrating symphonies of distributed computations.

Writing Your First MPI Program in C

Writing your first MPI program in C might sound daunting, but I promise it’s simpler than it appears. I remember writing mine and the sense of achievement it brought. So let me walk you through the process.

To start, make sure you have the MPI library installed on your system. Now let’s jump into the actual code. You’ll begin by including the necessary MPI header:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(&argc, &argv);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);

// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);

// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);

// Finalize the MPI environment.
MPI_Finalize();
}

The code you see initializes MPI and then prints a message from each process that includes its rank and the total number of processes.

To compile your program, use the MPI compiler wrapper, which on many systems is mpicc:

mpicc -o hello_mpi hello_mpi.c

Running the program requires mpiexec or mpirun, depending on your installation, with the number of processes np:

mpiexec -np 4 ./hello_mpi

This command runs your program across four processes. Simple, right?

But let’s talk about sending and receiving messages, because that’s the heart of MPI. Say you want process 0 to send an integer to process 1:

int number;
if (world_rank == 0) {
number = -1;
MPI_Send(&number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
} else if (world_rank == 1) {
MPI_Recv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Process 1 received number %d from process 0\n", number);
}

Keep in mind that MPI_Send and MPI_Recv are blocking functions, so process 1 will wait until it has received the integer before continuing - that’s key to avoiding concurrency issues.

Remember, this code is only the very basics of MPI in C. Later on, you’d delve into non-blocking communications, collective operations, and more advanced topics.

As for resources, online tutorials are great, but nothing beats the official MPI documentation for accuracy and depth. Also, sniffing around GitHub repositories with MPI projects can give real-life coding insights, which I find incredibly helpful.

Lastly, keep practicing; MPI can be particular and sometimes it’s fiddly. I spent hours debugging my early attempts because I misunderstood how ranks and communicators worked together. But like anything else in programming, persistence and patience pay off. Happy coding with MPI!

Advanced MPI Features and Techniques

When I first started diving into the deeper waters of MPI (Message Passing Interface), it felt like unlocking a new level of parallel programming. Beyond the basics, MPI has capabilities that allow for more efficient and fine-grained control of your parallel applications. Here, I’ll share some advanced features and techniques that will elevate your MPI game.

One of the more powerful features is the use of non-blocking communication. This means you can initiate a send or receive operation and then continue on with other work while the data is being transferred. It’s an excellent tool for overlapping computation with communication, which can significantly speed up your programs.

#include <mpi.h>

int main(int argc, char* argv[]) {
MPI_Init(&argc, &argv);

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

const int TAG = 0;
MPI_Request send_request, recv_request;

if(rank == 0) {
double send_buf = 123.456;
MPI_Isend(&send_buf, 1, MPI_DOUBLE, 1, TAG, MPI_COMM_WORLD, &send_request);
// Other work here while non-blocking send occurs ...
MPI_Wait(&send_request, MPI_STATUS_IGNORE);
} else if(rank == 1) {
double recv_buf;
MPI_Irecv(&recv_buf, 1, MPI_DOUBLE, 0, TAG, MPI_COMM_WORLD, &recv_request);
// Other work here while non-blocking receive occurs ...
MPI_Wait(&recv_request, MPI_STATUS_IGNORE);
}

MPI_Finalize();
return 0;
}

MPI also provides collective operations that are optimized for exchanging data across all processes in a communicator. For example, when I need to gather data from all processes to one process, I use MPI_Gather. But what if each process has different amounts of data to send? That’s where MPI_Gatherv (the “v” stands for “variable”) comes into play.

// Assuming each process has a buffer of doubles with varying length

MPI_Gatherv(
send_buffer, send_count, MPI_DOUBLE,
recv_buffer, recv_counts, displacements, MPI_DOUBLE,
root_process, MPI_COMM_WORLD
);

The recv_counts array specifies how many elements from each process will be received, and displacements array tells MPI where to place these elements in the receiving buffer.

Another helpful technique is derived datatypes. This allows you to define complex data structures that you can send and receive with a single MPI call, without needing to pack or unpack data.

typedef struct {
double x, y, z;
int id;
} Particle;

// Define the MPI datatype for Particle
MPI_Datatype particle_type;
MPI_Type_contiguous(4, MPI_DOUBLE, &particle_type);
MPI_Type_commit(&particle_type);

One more advanced feature is one-sided communication, where operations like MPI_Put and MPI_Get allow direct access to the memory of a remote process. This can lead to designs that resemble shared-memory programming.

double local_buf, remote_buf;

if(rank == target_rank) {
MPI_Win_create(&remote_buf, sizeof(double), sizeof(double), MPI_INFO_NULL, MPI_COMM_WORLD, &win);
} else {
MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
}

MPI_Win_fence(0, win);
if(rank == source_rank) {
MPI_Put(&local_buf, 1, MPI_DOUBLE, target_rank, 0, 1, MPI_DOUBLE, win);
}
MPI_Win_fence(0, win);

Always remember error handling, though. MPI functions return error codes, and you can use MPI_Error_string to get human-readable messages.

int errcode;
if ((errcode = MPI_Init(&argc, &argv)) != MPI_SUCCESS) {
char err_string[MPI_MAX_ERROR_STRING];
int resultlen;
MPI_Error_string(errcode, err_string, &resultlen);
fprintf(stderr, err_string);
exit(1);
}

These features only scratch the surface of what’s possible with MPI. For a full deep-dive, I recommend checking out official MPI documentation and sources like the MPI forum MPI Forum where the MPI standards are. With practice, you’ll be tailoring your MPI programs for optimal performance in no time. Happy coding!

Best Practices for MPI Programming in C

Having mastered the fundamental concepts of MPI in C and possibly churned out your first few applications, let’s consider the best practices that I’ve found invaluable through trial and error. MPI, or the Message Passing Interface, can be elegant but unforgiving with subtle nuances that, if overlooked, can lead to inefficient programs or worse, hard-to-debug errors.

Firstly, initializing and finalizing the MPI environment is crucial. Never forget this; your programs will simply not run correctly without it:

#include <mpi.h>

int main(int argc, char* argv[]) {
MPI_Init(&argc, &argv);

// Your code goes here...

MPI_Finalize();
return 0;
}

Throughout my early days using MPI, I learned the importance of understanding communication protocols. When sending and receiving messages, it’s easy for beginners to trip over blocking and non-blocking calls. Painstakingly debugging deadlocks taught me to religiously pair MPI_Send with MPI_Recv unless I’m explicitly working with a non-blocking send (MPI_Isend) or receive (MPI_Irecv), which require diligent request management and the use of MPI_Wait or MPI_Test to ensure proper completion of communication:

MPI_Request request;
MPI_Status status;
int number;
if (world_rank == 0) {
number = -1;
MPI_Isend(&number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
} else if (world_rank == 1) {
MPI_Irecv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
}

MPI_Wait(&request, &status);

Error handling in MPI can look verbose, but it’s your best friend. I soon realized the default error handler isn’t always helpful. By creating a custom error handler, I could get clearer insights into what went wrong. The MPI documentation and forums were an absolute savior when I was trying to decipher error messages.

Data types and buffering can also trip you up. I stick to sending contiguous blocks of data whenever possible since non-contiguous sends can get complicated. If you must send a complex data structure across processes, you’ll want to inspect MPI_Type_create_struct:

struct my_struct {
int id;
double position[3];
char name[32];
};

// Define MPI struct for my_struct in case it needs to be sent across processes

Always remember: parallel programming is about efficiency. Thus, keeping data movement to a minimum is a principle I preach. Keep computation close to data, and only communicate data when necessary. I even made a checklist to optimize the communication patterns and regularly revisit this to remind myself:

Is my data distributed optimally?
Can any communication be overlapped with computation?
Am I making use of collective operations like MPI_Bcast and MPI_Reduce when possible?

Testing and profiling your MPI applications is another skill you’ll want to pick up soon. Tools like mpitrace and MPIP have saved my code from being a bottleneck-ridden monster:

# Command to compile with tracing
mpicc -o mpi_program mpi_program.c -lmpitrace

Finally, MPI evolves. The community forums, GitHub repositories, and university resources have been invaluable for gleaning best practices and staying updated on advancements. Contributing to discussions on platforms like Hackernews or the MPI-users mailing list has given me not just answers, but a deeper understanding of the underlying mechanisms.

In conclusion, boiling down MPI programming to a single set of practices is challenging, given the vast and varied scenarios MPI encompasses. But the practices shared here are your gateway to not just operational MPI programs, but robust and efficient parallel applications. Now armed with these insights and strategies, you’ll find your journey with MPI in C far more navigable and enjoyable. Happy coding!