Skip to main content

more options


SPMD or Manager/Worker

Major Decision #2: SPMD or Manager/Worker?

If one has chosen the data parallel strategy, there is an additional decision to make.

  • Single Program, Multiple Data (SPMD)
    • All processes run the same program, operating on different data.
  • Manager/Worker
    • A single program (called the Manager) coordinates the work done on all the processes.
    • The Manager and Workers may all run the same program, or different programs.
    • "Embarrassingly parallel" is a special case.

Major Decision #2: SPMD or Manager/Worker?

If one has chosen the data parallel strategy, there is an additional decision to make.

  • Single Program, Multiple Data (SPMD)
    • All processes run the same program, operating on different data. This model is particularly appropriate for problems with a regular, predictable communication pattern. These tend to be scalable if all processes read/write to files and if global communication is avoided.
  • Manager/Worker
    • A single program (called the Manager) coordinates the work done on all the processes. These are called Workers. The Manager may or may not contribute to computation. This model has limited scalability due to the communication bottleneck caused by all Workers needing to communicate with a single Manager.
    • The Manager and Workers may all run the same program, or different programs. If they are running the same program, conditional (if) statements cause different tasks to run different code segments.
    • "Embarrassingly parallel" is a special case. If all tasks can be run completely independently from each other—i.e., they differ only in their input, and they are coupled only through their final output—then detailed parallel programming may be unnecessary. The Manager in this case could be a script (at the OS level) that initiates separate instances of a serial program on different processors, then collects and processes the final output.

Additional background material on SPMD and Manager/Worker can be found in the module Distributed Memory Programming (specifically section 6.2) and Performance Basics (specifically section 5.1).