Intro to Parallel Computing

Intro to Parallel Computing

Threads, processes and scaling across nodes: how work gets split up, and when that actually makes it faster. Video 5 of 13 in the series Research Computing on Explorer and AICR, built from Northeastern Research Computing's training material. Liam, in for Bear, narrates.

Chapters 0:00 Opening 0:10 The idea 0:38 Terms in this lecture 1:01 I · Why Parallel 1:27 II · Flynn's Four Kinds 1:51 III · Shared Or Distributed 2:38 IV · The Speed Limit 3:27 V · The Game Of Life 5:07 VI · GPU Computing 6:00 VII · Scaling PyTorch 7:19 VIII · Python, R, MATLAB 8:20 IX · Takeaways 8:52 Recap 9:30 Your turn

Your turn. Paste this into Claude: Here is a serial program I want to run faster on a cluster: [describe it and paste the main loop]. First, mark which parts are inherently serial and estimate my Amdahl ceiling. Then tell me which level fits: threads on one node, MPI across nodes, or a GPU, and why. Finally, write the Slurm header for that choice with --ntasks and --cpus-per-task explained. Then check two things yourself: did it name the serial part before it promised a speedup? do ntasks × cpus-per-task equal the cores you'd get?

Sources

• codes/, "Introduction to Parallel Computing", Northeastern Research Computing, presenter Ghanghoon "Will" Paik, HPC Performance Engineer; github.com/northeastern-rc-training/Intro-to-HPC-Scaling-Parallel-Computing-2026. Read 2026-10-06. Questions about the clusters: rchelp@northeastern.edu.

Made with the brutalist.art toolkit: Manim, Remotion and Kokoro, free and local. Narration is synthetic (Liam, in for Bear). This is an independent educational film, not an official Northeastern publication.

▶ The full series playlist — Brutalist https://www.youtube.com/playlist?list=PLG9H-C6rp5RU

What this video covers

• 0:00 — Opening
• 0:10 — The idea
• 0:38 — Terms in this lecture
• 1:01 — I · Why Parallel
• 1:27 — II · Flynn's Four Kinds
• 1:51 — III · Shared Or Distributed
• 2:38 — IV · The Speed Limit
• 3:27 — V · The Game Of Life
• 5:07 — VI · GPU Computing
• 6:00 — VII · Scaling PyTorch
• 7:19 — VIII · Python, R, MATLAB
• 8:20 — IX · Takeaways
• 8:52 — Recap
• 9:30 — Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

Opening

[0:00] Research Computing on Explorer and AICR. Intro to parallel computing. Threads, processes, and scaling across nodes. How work gets split up and when that actually makes it faster. Namast this is

The idea

[0:11] Liam in forbear. First a correction. Twice the processors is not twice the speed. The serial part of your code and the cost of processors talking to each other set the limit before the hardware does. This lecture walks from one core to many GPUs across nodes. Why we parallelize the four kinds of parallel machine shared against distributed memory, Amdoll's and Gustafson's laws, a real benchmark of Conway's game of life. How GPUs and PyTorch scale and the parallel tools in Python, R and Mat Lab.

Terms in this lecture

[0:39] Six terms. A core, one processing unit that runs instructions on its own. A node, one computer in the cluster with its own memory. A thread, a stream of work inside one process sharing that process's memory. A process, a running program with its own memory. MPI the message passing interface the standard way processes on different nodes send data to each other and speed up serial time divided by parallel time start with

I · Why Parallel

[1:02] Y simulations data processing model training sooner or later one core isn't fast enough so we split the work think of laundry serial wash dry fold one load then the next weights nine steps for three loads pipelining while load one dries load two washes five steps data parallelism three washers three dryers three loads at once three steps most of Today is data parallelism. Cut a big problem into small ones and solve them simultaneously.

II · Flynn's Four Kinds

[1:28] Flynn's taxonomy sorts machines by how instructions and data flow. Single or multiple instruction, single or multiple data. Cyst is an ordinary single core. SIMD is one instruction over many data points at once. That is a GPU. Mist is rare fault tolerant hardware. And MIMD is many instructions over many data. A multi-node cluster each task on its own data. Today's two are SIMD the GPU and MIMD the cluster. How processors reach

III · Shared Or Distributed

[1:52] Memory decides which tool you use. Shared memory. The cores of one node read one pool of ram and a core any address. Easier to program. The tool is open MP. The limit is one machine. Distributed memory. Each node has its own RAM and not can't read node B's. Programs send data over the network. The tool is MPI. The gain is that you scale across nodes. Most real HPC codes are hybrid. MPI between nodes open MP inside each. This maps straight onto slurm. N tasks is the number of processes the MPI ranks. CPUs per task is the cores each process gets for its threads. They multiply four tasks per node. Five CPUs per task is 20 cores per node. Pure open MP is one task with 20 CPUs. Pure MPI is 20 tasks with one CPU each. Hybrid sits

IV · The Speed Limit

[2:38] Between. Now the speed limit not everything parallelizes. Reading data writing results steps that depend on the last. Mdall's law speed up equals 1 over s plus p overn where s is the serial fraction, p the parallel fraction, and n the number of processors. If 10% of your code is serial, the most you can ever get is 10 times however many processors you add, the curve climbs, then flattens. This is strong scaling. The same problem more cores. Gustafson asked the other question. What if the problem grows with the cores? A researcher with 100 cores doesn't finish a 10-minute problem in 6 seconds. They solve a problem 100 times larger in the same 10 minutes. Speed up equals s plus p * n. That is weak scaling and it's how most real HPC work gets done. Both laws are right. They answer different questions. One tip before any code, plan what can be parallel. Retrofitting parallelism onto finished serial code is painful.

V · The Game Of Life

[3:27] Now a real case, Conway's game of life, a grid where each cell's next state depends on its eight neighbors. It is embarrassingly parallel because within one time step, every cell is independent of every other. A single core walks the grid row by row. Many cores can take the rows at once. With open MP, one directive above the loop asks the compiler to spread it across the cores of a node. The pragma collapses the two loops into one parallel space. The compiler creates and distributes the threads, submitting it one node, one task, 20 CPUs per task, and the thread count set from slurm's variable 10,000x 10,000 cells, 500 generations. When the grid outgrows one node's memory, split it across nodes, domain decomposition. The grid is cut into horizontal strips, one per MPI rank. But a cell on the edge of a strip needs neighbors

From the strip next door on another node. So before each step, the rank swap their boundary rows with MPI send and receive. Those borrowed rows are the halo or ghost zone. The hybrid job four nodes, four ranks per node, five threads per rank. Four* 4* 5 is 80 cores launched with MPI run in the task count from slurm. The real numbers, same grid, 500 iterations. Serial 250 seconds open MP on two cores 2.2 times faster four cores 4.3 8 8 1/2 only 7 and 1/2 slower than 8 16 10 20 12.4 4 then across nodes with MPI 40 cores 25 times 60 37 80 44 inside one node the scaling is close to ideal with no network in the way across nodes the communication cost shows and the curve bends under the ideal line exactly as AMDall predicted GPU a CPU is built

VI · GPU Computing

[5:09] For low latency tens of complex cores good at sequential logic and branching a GPU is built for throughput thousands of simple cores doing the same operation across a large array A few professional chefs cooking complicated dishes against a thousand line cooks each doing one simple task for a buffet. Every GPU program follows four steps. Allocate memory on the device. Copy the data from host to device. CPU to GPU. Launch a kernel. Thousands of threads run the same function on different data. Then copy the results back. Host is the CPU. Device is the GPU. A matrix multiply kernel. Each thread works out its own row and column from its block and thread indices and computes exactly one element of the result. For a thousand by,000 matrix that is over a million threads in parallel. You may never write CUDA yourself. PyTorch, coup, and rapids do it for you, but this is what they do. In

VII · Scaling PyTorch

[6:01] PyTorch, the pattern is put the tensors on the device and the library calls the CUDA kernels for you. Two 4,000 square matrices and one map submitted with one GPU on the GPU partition. More than one GPU on a node split one big operation across them. Partition by rows. Give every GPU its shard and a full copy of B. Launch the map moles. They run in parallel because kernel launches are a synchronous. Then synchronize each device submitted with two GPUs on the multi-poo partition cross nodes. PyTorch uses distributed data parallel. The vocabulary the back end is NCCL NVIDIA communication library and PI's cousin for GPUs. World size is the total number of

GPUs. rank is each GPU's ID. Local rank is its index inside its node. The communication test, every GPU holds a one, all reduce sums them. And if the result equals the world's size, every GPU is talking. The GPU benchmark of 4,000 square mm mole 100 times. One GPU 11.7 seconds. Two GPUs on one node 5 1/2 seconds. 2.1 times near linear over the fast in node link. Four GPUs across two nodes 5.1 seconds only 2.3 times. At this size, each GPU gets a tiny shard and the cross node sync costs more than the maths. The computation to communication ratio decides how far you can scale. You don't need C or CUDA to

VIII · Python, R, MATLAB

[7:21] Go parallel. In Python, MPI4 PI gives you MPI, a communicator, your rank and the size of the world. Each rank working on its own chunk and a reduced to rank zero. Launch it with MPI run. Also worth knowing, multiprocessing for one node, concurrent futures, dask and ray. In R, the parallel package ships with base R, make a cluster of all cores but one par L apply over your list. Stop the cluster or for each with do parallel for embarrassingly parallel loops. Register four cores and swap do for DUP par. In mat lab, parpole opens a pool of workers and parfer replaces a for loop. Mat Lab distributes the iterations. Choosing the tool open MP shared memory loop level parallelism on one node. MPI distributed multi-node simulations. MPI4 PI for multi-node Python. Multipprocessing for single node Python tasks. PyTorch DDP for training. Dask array for big data and pipelines both memory models. RS parallel and 4ch and mat lab parfer for embarrassingly parallel loops on one

IX · Takeaways

[8:20] Node. Four things to keep profile first. Find the bottleneck before you parallelize. Choose the level. Shared memory on one node. MPI across nodes. GPUs for data parallel maths. Communication costs are real. More nodes are not always faster. and plan for parallelism from the start questions RC help at northeastern.edu edu or office hours. And the three slurm recipes to keep open MP one node one task 20 CPUs MPI plus open MP four nodes four ranks each five threads each GPU one node two GPUs let's recap with claude pipelining

Recap

[8:54] And data parallelism and today was data parallelism sim is the GPU MIMD is the cluster shared [snorts] memory means open MP distributed means MPI amdall 10% serial caps you 10 times Gustafson grow the problem with the cores the game of life ran 12 times faster on 20 cores and 44 times on 80 across nodes with communication bending the curve. A GPU program allocates, copies in, launches the kernel, and copies back. Two GPUs gave 2.1 times, four across nodes only 2.3. Profile first. Choose the level, count the communication, and plan for parallelism. Your turn. Paste this into

Your turn

[9:31] Claude. Here's a serial program I want to run faster on a cluster. Describe it and paste the main loop. First mark which parts are inherently serial and estimate my AMD doll ceiling. Then tell me which level fits threads on one node MPI across nodes or GPU and Y. Finally write the slurm header for that choice with n tasks and C plus per task explained. Then check two things yourself. Did it name the serial part before it promised a speed up? And do its n tasks times CPUs per task equal the cores you'd actually get? Intro to parallel computing at Nick Bear Brown.