Optimizing GPU Usage on AICR

Optimizing GPU Usage on AICR

How to request the right GPU, measure what your job actually uses, and right-size it, so your work runs sooner and wastes nothing. Video 2 of 13 in the series Research Computing on Explorer and AICR, built from Northeastern Research Computing's training material. Liam, in for Bear, narrates.

Chapters 0:00 Opening 0:12 The idea 0:37 Terms in this lecture 1:02 I · Do You Have The GPU? 3:09 II · Seeing The GPU Live 4:22 III · The Profiling Toolchain 6:09 IV · Data Starvation 7:36 V · Memory vs Utilization 10:06 VI · More Than One GPU 11:34 VII · The Loop 13:24 VIII · Your Domain 14:02 Recap 14:32 Your turn

Your turn. Paste this into Claude: Here is my PyTorch training loop and my Slurm script: [paste both]. I run it on one GPU. Walk me through the optimization loop: what to measure for a 50-step baseline, how to profile 20 steps, and which of these to try first and why: DataLoader num_workers and pin_memory, a larger batch, BF16 autocast, gradient checkpointing, or a second GPU. Give me the exact code changes for your first pick only. Then check two things yourself: did it ask you to measure before it changed anything? did it change exactly one thing?

Sources AICR Training Workshop 2, presenter Arsalan Akhter, Research Computing Specialist, Northeastern University; public repo github.com/northeastern-rc-training/AICR-Optimizing-GPU-Usage (scripts/01–07, slurm/, setup_env.sh). Read in full 2026-10-06. Questions about the clusters: rchelp@northeastern.edu.

Made with the brutalist.art toolkit: Manim, Remotion and Kokoro, free and local. Narration is synthetic (Liam, in for Bear). This is an independent educational film, not an official Northeastern publication.

What this video covers

• 0:00 — Opening
• 0:12 — The idea
• 0:37 — Terms in this lecture
• 1:02 — I · Do You Have The GPU?
• 3:09 — II · Seeing The GPU Live
• 4:22 — III · The Profiling Toolchain
• 6:09 — IV · Data Starvation
• 7:36 — V · Memory vs Utilization
• 10:06 — VI · More Than One GPU
• 11:34 — VII · The Loop
• 13:24 — VIII · Your Domain
• 14:02 — Recap
• 14:32 — Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

Opening

[0:00] Research computing on Explorer and AICR. Optimizing GPU usage on AICR. How to request the right GPU. Measure what your job actually uses and right size it so your work runs sooner and wastes

The idea

[0:12] Nothing. Bonjour. This is Liam in forbear. First a correction. A GPU job that finished is not a GPU job that used the GPU. Many jobs leave the card waiting most of the time. This lecture is a set of measuring instruments. How to confirm you have the GPU. How to watch it live. How to profile it. How to fix the usual bottleneck. How to read memory against utilization, when to add GPUs, and the loop that ties it all together. Measure first, change second.

Terms in this lecture

[0:37] Six terms, GPU utilization, the share of the last second in which the GPU was running any work at all. RAM, the GPU's own memory, where your model and your batch live. Data loader, the part of PyTorch that reads and batches your data and hands it to the GPU. Profiler tool that says where the time went. BF16, a 16- bit number format that keeps the range of 32-bit floats. and DDP distributed data parallel one model copy per GPU. Start where the work can

I · Do You Have The GPU?

[1:04] Quietly go wrong. If the CUDA module isn't loaded, PyTorch runs on the CPU and raises no error at all. A typo in the GPU request lands you on the wrong kind of node and a leftover CUDA to visible devices setting from an earlier shell can hide the card you were given. AICR has two GPU families and four GPU partitions. B200 Devel and B200 batch with 192 GB of memory per card. RTX Devel and RTX batch with 96. Devel partitions run four hours and are for interactive work and profiling. Batch partitions run 24 hours and are for production. A minimal job script. The request first one node one task eight CPUs per task one GPU 32 GB of memory 4 hours

And the log files then the work make the logs folder clear the modules load CUDA and MinForge activate the workshop environment and run your training two lines deserve a pause me equals 32 GB is CPU memory not GPU memory the GPU's own memory comes with the GPU and CPUs per task is not a formality your data loading workers run on those cores Eight cores means seven workers and one main process. Never trust that you got what you asked for. The workshops verify script runs in under 5 seconds and checks five things. Is cute reachable? How many GPUs which model how much memory? What Q to visible devices says does Nvidia SMI agree? And a tensor round trip CPU to GPU

And back is available only means PIT torch can see a GPU. The round trip proves it can use it. About that variable, slurm sets Q to visible devices to tell your program which GPUs are yours. Zero means one GPU. Zero comma one means two, not set means you're outside slurmur in an interactive shell, which is normal. Never override it in your script unless you deliberately want a subset. One AICR specific rule, a background process watches GPU utilization, and a job that holds a GPU without running anything is canled. So don't request a GPU and then sit writing code. Run the verify script first. Profile in the develop partition in short sessions and keep the batch partitions for jobs that are already

II · Seeing The GPU Live

[3:09] Tuned. The fastest profiler is already on every node. NVIDIA SMI prints a snapshot. Watch at every second in a second terminal. Two numbers matter. GPU utilization 84% here means the GPU was running kernels for 84% of the last second. Healthy below 50. Something is wrong and memory 24 GB of 192 in use. 12% enormous room for a bigger batch. A quick quiz from the workshop. Utilization 12%, memory 91%, what's wrong? The model is loaded. The card is full and it's barely computing. Nothing on the CPU side is feeding it. That is data starvation. The most common problem of all, and it gets its own act. Hold these two numbers as two axes because people confuse them across utilization up memory. Top left is a loaded model that isn't running, a data pipeline problem. Bottom left is a GPU doing nothing at all. Bottom right is good throughput with memory to spare. Top right is the target. A big model working hard. 80 to 100% utilization. 70 to 90% memory. For a richer live view, NVI top a dashboard that updates continuously per process, GPU use, memory trend, power, and temperature. On a shared node, it answers the question that matters. Is it my Python process using the GPU or someone's leftover? Nvidia

III · The Profiling Toolchain

[4:23] Smi tells you that there's a problem. A profiler tells you where the time goes. There are layers. System level Nvidia SMI and NVI top is the GPU busy framework level the PIT torch profiler which operator or layer eats the GPU timeline and site systems every CPU thread kernel and memory copy in order and kernel level nsight compute is one kernel memory bound or computebound most problems show up on the first two rungs the habit that matters most on AICR profile and devel production in batch in a B200 devel L session, you run the verify script, run your code for 50 to 100 steps, profile, fix the bottleneck, and profile again until utilization is above 70%. Only then does the job go to be 200 batch unattended, and you read job stats when it finishes. The PyTorch profiler needs no extra install. Open a profile that records CPU and CUDA activity, records the tensor shapes, and writes

A trace you can open in Tensorboard. Then the pattern that matters. Profile a short window, never the full run. 20 steps is enough. Step the profiler each iteration and print the table sorted by total Q to time. The top row is where you look first. Reading it in order gaps between kernels on the timeline. The GPU is idle. The CPU is too slow. Large copy entries. Data shuttling between CPU and GPU. CPU total far above Q total. The work isn't really on the GPU. And data loader functions at the top of the CPU table. the pipeline. Again, when the order of events matters, CPU threads, kernels, memory cop is communication between GPUs, use Nsite systems, trace CUDA, NVTX, the OS runtime, CUDN, and CU blas write the report to scratch and copy it to your laptop for the Nsight GUI. Limit the capture to 50 or 100 steps. A full run makes gigabyte files. Now, the number

IV · Data Starvation

[6:10] One bottleneck. Picture a high-speed factory, the B200. It finishes a batch in 8 milliseconds. the supply truck. The CPU data loader takes 50 milliseconds to bring the next one. The factory stops and waits. Nothing is wrong with the factory. The supply chain can't keep up. And a B200 eats batches far faster than the cards you may be used to. So, a loader that barely kept up elsewhere will starve it here. The diagnostic is two timings. First, time 50 batches through the loader alone with no GPU work at all. Then, time 50 forward and backward passes on synthetic data, bypassing the loader entirely. If the loader time is larger than the compute time, you are data bound and no model optimization will help until that's fixed.

The workshop's quiz loader 45 milliseconds, compute 9. The GPU works for 9 and waits for 36. 80% of its time wasted. The fix usually lives in four data loader settings. Workers CPU processes loading in parallel. PIN memory page locked RAM for fast transfer. Persistent workers don't respond them every epoch and peretch factor. How many batches each worker loads ahead. Typical starting values workers CPUs per task minus one never zero. Pin memory always true on a GPU node. Persistent workers true whenever workers are above zero. Per fetch start at two. The single biggest win in the whole workshop is one line. Num workers from zero. The PIT torch default to four or more. That alone often takes a job from 20% utilization to 60 or 80. Two

V · Memory vs Utilization

[7:37] Percentages on the same screen that mean different things. Utilization. the share of the last second the GPU was running. Low means idle from data or synchronization. 100% sustained is fine. Memory, the share of RAM allocated. Low means you could hold a bigger batch. High is only bad near the edge where the job runs out of memory and crashes. So read them as a pair. Low and low, the GPU is doing nothing. Verify Q to check the data. Low utilization, high memory, loaded but not running. Fix the pipeline. High utilization, low memory, healthy with RAM unused. Increase the batch size. High and high. you're in good shape and an out of memory crash mixed precision first batch size moves both axis doubling it roughly doubles throughput as long as the GPU isn't already saturated and the memory is there on a B200 with 192 GB memory sitting at 15 or

20% means room for four to eight times the batch find the ceiling by binary search try a size see if it fits have the gap repeat then run production at 70 to 85% of that ceiling mixed precision is the next lever pietorch defaults to 32-bit floats Both AICRGPU families have tensor cores built for lower precision. The change is one wrapper autocast to be float 16 around the forward pass and the loss. Gradients stay in 32-bit on their own. Why BF16 and not FP16? A 16- bit number splits its bits between exponent which is range and mantisa which is precision. FP32 has eight exponent bits and 23 of mantissa. BF16 keeps the same eight exponent bits and keeps only seven for precision. So gradients don't underflow and you need no loss scaler. FP16 bets the other way. 10 bits of precision but only five of exponent which underflows. FP8 goes

Further still. Four and three every step down trades numerical quality for speed and memory. What BF16 buys on this hardware activation memory roughly have so much bigger batch fits. Matrix operations run two to four times faster on the tensor cores and it is mixed precision which is the safety net. The risky parts stay in 32-bit. Master weights the loss softmax normalization. AutoCast handles that for you. So the memory saving is partial and the speed up is largest for matrix multiplies. If memory is still too tight, gradient checkpointing trades compute for memory. Activations aren't stored in the forward pass. They're recomputed during back propagation. Roughly 60 to 70% less activation memory for about 30% more compute. In hugging face, it's one call. In custom modules, wrap the expensive layers. Use it when the batch is forced below eight or when you're fine-tuning a very large model on one card. Multi-GPU

VI · More Than One GPU

[10:08] Is the first thing people reach for when a job is slow and it's often the wrong move. More GPUs amplify the efficiency you already have. They don't create it. The 80% rule. If a single GPU is already above 80% you're compute bound and adding cards divides the work. If it's below 60, fix the bottleneck first because four starve GPUs waste four times as much and an I/Obound job gets worse. Since the cards now compete for the same files, here's what data parallel training does. Each GPU holds a full copy of the model and takes a different minute batch. After the backward pass, they all reduce. They average their gradients into one and each applies the same update. Inside one node over NVLink, that exchange is fast.

Across nodes, it's slower. So single node multi-GPU is usually much faster per card than spreading out. Before you ask for four GPUs, confirm your code is written for DDP. A plain script with GRE equals GPU 4 gets four cards and uses one. Check that distributed is available and initialized and that PyTorch sees as many devices as you requested. Then the request one task per GPU, four GPUs, eight CPUs for each GPU's workers and run it with torque run and measure the scaling throughput divided by the number of GPUs times the single GPU baseline gives efficiency. Above 80% the extra cards pay 70 to 80 is acceptable overhead. Below 70, communication or I/O is the real bottleneck and more GPUs won't help go back to the single card.

VII · The Loop

[11:35] Every section was one bottleneck. In practice, they come in that order of frequency. So, here's the loop. Baseline 50 steps. Record throughput and utilization. Profile 20 steps. Diagnose data, memory, compute, or communication. Fix one thing, only one, or you can't tell what helped. Rep-profile. Did the bottleneck move? Record the gain. Repeat until utilization is above 80 or the returns run out. The decision guide as a tree. Utilization under 50, it's the data pipeline. And if workers is zero, that first 50 to 80, look at memory. Under 40%, bigger batch. 40 to 85, try BF-16. Over 90, smaller batch or checkpointing. Memory fine profile above 80 with healthy memory. Consider more GPUs only if the single card is truly saturated and out of memory BF-16 than checkpointing. Then a smaller batch in that order. After a batch job finishes, read its report card. Job stats from Princeton Research Computing runs on AICR. Here's a real one from a

55minute fine-tuning job on an RTX card. GPU utilization 80% healthy. GPU memory 17% room for a bigger batch. CPU utilization 13%, eight cores requested, barely one used. And a note, the job needed 11% of the 8 hours it asked for. Request less time and the queue moves faster for everyone. Make this a habit under 60% GPU. Fix it before the next run. The presubmission checklist ideally run in a devel. The verify script passes. Nvidia SMI shows over 70% during a 50-step test. Workers is not zero. Pin memory is on. The batch uses at least 60% of RAM. BF-16 autocast is on. The script names the right partition. CPUs per task is at least four and matches workers plus one. The logs folder exists and you've profiled 20 steps and read the CUD to column. Then cancel the Devel running jobs per user is the limit and an idle one blocks you too. None of this is pi

VIII · Your Domain

[13:26] Torch only. Deep learning every section applies and in hugging face most fixes are one flag. Molecular dynamics verify the GPU type before a long run and multi-GPU scales well within a node. LLM inference RTX cards for smaller models B200s for the 70 billion class. Custom CUDA or Triton kernels everything applies plus N insight compute for the kernel itself. And help is one email away. RC help at northeastern.edu for hugging face transformers. The whole workshop is five lines of training arguments. BF16 on gradient checkpointing if you need it. Eight data loader workers and pinned memory. Let's

Recap

[14:03] Recap with Claude. Verify the GPU before anything else because idle allocations on AICR get cancelled. Watch NVIDIA SMI under 50% utilization means something is wrong. Profile 20 steps in the top row by Q to time is where to look. Low utilization with high memory is data starvation. Fix the number of workers. High utilization with low memory raise the batch size. Out of memory BF16 first. Add GPUs only when one is already above 80%. Measure the scaling and read job stats after every run. Your turn.

Your turn

[14:34] Paste this into Claude. Here's my PyTorch training loop and my slurm script. Paste both. I run it on one GPU. Walk me through the optimization loop. What to measure for a 50step baseline. How to profile 20 steps and which of these to try first and why. Data loader num workers and pin memory. A larger batch. BF16 autocast gradient checkpointing or a second GPU. Give me the exact code changes for your first pick only. Then check two things yourself. Did it ask you to measure before it changed anything? And did it change exactly one thing? Optimizing GPU usage on AICR at Nick Bear Brown.