Installing Software on an HPC Cluster

Installing Software on an HPC Cluster

How to get software running on a shared cluster without root: modules, Conda, Spack and EasyBuild, source builds, and Apptainer containers, and which one to reach for first. Video 8 of 13 in the series Research Computing on Explorer and AICR, built from Northeastern Research Computing's training material. Liam, in for Bear, narrates.

Chapters 0:00 Opening 0:14 The idea 0:43 Terms in this lecture 1:10 I · Why HPC Is Different 2:03 II · Environment Modules 2:51 III · Conda 4:22 IV · Spack 5:24 V · EasyBuild 6:56 VI · From Source 8:58 VII · Apptainer 10:29 VIII · Choosing The Tool 11:33 Recap 12:04 Your turn

Your turn. Paste this into Claude: I need to run [name the software] on a shared HPC cluster where I have no root access and a tight home quota. Walk me down the decision order: is it likely a module, a Conda package, a container image, a source build, or a Spack/EasyBuild job? Give me the exact commands for the first option that fits, where each file should live, and what I must not run on the login node. Then check two things yourself: run module avail [name] before believing any install is needed. every path it gave you is in your own space, never /usr/local.

Sources "Installing Software on an HPC Cluster", Northeastern Research Computing, presenter Ghanghoon "Will" Paik; github.com/northeastern-rc-training/Intro-to-Software-Installation-2026. Read 2026-10-06. Questions about the clusters: rchelp@northeastern.edu.

Made with the brutalist.art toolkit: Manim, Remotion and Kokoro, free and local. Narration is synthetic (Liam, in for Bear). This is an independent educational film, not an official Northeastern publication.

▶ The full series playlist — Brutalist https://www.youtube.com/playlist?list=PLG9H-C6rp5RU

What this video covers

• 0:00 — Opening
• 0:14 — The idea
• 0:43 — Terms in this lecture
• 1:10 — I · Why HPC Is Different
• 2:03 — II · Environment Modules
• 2:51 — III · Conda
• 4:22 — IV · Spack
• 5:24 — V · EasyBuild
• 6:56 — VI · From Source
• 8:58 — VII · Apptainer
• 10:29 — VIII · Choosing The Tool
• 11:33 — Recap
• 12:04 — Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

Opening

[0:00] Research Computing on Explorer and AICR. Installing software on an HPC cluster. How to get software running on a shared cluster without root modules. Spack and easy build source builds and apt containers. And which one to reach for

The idea

[0:14] First? Jambo. This is Liam in forbear. First a correction. On a shared cluster, you do not install software the way you do on a laptop. There's no pseudo, no root, no system package manager you are allowed to touch. Everything you install goes into your own space built against the cluster's own libraries. This lecture walks the main ways to get research software running on a shared cluster. Environment modules condu compiling from source with your own module files and appainer containers and then which one to reach for first. Six

Terms in this lecture

[0:43] Terms root the administrator account that can write to system directories you do not have it. A module a package and administrator already built loaded by name and version. User space your home scratch or project directories. The only places you can install a build from source compiling a program's code yourself with your own options. A module file, the small text file that tells the module command what to change. And a container, an application packed with its whole operating system into one image file. Why it is different? Your

I · Why HPC Is Different

[1:12] Laptop root access, one user, a local disk with no quota, and generic downloads that just work. the cluster user space only hundreds of people sharing the login node a network file system with tight quotas and builds that should link against the cluster's own compilers MPI and math libraries generic binary from the internet ignores all of that so it runs slower or will not launch across nodes two habits follow first do not build on the login node everyone who is connected shares it a long compile or a memoryhungry install there slows the system for all of them heavy builds go into an interactive job or a batch job second mine the file system your home directory is a network file system NFS, GPFS or Luster shared across every node. Quotas are tight and it is tuned for large reads, not the tens of thousands of tiny files aonda environment or a build tree creates. That is a real reason job launches are slow. Before installing anything, check

II · Environment Modules

[2:04] Whether it already exists. Most clusters expose pre-installed software through environment modules. This cluster uses the classic TCLbased environment modules, not LMOD, so a few commands differ from other tutorials. Module avails what is there. Avail with a name filters it. Show prints what a module would change. Load adds it. List shows what is loaded. Unload removes one. Purge clears everything. Names and versions are cluster specific. So always check avail first. What module load actually does. It does not install anything. It edits your shell's environment, the path variable and its library cousin. So the shell can find a package an administrator built once and shares with everyone. Load a module and path gains the package directory in front of user bin. unload it and the directory is gone. That is how you switch between versions cleanly with no conflicts. And the fastest install is the one you do not have to do. Is where

III · Conda

[2:52] Most researchers start, especially in Python and R. It installs into your own directory, handles non-Python libraries, too, and needs no root. On most clusters, you load a module rather than installing condi yourself. Create an environment with a Python version and packages. Activate it with source activate and check channels. prefer cond forge which has far broader and more current coverage and remove the defaults channel. Anaconda's terms restrict large organizations use of its default channels and many universities now avoid them. Now the pitfalls the part people skip and then file a ticket about. Solving an environment is heavy so do it in an interactive job not on the login node. Watch your home quota. An environment is tens of thousands of small files, but do not put environments on Scratch because Scratch is

Purged and the environment vanishes sometimes mid- project. Keep environments in persistent space, home or a project directory. The one piece you can safely send to Scratch is the package cache, which is regenerable, or just empty it now and then. In your rc, redirect only the package cache to scratch. If home is tight, create environments at a path in a project directory with -p and activate inside the job script. Never in dot bash rc. condesinit command writes a block into dot bash rc that autoactivates on every login. It slows every login and can break batch jobs. Comment that block out. The job script does the activating. Load the condom module source activate run and for reproducibility export the environment to a yiml file exact with builds and recreate it later from that

IV · Spack

[4:22] File. Spack is a package manager built for HPC. Unlike, it builds from source by default, and it lets you choose the compiler, the MPI implementation, and the build options, holding many versions side by side. One honest note first, Spack and Easy Build or heavy long compiles, big disc footprints, real upkeep. They are not the first thing to reach for. If a module or cond's core idea is the spec, package at version percent compiler plus variant HDF5 at 1.14.3% GCC 12.2 2 plus MPI spack resolves the whole dependency tree and builds each piece with exactly those choices. The version numbers here are illustrative. The workflow clone spack source it setup script to load it into your shell list and info a package to see versions and variants. Install it with the variant you want and spack load it. A clean install with many dependencies compiles for a long time. So run spack install inside a job with several cores. A spack environment group specs for one project. create it, activate it, add specs, install everything, and commit the spec.yamel it writes. Easy Build solves

V · EasyBuild

[5:25] A similar problem with a different philosophy. It builds from recipe files called easyconfigs, and you almost never write one. Thousands ship with it, so you install a package by name, and the recipe is fixed, which is the reproducibility selling point. A tool chain is a named bundle of compiler, MPI, and math libraries. GMP is GCC plus OpenMPI. There is no easy build module here. So you install it once into a cond environment. Then search for a recipe and run EB with three flags that are easy to miss. Those three flags modules tool equals environment modules because easy build assumes LOD by default and this cluster does not run it. Module syntax equals TCL because Lua module files will not load here. And install path in your own space because you cannot write to system directories. Put them

In a config file once. Then the key detail easy builds output is a module file. Point module at its directory with module use and load the result with module load like any pre-installed package. Spec versus easy build in four rows. How you install a spec on the command line versus a named recipe. Dependencies spec resolves a flexible graph. Easy build follows a fixed tool chain. Output spec load or a module versus always a module file. Strength flexible variance and versions versus strict repeatable tool chain based builds. And the candid note from experience, neither is the recommended first option on this cluster. Both are heavy to set up and painful to maintain and spack in particular can be frustrating when builds break. Reach for them only when you need source level control and have decided the upkeep is

VI · From Source

[6:56] Worth it. Sometimes nothing per built exists or you need a patch a specific version or a flag no package manager exposes. Then you compile from source. The project's readme or install file names its build system. The two you will meet most are autoto tools a configure script and CMake. Autotools configure with a prefix make with several cores make install CMake load the CMake module configure with an install prefix build install. Always read the project's own instructions first. What makes it an HPC build? Always set a prefix you cannot install to the systems user local directory. Load the clusters compiler and libraries as modules before you configure. So the build links against the optimized versions. That is the same reason spack and easy

Build produce fast binaries. Point parallel or numerical code at the clusters MPI and blas and lapac not a slow generic fallback and run the make step inside a job because - j8 is a multi-core workload. Five steps load modules configure with a prefix compile inside a job. Install into the prefix expose it on path or with a module file. After installing to a prefix the shell still does not know the software exists. export path and LD library path by hand and it works for this session. For something you reuse, write your own module file instead. Make a directory, tell module to look there with module use in your bash rc so it persists and write the file. This clusters module files are

TCL. The first line hash percent module must be exactly that. The magic cookie that marks the file as a module file. Two prepend path lines, one for path and one for the library path and a oneline description. The directory name becomes the module name and the file name becomes the version. Now module aail shows your build. Module load sets the paths for you and unload reverses everything cleanly. One module file becomes a personal stack. Each tool its own subdirectory, each version of file inside it. So my tool 1.0 and 2.0 sit side by side and you switch with one load exactly like the cluster's compilers. A module file can load other modules such as the write GCC and aversion file names. Default Appainer

VII · Apptainer

[8:59] Formerly Singularity brings containers to HPC. A container packs an application with its entire operating system environment into a single image file ASIF. So it runs the same way everywhere. Why not Docker? Docker normally needs root to run a security problem on a shared system. Appainer runs a container as you with no elevated privileges. Works with theuler and the shared file system. Runs GPU and MPI workloads and can pull and convert Docker images. So you are not locked out of that ecosystem. Appainer is installed systemwide. No module needed. Pull an existing image including from Docker registries into ASIF file. Exec runs one command inside it. Shell opens an interactive shell inside it. The SIF is one file. Copy it, archive it, share it, and it reproduces the same environment.

To build your own, you describe the image in a definition file, bootstrap from a Docker image, Ubuntu 204, a post section that installs Python and Numpy, and a run script that says what runs by default. Build images on your own machine, not on the cluster. Building a SIF from scratch generally needs root. So build where you have it, your laptop or workstation, then copy the finished image up with SCP and exec it on the cluster. The privileged step stays on a machine you control. Only the unprivileged run happens here. Containers shine when software is hard to install natively when a research group already publishes an image or when you want one portable artifact to move between clusters or site. The cost is image size and the upfront build.

VIII · Choosing The Tool

[10:30] Choosing the tool six options and none needs root environment modules pre-installed best for what the cluster already provides limited to what admins installed pre-built binaries best for Python and R and quick setups heavy on quota spack builds from source source level control and many variants heavy and slow to maintain easy build from source reproducible tool chain builds mostly an admin tool tool chain bound a source install full control patches and niche tools the most manual and the easiest to get wrong appainer an image for hard dependencies and portability. Built locally, then cop it up. Now, the practical order. One, check for module first. The best install is no install. Two, a Python or R package with a quota and login node precautions. Three, isolated portable or a nightmare of dependencies. Build a container on your own machine and run it here. Four, a patch, a specific version or a build flag nothing else exposes. Build from source. Five, source level control over compiler, MPI, or variants and willing to maintain it. then and mostly then spack or easy build. And if none of that fits, ask the research computing team.

Recap

[11:33] Let's recap with claude. No route in a shared file system. So every install goes in your own space off the login node. Mindful of quota. Check for module first. Module load only edits your environment. KN for Python and Ronda forge environments in persistent space. Activate in the job script. Spack and easy build from source and are the heavy advanced option not the default from source a prefix. the clusters, modules, and libraries. Compile inside a job in your own TCL module file. Appainer. Build the SIF locally. Copy it up. Run

Your turn

[12:04] It as you. Your turn. Paste this into Claude. I need to runame the software on a shared HPC cluster where I have no root access in a tight home quota. Walk me down the decision order. Is it likely a module, a cond package, a container image, a source build, or a spack/ easybuild job? Give me the exact commands for the first option that fits where each file should live and what I must not run on the login node. Then check two things yourself. Run module avail for the name before you believe any install is needed and confirm every path it gave you is in your own space, never the systems user local directory. Installing software on an HPC cluster at Nick Bear Brown.