Intro to Explorer and OOD

Intro to Explorer and OOD

The web front door to the Explorer cluster: launch apps, move files and start jobs from a browser, with no terminal required. Video 3 of 13 in the series Research Computing on Explorer and AICR, built from Northeastern Research Computing's training material. Liam, in for Bear, narrates.

Chapters 0:00 Opening 0:12 The idea 0:33 Terms in this lecture 0:54 I · What A Cluster Is 1:47 II · Getting An Account 2:07 III · Logging In 2:49 IV · Where Files Live 3:34 V · Moving Data 4:17 VI · Running Jobs 6:44 VII · Checking On A Job 7:25 VIII · Right-Sizing 7:50 IX · Software 8:23 Recap 8:50 Your turn

Your turn. Paste this into Claude: I'm new to a Slurm cluster. Here's what I want to run: [describe your program and data in two sentences]. Tell me which of my files belong in home, scratch and projects; write me an srun command to test it interactively with modest resources; then write the sbatch script for the real run, and tell me which commands I use to check on the job and to read its efficiency afterwards. Then check two things yourself: does every path it wrote start with a slash? did it tell you to test with srun before sbatch?

Sources "Introduction to Explorer and the OOD", Northeastern Research Computing training, presenter Serena Caplins, Associate Bioinformatician; public repo github.com/northeastern-rc-training/Intro-to-Explorer-OOD-2026. Read in full 2026-10-06. Questions about the clusters: rchelp@northeastern.edu.

Made with the brutalist.art toolkit: Manim, Remotion and Kokoro, free and local. Narration is synthetic (Liam, in for Bear). This is an independent educational film, not an official Northeastern publication.

What this video covers

• 0:00 — Opening
• 0:12 — The idea
• 0:33 — Terms in this lecture
• 0:54 — I · What A Cluster Is
• 1:47 — II · Getting An Account
• 2:07 — III · Logging In
• 2:49 — IV · Where Files Live
• 3:34 — V · Moving Data
• 4:17 — VI · Running Jobs
• 6:44 — VII · Checking On A Job
• 7:25 — VIII · Right-Sizing
• 7:50 — IX · Software
• 8:23 — Recap
• 8:50 — Your turn

Transcript

A lightly formatted transcript of the video, with timestamps so you can jump to any section on YouTube.

Opening

[0:00] Research Computing on Explorer and AICR. Intro to Explorer and Ood. The web front door to the Explorer cluster. Launch apps, move files, and start jobs from a browser with no terminal required. Ola,

The idea

[0:13] This is Liam in forbear. First, a correction. Explorer is not a big computer where you run your code. It's a cluster of computers and you ask it for a slice. This lecture is the whole introduction. What a cluster is, how you get an account, how you log in through the terminal or the open on demand website where your files live, how you move data, how you run and check a job, how to right size it, and how software works. Six terms. An HPC cluster, many

Terms in this lecture

[0:35] Computers wired together to work on one problem. A login node, the machine you land on for light work only. A compute node, a machine that does the heavy work handed to you on request. Slurm, the scheduler that takes those requests and decides who runs where. A partition, a named group of nodes with its own limits. and open on demand. UD, a website that gives you the cluster in a browser. Start with the shape of the

I · What A Cluster Is

[0:55] Thing. A cluster is many computers working together on problems too big for one desk. You log into a login node. Light work happens there, editing a file, looking around. When you need real computing power, the work moves to a compute node, and a scheduler called Slurm manages everyone's requests for those nodes. Explorer is Northeastern's cluster run by the research computing team, and it replaced the older Discovery cluster. 45,000 CPU cores, more than 525 GPUs, free of charge to every Northeastern faculty member and student. It lives at the Massachusetts Green Higherformance Computing Center in Holly Oak. Around the compute, a 10 GB link to the university network for moving data in and out, six pabytes of fast shared storage, and between the nodes themselves, either 10 GB Ethernet or infinity band at one or 200 Gbits a second. So, every size of workload has a

II · Getting An Account

[1:47] Place. Getting on for research. Every user belongs to a storage space owned by a PI or staff member. Storage is free up to 35 terabytes. If your lab has one, the PI adds you with the project command or you file an access request. If it doesn't, the PI requests a storage space first. For coursework, your instructor applies through the classroom access form and students get in through the

III · Logging In

[2:07] Course. Two ways in the terminal and the open ondemand website. Both reach the same cluster. The team's advice is to get comfortable with the terminal and we'll use both. From a Mac, open terminal and SSH to login.explorer.norththeastern.edu with your northeastern username. It asks for your password. The cursor doesn't move while you type it, which is normal. The welcome banner means you're on. On Windows, install an SSH client such as MOAXM or PUTty and point it at the same host, port 22. Or open a browser at ood.explorer.norththeastern.edu. You sign in with your northeastern username and password. Confirm with Duo and land on the open on demand dashboard. You're on the cluster now

IV · Where Files Live

[2:50] Where things live. Everyone has a home directory and a scratch directory. Researchers also have a project space. Students in a course have a courses space and may not have projects. Always write the full path starting with a slash. So a script means the same place wherever it runs. Each has a size. Home holds 75 GB and it fills fastest. When it does, you'll see quota exceeded. Scratch holds 50 terabytes, but it's purged on the first Tuesday of every month, so it's no place for anything you want to keep. Projects is the long-term home for big data and for folders of many small files. Of course, space holds one to five terabytes shared by the whole class. Paths in practice. LS lists a directory and cat prints a file. With an absolute path, you can read any file you have permission for from anywhere on the cluster. Moving data in from your

V · Moving Data

[3:35] Own machine. SCP cop is a file to the cluster through the transfer node. Make a small test file, copy it into your home directory, then log in and ls to see it arrive to put it in your project space instead. Change the destination path. Many small files zip or tar them first. Or use the website on the open on demand dashboard. Open files then home directory. The files app starts in your home directory and the change directory button takes you to scratch or projects. Upload opens a window where you browse for files or drop them in. The files app is limited to 10 GB of transfer. Bigger than that, use SCP for very large data sets. Globus and the same app downloads files, edits them, and makes new folders. Running jobs. So far, we've

VI · Running Jobs

[4:19] Only used the login nodes and the transfer node. Real computation has to move to a compute node. No scientific software, big transfers, or heavy work on the login node because its traffic slows the whole cluster for everyone. From the terminal srun gets you a compute node. The simplest form takes the defaults. Four hours, one node, one task, 2 GB of memory, two CPUs on the short partition. Flags ask for more here. 6 hours, 50 GB, 16 tasks. Every flag has a short form, too. A partition is how the team groups nodes by time limit, by node count, by CPU or GPU. on explorer short the default and sharing for CPU work GPU GPU interactive GPU short and sharing for GPU work to you apply for long and multi-poo once you can show your code uses them well and two only for courses courses and courses GPU which one if you're in a course

And have a courses directory use courses for CPU jobs and courses GPU for GPU jobs otherwise short for most CPU work GPU interactive or GPU short for interactive GPU sessions and the GPU partition for GPU batch jobs. A GPU is a processor built for thousands of parallel tasks, and it's what AI and much of modern science run on. To get one interactively, name the partition and the GRE. Naming the card type gives you exactly that card. Leaving it general gives you whatever is free first. Fine. For a quick test, but for repeated runs, use the same type or at least record which one you got. On the website, an interactive job is a form. For our studio, you pick the version, the partition, the hours, the CPUs, and the memory, and press launch. The defaults are fine to start. The session moves from cued to starting to running, and your time starts when

It's running. The card shows the host node, a cancel button, the time remaining, and a session ID. Cancel it when you're done. That frees the node for someone else. The jobs that do most of the real work are batch jobs set up, submitted, and run without you. Test your commands in an SRUN session first, then write a script. The sbatch lines name the job, the partition, one node, and 16 tasks, and ask for email on every event. Then load your software, activate your environment, and run. Submit it with sbatch. Slurm answers with a job ID. When the job runs, its output goes to a file named slurm. The job ID.out cat shows what the job printed. Use less to page through long files and cue to quit. If you're unsure how much to ask for, ask for a little and scale up when a job runs out of memory, CPU, or time. Is it

VII · Checking On A Job

[6:45] Running? SQ with your username lists your jobs. ID, partition name, state, R for running, time, and node. SQ for partition, pipe to WC counts how busy the queue is. 518 lines here. And dash- start estimates when awaiting job will begin. On the website, jobs, then active jobs shows the same list. And my interactive sessions shows each session with its host node. And in parenthesis, the seven-digit slurm job ID you give the research computing team if something goes wrong. Click the session ID link and you'll see the files open on demand made for the job. Output.log is the one to read when a session doesn't behave and the one the team will ask for in a ticket. Now the habit that makes you a

VIII · Right-Sizing

[7:27] Power user. Ask for close to the minimum you need because the more you request the longer you wait for it. After a job finishes se slurm efficiency shows what it actually used. Two real reports from the training. Job one 12 cores 8% CPU efficiency 10% of 50 GB used. It could have asked for far less. Job two, eight cores at 84% CPU efficiency. A much better fit. Try SE on your next finished

IX · Software

[7:50] Job. Software comes as modules, so you load only what you need when you need it because some conflict. Module avails them. Module load loads one with the exact name in case you saw. Don't put module load in your bashrc. It interferes with open on demand and makes your work harder to reproduce. Beyond modules, three routes, cond environments which can hold many kinds of scientific software, not only Python, containers which the cluster supports and building from source. And when you're stuck, email rc help at northeastern.edu. Come to office hours on Zoom or book a

Recap

[8:23] Consultation. Let's recap with Claude a cluster. You log into a login node and ask slurm for compute node. Two doors SSH to the login address or the open on demand website. Home holds 75 GB. Scratch holds 50 terabytes and is purged monthly. projects is for the long term. S run for live node sb batch for script sq to check on it. Partitions short by default the GPU partitions for GPUs courses for classes and seph after every job. So you request close to what you

Your turn

[8:50] Need. Your turn. Paste this into claude. I'm new to a slurm cluster. Here's what I want to run. Describe your program and data in two sentences. Tell me which of my files belong in home scratch and projects. Write me an srun command to test it interactively with modest resources. Then write the sbatch script for the real run and tell me which commands I use to check on the job and to read its efficiency afterwards. Then check two things yourself. Does every path it wrote start with a slash? And did it tell you to test with srun before you submit with sbatch? Intro to Explorer and OOD at Nick Bear Brown.