The setup
Most published pre-training runs assume a compute budget academic labs don't have — OLMoE-1B-7B saw 5 trillion tokens, Llama 3.2 even trillions more. But the methods behind them — data curation, architecture selection, optimizer choice, systems optimization — aren't inherently expensive. The interesting question is how far they carry you when compute is genuinely tight.
This semester we got to run that experiment. In EE 290/194, every group of students gets one node — 8×H100 GPUs, donated by NVIDIA for the entire semester. In the first two assignments, students are asked to pretrain a large language model from scratch within 2 weeks. This post is about how they approached it: what they prioritized, what they measured, and what came out.
The first two assignments build directly on each other. A1 teaches you how to build an efficient pre-training environment. The groups that took A1 seriously walked into A2 already knowing exactly how many tokens their systems work had bought them. Here are the three main lessons students learned in this full-stack pretraining task.
Lesson 1Fix your infra first
Every group hit the same wall. Nemotron-Nano-V3 is a hybrid Mamba/Attention/MoE model, and the open-source training configurations were mostly written for dense transformers, causing an extremely low baseline MFU between 0.18% and 1.4%. The GPUs were either idle, suffering from heavy communication overhead, or the parallelism configurations didn't take effect at all. Here are some of the strongest fixes from students:
| Group | Headline fix (with profiler evidence) | Result |
|---|---|---|
| 2 | Made AutoPipeline work for NemotronH, then pipeline-parallelized it (PR #1298) |
44× throughput† |
| 4 | Replaced HuggingFace's per-expert Python for-loop with a fused GroupedGEMM |
~1% → ~21% MFU, ~22× throughput |
| 5 | Wired pipeline + expert parallelism into the hybrid NemotronH layout (PR #1305) |
37× throughput (326 → 12,110 tok/s) |
| 6 | The EP (Expert Parallelism) mesh was built but never used; wrote an adapter to fix it | 0.58% → 12.8% MFU, ~95× throughput |
† MFU overcounts the Mamba/SSM layers (which run on CUDA cores, not tensor cores), so throughput speedups are the more honest cross-group signal — as Group 2 itself noted.
Group 2 got the hybrid checkpoint running under AutoPipeline, which had crashed because NemotronH doesn't expose its layers the way the framework expected. Then pipeline parallelism improved the throughput by ~95 ×. Group 4 inspected the HuggingFace source, found the experts were being run in a Python for-loop, and replaced it with a single fused GroupedGEMM. Group 5 patched the broken pipeline-stage construction and forward signatures for the hybrid layout and wired in DeepEP and pipeline parallelism, which improved throughput from 326 to 12,110 tok/s. Group 6 found that HuggingFace stores the 128 experts as an nn.ModuleList where the framework expected a stacked GroupedExperts tensor, so every isinstance check failed quietly and FSDP all-gathered all 128 experts each step. With a ~15-line adapter, they improved real-data throughput by ~95×.
Lesson 2Measure before you commit
Some groups spent real compute on small experiments first. Group 5 ran 18 experiments instead of one big run: a 3×2 grid of three data sources (raw ~55B tokens, curated ~5.3B, hybrid ~8.8B) crossed with dense vs. MoE, plus tiny scaling runs, for ~150 GPU-hours total. They found that MoE beats dense "for free" at matched per-token compute (HellaSwag +3–6 points, ARC-Easy +2.3–4.5). Moreover, Group 5 compared the contributions of data volume and data quality. With 55B tokens, raw unfiltered data won overall. However, at matched token budgets (~5.7B), the curated + synthetic hybrid won 5/7 benchmarks. Group 6 ran proxy evaluations at 2K training steps, comparing their hand-designed 9-category data mix against ClimbMix. ClimbMix won across the board. They also compared Muon vs. AdamW over 1000 steps — Muon converged faster. These short runs cost a fraction of the full training budget but determined the final configuration.
Lesson 3Then train
The actual training runs were the simplest phase of the project. By the time each group launched, the major decisions were already made. The top four base models are shown below:
| Group | Model (total / active) | Data | Tokens | Optimizer | Headline benchmarks | Val loss / PPL |
|---|---|---|---|---|---|---|
| 4 | 3.4B / 1.5B (16L) | NemotronCC-HQ | ~100B | AdamW | SciQ 84.9, HellaSwag 52.4, PIQA 70.9, ARC-E 58.3 | OOD PPL 8.8 |
| 5 | 1.4B / 390M (8 experts top-2) | FineWeb-Edu/PDFs/Wiki | 55B | AdamW | HellaSwag 49.1, PIQA 70.2, SciQ 79.6, ARC-E 59.6 | WikiText 17.6 ppl |
| 6 | 7B / ~1B (38L, 64 experts top-4) | ClimbMix | 73.4B | Muon | MMLU 37.7, HellaSwag 68.4, PIQA 79.6, ARC-C 48.0, ARC-E 73.5 | 2.08 / 8.0 |
| 8 | 4.5B / 1.02B (17L) | FineWeb-Edu (2-phase) | ~43B | Adam | HellaSwag 51.2, PIQA 70.9, WinoGrande 54.2, MMLU 27.1 | 1.89 / 6.6 |
Group 4 carried their A1 fused-GroupedGEMM kernel into A2 to sustain ~39K tok/s/GPU and train the largest token count of any group (~100B). Group 5 turned their 18-experiment study into a 1.4B-total / 390M-active MoE trained on 55B raw tokens. Group 6 produced the best model in the cohort, leading every shared benchmark and pulling clear on the hard reasoning tasks; the recipe was a 7B-total / ~1B-active hybrid Mamba/Attention/MoE model (38 layers, 64 experts, top-4) trained on 73.4B ClimbMix tokens with Muon and a Warmup-Stable-Decay schedule, and its architecture choices came straight out of A1. Group 8 conducted a disciplined four-phase architecture search and a two-phase data curriculum. Different teams optimized different parts of the recipe; Group 6's combination of capacity, data quality, and optimizer took the top result.
Credits
These results belong to the students of EE 290/194, who took eight GPUs and two weeks and turned them into real pre-trained language models from scratch.
And our deepest thanks to NVIDIA for donating the 8×H100 compute that made this work possible — and for the open-source training frameworks our students were glad to give back to.