Scalable AI

Instructors

Teaching Staff

NVIDIA infrastructure and technology

Staff list may expand; check Ed for updates.

Class Logistics

Communication

Ed only. Please post all course questions on Ed (course discussion board). We do not use Slack or other channels for course Q&A. If you have not been automatically added, the join code will be shared in lecture.

Once we finalize the enrollment, we will purge all non-students from the course forum to encourage free discussion among students.

Compute

Teams are 8 groups of 4. Each group receives 1 H100 node (8×H100) for the semester (8 total nodes on GCP). Each group is assigned a single static external IP address for SSH access.

Course questionnaire (Week 1): a short onboarding questionnaire released in the first lecture; enrollment/compute allocations will be finalized by the end of the second week.

Everything in one place: lecture plan, dated timeline, deadlines/milestones, and recommended readings. Dates below are for Fall 2026. Lecture plan subject to change.

Date	Lecture / Events	Recommended Readings / Resources	Notes
Part 1: Architecture. Defining the Computational Graph and Scaling Strategies
Aug 26	L1. Course Overview and the Modern AI Stack 📄 Slides Key topics Course goals, structure, and expectations The full-stack view: hardware ↔ software ↔ optimization Lifecycle map: architecture → training → post-training → inference → applications	"The Bitter Lesson" by Rich Sutton "The Hardware Lottery" by Sara Hooker "Training Compute-Optimal Large Language Models" by Hoffmann et al. (Chinchilla) "An introduction to transformers" by Turner
Aug 31	L2. All About Performance (Part 1) 📄 Slides Key topics Matrix multiplication as the basis of deep learning The Roofline model and memory bandwidth bottlenecks Component-level cost analysis (compute vs. memory vs. communication) The economics of tokens: training vs. serving tradeoffs Also: A1 released (Performance Engineering Assignment) Release	How to Scale Your Model (Scaling Book), Chapters 1, 4, and 7 BRRR Intro GPU performance engineering resources How To Write A Fast Matrix Multiplication From Scratch With Tensor Cores
Sep 2	L3. All About Performance (Part 2) 📄 Slides	—
Sep 7	Labor Day No class	—
Sep 9	L4. Architectures to Break Bottlenecks: MoE, Sparse & Long-Context Architectures 📄 Slides	Mixture of Experts Explained (Hugging Face) Switch Transformers (Fedus et al.) DeepSeek-V3 Technical Report (MoE + MLA at scale) YaRN: Efficient Context Window Extension (Peng et al.)
Sep 14	L5. Parallelism Strategies 📄 Slides Key topics Parallelism mechanics (e.g., multi-dimensional parallelism) Interconnect topology and communication costs Automated orchestration and practical scaling patterns	Pipeline-parallelism schedule visualizer (interactive) The Ultra-Scale Playbook (Hugging Face) Megatron-LM: Training Multi-Billion Parameter Models (Shoeybi et al.) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al.)
Sep 16	L5. Parallelism Strategies (continued) L6. FlashAttention + Sonic MoE	FlashAttention (Dao et al.) FlashAttention-2 (Dao) SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations (Guo et al.)
Sep 21	L7. NeMo AutoModel	NeMo AutoModel (GitHub)
Part 2: Pre-Training of Language Models. The Engineering of “Base Models”
Sep 23	L8. Introduction to Pre-Training Key topics Tokenization fundamentals The causal language modeling (CLM) task Pre-training fundamentals and scaling considerations How to evaluate pre-trained models Also: Hypothesis statement v1 due Milestone	—
Sep 28	L9. Pre-Training (Part 2): Data for Pre-Training Key topics Data curation: deduplication, quality filtering, and mixture design Curating corpora at scale with NeMo Curator Also: A1 due Deadline A2 released (Pre-Training Assignment) Release	—
Sep 30	L10. Modern Architectures + Modern Optimizers Key topics SGD-family methods and adaptive optimizers (Adam/AdamW) Learning rate schedules, warmup, and stability Gradient clipping, normalization, and numerics Batch size, effective step size, and scaling behavior	—
Oct 5	L11. MARS Lecture	—
Oct 7	L12. Case Study: The Pre-Training of Nano-V3 Also: Project proposal v2 due Milestone	—
Part 3: Post-Training of Language Models. Specializing Models Through SFT and RL
Oct 12	L13. Intro To the LLM Post-Training Lifecycle and Evaluation Key topics Differences between pre-training and post-training data Chat templating and training fundamentals Post-training benchmarks and evaluation practices	—
Oct 14	L14. The Data Powering Post-Training: SFT Data Engineering and RL Environments + Details in Specific Domains Key topics Synthetic data generation pipelines Rejection sampling and quality scoring Skills mixtures and data balancing RL environment construction and reward design basics	—
Oct 19	L15. Building SFT Datasets: Generation, Tooling, and NeMo Data Designer	—
Oct 21	L16. Using the NeMo RL Stack for RL Post-Training + NeMo Gym Also: A2 due Deadline A3 released (Post-Training Assignment) Release	—
Oct 26	L17. Case Study: Post-Training of Nemotron-NanoV3	—
Part 4: Efficient Inference. Deployment Preparation and High-Performance Frameworks
Oct 28	L18. Fundamentals and Overview of High-Performance Inference Frameworks Key topics A systems overview of modern inference engines, batching/scheduling, memory management, and throughput/latency tradeoffs.	—
Nov 2	L19. Efficient Inference (Part 2): Inference Engines	—
Nov 4	L20. High-Performance Inference in Practice: Dynamo, TRT-LLM, vLLM, and SGLang Key topics Serving engines and control runtimes for LLM applications: throughput optimization, KV cache management, and efficient scheduling. Also: A3 due (Post-Training Assignment) Deadline A4 released (Inference and Serving Assignment) Release	—
Nov 9	L21. Grok LPX Also: Midterm check-in report due Milestone	—
Nov 11	Veterans Day No class	—
Part 5: LLM Applications and Use Cases. Building Real-World AI Systems
Nov 16	L22. Fundamentals of Context Engineering Key topics Designing prompts, retrieval, memory, and tool usage under latency/cost constraints and reliability targets.	—
Nov 18	L23. Harness Design Key topics Methods for engineering software harnesses for modern LLMs.	—
Nov 23	L24. Safety and Security (Part 1): Deployment Guardrails	—
Nov 25	Non-Instructional Day No class	—
Nov 30	L25. Safety and Security (Part 2): Open-Weight Safety	—
Part 6: Research Greenfields. Emerging Research Directions
Dec 2	L26. Greenfields in Performance & Accelerators Also: A4 due (Inference and Serving Assignment) Deadline	—
Dec 7	L27. Greenfields in Pre-Training and Post-Training Also: Draft project report due (submit on Gradescope) Milestone	—	Peer review cycle
Dec 9	L28. Greenfields in Inference Also: Final project presentations (short talks) Milestone	—
Dec 10	Peer review due Milestone Submit on Gradescope	—
Dec 11	Poster session Milestone No formal lecture	—
Dec 16	Final deliverables due Milestone Code + final report	—
Dec 18	Impact update Milestone Proof of impact document	—

Grading Breakdown

This course has no exams. Evaluation is based on assignments, a semester-long research project, an explainer video, and participation.

Component	Weight
Research project	50%
Assignments	35%
Explainer video	10%
Attendance & participation	5%

Commitment & Letter Grading

This course requires sustained weekly engagement (group work, compute usage, and in-person checkoffs). It must be taken for the maximum number of units. Letter grade only (no Satisfactory/Non-Satisfactory).

Assignments

There are four group-based assignments, each spanning roughly 3–4 weeks. Assignments include conceptual questions and hands-on experiments (implementation, profiling, scaling, analysis) with an in-person oral checkoff/presentation per assignment.

All assignment release/due dates are listed in the Consolidated Schedule.

• A1: Performance Engineering Assignment
• A2: Pre-Training Assignment
• A3: Post-Training Assignment
• A4: Inference and Serving Assignment

Late Policy

You have 3 total slack days across the semester for assignments. At most one slack day may be applied to any single assignment (extends deadline by 24 hours). No other late work is accepted without an approved exception.

Enhancements

Your assignments will also be graded on enhancements -- contribute meaningfully by adding questions, or reframing the programmatic part to be more challenging. Each group only needs to enhance one assignment. Out of the 35% of your grade that is reserved for assigments, 10% will be graded based on enhancements, and the other 25% will be graded based on the core assignment.

Research Project

The research project is group-based and open-ended, with an emphasis on producing something useful to the community. Expect 1–2 project check-ins with course staff over the course of the semester.

All project milestones and dates are listed in the Consolidated Schedule.

• Technical quality (40%): hypothesis clarity, depth, correctness, rigor, and artifact quality (code + report + other deliverables).
• Impact (10%): evidence of external usefulness (e.g., PR merged into open source, reproducible benchmark, public report, arXiv preprint).
• Peer review: each group provides a structured review of another group’s report.

Explainer Video

Each student records a short explainer video on a topic from the course: record yourself at a whiteboard giving a clear, simple explanation of a concept from lecture, where you go further than we had time to.

These are made for your classmates, so make them as helpful as you can. Full credit requires going into more depth on the topic than lecture does, not restating it. Each video is due a month after its topic is introduced in lecture. No video editing is required.

Attendance & Participation

Attendance is required. Participation includes contributing to in-class discussions and project check-ins, posting/answering on Ed, and providing constructive comments on lecture notes and slides.

Academic Integrity

Collaboration is encouraged within your group. Do not copy code or reports from other groups. If you use external code or AI assistants, cite the source clearly and ensure you understand the work you submit.

Infrastructure Notes

This course uses shared GPU infrastructure. For access/quota/networking issues, post on Ed and include timestamps, job IDs, logs, and a short repro description. Multi-node experiments require coordinating with another team to share nodes for a bounded window of time.

Submission platform (Gradescope): all written and code submissions are via Gradescope (entry code posted on Ed), unless explicitly stated otherwise. Oral presentations/checkoffs require sign-up and in-person attendance.