Scalable AI

• Course goals, structure, and expectations
• The full-stack view: hardware ↔ software ↔ optimization
• Lifecycle map: architecture → training → post-training → inference → applications

• Matrix multiplication as the basis of deep learning
• The Roofline model and memory bandwidth bottlenecks
• Component-level cost analysis (compute vs. memory vs. communication)
• The economics of tokens: training vs. serving tradeoffs

• Matrix multiplication as the basis of deep learning
• The Roofline model and memory bandwidth bottlenecks
• Component-level cost analysis (compute vs. memory vs. communication)
• The economics of tokens: training vs. serving tradeoffs

• Parallelism mechanics (e.g., multi-dimensional parallelism)
• Interconnect topology and communication costs
• Automated orchestration and practical scaling patterns

Practical instruction for performance profiling, arithmetic intensity, and optimization workflows.

• Tokenization fundamentals
• The causal language modeling (CLM) task
• Pre-training fundamentals and scaling considerations
• How to evaluate pre-trained models
• Data curation: deduplication, quality filtering, and mixture design

Hands-on curating work with NeMo Curator: building usable pre-training datasets.

A deep dive into pre-training data, evaluation, ablations, and the final “hero run” design choices.

• SGD-family methods and adaptive optimizers (Adam/AdamW)
• Learning rate schedules, warmup, and stability
• Gradient clipping, normalization, and numerics
• Batch size, effective step size, and scaling behavior

Moving beyond AdamW to emerging approaches (e.g., Muon, SOAP), and how to reason about their tradeoffs at scale.

Patterns and tooling for scalable synthetic data generation pipelines that can power pre-training, including quality controls and reproducibility.

• Differences between pre-training and post-training data
• Chat templating and training fundamentals
• Post-training benchmarks and evaluation practices

• Synthetic data generation pipelines
• Rejection sampling and quality scoring
• Skills mixtures and data balancing
• RL environment construction and reward design basics

A practical walkthrough of NeMo Data Designer for building and iterating on high-quality post-training datasets.

RL Algorithms and their application to post-training.

A detailed case study of post-training decisions, data strategy, and evaluation tradeoffs in practice.

• Speculative decoding: concepts and systems implications
• Advanced data types and quantization paradigms
• Calibration strategies and accuracy tradeoffs
• Pruning and neural architecture search (NAS) for inference efficiency

A systems overview of modern inference engines, batching/scheduling, memory management, and throughput/latency tradeoffs.

Compilation and runtime strategies for production inference, including graph capture, kernel selection, and end-to-end profiling.

Serving engines and control runtimes for LLM applications: throughput optimization, KV cache management, and efficient scheduling.

Designing prompts, retrieval, memory, and tool usage under latency/cost constraints and reliability targets.

Building agentic systems: planning, tool use, orchestration, evaluation, and failure modes.

Deployment-time safety: guardrails, red teaming, monitoring, and risk mitigation in real systems.

Alternative generative paradigms beyond autoregressive decoding and how they change training/inference tradeoffs.

Methods for engineering software harnesses for modern LLMs.

Multi-agent coordination, communication, and architecture patterns for scalable AI systems.