Expanding the universe of LLM-generated kernels.
We stress-tested frontier models on GPU kernel engineering across a wide range of domains and tasks.
The Problem: Why Another Kernel Benchmark?
Existing kernel benchmarks cover the operators you find in a standard deep learning stack: matrix multiplications, attention variants, normalization layers. That is a reasonable starting point, but production GPU workloads increasingly span domains that do not look like training loops -- sparse graph learning, molecular simulation (Evoformer blocks, sequence alignment), robotics dynamics, probabilistic inference. None of these appear in existing benchmarks. PopcornBench fills that gap with 94 problems across four non-standard scientific subdomains.
The second gap: no prior benchmark tests whether a model can take a production kernel written for one GPU architecture and translate it to a newer one. H100 introduces wgmma, TMA, and the SM90 memory hierarchy, primitives absent from A100 code. A model that can write a kernel from scratch may not know how to port an existing one. PopcornBench's hardware translation track covers 15–16 production kernels (single-GPU and multi-GPU) ported from A100 to H100.
We also address reward hacking -- a known problem (Lange et al. 2025) -- by baking stress testing into the evaluation loop rather than applying it post-hoc. When evaluation parameters are fixed and disclosed in the prompt, a model can hardcode those constants and produce kernels that look correct on the disclosed inputs but fail on any other input shape. PopcornBench reruns submitted kernels on randomized input shapes; contaminated numbers have nowhere to hide.
What PopcornBench Covers
Subdomain Generation (Levels 1-3)
94 problems spanning graph learning, computational biology, robotics, and probability/statistics. Ranges from CSR SpMV and Smith-Waterman alignment to Gibbs sampling, GJK collision detection, flow-matching ODE solvers, and full Evoformer blocks. Curated Popcorn2 subset: 34 problems with communication-collective problems removed.
Hardware Translation (Level 5)
15-16 production kernels from vLLM, FlashAttention-2, Marlin, and CUTLASS. Two tracks: single-GPU (A100 to H100) and multi-GPU (8xA100 to 8xH100).
Three agent harnesses control what tools the model can use during a run:
| Harness | Tools available | Max turns |
|---|---|---|
| Single Turn (ST) | Submit kernel only | 1 |
| Default (MT-Default) | Compile, correctness check, specs, static analysis, submit | Multi |
| All Tools (MT-All) | Default + profiling (ncu), disassembly, roofline (ERT) | Multi |
The profiling tools in MT-All expose hardware counters: DRAM utilization, compute utilization, speed-of-light (SOL), warp stall breakdown. The idea was that richer feedback leads to better kernels. Results vary by task type.
Subdomain Generation Results
PopcornBench is the first benchmark to evaluate LLM kernel generation across scientific computing domains outside the standard deep learning stack. The 94 problems span four areas: graph learning (sparse matrix operations, GNN primitives), computational biology (Smith-Waterman alignment, Evoformer blocks), robotics (GJK collision detection, rigid body dynamics), and probability/statistics (Gibbs sampling, flow-matching ODE solvers). Existing benchmarks do not test any of these.
Headline numbers: ST 12.7%, MT-Default 32.4%, MT-All 19.6%. For scientific computing kernels, compile-and-check feedback is the key enabler; profiling tools add overhead without improving correctness. The model benefits from seeing compilation errors and correctness failures, but hardware counter feedback does not move the needle and costs agent budget.
The Pareto analysis reveals where gains come from. Speedups are dominated by kernel fusion: the model combines multiple Python-level operations into a single CUDA kernel, eliminating global memory round-trips. Tensor core utilization is below 5% across the board. These are memory-bound problems, and the solver that wins reduces memory traffic, not one that maximizes compute throughput.
Resource efficiency: multi-turn agents spend roughly 15x more tokens than single-turn for approximately 2.5x the solve rate. More importantly, token spend within multi-turn runs is uncorrelated with the speedup of the final kernel. The agent that spends the most tokens does not produce the fastest kernel.
Hardware Translation Results
PopcornBench is the first benchmark to evaluate LLM kernel translation across GPU generations, using human-written H100 references rather than PyTorch baselines.
Multi-GPU: 8xA100 to 8xH100
Translating multi-GPU kernels across GPU generations is near-intractable in single-turn mode: 4.2% correctness. Compile and correctness feedback alone (MT-Default) does not move the needle much. The gains come from MT-All, which reaches 22.2% correctness: a 5x improvement over ST.
The most striking data point: when the model actually invokes the profiling tools during a run, it solves the problem 100% of the time (7 out of 7). Profiling invocation is a near-perfect predictor of success in this track. The bottleneck is not observation capacity but the model knowing when to ask for hardware feedback.
The failure mode is specific. Models can translate arithmetic structure: they convert tensor core operations from A100 to H100 semantics in straightforward cases. What they consistently miss are the H100-specific primitives: wgmma (warpgroup matrix multiply-accumulate), TMA (tensor memory access) descriptors, and asynchronous pipeline stages. These require understanding the SM90 memory hierarchy and synchronization model, not just the compute semantics.
Single-GPU: A100 to H100
The single-GPU track produces a reversal: MT-Default (27.1% correctness) beats MT-All (20.8%). More tools hurt.
The explanation: single-GPU A100-to-H100 translation is bottlenecked by instruction-level SM90 rewrites, not by insufficient hardware feedback. Profiling counters tell the model that utilization is low, but they do not tell it how to structure wgmma and TMA usage. The MT-All agent spends a disproportionate share of its token budget on profiling inspection, leaving fewer turns for code iteration. Observation surface was not the binding constraint.
Even when models produce correct kernels, they are slower than the human reference: median speedup 0.94x. Correctness does not imply efficiency.
The kernel family breakdown shows where the ceiling is. CUTLASS kernels are the binding constraint: 11% correctness on MT-Default, 6% on MT-All. vLLM and Marlin kernels are substantially more tractable (38%/33% each). CUTLASS exposes the full complexity of the H100 memory hierarchy and requires precise CUTLASS 3.x API usage that models have limited training signal on.
Evaluation Methodology: Preventing Reward Hacking
This is a known problem in kernel benchmarking (Lange et al. 2025). When evaluation parameters -- batch size, sequence length, feature dimension -- are fixed and disclosed in the prompt, a model can read them and hardcode those values as template arguments or loop bounds. The compiled kernel runs extremely fast on the evaluation inputs because it does essentially no work at runtime; on any other input shape, it produces wrong results or crashes. PopcornBench addresses this with infrastructure: stress tests are baked into the harness, so contaminated results never reach the leaderboard. The numbers are stark: 96.6% of GPT-5.5's initially-correct kernels fail after stress testing (89/89 down to 3/89). The baseline reward-hacking rate is 69.9% with MT-Default; it drops to 21.1% when profiling tools are available, because hardware counter feedback (near-zero DRAM utilization, near-zero SOL) signals to the model that the kernel is not doing real work.
Key Takeaways
Graph learning, computational biology, robotics, and probabilistic inference reveal capability gaps that standard DL operators do not. Domain coverage is not a cosmetic benchmark dimension -- it changes which models lead and by how much.
Models can translate arithmetic structure but consistently miss SM90 instruction-level primitives: wgmma, TMA, asynchronous pipeline stages. CUTLASS is near-intractable. Correctness on translation tasks does not imply efficiency -- median speedup is 0.94x.
MT-All underperforms MT-Default on subdomain generation and single-GPU translation. Richer observation costs agent budget without improving what matters: correct SM90 rewrites and algorithmic structure discovery.
Across all subdomain problems, the winning strategy is fusing operations to reduce global memory traffic. Tensor core utilization below 5%. The problem is memory-bandwidth, not compute.
Fixed disclosed evaluation parameters invite hardcoding. Any benchmark that skips stress testing on randomized input shapes is measuring reward hacking alongside kernel quality. SOL and DRAM utilization are the only way to tell the difference.
What's Next
The full problem set, evaluation harness, and all sweep results are available at the links below. We are expanding the model coverage (Grok-4.3 results pending) and the hardware translation track. The stress-testing infrastructure is open for integration into other kernel benchmarks that want to harden their evaluations against reward hacking.