First-hand information for everyone
A team from NVIDIA has published a detailed recipe for a relatively new stage in large-model development called “midtraining,” and shown that it can measurably improve how well base language models later become agentic coding assistants. Midtraining sits between conventional pretraining and the supervised and reinforcement post-training steps (SFT/RL). The goal is to give models exposure to the long, structured artifacts and interaction patterns that real software engineering requires — whole repositories, pull requests, commit histories, build logs — together with a bounded share of agent trajectories (model–tool interactions).
Why is this useful? Typical pretraining is tuned for efficiency and uses relatively short contexts (a few thousand tokens), and SFT often consists largely of synthetic chat-style examples. That mismatch leaves models underprepared for long-horizon engineering tasks and for working with tools and environment outputs. The authors argue midtraining fills the gap: it trains models on documents that are tens or hundreds of thousands of tokens long and on a mix of natural software-engineering material plus a modest amount of agent traces, so the model learns to reason and act across long contexts before SFT.
What they did. The paper proposes an open midtraining blend and training schedule. Key design choices include training at a 256,000-token context window, running for hundreds of billions of tokens in total, keeping natural software engineering sources as the large majority of the data, and reserving roughly 10–15% of the midtraining blend for agent traces (trajectories). They also keep 20% of the training signal as replay from the original pretraining data to avoid abrupt distribution shifts, and they mask tool responses (environment outputs) from the loss when using traces. The authors report ablations that motivated several of these choices.
Measured benefits on Nemotron models. The team evaluated their recipe on two Nemotron base models of different sizes. For the 30B-parameter “Nemotron 3.5 Lightning” model, 500 billion midtraining tokens (followed by identical SFT) lifted performance on SWE-bench Verified from 60.7% to 64.5% pass@1, SWE-bench Multilingual from 55.8% to 62.8%, and SWE-bench Pro from 32.0% to 35.5%. For the larger 120B-parameter “Nemotron 3.5 Super,” 280 billion midtraining tokens improved DeepSWE — a challenging long-horizon benchmark — from 11.7% to 15.7% and raised SWE-bench Verified from 72.9% to 74.3% after identical SFT. The authors also observed that midtraining tends to make models more robust across different agent harnesses (the surrounding agent code that runs the model).
Which parts mattered most? The paper reports ablations showing that the long, natural software-engineering data is important, but that including a limited share of agent traces drives much of the downstream improvement after post-training. They also found that training on tool responses (the environment’s outputs) hurt post-SFT agent performance in their setup, and that moderate replay of pretraining data and a conservative learning-rate tuning helped avoid regressions in general capabilities.
Limits and caveats. The authors are clear about limitations. Their experiments are on a single model family (Nemotron) and mostly target coding/agentic tasks; results may not generalize across architectures or non-coding domains. Benchmarks for agentic behavior are noisy and comparisons reported in the paper are generally single runs, so small score differences should be interpreted cautiously. They also did not compare spending the same compute budget on longer pretraining or SFT instead of a separate midtraining stage, and reinforcement-learning results on top of SFT were limited in these experiments.
What this means going forward. The paper provides a transparent, reproducible blueprint for an intermediate training stage that other teams can test. If the effects hold across more models and settings, midtraining could become a standard way to prepare base models for later tool-enabled and long-horizon tasks without making SFT so long that it harms later RL fine-tuning. The authors highlight future work on expanding and diversifying trace data and exploring how other post-training objectives might be shifted into midtraining.
In short, this work does not claim to be a final answer, but it does offer concrete, measured guidance for teams building agentic models: expose base models to long, naturally structured engineering data and a controlled share of agent interactions at an intermediate stage, tune replay and learning-rate choices conservatively, and verify downstream post-training gains rather than relying only on base-model proxy metrics.