T1 trains a 122B Mixture-of-Experts terminal agent with RL across 300-turn tasks

Researchers introduced T1, a Mixture-of-Experts model with 122 billion total parameters trained with reinforcement learning to operate a real shell inside a cloud sandbox, according to the abstract of arXiv paper 2609.11042, posted under cs.LG. The model runs as many as 300 or more tool-call turns on a single task, and it is rewarded by executing each task's own verifier rather than by a learned reward model or a human-written rubric. The paper frames the work around the shift in agent usage toward long-horizon tasks such as coding and scientific discovery, with terminal work singled out as especially important.

That framing sets up the problem T1 is built to solve. A trajectory spanning hundreds of shell commands gives reinforcement learning two failure modes at once: the value estimates an actor-critic setup depends on have little to latch onto early in training, and the tokens the trainer grades stop matching the tokens the sampler actually produced. The paper reports both symptoms and the fixes applied to each.

The first fix is an aggressive warm start, which the authors use to stabilize actor-critic training. Alongside it they add a dense process reward that scores trajectories by the absolute number of passing verifiers, so a rollout that clears three of a task's checks earns a different signal than one that clears none, instead of collapsing to a single terminal verdict.

The second fix covers optimization. Under TITO construction, training runs on the exact sampled token identifiers, with drift repaired at turn boundaries. Rollout routing replay, which the paper abbreviates R3, records the sampler's per-token expert choices at every MoE layer and replays them during training. The two techniques cut the training-to-inference log-probability difference from 0.021 to 0.013.

The third piece is the data. T1 trains on what the paper describes as a fully out-of-distribution corpus: isolated seeds and synthesized tasks kept disjoint from Terminal-Bench 2.1. The stated purpose is to remove the easiest explanation for a good score, that the model saw the tasks it is measured on, so that any gain traces to genuine capability transfer rather than benchmark overfitting.

Read as a whole, the recipe is the contribution as much as the model is. Warm start, dense verifier-count reward, exact-token training, router replay and a held-out corpus are each aimed at a specific instability in long-horizon agentic reinforcement learning, and the authors present them together rather than as separate findings.

What the excerpt does not contain is the result table. The abstract text available here ends mid-sentence at "with exact," so there are no headline scores, no comparison against other terminal agents, no active-parameter count or compute budget, and no statement about weights or code release. No other coverage of the paper appears in the material supplied; the single arXiv entry is the only source for this account. T1's benchmark performance and the detail in the truncated section remain the open questions.