First-hand information for everyone
Main discovery Agentic search episodes are deterministic functions of the policy’s decisions and the sampling noise. This makes “twin worlds” realizable: you can replay the same downstream randomness while changing a single decision. Doing so isolates the decision’s causal effect on the final outcome (the same noise, a different decision). (D1, D2)
How TWICE works TWICE (TWIn-world Credit Estimation) implements these twin worlds. It keys the per-token Gumbel noise (random numbers used when sampling tokens) by episode, turn, line, and position. That lets a counterfactual twin inherit the factual episode’s downstream noise while remaining an exact sample from the policy. (M1)
A twin is created by drawing an alternative for a visited turn, rolling out the rest of the episode with the factual keys, and computing the paired outcome difference (factual minus twin). That difference becomes a turn-level advantage. TWICE uses those advantages in PPO/GRPO-style updates on actor-sampled trajectories. Only the trained actor is deployed at inference. (M2)
TWICE also budgets auxiliary rollouts. It branches decisions with probability proportional to the square root of (predicted discordance ÷ cost), subject to clipping and inverse‑probability reweighting (the square‑root budget rule in Prop.4 and Eq.(2)). This gives more twins to cheap late decisions and to prompts with mixed outcomes, while keeping the estimator unbiased. (M3)
Concrete results Training followed the Search-R1 protocol. Retrieval used the top-3 passages from a fixed 2018 Wikipedia dump, each episode allowed at most four search calls, and exact-match was used as the reward. All methods were compared on the same data and update protocol. (C1, C2)
On seven question-answering benchmarks, TWICE improved the average exact match of GRPO by 5.3 points with Qwen2.5-3B and by 4.5 points with Qwen2.5-7B. It also outperformed tree-search and counterfactual-branching baselines at the reported budgets. (C3)
In an audit of 1,036 visited decisions with Qwen2.5-3B, TWICE reduced wrong-sign credit to 6.9% (compared with 21.7% for an independent-counterfactual baseline, CRR). TWICE achieved the highest Pearson correlation (0.72) with an oracle advantage estimated from 128 rollouts per decision. It required 152 seconds per update versus 262 seconds for VinePPO, using 68% fewer auxiliary tokens. (C4, C3)
Why this matters Sharing downstream randomness makes the twin’s outcome difference isolate the decision’s effect. That reduces the shared-luck variance that plagues independent counterfactuals. The square-root budget focuses compute where it helps most. Together, these reduce wrong-sign credit and improve sample efficiency in the reported QA experiments. (D2, M2, M3, C4)
Supported caveats and limits TWICE requires replayable tools so continuations can be rerun with the same downstream noise. Retrieval from a fixed corpus, sandboxed code, and simulators qualify. Live web search or stateful APIs need caching to qualify; without replayability the coupling weakens, though credit estimates remain unbiased. (L1)
The benefit depends on the twin correlation (how much the twin and factual continuations stay similar). Correlation drops at high sampling temperature and for long continuations where divergence compounds. The formal sample-complexity advantage was stated for binary utilities; graded rewards inherit variance benefits but not necessarily the same rate. TWICE estimates whole-turn advantages; it does not split credit inside a turn between reasoning tokens and the action. Experiments were limited to question answering and two text environments, models up to 14B parameters, and at most eight search calls. Long-horizon agents with many calls were not tested in the reported experiments. (L2, L1)