Refined reproducible artifacts boost experiment success by up to 23 % and lower AI‑model API costs by almost 20 %
arXiv: 2611.08452v1
A new framework, called ReproBridge, builds a single, refined artifact that bundles a cleaned‑up codebase, an environment set‑up script, launch helpers, and a step‑by‑step guide. ReproBridge runs an automated auditor to confirm that the experiment reproduces correctly. A benchmark on 36 research papers contains 1,029 downstream tasks, each linked to five human‑verified target scores.
When this refined artifact (RRA) is used instead of the raw released code, the mean success rate jumps from 72 % to 92 % on average. Across four large language‑model back‑ends — Sonnet 4.6, Sonnet 5, Opus 5 and Gemini 3.8 Flash — the increases are 19.3, 16.8, 5.5 and 23.2 percentage points, respectively. Simultaneously, the total tokens spent on API calls fall by 19.8 % to 18.3 %, and the dollar cost is reduced by 20.6 % to 23.7 %.
Table showing for each model the raw and refined reproducible artifact reproduction success rates, token counts and API USD costs, along with the gain in success percentage points and reductions in tokens and USD.
Success (%)
Tokens (B)
API (USD)
Model
Raw
RRA
Gain
Raw
RRA
Red.
Raw
RRA
Red.
Sonnet 4.6
72.3±1.1
91.6±3.4
+19.3
8.48±0.38
6.80±0.76
19.8
3,165±142
2,512±286
20.6
Sonnet 5
76.5±2.8
93.3±1.4
+16.8
4.98±0.55
4.52±0.24
9.2
1,726±175
1,317±102
23.7
Opus 5
89.2±1.2
94.7±2.2
+5.5
3.01±0.16
2.82±0.29
6.3
3,480±165
2,865±312
17.7
Gemini 3.8
67.6±1.9
90.8±1.1
+23.2
2.24±0.26
1.83±0.09
18.3
392±41
326±16
16.8
Table 3. Mean reproduction success percentages and corresponding percent‑point gains, token reductions, and API‑USD savings for each model under raw code and refined reproducible artifact (RRA).Wang, Yubo, Meng, Rui, Nam, Jaehyun, Pfister, Tomas, Yoon, Jinsung · arXiv:2611.08452v1 · Table 3: · CC BY 4.0
The gains hold when the tasks are evaluated three times and when a wide range of real‑world papers are included. The refined artifact also limits the spread of success rates between models, reducing the difference from 21.6 pp to 3.9 pp.
The study is limited to 36 papers and 1,029 tasks; it does not cover the full range of research in machine‑learning conferences. The procedures can preserve incorrect assumptions, and the automatic audit may miss some violations. Future work should expand the dataset and refine the audit logic.
ReproBridge thus demonstrates that preserving the full execution knowledge in a reproducible package can make experiments more reliable and less costly for downstream users.