First-hand information for everyone
YANAMI introduces the first large‑scale multimodal dialogue benchmark that explicitly tracks changes in visual and textual state across turns. The dataset contains 400 k multi‑turn dialogues that weave together image‑based and text‑based inputs, demanding that models keep, update and revise a shared multimodal state.
The core dataset construction incorporates six key state‑management capabilities—Reference Resolution, Context Retrieval, Consistency Maintenance, Spatial and Physical Consistency, Constraint Keeping, and Correction‑Based State Updating—within four scenario types: Multi‑Image Fusion, Image Editing, World Modeling, and Visual Reasoning. Each dialogue is designed so that stateful interactions must be reflected in all turns.
Fine‑tuning the Unified Multimodal Model BAGEL on YANAMI‑400K produces consistent gains on the YANAMI‑Bench evaluation suite. The fine‑tuned BAGEL scores 13.23 % higher in multimodal stateful interaction than the baseline while preserving its general visual understanding. The benchmark also shows that current open‑source and closed‑source unified multimodal models score an overall average of 67.30–67.41, underscoring a large gap to the best closed‑source model.
| Index | Model | Size | Modality | Format | Fusion | Editing | Modeling | Reasoning | Overall |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-Image-2 (OpenAI, 2026b) | – | 86.78 | 90.54 | 81.40 | 73.46 | 83.05 | ||
| 2 | Nano Banana 2 (Google, 2026a) | – | 84.73 | 87.32 | 74.74 | 76.93 | 80.93 | ||
| 3 | Step1X-v1p2 (Liu et al., 2025) | 12B | 56.08 | 63.36 | 51.66 | 48.00 | 54.77 | ||
| 4 | Qwen-Edit (Wu et al., 2025a) | 20B | 49.66 | 54.70 | 28.14 | 37.72 | 42.55 | ||
| 5 | FLUX.2 [dev] (Labs, 2025) | 32B | 72.44 | 74.54 | 65.54 | 56.68 | 67.30 | ||
| 6 | UniPic 3.0 (Wei et al., 2026a) | 20B | 63.44 | 70.66 | 56.96 | 48.82 | 59.97 | ||
| 7 | OmniGen (Xiao et al., 2025) | 3.8B | 48.24 | 58.12 | 41.82 | 33.94 | 45.53 | ||
| 8 | OmniGen2 (Wu et al., 2025c) | 7B | 50.10 | 51.70 | 55.12 | 47.62 | 51.13 | ||
| 9 | Janus-Pro (Chen et al., 2025d) | 7B | 45.94 | 41.94 | 42.56 | 31.86 | 40.58 | ||
| 10 | Show-o2 (Xie et al., 2026a) | 7B | 40.82 | 45.12 | 41.78 | 29.42 | 39.29 | ||
| 11 | Ovis-U1 (Wang et al., 2025a) | 3B | 68.72 | 46.24 | 57.72 | 43.42 | 54.03 | ||
| 12 | Emu3.5 (Cui et al., 2025) | 34B | 74.48 | 60.10 | 74.06 | 60.98 | 67.41 | ||
| 13 | InternVL-U (Tian et al., 2026) | 4B | 66.72 | 66.12 | 66.48 | 46.52 | 61.46 | ||
| 14 | UniAR (Peng et al., 2026) | 10B | 57.06 | 41.82 | 41.80 | 35.92 | 44.15 | ||
| 15 | BAGEL (Deng et al., 2025) | 7B | 62.44 | 67.40 | 68.16 | 62.06 | 65.02 | ||
| + YANAMI-400K | 7B | 78.98 ↑ | 84.96 ↑ | 73.26 ↑ | 75.80 ↑ | 78.25 ↑ |
Visual performance is uneven across scenarios; “Visual Reasoning” is the hardest, while “Image Editing” and “World Modeling” see better results. These findings confirm that even strong models struggle to maintain, update and visually realize evolving multimodal dialogue states.
Significant limitations remain. Existing unified multimodal models still fail to faithfully translate updated states into accurate image edits and to maintain consistency across turns. The persistent performance gap between understanding the state and rendering it visually suggests that future models should target more robust internal state tracking.
Figure 1 illustrates YANAMI’s multimodal dialogue structure, highlighting how user and assistant turns build and modify a shared state. Table 2 (shown as Visual 6) summarizes model scores across interaction scenarios, highlighting the average performance gaps and the areas where stateful reasoning is most challenging. The benchmark therefore provides a unified foundation for training and evaluating future multimodal systems towards more complete, iterative state management.
