arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Coda‑DaVinci learns to switch between direct video generation and proxy‑guided synthesis for more consistent and visually realistic animations | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Computer VisionEnglishPublished

Coda‑DaVinci learns to switch between direct video generation and proxy‑guided synthesis for more consistent and visually realistic animations

November 22, 2026arXiv: 2611.26595v1

Coda‑DaVinci is a new agentic framework that decides when to let a text‑to‑video model generate a scene from scratch and when to guide it with an explicit 3‑D proxy. The system adapts the proxy’s granularity—coarse models for human bodies and medium‑grained models for objects—and runs physics simulations when the scene needs realistic interactions.

Illustration of the Coda‑DaVinci design, featuring a decision node that routes between direct video generation and proxy‑guided generation, proxy refinement levels for human bodies and objects, and a physics engine guiding the final video.
Figure 1. Figure 1 shows the end‑to‑end design of the Coda‑DaVinci framework: a routing decision that chooses either direct generation or proxy‑guided synthesis; the selection of proxy granularity (coarse for humans, medium for objects); and the integration of physics simulation to support realistic interactions.Ji, Yiyan, Wang, Qixun, Shi, Yang, Zhang, Zhuoran, Xu, Yifan, Liu, Tengfei et al. · arXiv:2611.26595v1 · Figure 1: ·
CC BY 4.0

The framework was evaluated on three public benchmarks (WorldMemBench, VGIF‑Bench, and Principia). Across all tests, the configuration G1/P1/PE2—medium‑grained proxies, physics simulation enabled, and enhanced prompting—achieves the highest average human‑judged score for consistency, instruction following, physical plausibility and naturalness. The data show that agents using coarse proxies tend to produce more natural motion, while medium‑grained proxies improve overall consistency.

Table of per‐metric evaluation results for each configuration and benchmark, highlighting the G1/P1/PE2 row as the top performer.
WorldMemBenchVGIF-BenchPrincipiaAverage
ConfigurationC↑I↑P↑N↑C↑I↑P↑N↑C↑I↑P↑N↑C↑I↑P↑N↑
Direct generation2.1821.2422.4852.3332.4831.8282.2072.0002.5622.4062.0002.1562.4041.8192.2342.170
G0/P0/PE02.7583.0002.6062.4552.4292.2761.8971.4142.9692.9382.5622.5942.7312.7552.3722.181
G1/P0/PE02.6063.0002.5762.4552.6902.2071.4481.0342.9693.0002.6562.7192.7552.7552.2552.106
G2/P0/PE02.7883.0002.6062.4552.5522.0691.4481.0002.9062.8752.5312.5622.7552.6702.2232.043
G0/P1/PE02.6062.9702.7582.6972.4482.2761.7591.3452.9692.9692.5312.6252.6812.7552.3722.255
G1/P1/PE02.8183.0002.7882.6062.5862.2761.6901.2072.9692.9382.6882.7812.7982.7552.4152.234
G2/P1/PE02.7273.0002.7272.5452.4142.2411.3790.9312.9692.9382.7502.5622.7132.7452.3192.053
G1/P0/PE12.4552.9702.6362.6062.4142.1721.3450.8972.0622.4692.4382.2812.3092.5532.1701.968
G1/P0/PE22.7883.0002.5762.5452.5862.3101.4481.0342.9382.9692.5312.6562.7772.7772.2132.117
G1/P1/PE12.1822.6972.5152.2732.7242.8971.8281.3793.0003.0002.1112.2782.5622.8372.1751.950
G1/P1/PE22.8182.9702.7272.5152.9662.9312.0341.4483.0003.0002.5002.5942.9262.9682.4362.213
Table 1. Table 1 reports the mean human‑judged scores for consistency, instruction following, physical plausibility, and naturalness across the three benchmarks; the configuration G1/P1/PE2 (medium‑grained proxy, physics enabled, enhanced prompting) achieves the highest average score of 2.97 on WorldMemBench, 2.93 on VGIF‑Bench, and 2.79 on Principia, leading to the overall best four‑metric mean.Ji, Yiyan, Wang, Qixun, Shi, Yang, Zhang, Zhuoran, Xu, Yifan, Liu, Tengfei et al. · arXiv:2611.26595v1 · Table 1: · CC BY 4.0

While the proxy‑guided approach outperforms straight generation on these metrics, it has limits. In long‑video experiments, the system sometimes fails to maintain scene stability or return to the prescribed setting after an instruction, and it struggles to keep objects correctly placed through successive interactions. Direct generation, on the other hand, often loses scene state fidelity between steps, leading to inconsistent object placements and properties.

Overall, Coda‑DaVinci demonstrates that a task‑adaptive mix of proxy modeling and direct generation can improve visual realism and physical plausibility, but further work is needed to handle long‑duration consistency and strict scene adherence.

Source paper

Coda-DaVinci: Reimagining Video Generation with Code2Video

Computer Vision
View on arXiv