arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Researchers map four layers of spatial awareness in popular visual foundation models, revealing gaps in prediction and action planning | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Computer VisionEnglishPublished

Researchers map four layers of spatial awareness in popular visual foundation models, revealing gaps in prediction and action planning

October 31, 2026arXiv: 2610.40636v1

Scientists examined ten pretrained visual models that learn from videos by freezing their internal features and training simple read‑out probes. Using streaming egocentric footage of static indoor rooms, the probes extracted information from prefixes that grew in length. The analysis identified four distinct spatial capabilities:

Level 1 – spatial structure, which captures geometry and object identity across views. Level 2 – object persistence, the ability to remember where objects were after they leave the field of view. Level 3 – predictive spatial update, anticipating future observations after an action. Level 4 – actionability, using the internal state and a goal to select an action that moves the observer toward that goal. The figure shows a schematic of a moving observer building this hierarchical state.

Diagram depicting a living‑room scene with a red dot indicating the observer’s position and arrows illustrating the building of internal spatial structure, persistence of objects, prediction of future views, and action selection toward a goal.
Figure 1. Figure 1 shows the internal spatial state constructed by a moving observer from partial views of an environment, along with the four identified spatial capability levels—spatial structure, object persistence, predictive spatial update, and actionability.
Yin, Baiqiao, Cai, Zhongang, Wang, Ruisi, Liang, Qiwei, Yang, Lei, Liu, Ziwei · arXiv:2610.40636v1 · Figure 1: · CC BY 4.0

Performance varied across models and tasks. , a measure of how accurately the state keeps track of past objects. DINOv2 led depth prediction, showing that good visibility‑scene metrics do not guarantee persistence performance. A table of results summarizes these numbers.

Table summarizing metrics for each model, including depth loss, AUC, instance loss, balanced accuracy, angular error, global retrieval at rank‑1, and path integration, with Fast3R and VideoMAE showing the smallest angular errors and DINOv2 showing the lowest depth loss.
Metric geometryInstance identityView consistencyObject persistenceDirect predictionPath integration
Model (layer)Depth loss ↓AUC@30 ↑Instance loss ↓Bal. acc. ↑Ang. err. (∘) ↓Global R@1 ↑Global R@1 ↑
DINOv2-L (0)0.14220.04121.38280.955765.380.23730.0900
Fast3R (24)0.13860.38701.60720.980832.790.01330.0190
VideoMAE-L (23)0.29680.20264.15710.911836.450.05310.0685
V-JEPA2-L (23)0.27080.18542.92110.925645.310.06250.1588
Qwen2.5-VL (−1)0.22440.02692.94760.938568.590.17120.0903
Wan (20)0.36090.06504.93010.821562.380.02600.0405
CogVideoX (20)0.43400.02766.69920.800170.620.00880.0160
Aether (1)0.52700.01857.18470.692861.180.00240.0042
MiniMax-H3 (30)0.42520.04776.22650.825555.970.0249†0.0117†
LingBot-video (30)0.43140.02325.58750.785870.300.03020.0429
Table 2. Table 2 presents quantitative results for all ten representations, highlighting that Fast3R and VideoMAE achieved the lowest angular error for object persistence, while DINOv2 topped depth prediction, illustrating that visibility‑scene metrics do not guarantee persistence performance.Yin, Baiqiao, Cai, Zhongang, Wang, Ruisi, Liang, Qiwei, Yang, Lei, Liu, Ziwei · arXiv:2610.40636v1 · Table 2: · CC BY 4.0

However, probes trained to read out real future features did not hold up on the models’ own predicted features. When the same probe was applied unchanged to native predictions, error rose from 0.182 to 0.246 in normalized depth MAE, and retraining the probe improved it only to 0.223.

Even when goal information was supplied, action predictions rarely improved progress. This suggests a gap between a model’s sensitivity to goals and its actionable utility.

Together, these findings demonstrate that visual foundation models encode multiple spatial layers, but the fidelity of persistence and action planning remains uneven and vulnerable to the prediction pipeline.

Source paper

Probing Spatial State Representations in Visual Foundation Models

Computer VisionArtificial Intelligence
View on arXiv