Visual AI Models Show Uneven Ability to Maintain Spatial State

What the study found

In an arXiv preprint, Baiqiao Yin and colleagues examined ten representations using probes that read features from models whose parameters were held fixed. The models received streams of first-person observations of static indoor environments. The authors assessed four levels of capability: spatial structure, object persistence, predictive spatial updating and actionability.

The authors report that performance varied across tasks, network layers and the length of the observation history. Strong readout of the visible scene did not ensure accurate tracking of objects beyond view or retrieval of future features. Predicted future features retained recoverable spatial structure, but a spatial reader trained on real features did not transfer unchanged to those predictions.

Predicted actions changed when the goal changed, yet providing the correct goal did not reliably improve progress. The authors also report that object persistence and actionability built on camera-pose readout, while feature prediction built on visible-scene readout.

Why it matters

The findings suggest that decoding information about a scene is only one test of spatial competence. A representation may also need to retain information beyond the current view, update it as the observer moves and make it useful for goal-directed action. The authors used matched controls to distinguish readout accuracy from reliance on visual history and motion.

Limits and unknowns

The evidence described in the abstract concerns static indoor environments; it does not establish performance in changing scenes or other settings. The abstract does not name the ten representations, give numerical results or describe the probe designs, controls and action tests in enough detail to assess their implementation. The work is a preprint.