MindTopo reveals VLMs’ spatial reasoning abilities

Microsoft Research has developed MindTopo, a new benchmark designed to assess whether multimodal large language models (VLMs) can understand and manipulate topological relationships such as connectivity, enclosure, order, separation, and knots. Unlike traditional spatial evaluations focusing on Euclidean properties like distance and size, MindTopo emphasizes structural relationships that persist despite deformation or movement, reflecting a more human-like spatial understanding.

The benchmark categorizes tasks into five key areas: continuity, separation, order, enclosure, and knots. These are tested through reasoning tasks, where models analyze static scenes and answer questions about their topological properties, and planning tasks, where models interact with simulated environments to create or alter these relationships through actions. The environments are generated from controlled simulators, providing precise ground truth and adjustable difficulty levels.

Evaluation across various proprietary and open-source models shows that while models perform reasonably well on static reasoning tasks, their ability to maintain and manipulate topological relationships during interactive tasks remains limited. Performance consistently falls below human levels, especially when multiple actions are involved, and errors often stem from perception failures or loss of track of relationships during scene changes.

The research also explored whether visual generation tools like image and video synthesis could aid models in understanding topology. Results indicate that while static images can sometimes help, dynamic sequences often introduce errors, especially when topology changes or constraints are violated.

The findings underscore the importance of developing models that explicitly encode topological states or incorporate world models capable of preserving structural relationships over time. Such advancements are crucial for robotics, interactive environments, and AI systems that require reliable spatial reasoning and decision-making.

Microsoft Research emphasizes that closing this gap in topological understanding could significantly enhance AI's ability to operate effectively in complex, real-world scenarios where maintaining structural relationships is essential.