Teaching AI to read a map
MapTrace addresses a major weakness in multimodal language models: recognizing objects on maps is easier for them than understanding connectivity, obstacles, and valid routes. The authors propose a synthetic-data pipeline that generates maps, identifies walkable areas, constructs navigation graphs, and verifies computed paths with AI critics. They report releasing 2 million map question-answer pairs and show that fine-tuning on a much smaller subset improves route tracing on unseen real-world maps. ## The Challenge: Weak Spatial Grounding - MLLMs may recognize locations and objects in an image but still draw routes through walls, buildings, enclosures, or shops. - Effective navigation requires understanding: - Which regions are traversable - How paths connect - That routes are ordered sequences of connected points - The geometric and topological relationships between map features - Existing image-text training rarely teaches this “spatial grammar.” - Manual pixel-level route annotation would be expensive and difficult to scale. - Many useful maps of malls, museums, and theme parks are proprietary, limiting access to real-world training data. ## A Scalable Synthetic-Data Pipeline MapTrace uses generative AI to create diverse maps and automatically produce valid route annotations. ### Generating Diverse Maps - An LLM creates detailed prompts for environments such as: - Zoos with interconnected habitats - Shopping malls with food courts - Fantasy theme parks with themed areas - A text-to-image model renders the prompts as map images. - This approach provides control over map diversity and complexity. ### Identifying Walkable Areas with a Mask Critic - Pixels are clustered by color to produce candidate masks representing possible walkways. - An MLLM reviews each mask alongside the original map. - The “Mask Critic” rejects masks that do not represent realistic, connected traversable regions. - Accepted areas may include sidewalks, crosswalks, and pedestrian paths. ### Converting Maps into Navigation Graphs - The selected traversable mask is converted into a pixel-based graph. - Walkway intersections become nodes, while connected stretches become edges. - This graph captures the map’s connectivity and enables computational route planning. ### Generating and Validating Routes - Thousands of random start and end points are sampled for each map. - Dijkstra’s algorithm computes the shortest path between each pair. - A “Path Critic” checks the overlaid route to ensure it: - Stays within traversable regions - Avoids obstacles - Follows a logical human route - Routes approved by the critic become training examples. ## Dataset and Evaluation - The pipeline generated 2 million annotated map question-answer pairs. - The authors note that generated maps sometimes contain incorrect text, but the study focuses primarily on path fidelity. - They fine-tuned models including Gemma 3 27B and Gemini 2.5 Flash on 23,000 generated paths. - Performance was evaluated on MapBench, which contains unseen real-world maps. - Route accuracy was measured using normalized dynamic time warping (NDTW), which compares predicted and reference coordinate sequences while accounting for differences in sampling and travel speed. - Lower NDTW scores indicate closer agreement with the reference route. ## Conclusion The work suggests that targeted synthetic training data can teach MLLMs map-based spatial reasoning that is largely missing from general pretraining. The released dataset and pipeline provide a foundation for improving visual navigation, while better image-generation models should reduce remaining typography and rendering artifacts.
Read original(opens in new tab)