Teaching AI to read a map

MapTrace addresses a major weakness in multimodal language models: recognizing objects on maps is easier for them than understanding connectivity, obstacles, and valid routes. The authors propose a synthetic-data pipeline that generates maps, identifies walkable areas, constructs navigation graphs, and verifies computed paths with AI critics. They report releasing 2 million map question-answer pairs and show that fine-tuning on a much smaller subset improves route tracing on unseen real-world maps.

The Challenge: Weak Spatial Grounding

• MLLMs may recognize locations and objects in an image but still draw routes through walls, buildings, enclosures, or shops.
• Effective navigation requires understanding:
  • Which regions are traversable
  • How paths connect
  • That routes are ordered sequences of connected points
  • The geometric and topological relationships between map features
• Existing image-text training rarely teaches this “spatial grammar.”
• Manual pixel-level route annotation would be expensive and difficult to scale.
• Many useful maps of malls, museums, and theme parks are proprietary, limiting access to real-world training data.

A Scalable Synthetic-Data Pipeline

MapTrace uses generative AI to create diverse maps and automatically produce valid route annotations.

Generating Diverse Maps

• An LLM creates detailed prompts for environments such as:
  • Zoos with interconnected habitats
  • Shopping malls with food courts
  • Fantasy theme parks with themed areas
• A text-to-image model renders the prompts as map images.
• This approach provides control over map diversity and complexity.

Identifying Walkable Areas with a Mask Critic

• Pixels are clustered by color to produce candidate masks representing possible walkways.
• An MLLM reviews each mask alongside the original map.
• The “Mask Critic” rejects masks that do not represent realistic, connected traversable regions.
• Accepted areas may include sidewalks, crosswalks, and pedestrian paths.

Converting Maps into Navigation Graphs

• The selected traversable mask is converted into a pixel-based graph.
• Walkway intersections become nodes, while connected stretches become edges.
• This graph captures the map’s connectivity and enables computational route planning.

Generating and Validating Routes

• Thousands of random start and end points are sampled for each map.
• Dijkstra’s algorithm computes the shortest path between each pair.
• A “Path Critic” checks the overlaid route to ensure it:
  • Stays within traversable regions
  • Avoids obstacles
  • Follows a logical human route
• Routes approved by the critic become training examples.

Dataset and Evaluation

• The pipeline generated 2 million annotated map question-answer pairs.
• The authors note that generated maps sometimes contain incorrect text, but the study focuses primarily on path fidelity.
• They fine-tuned models including Gemma 3 27B and Gemini 2.5 Flash on 23,000 generated paths.
• Performance was evaluated on MapBench, which contains unseen real-world maps.
• Route accuracy was measured using normalized dynamic time warping (NDTW), which compares predicted and reference coordinate sequences while accounting for differences in sampling and travel speed.
• Lower NDTW scores indicate closer agreement with the reference route.

Conclusion

The work suggests that targeted synthetic training data can teach MLLMs map-based spatial reasoning that is largely missing from general pretraining. The released dataset and pipeline provide a foundation for improving visual navigation, while better image-generation models should reduce remaining typography and rendering artifacts.