“Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning” separates two things that are usually measured together: being correct about the world, and being faithful to the context supplied.

To do so, it fixes the context. For a point on a map, a short “spatial brief” is computed from OpenStreetMap and GHS-POP over the area reachable on foot: residents, density, buildings, street length, points of interest by category. The model receives finished numbers and only has to interpret them. Because the brief is known, every claim in an answer can be checked against it.

The study applies this to three contrasting places, each tied to a theme in urban research that the models are likely to have met in training. The first is a neighbourhood on Chicago’s West Side, where 7,128 residents live within an 800 m walk that contains no supermarket or grocery store. The second is the historic core of Paris near Le Marais, with about 26,800 residents per km² and 971 points of interest within 400 m. The third is Hanoi’s Old Quarter, a district known for shops and tourism that nonetheless holds over 7,000 residents within a 300 m walk. The three are ordered by difficulty: Chicago asks the model to notice an absence, Paris to interpret a number, and Hanoi to hold a high resident count against a strong reputation.

Scoring answers only against the truth would rate many of these models as reliable. For decision-support uses, the practical lesson is that a larger model that yields to its user is a worse choice than a smaller one that holds to the data.

This matters most in geography, where what a model believes about a place is often a reputation rather than a fact: the Marais as a quiet historic quarter, Hanoi’s Old Quarter as shops and tourists rather than residents. Planners, analysts and public agencies are starting to put local data in front of these models precisely to get past such assumptions. If the model drops that data as soon as a confident user repeats the stereotype, the retrieval step has achieved nothing.

The paper also shows that this risk cannot be read off a model card. Parameter count does not predict it, a reasoning mode does not fix it, and one good answer does not rule it out. It has to be measured, on the data and the questions the model will actually face.