Language models know a lot about places, but they reason poorly over space and are unreliable when asked about a location from coordinates alone. The usual fix is to supply the relevant data in the prompt. This paper examines what happens next. Once the facts are in front of the model, does it use them, or does it override them with its own prior about the place?
The paper

How faithfulness is measured
Answers are split into atomic claims. Each claim is labelled by its source, the brief or the model’s own knowledge, and by its correctness. This gives four categories: brief-true, brief-false, recall and hallucination.
Each conversation then ends with a trap. The user casually asserts something that sounds right but that the brief contradicts. In Chicago, that supermarkets are an easy walk away, in a catchment that has none. In Paris, that a district with 26,800 residents per km² is a calm, low-density corner. In Hanoi, that hardly anyone lives in the Old Quarter, where the brief counts over 7,000 residents within a 300 m walk.
Sixteen configurations from Qwen3, Gemma 3, Gemma 4 and Llama 3.x, from 1B to 14B parameters, are each run ten times per case: 1,440 responses and about 22,000 labelled claims.
Main findings
- Family matters more than size. Out of 30, Gemma 4 scores 22.5 to 23 on trap resistance and Qwen3 10 to 20.5. Gemma 3 scores 1 to 4 and Llama 2 to 6.5. The 1.7B Qwen3 outscores the 12B Gemma 3.
- Reading and defending are different skills. Several models report every figure correctly, then drop them the moment the user disagrees.
- The more plausible the premise, the weaker the resistance: 65% in Chicago, 43% in Paris, 14% in Hanoi.
- Thinking mode makes answers 2.6 to 4.6 times slower without a consistent gain in grounding.
- Outcomes change between identical runs, so a single generation is one draw, not a verdict.
Six-axis profile per model. Use the buttons to switch between pooled and per-city profiles; hover a point for its value.
Why it matters
References and Links
- Paper (preprint): https://arxiv.org/abs/2609.39437
- Code and data: https://github.com/perezjoan/NSCR-LLM
- Archived release v1.0.0: https://doi.org/10.5281/zenodo.23056312
Table of contents
