Module 3 is the inference core of the SAGAI (Streetscape Analysis with Generative Artificial Intelligence) framework. Its role is to transform large collections of street-level images into structured, quantitative outputs using vision–language models (VLMs), enabling systematic streetscape analysis and subsequent geospatial aggregation.

With SAGAI v1.1, Module 3 has been released in a new major version (Module 3 v2.0) that introduces a fully standardized and maintenance-safe inference architecture. This update reflects both the maturation of multimodal model ecosystems and the need for long-term reproducibility in large-scale urban analysis pipelines.

Earlier iterations of Module 3 were developed during a period of rapid evolution in both LLaVA research codebases and execution environments such as Google Colab. As multimodal models transitioned toward Transformers-native implementations distributed via Hugging Face, assumptions embedded in earlier hybrid workflows became increasingly difficult to sustain.

Module 3 v2.0 addresses this evolution by aligning the entire inference pipeline with official Hugging Face multimodal APIs. Model loading, prompt formatting, image–text fusion, and generation are now handled through maintained and versioned components, ensuring compatibility across environments, models, and future updates.

This document details the architectural context motivating the update, the design choices behind the refactored inference engine, and the rationale for releasing Module 3 v2.0 as a long-term, stable component of SAGAI v1.1.

The initial implementation of Module 3 (SAGAI v1.0) relied on a hybrid architecture that mixed two incompatible sources of LLaVA code, combined with a rapidly evolving execution environment in Google Colab. This design choice made the pipeline fragile and ultimately unsustainable.

First, the pipeline simultaneously depended on the LLaVA GitHub repository (haotian-liu/LLaVA) and on Hugging Face–hosted model checkpoints. The GitHub repository is a research-oriented codebase under active development. Its internal APIs, class structures, and utilities evolve rapidly and are not version-locked. Constructors, module paths, and helper functions may change or disappear without notice, and the repository is not designed to maintain backward compatibility across releases.

At the same time, pretrained model weights were downloaded from Hugging Face. These checkpoints follow the Transformers-native multimodal format, using Hugging Face–specific configuration files, processors, and model classes (e.g., LlavaNextForConditionalGeneration, AutoProcessor, and chat templates). This architecture is fundamentally different from the internal design assumed by the GitHub LLaVA code, which relies on custom token insertion, internal vision tower management, and non-Transformers abstractions.

As a result, the pipeline operated in a structural mismatch: GitHub code expected architectural fields, model attributes, and tokenizer behavior that were not present in Hugging Face checkpoints, while Hugging Face checkpoints expected model wrappers and configuration logic that the GitHub code did not provide.

This fragility was exposed when Google Colab upgraded its backend environment in early 2025. Major changes included Python 3.12, NumPy ≥ 2.0 (introducing ABI-breaking changes for compiled extensions), newer PyTorch releases (≥ 2.2), and updated system libraries. These updates caused widespread failures in binary dependencies and research codebases that were not aligned with the new runtime.

In practice, this led to errors such as NumPy ABI incompatibilities, PyTorch extension failures, missing or renamed modules, and import errors in LLaVA GitHub utilities. Because the pipeline depended on both unstable research code and binary-sensitive extensions, even minor environment updates were sufficient to break execution.

Module 3 has been fully refactored to remove any dependency on the original LLaVA GitHub repository. The inference pipeline now relies exclusively on Hugging Face–native LLaVA models and APIs, ensuring long-term stability and compatibility with evolving software environments.

In the previous architecture, the script depended on cloning the LLaVA GitHub repository, installing it in editable mode, and importing internal modules (llava.*). Prompts were manually assembled using LLaVA-specific multimodal tokens (e.g., <im_start>, <image>), custom separators, and internal utilities. Image tokens and embeddings were explicitly inserted into the prompt, tightly coupling the forward pass to a specific implementation of the LLaVA codebase. As a result, updates to Google Colab, PyTorch, NumPy, or the LLaVA repository frequently introduced breaking changes.

The current implementation removes all such dependencies. Prompt formatting and multimodal input construction are now handled entirely through Hugging Face abstractions. Prompts are formatted using processor.apply_chat_template(), while images and text are combined using processor(images=…, text=…). Image embedding alignment, multimodal token placement, and chat formatting are fully managed by the Hugging Face processor and model configuration. Inference is performed using the standard model.generate() API, without any custom token handling or internal utilities.

This refactoring makes the SAGAI inference engine model-agnostic within the Hugging Face LLaVA ecosystem. The same forward pass is compatible with LLaVA-NeXT (v1.6), LLaVA-Interleave, LLaVA-OneVision, and future Hugging Face LLaVA releases that expose a processor and chat template. Switching between models or architectures requires only changing the model_id, with no modification to prompt logic or inference code.

To ensure reliable downstream analysis, Module 3 also includes a dedicated numeric output stabilization step. After decoding the model response, any prompt echoes or metadata—including residual [INST] … [/INST] segments—are removed. The final output is parsed using a simple regular expression to retain only numeric values (e.g., 0, 1, 2, 1.5). This guarantees clean, machine-readable outputs and a stable CSV format across all supported models.

Model loading has been simplified and standardized using Hugging Face–approved APIs. Both the processor and the model are instantiated directly from Hugging Face model cards via from_pretrained, with optional 4-bit quantization enabled through load_in_4bit=True. This eliminates the need for manual vision-tower initialization, deprecated classes, or custom C++ operators, and avoids common incompatibilities related to PyTorch, CUDA, or NumPy upgrades in Google Colab. Official Hugging Face code paths ensure that pretrained weights are always matched with the correct implementation.

Optional authentication using a Hugging Face access token is supported to avoid rate limits and improve download reliability when working with large checkpoints, though public models remain accessible without authentication.

Overall, this refactoring significantly improves robustness, reproducibility, and maintainability, while enabling systematic experimentation across multiple LLaVA variants and quantization settings within a unified inference framework.

The refactored inference system in Module 3 is designed as a long-term, maintenance-safe release. This is achieved by aligning the entire pipeline with Hugging Face’s officially supported multimodal APIs and model distribution mechanisms.

First, the new architecture is robust to Google Colab environment updates. All critical dependencies—Python (≥3.12), NumPy (≥2.0), PyTorch (2.x), CUDA wheels, and BitsAndBytes quantization—are now managed through Hugging Face Transformers and its dependency resolution. Because the model code, processor logic, and quantization pathways are maintained upstream, updates to Colab or its underlying libraries no longer break the inference pipeline. As long as Hugging Face continues to support the model card, the code remains functional without manual intervention.

Second, the system relies exclusively on official Hugging Face–maintained components. Core classes such as LlavaNextForConditionalGeneration, LlavaNextProcessor, chat templates, and multimodal preprocessing logic are all part of the Transformers library. These components are actively maintained, versioned, and tested by Hugging Face, providing a level of stability and backward compatibility that is not guaranteed when relying on research repositories or development branches.

Third, the new setup significantly improves reproducibility. Each run explicitly references a fixed Hugging Face model checkpoint via the model_id, ensuring that the same weights, architecture, and prompt template are used across sessions and machines. In addition, generation parameters (sampling strategy, temperature, nucleus sampling, and output length) are explicitly defined, enabling consistent and repeatable results across runs.

Fourth, the architecture is easy to extend and experiment with. Switching between different LLaVA variants now requires changing a single configuration line (model_id). The same inference code supports LLaVA 1.5 models, LLaVA-NeXT (v1.6), Interleave models, OneVision models, and larger checkpoints (e.g., 13B or 34B), including variants based on Mistral, Vicuna, Qwen, or Yi backbones. No changes to prompt construction or forward-pass logic are required.

Finally, the multimodal pipeline is now cleanly abstracted and internally consistent. Hugging Face handles all low-level details, including image preprocessing, chat formatting, positional embeddings, image sequence length management, and attention masking. This eliminates a large class of subtle bugs related to tensor alignment and multimodal token placement, while ensuring that the vision and language components remain synchronized across model updates.

• Streetscape Analysis with Generative AI (SAGAI) on Github with v1.1 update. https://github.com/perezjoan/SAGAI

• Perez, J and Fusco, G. (2025) Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063, 18p. Available at: https://www.sciencedirect.com/science/article/pii/S1195103625000199

CSR, or Complete Spatial Randomness, is a theoretical model used in spatial statistics to describe a random distribution of points in space. Essentially, in a CSR process, points are distributed randomly across the study area, with no clustering or spatial patterns. Each point is independent of the others and has an equal chance of occurring at any location within the study area.

When comparing observed spatial patterns to CSR, it is important to note that if the observed pattern exhibits a mean nearest neighbor distance that is significantly smaller than the mean nearest neighbor distance of a random distribution (CSR), it suggests clustering. In other words, this means that points in the observed pattern are closer to each other on average than would be expected under a random process. Conversely, if the mean nearest neighbor distance of the observed pattern is significantly larger than that of a random distribution, it suggests dispersion. Consequently, this indicates that points in the observed pattern are farther apart on average than would be expected under a random process.

Calculating Nearest Neighbor Distance (NND) and comparing it with CSR can be useful in various fields where spatial patterns play a crucial role, such as ecology (distribution of species) or urban planning. Let’s see together how to calculate a nearest neighbor distance from a given point pattern and compare it to a random distribution (CSR).

To perform this analysis, we will use two building layers. You can obtain OSM building data from various locations worldwide using the Overpass Turbo tool. The sample data used in this demonstration consists of two OSM building layers compiled into a Geopackage file.

This sample data pertains specifically to the Italian cities of Grosseto and Sinalunga, each containing approximately 7500 buildings.

import geopandas as gpd
import numpy as np
from scipy.spatial.distance import cdist
import matplotlib.pyplot as plt

# Read the layers for the two Italian cities
gdf_1 = gpd.read_file("Italian_cities.gpkg", layer = "Grosseto")
gdf_1 = gdf_1.to_crs(epsg=6875)
gdf_2 = gpd.read_file("Italian_cities.gpkg", layer = "Sinalunga")
gdf_2 = gdf_2.to_crs(epsg=6875)

The provided code imports necessary libraries and extracts each layer from the GeoPackage. For detailed guidance on configuring your Python environment and managing libraries, you can consult this post. Additionally, if you seek information on GeoPackage usage and layer importation into Python, you can refer to this post.

# Create a figure with two subplots
fig, axs = plt.subplots(1, 2, figsize=(20, 10))

# Plot gdf_1 in the first subplot
gdf_1.plot(ax=axs[0], color='blue', edgecolor='black')
axs[0].set_title("Map View of Grosseto")
axs[0].set_xlabel("Longitude")
axs[0].set_ylabel("Latitude")

# Plot gdf_2 in the second subplot
gdf_2.plot(ax=axs[1], color='red', edgecolor='black')
axs[1].set_title("Map View of Sinalunga")
axs[1].set_xlabel("Longitude")
axs[1].set_ylabel("Latitude")

# Adjust layout
plt.tight_layout()

# Display the plot
plt.show()

This code snippet creates a figure with two subplots. Specifically, in the first subplot, the map view of Grosseto is plotted, while Sinalunga is shown in the second plot. At first glance, buildings in both maps look clustered. However, it is difficult to visually assess which pattern is more clustered or dispersed than the other.

# Calculate the centroids of gdf_1
gdf_1_points = gdf_1.centroid
points_geom = gpd.GeoDataFrame(geometry=gdf_1_points)
# Extract coordinates from the centroids
coordinates = np.array([[points.coords[0][0], points.coords[0][1]] for points in points_geom.geometry])

# Calculate the bounding box of the GeoDataFrame
min_x, min_y, max_x, max_y = gdf_1.total_bounds
np.random.seed(123)  # Set seed for reproducibility

# Generate random points within the bounding box
random_points_x = np.random.uniform(min_x, max_x, 1000)
random_points_y = np.random.uniform(min_y, max_y, 1000)
# Create a GeoDataFrame from the random points
random_points_geom = gpd.GeoDataFrame(geometry=gpd.points_from_xy(random_points_x, random_points_y))

# Extract coordinates of the random points
coordinates_random = np.array([[point.x, point.y] for point in random_points_geom.geometry])

# Plot the random points
ax = random_points_geom.plot(markersize=1)
ax.set_title('Random Points Plot')
plt.show()

# Function to calculate nearest neighbor distances
def nearest_neighbor_distances(points):
    # Calculate pairwise distances between points
    distances = cdist(points, points)
    np.fill_diagonal(distances, np.inf)  # Exclude distance to itself
    # Calculate nearest neighbor distances
    nearest_distances = np.min(distances, axis=1)
    return nearest_distances

# SD on the nearest neighbor distances
def nearest_neighbor_sd(points):
    # Calculate nearest neighbor distances
    nearest_distances = nearest_neighbor_distances(points)
    # Calculate standard deviation of nearest neighbor distances
    sd = np.std(nearest_distances)
    return sd

The nearest_neighbor_distances function calculates the nearest neighbor distances for a given set of points in a two-dimensional space. Specifically, it computes the pairwise distances between all points using the cdist function from SciPy’s spatial module. After that, by filling the diagonal of the distance matrix with infinity, it excludes distances to the points themselves. Consequently, it finds the minimum distance for each point, representing its nearest neighbor distance. Finally, the function returns an array containing the nearest neighbor distance for each point. This array is then used by the nearest_neighbor_sd function, which computes the standard deviation of nearest neighbor distances for a set of points.

mean_observed_nn = np.mean(nearest_neighbor_distances(coordinates))
mean_random_nn = np.mean(nearest_neighbor_distances(coordinates_random))
std_random_nn = nearest_neighbor_sd(coordinates_random)

# Calculate threshold of within CSR (-1/2 +1/2 sd)
lower_bound = mean_random_nn - 0.5 * std_random_nn
upper_bound = mean_random_nn + 0.5 * std_random_nn

# Assess clustering or dispersion
if mean_observed_nn < lower_bound:
    print(f"The observed point pattern (mean: ) is significantly clustered compared to the random pattern (mean: ).")
elif mean_observed_nn > upper_bound:
    print(f"The observed point pattern (mean: ) is significantly dispersed compared to the random pattern (mean: ).")
else:
    print(f"The observed point pattern (mean: ) is consistent with CSR (mean random: ).")

The observed point pattern (mean: 24.73) is significantly clustered compared to the random pattern (mean: 47.95).

The code calculates the mean nearest neighbor distances for both an observed set of coordinates and a randomly generated set. Additionally, it computes the standard deviation (sd) of the nearest neighbor distances for the randomly generated set. These values are then used to establish thresholds for Complete Spatial Randomness (CSR). To detect clustered and dispersed patterns, lower and upper bounds are set using statistical principles: approximately 68% of data points fall within one standard deviation of the mean in a normal distribution. Hence, these bounds are established using half the standard deviation away from the mean of the random distances.

It is then possible to assess whether the observed point pattern exhibits clustering, dispersion, or adherence to CSR. If the mean nearest neighbor distance of the observed pattern falls below the lower bound, it indicates significant clustering compared to the random pattern. Conversely, if it exceeds the upper bound, it suggests significant dispersion. Otherwise, if the mean observed distance lies within the bounds, the point pattern is deemed consistent with CSR. The buildings of the city of Grosseto exhibit a significantly clustered pattern as compared to a random pattern. But how another city like Sinalunga compares to Grosseto?

# Calculate the centroids of the GeoDataFrame
gdf_2_points = gdf_2.centroid
# Create a GeoDataFrame from the centroids
points_geom2 = gpd.GeoDataFrame(geometry=gdf_2_points)
# Extract coordinates from the centroids
coordinates2 = np.array([[points.coords[0][0], points.coords[0][1]] for points in points_geom2.geometry])
mean_observed_nn2 = np.mean(nearest_neighbor_distances(coordinates2))
# Compare mean nearest neighbor distances
if mean_observed_nn2 < mean_observed_nn:
    print("The values for the second city are more clustered.")
    print(f"Mean nearest neighbor distance for observed points in the first city: ")
    print(f"Mean nearest neighbor distance for observed points in the second city: ")
else:
    print("The values for the second city are more dispersed.")
    print(f"Mean nearest neighbor distance for observed points in the first city: ")
    print(f"Mean nearest neighbor distance for observed points in the second city: ")

The values for the second city are more clustered.
Mean nearest neighbor distance for observed points in the first city: 24.73
Mean nearest neighbor distance for observed points in the second city: 19.96

The code compares the mean nearest neighbor distances between the first city and the second city. If the mean distance for the second city is less than that of the first city, indicating greater clustering, a message is printed to convey this observation along with the respective mean distances for both cities. Conversely, if the mean distance for the second city is greater, suggesting more dispersion, another message is printed with the corresponding mean distances for both cities. The results suggest that the buildings of Sinalunga are more clustered than the ones in Grosseto.

# Plot histograms of nearest neighbor distances
plt.figure(figsize=(10, 6))
plt.hist(nearest_neighbor_distances(coordinates), bins=50, color='blue', alpha=0.5, label='Grosseto')
plt.hist(nearest_neighbor_distances(coordinates2), bins=50, color='green', alpha=0.5, label='Sinalunga')
plt.hist(nearest_neighbor_distances(coordinates_random), bins=50, color='red', alpha=0.5, label='Random CSR')
plt.xlabel('Nearest Neighbor Distance')
plt.ylabel('Frequency')
plt.title('Nearest Neighbor Distance Distribution')
plt.legend()
plt.show()

• Point Pattern Analysis – Geographic Data Science
• Yuan, Y., Qiang, Y., Bin Asad, K., and Chow, T. E. (2020). Point Pattern Analysis. The Geographic Information Science & Technology Body of Knowledge (1st Quarter 2020 Edition), John P. Wilson (ed.). DOI: 10.22224/gistbok/2020.1.13.
• Usui, H and Perez, J. (2020) « Are patterns of vacant lots random? Evidence from empirical spatiotemporal analysis in Chiba prefecture, east of Tokyo », Environment and Planning B: Urban Analytics and City Science, 49(3). Read more

In this guide you’ll find clear instructions on setting up Python with Anaconda for spatial analysis. Then, we’ll cover installing Python alongside Anaconda and adding essential dependencies like GeoPandas via the Anaconda Prompt. Lastly, we’ll explore using the Jupyter Notebook for practical application. By the end, you’ll be ready to start your journey in Python-based spatial analysis, AI, and machine learning.

Anaconda is a powerful tool for managing Python environments. Indeed, Anaconda simplifies package management through its integrated package manager, conda. In addition, it comes bundled with a set of pre-installed libraries commonly used in data science and spatial analysis, such as NumPy, Pandas or Matplotlib. Therefore, this eliminates the need for manual installation and ensures immediate access to these libraries. Anaconda is also cross-platform compatible (available for Windows, macOS, and Linux) thus providing consistent Python environments across different operating systems. Furthermore, it seamlessly integrates with popular development environments like Jupyter Notebook, known amongst other things for its user-friendly interface. Visit the Anaconda website and download the Anaconda distribution related to your operating system (Windows, macOS, or Linux). Once Anaconda is installed, you’ll have access to the Anaconda Navigator, Anaconda Prompt, and other useful tools for managing Python environments.

Even if the Python distribution installed by Anaconda comes with numerous pre-installed libraries, additional dependencies will be required to enable performing advanced manipulations, and this is especially true for manipulating spatial data. So, let’s install the following three dependencies to manipulate spatial data : GeoPandas : allow spatial operations on geometric types ; Pyogrio : interoperability between spatial data formats and Contextily : retrieve tile maps from the internet.
Then, on Windows, click on the Start Menu and type “Anaconda Prompt” in the search bar and open it. This will open a new command prompt window with Anaconda enabled. Within the command prompt, run the following commands one by one to install the aforementioned dependencies.

conda install -c conda-forge geopandas 
conda install -c conda-forge pyogrio 
conda install -c conda-forge contextily

If you’ve previously installed the libraries, executing these lines will not only update the libraries themselves but also their dependencies. Additionally, any other dependencies not encompassed within the Anaconda distribution of Python can be installed using the same commands. For instance, for machine learning purposes, you can install ‘XGBoost’ (eXtreme Gradient Boosting) using these lines as well.

Now that you’ve installed Python and the required dependencies, you can open the Anaconda Navigator and launch the Jupyter Notebook from the Home tab, as follows:

Jupyter Notebook is an open-source web application that allows users to create and share documents containing live code, visualizations, and narrative text. Upon launching the notebook, you’ll be directed to the Jupyter Notebook explorer in your default web browser, where you can create folders and new notebooks with a simple right-click action, as demonstrated below.

Afterward, you can rename your notebook (indicated by arrow 1 below). In a notebook, users can compose and execute code in segmented blocks. The figure below illustrates one such block (indicated by arrow 2 below).

Your notebook is automatically linked to the files within the folders where it resides. Let’s add some data to this folder and import it into the notebook (or Python environment). Begin by downloading the dataset provided below. This dataset comprises a GeoPackage file containing two building layers corresponding to two small cities in Italy: Grosseto and Sinalunga. For further insight related the GeoPackage format, you can refer to this post. Then, once you have downloaded, place it in the same directory as your notebook.

Then, you can run the code below to import a layer in Python using GeoPandas. In this example, we are importing a layer of building related to the italian city of Grosseto.

Finally, you can run the code below to plot the building layer with a basemap from OSM using contextily.

The map above was generated using the Python code provided — demonstrating how to overlay geospatial data with a custom basemap using contextily. Fell free to provide feedbacks on our blog posts by contacting us.