SAGAI (Streetscape Analysis with Generative Artificial Intelligence) is an open-source workflow for scoring and mapping street-level urban environments using vision-language models and open geospatial data. Since its initial release, SAGAI has been structured as a set of independent Colab notebooks, one per pipeline stage, each relying on its own dependencies and documentation.

SAGAI v2.0 is a major release that consolidates the entire pipeline into a single notebook and replaces the custom inference code with the UVLM package. Where previous versions were tied to a single LLaVA checkpoint with handwritten inference logic, SAGAI v2.0 delegates all vision-language model loading, prompting, and evaluation to UVLM’s unified interface. This makes the scoring engine model-agnostic: users can select from 11 VLM checkpoints spanning the LLaVA-NeXT and Qwen2.5-VL families, compare their performance on identical tasks, and benefit from features such as consensus validation, reasoning traces, and truncation diagnostics; all within the same notebook.

Beyond the inference engine, v2.0 introduces structural and functional changes across the entire pipeline: a unified six-block architecture, interactive HTML mapping via Folium, view-direction filtering for aggregation, and the ability to load an existing polygon as a study area boundary instead of defining a bounding box manually.

This post details the architectural changes, the UVLM integration, and the new features introduced in SAGAI v2.0.

Previous SAGAI releases were organized as four independent Colab notebooks — one for street sampling, one for image retrieval, one for VLM inference, and one for aggregation and mapping — each accompanied by a separate NOTICE file documenting its dependencies and usage. This modular design was useful for development but introduced friction in practice: users had to manage file paths between notebooks, track four separate environments, and consult multiple documentation files.

SAGAI v2.0 merges all four stages into a single notebook (SAGAI.ipynb) structured as six sequential blocks. The pipeline flows from study area definition through street sampling, image downloading, VLM scoring, and mapping, with all intermediate data passed directly between blocks in the same runtime session. The separate per-module NOTICE files and the standalone requirements file (requirements_sagai_module_3_v1-0.txt) have been removed — dependency management is now handled automatically by the UVLM package installation.

In previous versions, the study area was defined exclusively by a bounding box in WGS84 coordinates. SAGAI v2.0 retains this option but adds the ability to draw your own polygon or to load an existing polygon; for example, a GeoPackage representing a neighborhood, municipality, or custom boundary. When a polygon is provided, the street sampling step extracts the OpenStreetMap network within that geometry rather than a rectangular extent. This makes it straightforward to work with irregular administrative boundaries or user-defined study zones without manually computing bounding coordinates.

The most significant change in SAGAI v2.0 is the replacement of the inline inference code with the UVLM package. In previous versions, Blocks 3 through 5 contained custom code for loading a single LLaVA checkpoint, constructing prompts, running inference, and parsing outputs. This logic was tightly coupled to one model architecture and required manual maintenance when Hugging Face APIs or model formats changed.

SAGAI v2.0 installs UVLM directly from its GitHub repository at the start of the notebook. All model loading, prompt formatting, inference execution, response parsing, and batch processing are delegated to UVLM’s API. The inline inference code has been entirely removed.

Through UVLM, SAGAI v2.0 supports 11 VLM checkpoints across two model families:

• LLaVA-NeXT — Mistral 7B, Vicuna 7B, Vicuna 13B, 34B, LLaMA3 8B, 72B, 110B
• Qwen2.5-VL — 3B Instruct, 7B Instruct, 32B Instruct, 72B Instruct

UVLM’s dual-backend abstraction automatically detects the model family and routes inference to the correct pipeline — LlavaNextProcessor for LLaVA models, AutoProcessor with process_vision_info for Qwen models — so users switch between architectures by changing a single model selection, with no modification to the rest of the notebook.

Quantization is handled through UVLM’s built-in support for 4-bit, 8-bit, and FP16 precision via BitsAndBytes. Models up to 34B parameters can run on a single Colab GPU (T4 or A100) with 4-bit quantization.

UVLM provides a widget-based prompt builder that SAGAI v2.0 exposes directly in the notebook. Users can define up to 10 analysis tasks per run, each with its own prompt, response type (numeric, category, boolean, or text), and label. This replaces the previous approach of selecting from a small set of hardcoded tasks (T1, T2, T3) or manually editing prompt strings in the code.

Tasks are configured interactively before execution and applied uniformly across all images in the batch. Each task produces its own column in the output CSV file.

SAGAI v2.0 inherits UVLM’s consensus validation mechanism. Each analysis task can be run 2 to 5 times per image, and the final score is determined by majority voting across the repeated inferences. NA values from failed parses are filtered before voting. An agreement ratio is recorded alongside the final score, providing a built-in measure of prediction reliability without any external validation step.

UVLM supports two approaches to chain-of-thought (CoT) reasoning, both available in SAGAI v2.0. Users can write task prompts that explicitly request step-by-step reasoning and adjust the token budget (up to 1,500 tokens) to allow the model sufficient generation space. Alternatively, a built-in CoT reference mode can be enabled per task, which triggers a standardized reasoning template with a fixed 1,024-token budget. In both cases, the reasoning trace is stored in a dedicated column in the output CSV for inspection.

Truncation detection is performed automatically after every inference call. The exact number of generated tokens is compared against the token limit, and truncated responses are flagged in per-task CSV columns. This allows users to identify tasks where the token budget is insufficient without post-hoc analysis.

Previous SAGAI versions generated static thematic maps using Matplotlib. SAGAI v2.0 replaces these with interactive HTML maps built with Folium. Point-level and street-segment-level scores are rendered as interactive layers that can be panned, zoomed, and queried directly in the browser. This is particularly useful for exploratory analysis and for sharing results with collaborators who do not use GIS software.

Google Street View images are typically downloaded in multiple compass directions at each sampling point (e.g., front, back, left, right). In previous versions, all views were aggregated together when computing point- or street-level scores. SAGAI v2.0 introduces a view filter that allows users to select which directions to include in the aggregation — for example, scoring only left-side and right-side views to focus on building facades, or only front views to capture the pedestrian perspective along the street axis. This filter is applied at the aggregation stage and does not affect the scoring step itself.

The batch execution engine inherited from UVLM provides resume-safe processing with checkpoint saving every 3 images. If a Colab session is interrupted — due to a timeout, a runtime reset, or a connectivity issue — the notebook can be re-executed and will automatically skip already-processed images. New tasks added between runs trigger automatic CSV schema upgrading, so the output file grows incrementally without losing previous results.

• SAGAI v2.0 on GitHub: https://github.com/perezjoan/SAGAI
• UVLM on GitHub: https://github.com/perezjoan/UVLM
• Perez, J. and Fusco, G. (2025). Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063. https://www.sciencedirect.com/science/article/pii/S1195103625000199
• Perez, J. and Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893. https://arxiv.org/abs/2603.13893

Module 3 is the inference core of the SAGAI (Streetscape Analysis with Generative Artificial Intelligence) framework. Its role is to transform large collections of street-level images into structured, quantitative outputs using vision–language models (VLMs), enabling systematic streetscape analysis and subsequent geospatial aggregation.

With SAGAI v1.1, Module 3 has been released in a new major version (Module 3 v2.0) that introduces a fully standardized and maintenance-safe inference architecture. This update reflects both the maturation of multimodal model ecosystems and the need for long-term reproducibility in large-scale urban analysis pipelines.

Earlier iterations of Module 3 were developed during a period of rapid evolution in both LLaVA research codebases and execution environments such as Google Colab. As multimodal models transitioned toward Transformers-native implementations distributed via Hugging Face, assumptions embedded in earlier hybrid workflows became increasingly difficult to sustain.

Module 3 v2.0 addresses this evolution by aligning the entire inference pipeline with official Hugging Face multimodal APIs. Model loading, prompt formatting, image–text fusion, and generation are now handled through maintained and versioned components, ensuring compatibility across environments, models, and future updates.

This document details the architectural context motivating the update, the design choices behind the refactored inference engine, and the rationale for releasing Module 3 v2.0 as a long-term, stable component of SAGAI v1.1.

The initial implementation of Module 3 (SAGAI v1.0) relied on a hybrid architecture that mixed two incompatible sources of LLaVA code, combined with a rapidly evolving execution environment in Google Colab. This design choice made the pipeline fragile and ultimately unsustainable.

First, the pipeline simultaneously depended on the LLaVA GitHub repository (haotian-liu/LLaVA) and on Hugging Face–hosted model checkpoints. The GitHub repository is a research-oriented codebase under active development. Its internal APIs, class structures, and utilities evolve rapidly and are not version-locked. Constructors, module paths, and helper functions may change or disappear without notice, and the repository is not designed to maintain backward compatibility across releases.

At the same time, pretrained model weights were downloaded from Hugging Face. These checkpoints follow the Transformers-native multimodal format, using Hugging Face–specific configuration files, processors, and model classes (e.g., LlavaNextForConditionalGeneration, AutoProcessor, and chat templates). This architecture is fundamentally different from the internal design assumed by the GitHub LLaVA code, which relies on custom token insertion, internal vision tower management, and non-Transformers abstractions.

As a result, the pipeline operated in a structural mismatch: GitHub code expected architectural fields, model attributes, and tokenizer behavior that were not present in Hugging Face checkpoints, while Hugging Face checkpoints expected model wrappers and configuration logic that the GitHub code did not provide.

This fragility was exposed when Google Colab upgraded its backend environment in early 2025. Major changes included Python 3.12, NumPy ≥ 2.0 (introducing ABI-breaking changes for compiled extensions), newer PyTorch releases (≥ 2.2), and updated system libraries. These updates caused widespread failures in binary dependencies and research codebases that were not aligned with the new runtime.

In practice, this led to errors such as NumPy ABI incompatibilities, PyTorch extension failures, missing or renamed modules, and import errors in LLaVA GitHub utilities. Because the pipeline depended on both unstable research code and binary-sensitive extensions, even minor environment updates were sufficient to break execution.

Module 3 has been fully refactored to remove any dependency on the original LLaVA GitHub repository. The inference pipeline now relies exclusively on Hugging Face–native LLaVA models and APIs, ensuring long-term stability and compatibility with evolving software environments.

In the previous architecture, the script depended on cloning the LLaVA GitHub repository, installing it in editable mode, and importing internal modules (llava.*). Prompts were manually assembled using LLaVA-specific multimodal tokens (e.g., <im_start>, <image>), custom separators, and internal utilities. Image tokens and embeddings were explicitly inserted into the prompt, tightly coupling the forward pass to a specific implementation of the LLaVA codebase. As a result, updates to Google Colab, PyTorch, NumPy, or the LLaVA repository frequently introduced breaking changes.

The current implementation removes all such dependencies. Prompt formatting and multimodal input construction are now handled entirely through Hugging Face abstractions. Prompts are formatted using processor.apply_chat_template(), while images and text are combined using processor(images=…, text=…). Image embedding alignment, multimodal token placement, and chat formatting are fully managed by the Hugging Face processor and model configuration. Inference is performed using the standard model.generate() API, without any custom token handling or internal utilities.

This refactoring makes the SAGAI inference engine model-agnostic within the Hugging Face LLaVA ecosystem. The same forward pass is compatible with LLaVA-NeXT (v1.6), LLaVA-Interleave, LLaVA-OneVision, and future Hugging Face LLaVA releases that expose a processor and chat template. Switching between models or architectures requires only changing the model_id, with no modification to prompt logic or inference code.

To ensure reliable downstream analysis, Module 3 also includes a dedicated numeric output stabilization step. After decoding the model response, any prompt echoes or metadata—including residual [INST] … [/INST] segments—are removed. The final output is parsed using a simple regular expression to retain only numeric values (e.g., 0, 1, 2, 1.5). This guarantees clean, machine-readable outputs and a stable CSV format across all supported models.

Model loading has been simplified and standardized using Hugging Face–approved APIs. Both the processor and the model are instantiated directly from Hugging Face model cards via from_pretrained, with optional 4-bit quantization enabled through load_in_4bit=True. This eliminates the need for manual vision-tower initialization, deprecated classes, or custom C++ operators, and avoids common incompatibilities related to PyTorch, CUDA, or NumPy upgrades in Google Colab. Official Hugging Face code paths ensure that pretrained weights are always matched with the correct implementation.

Optional authentication using a Hugging Face access token is supported to avoid rate limits and improve download reliability when working with large checkpoints, though public models remain accessible without authentication.

Overall, this refactoring significantly improves robustness, reproducibility, and maintainability, while enabling systematic experimentation across multiple LLaVA variants and quantization settings within a unified inference framework.

The refactored inference system in Module 3 is designed as a long-term, maintenance-safe release. This is achieved by aligning the entire pipeline with Hugging Face’s officially supported multimodal APIs and model distribution mechanisms.

First, the new architecture is robust to Google Colab environment updates. All critical dependencies—Python (≥3.12), NumPy (≥2.0), PyTorch (2.x), CUDA wheels, and BitsAndBytes quantization—are now managed through Hugging Face Transformers and its dependency resolution. Because the model code, processor logic, and quantization pathways are maintained upstream, updates to Colab or its underlying libraries no longer break the inference pipeline. As long as Hugging Face continues to support the model card, the code remains functional without manual intervention.

Second, the system relies exclusively on official Hugging Face–maintained components. Core classes such as LlavaNextForConditionalGeneration, LlavaNextProcessor, chat templates, and multimodal preprocessing logic are all part of the Transformers library. These components are actively maintained, versioned, and tested by Hugging Face, providing a level of stability and backward compatibility that is not guaranteed when relying on research repositories or development branches.

Third, the new setup significantly improves reproducibility. Each run explicitly references a fixed Hugging Face model checkpoint via the model_id, ensuring that the same weights, architecture, and prompt template are used across sessions and machines. In addition, generation parameters (sampling strategy, temperature, nucleus sampling, and output length) are explicitly defined, enabling consistent and repeatable results across runs.

Fourth, the architecture is easy to extend and experiment with. Switching between different LLaVA variants now requires changing a single configuration line (model_id). The same inference code supports LLaVA 1.5 models, LLaVA-NeXT (v1.6), Interleave models, OneVision models, and larger checkpoints (e.g., 13B or 34B), including variants based on Mistral, Vicuna, Qwen, or Yi backbones. No changes to prompt construction or forward-pass logic are required.

Finally, the multimodal pipeline is now cleanly abstracted and internally consistent. Hugging Face handles all low-level details, including image preprocessing, chat formatting, positional embeddings, image sequence length management, and attention masking. This eliminates a large class of subtle bugs related to tensor alignment and multimodal token placement, while ensuring that the vision and language components remain synchronized across model updates.

• Streetscape Analysis with Generative AI (SAGAI) on Github with v1.1 update. https://github.com/perezjoan/SAGAI

• Perez, J and Fusco, G. (2025) Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063, 18p. Available at: https://www.sciencedirect.com/science/article/pii/S1195103625000199

GeoPackage (GPKG) is an open and non-proprietary data format that allows different layers, both spatial and non-spatial, to be stored within the same file.
These layers can include:

• Spatial Layers with vector data such as points, lines, and polygons representing geographic features; Raster Data with gridded data representing continuous phenomena like elevation, land cover, or satellite imagery.
• Non-Spatial Layers such as tabular data: Attribute tables associated with spatial features, containing information such as attribute values, metadata, or statistical data or metadata: Descriptive information about the dataset, including authorship, data sources, coordinate reference systems, and data quality indicators.

GeoPackage’s ability to accommodate various types of spatial and non-spatial data in a single file makes it a versatile and efficient format for storing geospatial information. In this blog post, we are going to read and save geopackage’s layers using python and R.

Let’s download a simple Geopackage file. The following file comprises a GeoPackage file containing two building layers corresponding to two small cities in Italy: Grosseto and Sinalunga. First, you have to put the geopackage into your working directory. Then, you can run the code below to import the Grosseto layer and print the first rows of the dataset.

#Load the sf library
library(sf)

# Read the Grosseto layer of buildings
Grosseto = st_read("Italian_cities.gpkg", layer = "Grosseto")

# Print the first rows
head(Grosseto)

Let’s create another layer by doing a simple buffer over the buildings of Grosseto. The st_buffer function from the sf package allows doing buffer quickly. In addition, the code belows perform a buffer of 10 meters around the buildings of Grosseto. Then, the newly created layer is saved as an additional layer into the original geopackage using the st_write function.

# Perform a buffer of 10 meters
Grosseto_10B <- st_buffer(Grosseto, 10)

# Save the newly created layer
st_write(Grosseto_10B, "Italian_cities.gpkg", layer = "Grosseto_10B")

If you are new to Python, you can refer to this post to set up your Python environment with Anaconda and the Jupyter Notebook for spatial analysis. We will work with the same Geopackage file than for the R application. First, place the Geopackage in the same directory as your notebook. You can now run the code below to import a layer in Python using the GeoPandas library. In this example, we are importing a layer of building related to the italian city of Grosseto. In order to check that the layer has been imported, you can print the first rows using the .head function.

import geopandas as gpd
Grosseto = gpd.read_file("Italian_cities.gpkg", layer = "Grosseto")
Grosseto.head()

You can use the ‘to_file‘ method provided by GeoPandas to save a new layer in the GeoPackage. For example, the code below reproject the geometries in a projected CRS, perform a buffer of 10 meters and save a new layer named « buffered_Grosseto » in the GeoPackage « Italian_cities ».

# Re-project geometries to a projected CRS
Grosseto = Grosseto.to_crs(epsg=6875)

# Create a buffer of 10 meters around the geometries
buffered_Grosseto = Grosseto.buffer(10)

# Save the buffered layer to the GeoPackage file
buffered_Grosseto.to_file("Italian_cities.gpkg", driver="GPKG", layer="buffered_Grosseto")

I hope you enjoyed this short tutorial about how to work with the geopackage format in Python and R. Don’t hesitate to comment and provide feedbacks by engaging with this post.

R is an open-source statistical programming language used in statistical analysis but also in spatial analysis, artificial intelligence (AI), and machine learning (ML) applications. When coupled with RStudio, an integrated development environment (IDE) for R, it becomes user-friendly with an interactive interface. In this guide, we will walk you through the initial steps of setting up R and RStudio along with installing essential packages and testing them with spatial data.

Before diving into data mining, you need to set up the programming environment. Start by downloading and installing R from the Comprehensive R Archive Network (CRAN) website (https://cran.r-project.org/). Choose the appropriate version for your operating system and follow the installation instructions. Once R is installed, proceed to install RStudio, which provides a user-friendly interface for R programming. You can download RStudio from the official website (https://www.rstudio.com/products/rstudio/download/) and install it on your system.

The interface is divided into four parts, each fulfilling a specific role in the workflow. The Source section houses scripts, while the Console executes code and displays feedback. In Environments, users can review imported data and created objects. Finally, the Output section presents graphs, maps, and other outputs. This output section also facilitates file browsing and access to the help sections of the packages.

R’s functionality can be extended through packages, which are collections of R functions, data, and compiled code. One such essential package for spatial analysis is sf, which provides simple features (sf) for handling and analyzing spatial data. To install the sf package, open RStudio and execute the following command in the console:

install.packages("sf")

Once the sf package is installed, load it into your R session using the library() function:

library(sf)

Now, you have access to a wide range of spatial functions and data structures provided by the sf package, allowing you to manipulate and analyze spatial data efficiently.

Reading a csv file in R is straightforward using the read.csv function. Within this function, you can set a custom delimiter using the sep argument, and point at the presence of a header (first line as column titles). Don’t forget to put your file in your working directory. Alternatively, you can provide the full path to your file if your csv file is not located in your working directory. This website provides samples of csv file to download.

# Example 1: File in the working directory
read_csv = read.csv('file.csv', sep=',', header=FALSE)

# Example 2 : path to file
read_csv = read.csv('/Users/admin/file.csv', sep=',', header=FALSE)

To test the functionality of the sf package, let’s import a GeoPackage (GPKG) layer containing spatial data and visualize it on a map. You can download sample GPKG data from various sources such as governmental GIS portals or open data repositories. Assuming you have a GPKG file named example_data.gpkg, use the st_read() function from the sf package to read the spatial data into R. If you want to know more about the GeoPackage format, and try with a real Geopackage file, you can have a look at this post.

data <- st_read("path/to/example_data.gpkg")

Replace "path/to/example_data.gpkg" with the actual path to your GPKG file. If your geopackage file contains more than one layer, you can choose which layer to import using the layer = "layer_name" argument. Once the data is imported, you can create a simple map to visualize it using the plot() function

plot(data)

This will generate a basic plot displaying the spatial features contained in the GPKG layer.
R combined with RStudio provides a powerful environment for spatial analysis, AI, and ML tasks. By following the steps outlined in this guide, you’ve prepared a perfect environment for exploring more advanced techniques. Stay tuned for more tutorials and insights on leveraging R for spatial analysis. Happy coding

QGIS, a leading open-source Geographic Information System (GIS), provides an interesting suite of tools for geospatial analysis. Have you ever wondered about controlling QGIS with a Python script ? About calling a specific tool within QGIS from Python? This capability can be useful to integrate processing already available in QGIS within a script of your own without opening the QGIS software directly. In this blog post, we’ll explore how to call QGIS from a Python script in the Jupyter Notebook. Let’s dive in.

If you’re new to Python or Jupyter Notebook, check out our introductory guide to set up a Python environnement using Anaconda. In this tutorial, we assume that you are using the Anaconda distribution of Python. The first step is to open your Anaconda prompt windows and create a Conda environment that includes both QGIS and Jupyter Notebook. On Windows, click on the Start Menu and type “Anaconda Prompt” in the search bar and open it. Within the command prompt, run the line below:

Now, you can activate the newly created Conda environment using the following command, also in the Anaconda prompt windows. Once you activate the qgis_jupyter environment, you’ll have access to QGIS and Jupyter Notebook within the same environment, allowing you to use QGIS functionality directly from your Jupyter Notebooks. If you have an issue with the code above, it means that you need to install QGIS in your newly created environment. Thus, run this conda install -c conda-forge qgis line of code before proceeding further.

And finally, open a Jupiter Notebook within the environnement you just activated

Now that your notebook is open, you can run the code snippet below to set up the interaction with QGIS. Note that even if you don’t need to open the QGIS standalone, you need to have QGIS installed on your machine to interact with it. That’s why we need to specify in the code below the installation directory path of QGIS and initialize it. This ensures that the Python interpreter can locate the necessary QGIS libraries. Additionally, the script initializes the next processing algorithms that will be performed.

The layer contains the building footprints of Grosseto. Provided you put the GeoPackage in the folder where your notebook is located, you can already import and plot the layer, as shown below. For this example, we are working on a subset of 100 buildings.

Now, let’s execute a straightforward algorithm, such as « dissolve, » by invoking the QGIS tool directly within the notebook and visualize the outcome to verify its success. Ensure to specify the location of the Geopackage in the code, along with defining the input and output layers. Subsequently, call the algorithm using the native:dissolve identifier, and adjust its parameters as necessary. The result, named Dissolved_Subset_Grosseto is saved as a new layer in the GeoPackage and ploted in the notebook.

As you can see below the buildings have been perfectly dissolved using the native QGIS algorithm.

This code snippet below demonstrates a workflow in Python using the buffer QGIS algorithm. First, we reproject the input layer to a Cartesian Coordinate Reference System (CRS) suitable suitable for the buffer function. Subsequently, the script iterates through a series of distances, executing the buffer algorithm four times with varying distances (10, 20, 50, and 100 meters). Each buffer operation generates a new layer with the GeoPackage.

Let’s map the different buffers. The code below import the layers we just created with a loop, and plot them in reverse order in order to visualize the overlap using Matplotlib.

Feel free to engage with this post by commenting, asking questions, or providing feedback.