In v3.1.0 we added Qwen3-VL and made a promise: thanks to the new FAMILY_GROUPS registry, future model families would appear in the notebooks automatically, with no interface edits at all. Version 3.2.0 is that promise kept. InternVL3.5 — the latest generation of OpenGVLab’s InternVL line, released in August 2025 — is now the fourth family in the registry, and neither notebook changed by a single line to display it.

UVLM integrates the six Transformers-native -HF checkpoints, which run through the standard Transformers stack without any custom remote code. None of them is gated: no Hugging Face token required.

The registry now totals 21 checkpoints across 4 families, from 1B to 110B parameters — and the 1B entry replaces Qwen3-VL 2B as the smallest model UVLM has ever supported.

InternVL3.5 is not a variation on the Qwen conventions — it uses the standard Transformers pattern in which the chat template tokenizes directly (apply_chat_template(tokenize=True)), the generated tokens are sliced off after the prompt, and only the generated portion is decoded. That makes it the third distinct inference path in UVLM, alongside LLaVA’s string-based cleaning and Qwen’s separate vision preprocessing with token trimming. As always, all three converge at the same unified response parser — from the user’s side, InternVL3.5 is simply one more family in the dropdown.

Loading follows the same BF16-aware logic introduced in v3.1.0: BF16 automatically on GPUs with native support (RTX 30-series and newer, L4, A100), FP16 fallback otherwise.

While validating the new backend, we caught a fossil from UVLM’s two-backend era: the notebooks used a hardcoded rule that sent every non-Qwen2.5 model’s results to Score_Analysis_LLaVA.csv. With four families, that meant different models could silently append into the same CSV — and resume mode could not tell them apart. As of v3.2.0, output filenames are derived from the loaded checkpoint (e.g. Score_Analysis_InternVL3_5-8B-HF.csv), so each model writes its own file and resume mode and schema upgrades are per-model by construction. If you benchmark several models on the same image folder, this is the release that keeps your results honest.

Nothing changes in the workflow — install (or upgrade) and the new family is there:

pip install --upgrade --force-reinstall --no-deps
git+https://github.com/perezjoan/UVLM.git

Or open the Colab notebook — it always installs the latest version automatically. The three-block workflow (load → configure tasks → run batch), consensus validation, chain-of-thought mode, and truncation detection all work with InternVL3.5 out of the box. No dependency changes since v3.1.0. Tested locally on Windows 11 with an RTX 5060 laptop GPU, where the 1B model loads in about 15 seconds once cached.

One field note from validation, and a nice illustration of why UVLM separates format reliability from accuracy: under temperature sampling, InternVL3.5 1B answered a counting task with “There are two vehicles in the picture” — correct, but unparseable as an integer, so it was recorded as NA by design. Under greedy decoding with a strict format instruction, the same model returned a clean integer. Small models follow instructions best when you ask firmly and decode greedily.

The third family addition promised in v3.1.0 — the Gemma multimodal line — is coming next, and it will be a bigger step than a minor version: Gemma 4 requires the Transformers v5 line, which means UVLM’s next release will be a major version with a documented migration. Same discipline as always: one backend at a time, validated before released.

Full change log in VERSIONS.txt · Source and releases on GitHub · If you use UVLM in research, please cite our Software paper.

UVLM was built around one idea: compare Vision-Language Models across architectures using identical prompts and evaluation protocols, without writing model-specific code. Until now that meant two families: LLaVA-NeXT and Qwen2.5-VL. Version 3.1.0 adds a third: Qwen3-VL, the successor to the Qwen2.5-VL family that anchored our published benchmark.

Qwen3-VL is the latest vision-language generation from Alibaba’s Qwen team, first released in September 2025 with the 235B-A22B flagship, followed shortly after by the compact dense checkpoints that matter for most research budgets. UVLM v3.1.0 integrates the four dense Instruct sizes:

The registry now totals 15 checkpoints across 3 families, from 2B to 110B parameters — and the 2B entry is the smallest model UVLM has ever supported, which makes it an interesting new baseline for large-scale, low-cost batch analysis.

Technically, Qwen3-VL keeps the Qwen inference conventions (chat template → separate vision preprocessing → generation → token trimming), so it plugs into UVLM’s existing Qwen pipeline. What changes under the hood: the model loads through Transformers’ generic AutoModelForImageTextToText class, requires transformers ≥ 4.57 and qwen-vl-utils ≥ 0.0.14, and resizes images to multiples of 32 pixels rather than 28. All of this is handled inside the package — from the user’s side, it is simply one more family in the dropdown.

With three families and fifteen checkpoints, a single flat dropdown was getting crowded. Both notebooks (Colab and local) now use a two-level selector: choose the family first — LLaVA-NeXT, Qwen2.5-VL, or Qwen3-VL — and the model list refreshes automatically.

The selector is built from a new FAMILY_GROUPS mapping in the registry, which means future families will appear in the widgets automatically, with no notebook edits at all.

Qwen3-VL checkpoints are trained in BF16. On GPUs with native BF16 support (RTX 30-series and newer, L4, A100), the loader now selects BF16 automatically, falling back to FP16 on older cards and FP32 on CPU. If you followed our earlier benchmark work, you may remember the FP16 numerical-overflow crashes we documented with BF16-trained checkpoints on T4 hardware — this release is the first step toward closing that class of problem at the loader level.

Local Jupyter users get a long-overdue improvement: the model-loading progress (download bars, device map, timings) is now displayed in a log area under the Load button. Previously, output emitted inside the widget callback was silently swallowed in local Jupyter — Colab was never affected. The release also silences the torch_dtype deprecation warnings from recent Transformers versions and synchronizes the package version metadata.

Nothing changes in the workflow — install (or upgrade) and the new family is there:

pip install --upgrade --force-reinstall --no-deps git+https://github.com/perezjoan/UVLM.git

Or open the Colab notebook — it always installs the latest version automatically. The three-block workflow (load → configure tasks → run batch), consensus validation, chain-of-thought mode, and truncation detection all work with Qwen3-VL out of the box. Tested locally on Windows 11 with an RTX 5060 laptop GPU, where the 2B model loads in well under a minute once cached.

v3.1.0 is the first of a series of family additions. Next on the roadmap: InternVL3.5 (via the Transformers-native -HF checkpoints) and the Gemma multimodal line. Each family will land as its own validated release — same discipline, one backend at a time.

Full change log in VERSIONS.txt · Source and releases on GitHub · If you use UVLM in research, please cite our Software paper.

SAGAI (Streetscape Analysis with Generative Artificial Intelligence) is an open-source workflow for scoring and mapping street-level urban environments using vision-language models and open geospatial data. Since its initial release, SAGAI has been structured as a set of independent Colab notebooks, one per pipeline stage, each relying on its own dependencies and documentation.

SAGAI v2.0 is a major release that consolidates the entire pipeline into a single notebook and replaces the custom inference code with the UVLM package. Where previous versions were tied to a single LLaVA checkpoint with handwritten inference logic, SAGAI v2.0 delegates all vision-language model loading, prompting, and evaluation to UVLM’s unified interface. This makes the scoring engine model-agnostic: users can select from 11 VLM checkpoints spanning the LLaVA-NeXT and Qwen2.5-VL families, compare their performance on identical tasks, and benefit from features such as consensus validation, reasoning traces, and truncation diagnostics; all within the same notebook.

Beyond the inference engine, v2.0 introduces structural and functional changes across the entire pipeline: a unified six-block architecture, interactive HTML mapping via Folium, view-direction filtering for aggregation, and the ability to load an existing polygon as a study area boundary instead of defining a bounding box manually.

This post details the architectural changes, the UVLM integration, and the new features introduced in SAGAI v2.0.

Previous SAGAI releases were organized as four independent Colab notebooks — one for street sampling, one for image retrieval, one for VLM inference, and one for aggregation and mapping — each accompanied by a separate NOTICE file documenting its dependencies and usage. This modular design was useful for development but introduced friction in practice: users had to manage file paths between notebooks, track four separate environments, and consult multiple documentation files.

SAGAI v2.0 merges all four stages into a single notebook (SAGAI.ipynb) structured as six sequential blocks. The pipeline flows from study area definition through street sampling, image downloading, VLM scoring, and mapping, with all intermediate data passed directly between blocks in the same runtime session. The separate per-module NOTICE files and the standalone requirements file (requirements_sagai_module_3_v1-0.txt) have been removed — dependency management is now handled automatically by the UVLM package installation.

In previous versions, the study area was defined exclusively by a bounding box in WGS84 coordinates. SAGAI v2.0 retains this option but adds the ability to draw your own polygon or to load an existing polygon; for example, a GeoPackage representing a neighborhood, municipality, or custom boundary. When a polygon is provided, the street sampling step extracts the OpenStreetMap network within that geometry rather than a rectangular extent. This makes it straightforward to work with irregular administrative boundaries or user-defined study zones without manually computing bounding coordinates.

The most significant change in SAGAI v2.0 is the replacement of the inline inference code with the UVLM package. In previous versions, Blocks 3 through 5 contained custom code for loading a single LLaVA checkpoint, constructing prompts, running inference, and parsing outputs. This logic was tightly coupled to one model architecture and required manual maintenance when Hugging Face APIs or model formats changed.

SAGAI v2.0 installs UVLM directly from its GitHub repository at the start of the notebook. All model loading, prompt formatting, inference execution, response parsing, and batch processing are delegated to UVLM’s API. The inline inference code has been entirely removed.

Through UVLM, SAGAI v2.0 supports 11 VLM checkpoints across two model families:

• LLaVA-NeXT — Mistral 7B, Vicuna 7B, Vicuna 13B, 34B, LLaMA3 8B, 72B, 110B
• Qwen2.5-VL — 3B Instruct, 7B Instruct, 32B Instruct, 72B Instruct

UVLM’s dual-backend abstraction automatically detects the model family and routes inference to the correct pipeline — LlavaNextProcessor for LLaVA models, AutoProcessor with process_vision_info for Qwen models — so users switch between architectures by changing a single model selection, with no modification to the rest of the notebook.

Quantization is handled through UVLM’s built-in support for 4-bit, 8-bit, and FP16 precision via BitsAndBytes. Models up to 34B parameters can run on a single Colab GPU (T4 or A100) with 4-bit quantization.

UVLM provides a widget-based prompt builder that SAGAI v2.0 exposes directly in the notebook. Users can define up to 10 analysis tasks per run, each with its own prompt, response type (numeric, category, boolean, or text), and label. This replaces the previous approach of selecting from a small set of hardcoded tasks (T1, T2, T3) or manually editing prompt strings in the code.

Tasks are configured interactively before execution and applied uniformly across all images in the batch. Each task produces its own column in the output CSV file.

SAGAI v2.0 inherits UVLM’s consensus validation mechanism. Each analysis task can be run 2 to 5 times per image, and the final score is determined by majority voting across the repeated inferences. NA values from failed parses are filtered before voting. An agreement ratio is recorded alongside the final score, providing a built-in measure of prediction reliability without any external validation step.

UVLM supports two approaches to chain-of-thought (CoT) reasoning, both available in SAGAI v2.0. Users can write task prompts that explicitly request step-by-step reasoning and adjust the token budget (up to 1,500 tokens) to allow the model sufficient generation space. Alternatively, a built-in CoT reference mode can be enabled per task, which triggers a standardized reasoning template with a fixed 1,024-token budget. In both cases, the reasoning trace is stored in a dedicated column in the output CSV for inspection.

Truncation detection is performed automatically after every inference call. The exact number of generated tokens is compared against the token limit, and truncated responses are flagged in per-task CSV columns. This allows users to identify tasks where the token budget is insufficient without post-hoc analysis.

Previous SAGAI versions generated static thematic maps using Matplotlib. SAGAI v2.0 replaces these with interactive HTML maps built with Folium. Point-level and street-segment-level scores are rendered as interactive layers that can be panned, zoomed, and queried directly in the browser. This is particularly useful for exploratory analysis and for sharing results with collaborators who do not use GIS software.

Google Street View images are typically downloaded in multiple compass directions at each sampling point (e.g., front, back, left, right). In previous versions, all views were aggregated together when computing point- or street-level scores. SAGAI v2.0 introduces a view filter that allows users to select which directions to include in the aggregation — for example, scoring only left-side and right-side views to focus on building facades, or only front views to capture the pedestrian perspective along the street axis. This filter is applied at the aggregation stage and does not affect the scoring step itself.

The batch execution engine inherited from UVLM provides resume-safe processing with checkpoint saving every 3 images. If a Colab session is interrupted — due to a timeout, a runtime reset, or a connectivity issue — the notebook can be re-executed and will automatically skip already-processed images. New tasks added between runs trigger automatic CSV schema upgrading, so the output file grows incrementally without losing previous results.

• SAGAI v2.0 on GitHub: https://github.com/perezjoan/SAGAI
• UVLM on GitHub: https://github.com/perezjoan/UVLM
• Perez, J. and Fusco, G. (2025). Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063. https://www.sciencedirect.com/science/article/pii/S1195103625000199
• Perez, J. and Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893. https://arxiv.org/abs/2603.13893

When we released UVLM in March 2026, it was a Google Colab notebook. You opened it in your browser, picked a model, typed your prompts, and ran your images — all without installing anything. That simplicity was the point: a tool that anyone could use to load and compare Vision-Language Models, regardless of their technical setup.

But we kept hearing the same requests. Can I run this on my own machine? Can I call UVLM from a script? Can I integrate it into an existing pipeline? The answer was always the same: not easily. The entire tool lived inside a single notebook, with all the logic packed into three massive code cells. Moving it anywhere else meant copy-pasting thousands of lines and untangling global variables.

Version 3.0.0 changes that. UVLM is now a proper Python package.

The core logic — model loading, dual-backend inference, response parsing, consensus validation, batch processing — has been extracted from the notebook into eight standalone Python modules. These modules have no dependency on Google Colab, no global variables, and no widget code. They are plain Python functions that accept arguments and return results.

The package is installed from GitHub in one line:

pip install git+https://github.com/perezjoan/UVLM.git

On Google Colab, this happens automatically in the first cell of the Colab notebook. On your local machine, you run it once in a terminal and you are done.

Nothing changed in how UVLM analyses images. The same 11 model checkpoints are supported (LLaVA-NeXT and Qwen2.5-VL, from 3B to 110B parameters). The same parsing logic, the same consensus validation, the same truncation detection. If you had a workflow built on v2.2.2, the outputs will be identical.

Google Colab — Zero Install

This is the same experience as before. Open the Colab notebook, select a GPU runtime, and start working. The notebook installs the UVLM package automatically. Images are loaded from Google Drive. Nothing has changed for Colab users, except that the code running behind the widgets is now cleaner and easier to maintain.

Local Jupyter Notebook — Your GPU, Your Data

If you have an NVIDIA GPU on your workstation (or access to a GPU server), you can now run UVLM locally. The local Jupyter notebook provides the same widget-based interface — model selection dropdown, prompt builder form, batch execution button — but images are read from your local filesystem and results are saved locally. No Google account needed, no data leaves your machine.

This matters for researchers working with sensitive imagery (medical, security, proprietary datasets) or for anyone who wants faster and more reliable model loading than what Colab’s network provides.

Python Script — Full Programmatic Control

For integration into larger pipelines, UVLM now exposes a clean API. Three lines of code replace the entire notebook workflow:

from uvlm import load_model, run_inference, parse_response
ctx = load_model("[Qwen] Qwen2.5-VL 7B Instruct", precision="4bit")
raw, tokens = run_inference("photo.jpg", "Count the cars", ctx)
result = parse_response(raw, "numeric")

The `load_model()` function returns a context dictionary containing the model, processor, backend type, and device information. This dictionary is passed to every subsequent function — no global state, no hidden side effects. You can load multiple models in the same session and switch between them by passing different context objects.

For batch processing, `run_batch()` handles the full pipeline:

from uvlm import load_model
from uvlm.batch import run_batch

ctx = load_model("[Qwen]  Qwen2.5-VL 7B Instruct", precision="4bit")
df = run_batch(
    model_ctx=ctx,
    task_specs=my_tasks,
    image_folder="./images",
    output_path="./results.csv",
)

The monolithic notebook has been split into eight modules, each with a single responsibility:

registry.py holds the model dictionary — 11 checkpoints with their backend type and HuggingFace checkpoint ID. Adding a new model is one line in a dictionary.

loader.py contains the `load_model()` function. It handles quantisation configuration (4-bit, 8-bit, FP16), device placement (single GPU, auto, CPU offload), and the LLaVA vs Qwen branching logic. It returns a dictionary — not a set of global variables.

inference.py contains `run_inference()`, the dual-backend forward pass. It accepts a model context dictionary and returns the raw response plus the exact token count as a tuple. The full LLaVA response cleaning logic and the full Qwen token-trimming pipeline are preserved exactly as they were.

parsers.py holds the four response parsers (numeric, category, boolean, text) and the advanced reasoning parser. These are pure functions with zero dependencies beyond Python’s standard library.

consensus.py contains the majority voting logic. batch.py handles folder iteration, CSV writing, resume mode, and schema upgrading. prompts.py stores the task type definitions and the chain-of-thought templates. utils.py provides seed management, environment detection, and HuggingFace token retrieval.

On Colab: Open the notebook from GitHub and run the three blocks as before. The package installs itself.

Locally: First, install PyTorch with CUDA support matching your GPU driver (check with `nvidia-smi`). For example, with CUDA 12.8+:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

Then open the local Jupyter notebook.

You get the same dropdown menus, the same prompt builder form, the same batch execution. The only difference is that you type a local path for your image folder instead of a Google Drive path.

For HuggingFace authentication (needed for some gated models like LLaMA3-based checkpoints), either set the `HF_TOKEN` environment variable or run `huggingface-cli login` once in your terminal.

The package architecture makes it much easier to add new VLM families. InternVL, BLIP-2, CogVLM, DeepSeek-VL, and Molmo are planned for future releases — each one requires implementing the backend-specific sections of the inference function and adding entries to the registry, without touching the rest of the codebase.

We are also working on multi-GPU batching for parallel inference across images, video frame analysis support, and integration with the SAGAI workflow for automated streetscape analysis.

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Software & Algorithms › UVLM

Previous blog post: Introducing UVLM: A Free Tool to Compare AI Models That Understand Images

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Imagine you have thousands of street photographs and you need to answer the same questions about each one: how many cars are parked? Is there a sidewalk? How long is the building frontage? Hiring someone to go through every image manually would take weeks. Training a custom computer vision model would take months. But what if you could simply ask an AI model these questions in plain English — and get structured, usable answers back?

That is exactly what Vision-Language Models do. And today, we are releasing UVLM — an open-source tool that makes it easy to load, test, and compare these models, all from a single notebook in your browser.

Vision-Language Models (VLMs) are AI systems that can look at an image and answer questions about it in natural language. Unlike traditional computer vision, which requires training a separate model for every task (one for counting cars, another for detecting sidewalks, a third for classifying buildings), a VLM handles all of these through text prompts. You write a question, attach a photo, and the model responds.

For example, you can ask a VLM: “Count all motor vehicles visible in this image” and it will answer “3”. You can ask the same model “Is there a sidewalk along the street frontage?” and it will answer “yes”. You can even ask it to estimate the length of a building facade in meters — a task that requires the model to identify reference objects (like parked cars), estimate their size, and reason about perspective. All of this from a single model, with no retraining and no labelled dataset.

The catch is that there are many VLM families available (LLaVA, Qwen, InternVL, BLIP-2, and more), and each one works differently under the hood. They use different image encoders, different tokenisation strategies, and different code to run. If you want to know which model is best for your specific task, you normally have to write separate code for each one — a tedious and error-prone process.

UVLM (Universal Vision-Language Model Loader) is a free, open-source tool that lets you load, configure, and compare multiple VLM architectures using the same prompts and the same evaluation protocol — without writing any model-specific code. It runs entirely in Google Colab, which means you do not need to install anything on your computer or own a GPU. A free Google account is all you need.

The idea is simple: you pick a model from a dropdown menu, type your analysis questions into a form, point the tool at a folder of images, and hit run. UVLM handles all the technical details — the processor classes, the tokenisation, the generation settings, the output parsing — and delivers a clean CSV file with one row per image and one column per task. If you want to try a different model, you just switch the dropdown and run again. Same prompts, same images, same output format. Now you can compare.

To demonstrate what UVLM can do, we benchmarked 8 different models on 120 street-level photographs of French urban frontages. Each image was analysed on five tasks: counting vehicles, detecting sidewalks, counting pedestrian entrances, estimating the street frontage length in meters, and classifying the vegetation type. That is 16 model configurations (each model tested in standard and advanced reasoning modes), 120 images, and 5 tasks per image — all processed and compared through UVLM.

The results were revealing. The largest model (LLaVA 34B, with 34 billion parameters) actually ranked last overall. A much smaller model (LLaVA Vicuna 7B) outperformed it significantly and ran on a free Google Colab GPU. The best overall results came from Qwen 32B with chain-of-thought reasoning enabled, which achieved 88% proximity to human expert annotations across all five tasks. Without UVLM, discovering these differences would have required writing and debugging eight separate inference pipelines.

UVLM was designed for anyone who works with images and wants to extract structured information from them at scale — without becoming a machine learning engineer. If you are an urban planner evaluating streetscape quality across a city, UVLM lets you score thousands of street photographs using natural language prompts. If you are an environmental researcher classifying vegetation from field photographs, UVLM lets you test which AI model gives the most reliable results for your specific classification scheme. If you are an infrastructure inspector processing damage assessment photographs, UVLM lets you set up automated counting and scoring tasks and run them across your entire image archive.

The tool is also valuable for AI researchers who need a controlled benchmarking environment. Because UVLM ensures that every model receives exactly the same prompt and is evaluated with the same metrics, it produces fair, reproducible comparisons. The consensus validation feature (running each task multiple times and taking a majority vote) addresses the inherent randomness of AI outputs, and the truncation detection feature flags when a model’s response was cut off before it could finish — a common but often invisible source of errors.

Getting started takes about five minutes. Open the UVLM notebook from GitHub (the link is below), connect to a GPU runtime in Google Colab, and run the first block to load a model. The second block gives you a form where you type your analysis questions — no coding required. The third block processes your images and saves the results as a CSV file on your Google Drive.

The tool currently supports 11 model checkpoints from two major families (LLaVA-NeXT and Qwen2.5-VL), ranging from 3 billion to 110 billion parameters. Models up to 34B can run on a single free-tier Colab GPU with 4-bit quantisation. Advanced features include consensus validation (2–5 runs per task with majority voting), chain-of-thought reasoning for complex tasks, and automatic truncation detection.

UVLM is released under the Apache 2.0 open-source licence. You can use it, modify it, and build on it for any purpose — academic or commercial.

Links

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Softwares & Algorithms › UVLM

Benchmark dataset: Zenodo — 120 street-view images

Citation

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Module 3 is the inference core of the SAGAI (Streetscape Analysis with Generative Artificial Intelligence) framework. Its role is to transform large collections of street-level images into structured, quantitative outputs using vision–language models (VLMs), enabling systematic streetscape analysis and subsequent geospatial aggregation.

With SAGAI v1.1, Module 3 has been released in a new major version (Module 3 v2.0) that introduces a fully standardized and maintenance-safe inference architecture. This update reflects both the maturation of multimodal model ecosystems and the need for long-term reproducibility in large-scale urban analysis pipelines.

Earlier iterations of Module 3 were developed during a period of rapid evolution in both LLaVA research codebases and execution environments such as Google Colab. As multimodal models transitioned toward Transformers-native implementations distributed via Hugging Face, assumptions embedded in earlier hybrid workflows became increasingly difficult to sustain.

Module 3 v2.0 addresses this evolution by aligning the entire inference pipeline with official Hugging Face multimodal APIs. Model loading, prompt formatting, image–text fusion, and generation are now handled through maintained and versioned components, ensuring compatibility across environments, models, and future updates.

This document details the architectural context motivating the update, the design choices behind the refactored inference engine, and the rationale for releasing Module 3 v2.0 as a long-term, stable component of SAGAI v1.1.

The initial implementation of Module 3 (SAGAI v1.0) relied on a hybrid architecture that mixed two incompatible sources of LLaVA code, combined with a rapidly evolving execution environment in Google Colab. This design choice made the pipeline fragile and ultimately unsustainable.

First, the pipeline simultaneously depended on the LLaVA GitHub repository (haotian-liu/LLaVA) and on Hugging Face–hosted model checkpoints. The GitHub repository is a research-oriented codebase under active development. Its internal APIs, class structures, and utilities evolve rapidly and are not version-locked. Constructors, module paths, and helper functions may change or disappear without notice, and the repository is not designed to maintain backward compatibility across releases.

At the same time, pretrained model weights were downloaded from Hugging Face. These checkpoints follow the Transformers-native multimodal format, using Hugging Face–specific configuration files, processors, and model classes (e.g., LlavaNextForConditionalGeneration, AutoProcessor, and chat templates). This architecture is fundamentally different from the internal design assumed by the GitHub LLaVA code, which relies on custom token insertion, internal vision tower management, and non-Transformers abstractions.

As a result, the pipeline operated in a structural mismatch: GitHub code expected architectural fields, model attributes, and tokenizer behavior that were not present in Hugging Face checkpoints, while Hugging Face checkpoints expected model wrappers and configuration logic that the GitHub code did not provide.

This fragility was exposed when Google Colab upgraded its backend environment in early 2025. Major changes included Python 3.12, NumPy ≥ 2.0 (introducing ABI-breaking changes for compiled extensions), newer PyTorch releases (≥ 2.2), and updated system libraries. These updates caused widespread failures in binary dependencies and research codebases that were not aligned with the new runtime.

In practice, this led to errors such as NumPy ABI incompatibilities, PyTorch extension failures, missing or renamed modules, and import errors in LLaVA GitHub utilities. Because the pipeline depended on both unstable research code and binary-sensitive extensions, even minor environment updates were sufficient to break execution.

Module 3 has been fully refactored to remove any dependency on the original LLaVA GitHub repository. The inference pipeline now relies exclusively on Hugging Face–native LLaVA models and APIs, ensuring long-term stability and compatibility with evolving software environments.

In the previous architecture, the script depended on cloning the LLaVA GitHub repository, installing it in editable mode, and importing internal modules (llava.*). Prompts were manually assembled using LLaVA-specific multimodal tokens (e.g., <im_start>, <image>), custom separators, and internal utilities. Image tokens and embeddings were explicitly inserted into the prompt, tightly coupling the forward pass to a specific implementation of the LLaVA codebase. As a result, updates to Google Colab, PyTorch, NumPy, or the LLaVA repository frequently introduced breaking changes.

The current implementation removes all such dependencies. Prompt formatting and multimodal input construction are now handled entirely through Hugging Face abstractions. Prompts are formatted using processor.apply_chat_template(), while images and text are combined using processor(images=…, text=…). Image embedding alignment, multimodal token placement, and chat formatting are fully managed by the Hugging Face processor and model configuration. Inference is performed using the standard model.generate() API, without any custom token handling or internal utilities.

This refactoring makes the SAGAI inference engine model-agnostic within the Hugging Face LLaVA ecosystem. The same forward pass is compatible with LLaVA-NeXT (v1.6), LLaVA-Interleave, LLaVA-OneVision, and future Hugging Face LLaVA releases that expose a processor and chat template. Switching between models or architectures requires only changing the model_id, with no modification to prompt logic or inference code.

To ensure reliable downstream analysis, Module 3 also includes a dedicated numeric output stabilization step. After decoding the model response, any prompt echoes or metadata—including residual [INST] … [/INST] segments—are removed. The final output is parsed using a simple regular expression to retain only numeric values (e.g., 0, 1, 2, 1.5). This guarantees clean, machine-readable outputs and a stable CSV format across all supported models.

Model loading has been simplified and standardized using Hugging Face–approved APIs. Both the processor and the model are instantiated directly from Hugging Face model cards via from_pretrained, with optional 4-bit quantization enabled through load_in_4bit=True. This eliminates the need for manual vision-tower initialization, deprecated classes, or custom C++ operators, and avoids common incompatibilities related to PyTorch, CUDA, or NumPy upgrades in Google Colab. Official Hugging Face code paths ensure that pretrained weights are always matched with the correct implementation.

Optional authentication using a Hugging Face access token is supported to avoid rate limits and improve download reliability when working with large checkpoints, though public models remain accessible without authentication.

Overall, this refactoring significantly improves robustness, reproducibility, and maintainability, while enabling systematic experimentation across multiple LLaVA variants and quantization settings within a unified inference framework.

The refactored inference system in Module 3 is designed as a long-term, maintenance-safe release. This is achieved by aligning the entire pipeline with Hugging Face’s officially supported multimodal APIs and model distribution mechanisms.

First, the new architecture is robust to Google Colab environment updates. All critical dependencies—Python (≥3.12), NumPy (≥2.0), PyTorch (2.x), CUDA wheels, and BitsAndBytes quantization—are now managed through Hugging Face Transformers and its dependency resolution. Because the model code, processor logic, and quantization pathways are maintained upstream, updates to Colab or its underlying libraries no longer break the inference pipeline. As long as Hugging Face continues to support the model card, the code remains functional without manual intervention.

Second, the system relies exclusively on official Hugging Face–maintained components. Core classes such as LlavaNextForConditionalGeneration, LlavaNextProcessor, chat templates, and multimodal preprocessing logic are all part of the Transformers library. These components are actively maintained, versioned, and tested by Hugging Face, providing a level of stability and backward compatibility that is not guaranteed when relying on research repositories or development branches.

Third, the new setup significantly improves reproducibility. Each run explicitly references a fixed Hugging Face model checkpoint via the model_id, ensuring that the same weights, architecture, and prompt template are used across sessions and machines. In addition, generation parameters (sampling strategy, temperature, nucleus sampling, and output length) are explicitly defined, enabling consistent and repeatable results across runs.

Fourth, the architecture is easy to extend and experiment with. Switching between different LLaVA variants now requires changing a single configuration line (model_id). The same inference code supports LLaVA 1.5 models, LLaVA-NeXT (v1.6), Interleave models, OneVision models, and larger checkpoints (e.g., 13B or 34B), including variants based on Mistral, Vicuna, Qwen, or Yi backbones. No changes to prompt construction or forward-pass logic are required.

Finally, the multimodal pipeline is now cleanly abstracted and internally consistent. Hugging Face handles all low-level details, including image preprocessing, chat formatting, positional embeddings, image sequence length management, and attention masking. This eliminates a large class of subtle bugs related to tensor alignment and multimodal token placement, while ensuring that the vision and language components remain synchronized across model updates.

• Streetscape Analysis with Generative AI (SAGAI) on Github with v1.1 update. https://github.com/perezjoan/SAGAI

• Perez, J and Fusco, G. (2025) Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063, 18p. Available at: https://www.sciencedirect.com/science/article/pii/S1195103625000199