Cloud chat assistants are convenient, but they come with trade-offs: your queries leave your machine, you depend on someone else’s uptime and pricing, and the model’s behaviour can change under you without warning. For research, sensitive data, or simply full control, running a model locally is an appealing alternative. The good news is that modern quantization has made this accessible on modest consumer hardware. A capable 7–8 billion parameter model now fits comfortably on an 8 GB laptop GPU. This tutorial walks through the entire process end to end, using an NVIDIA Blackwell card (RTX 5060, 8 GB) as the worked example — though the approach applies to any recent NVIDIA GPU.

Everything starts with a clean, isolated environment. Mixing deep-learning dependencies into your base Python installation is a recipe for version conflicts, so we create a dedicated Conda environment for this project alone. If you have followed our earlier Anaconda setup guide, this will feel familiar.

Open the Anaconda Prompt and create a fresh environment:

The single most important, and most overlooked, step is installing the correct build of PyTorch for your specific GPU. This is where most local-LLM attempts fail silently. NVIDIA GPUs each have a “compute capability” (an architecture identifier such as sm_86, sm_90, sm_120), and a PyTorch binary only works if it was compiled with kernels for your card’s architecture. Install the wrong build and you will see CUDA reported as “available” while every actual GPU operation crashes — a particularly confusing failure mode.

The newest Blackwell cards (the RTX 50-series, including our RTX 5060) use compute capability sm_120, which older PyTorch wheels do not support. For these cards you need a build compiled against CUDA 12.8 or newer:

If you are on an older card (RTX 30- or 40-series), the standard CUDA 12.x wheels are fine. The general rule: match the PyTorch CUDA build to your GPU generation, and when a brand-new card isn’t yet supported in the stable channel, reach for the nightly build of the matching CUDA version.

Now verify it properly. Do not trust torch.cuda.is_available() alone — it can return True even when no compatible kernels exist. Instead, force an actual computation onto the GPU:

A clean OK cuda:0 (12, 0) with no warnings means real GPU compute is working. That is your green light. With the engine confirmed, install the rest of the stack and register the environment as a dedicated Jupyter kernel so the notebook always uses exactly these packages:

Finally, launch JupyterLab from your project directory so your notebook is rooted where you want it rather than in a system folder:

Open the localhost address provided by jupyter on your navigator and once inside, select the “Python (localllm)” kernel. We recommend JupyterLab over the classic Notebook here: it renders interactive widgets reliably out of the box, which matters for the chat interface we build in Section 3.

A model’s weights have to live in memory, and for modern LLMs they are large. A 7–8 billion parameter model in full 16-bit precision needs roughly 14–16 GB — too much for an 8 GB card. The solution is quantization: storing each weight in 4 bits instead of 16. This shrinks an 8B model to around 5 GB with only a minor quality cost, which is what makes local inference on consumer hardware possible at all.

We use Hugging Face Transformers together with the bitsandbytes library, which quantizes the model to 4 bits on the fly as it loads. This keeps everything inside your Python kernel — the model object lives in your notebook, you load directly from Hugging Face with optional token authentication, and you can inspect internals if you wish. Hugging Face acts as the model registry: the first load downloads the weights and caches them to disk (under your user folder’s .cache/huggingface), and every subsequent load reads from that local cache with no network access.

A note on model choice. There is no single “best” small model; it depends on your task and your memory budget. Here is a practical comparison for an 8 GB card:

• Qwen3 4B Instruct — the lightweight workhorse. Around 2.7 GB in 4-bit, very fast, strong reasoning and multilingual ability for its size. Ideal as a daily driver for quick questions.
• Dolphin 3.0 (Llama 3.1 8B) — a larger, more capable general-purpose model at around 5–5.5 GB in 4-bit. Built on Llama 3.1 and instruction-tuned by Cognitive Computations, it is designed to put alignment under the user’s control, making it well suited to research contexts where you define the system prompt and behaviour yourself.
• Other strong candidates — Phi-4-mini for very light tasks, and Gemma-class models for multilingual writing, depending on what fits your remaining VRAM.

The rule of thumb: pick the smallest model that does your job well. A 4B model runs noticeably faster than an 8B simply because there are fewer parameters to push through per token, so match model size to task.

The loading code configures 4-bit quantization and reads the model from Hugging Face. We wrap it in a small dropdown so you can switch models without rewriting code:

If you load a model built on a gated base (such as Llama), you may need to authenticate once with a Hugging Face token via huggingface_hub.login(). Most fine-tuned community models, including the two above, load without one.

A loaded model is, by itself, stateless. It has no memory of anything you said previously — each call only sees the text you hand it. To create the experience of a conversation, we must keep the history and re-send it on every turn. Understanding this is the key to using local models well, and it requires distinguishing three concepts that are easy to confuse.

The context window is the model’s hard architectural limit: the maximum number of tokens it can attend to at once, counting both the prompt and the output together. Llama 3.1-based models support up to 128k tokens. The conversation memory is not a property of the model at all — it is simply the running list of past turns that we re-inject into the prompt each time, and it consumes part of the context window. The max new tokens setting is a cap we choose on how many tokens the model may generate in a single reply; it controls output length only and does not affect what the model can read.

So the relationship is: the prompt (system message + accumulated history + your new question) plus the reserved output space must all fit inside the context window. The context window is the room; memory is the furniture already in it; max new tokens is the space you set aside for the answer.

A common misconception is that a larger context window makes the model faster. It is the opposite. A bigger active context costs more VRAM (the key-value cache grows) and runs slower, because each newly generated token must attend over every preceding token. Speed comes from keeping the active context small — short prompts and trimmed history — not large. Reducing max new tokens does not speed up generation either; it simply stops the reply earlier, often mid-thought, since the model does not plan around the limit. The right way to get shorter, faster answers is to instruct the model to be concise via a system prompt, so it produces a complete but brief response.

The widget below puts these ideas into practice. It keeps a chat_history list (the memory), trims it to a fixed number of recent turns (capping context growth), and displays a live VRAM gauge so you can see your headroom and know when to reset. Re-running the cell clears the history — that is your reset.

A few design notes. We use greedy decoding (do_sample=False) rather than random sampling: it is marginally faster and fully reproducible, with no meaningful quality loss for factual exchanges. The MAX_TURNS value is your direct control over how much the model “remembers” versus how lean and fast it stays. And the VRAM gauge turns green, amber, or red as memory fills, giving you a clear signal of when to start a fresh conversation.

What we have built is small but genuinely yours. Every conversation lives only in your computer’s memory, inside the running notebook kernel. Nothing is written to disk, nothing is sent anywhere, and nothing is logged. Close the kernel and the entire conversation simply vanishes — the only thing that persists is the downloaded model weights in your local cache. For sensitive research data, confidential analysis, or simply peace of mind, this is a meaningful difference from any cloud service.

Beyond privacy, running locally brings other advantages. You are not subject to per-token billing or rate limits, so you can experiment freely. You are insulated from silent model changes and deprecations — your model behaves the same tomorrow as it does today. And with community fine-tunes such as Dolphin, you control the system prompt and the model’s alignment yourself, rather than inheriting a one-size-fits-all policy. With fewer built-in guardrails, these models will engage with a wider range of legitimate research and technical questions, which can be valuable in specialist domains where general-purpose assistants are overly cautious — a freedom that naturally comes with the responsibility to use it sensibly.

This is only the foundation. In future posts we will extend this local assistant in several directions. We will give it web browsing, so it can retrieve current information rather than relying solely on its training. We will explore an expert mode, pre-loading the context with domain knowledge — for instance a corpus of spatial-analysis references — so the assistant answers as a specialist in your field. And we will look at containerizing the whole setup with Docker so it can be deployed on a dedicated GPU server or in the cloud, turning this notebook prototype into a private assistant you can embed directly in your own website.

For now, you have a capable, private language model running on hardware you already own, set up in about half an hour. Learn it, build on it, and apply it to your own work.

SAGAI (Streetscape Analysis with Generative Artificial Intelligence) is an open-source workflow for scoring and mapping street-level urban environments using vision-language models and open geospatial data. Since its initial release, SAGAI has been structured as a set of independent Colab notebooks, one per pipeline stage, each relying on its own dependencies and documentation.

SAGAI v2.0 is a major release that consolidates the entire pipeline into a single notebook and replaces the custom inference code with the UVLM package. Where previous versions were tied to a single LLaVA checkpoint with handwritten inference logic, SAGAI v2.0 delegates all vision-language model loading, prompting, and evaluation to UVLM’s unified interface. This makes the scoring engine model-agnostic: users can select from 11 VLM checkpoints spanning the LLaVA-NeXT and Qwen2.5-VL families, compare their performance on identical tasks, and benefit from features such as consensus validation, reasoning traces, and truncation diagnostics; all within the same notebook.

Beyond the inference engine, v2.0 introduces structural and functional changes across the entire pipeline: a unified six-block architecture, interactive HTML mapping via Folium, view-direction filtering for aggregation, and the ability to load an existing polygon as a study area boundary instead of defining a bounding box manually.

This post details the architectural changes, the UVLM integration, and the new features introduced in SAGAI v2.0.

Previous SAGAI releases were organized as four independent Colab notebooks — one for street sampling, one for image retrieval, one for VLM inference, and one for aggregation and mapping — each accompanied by a separate NOTICE file documenting its dependencies and usage. This modular design was useful for development but introduced friction in practice: users had to manage file paths between notebooks, track four separate environments, and consult multiple documentation files.

SAGAI v2.0 merges all four stages into a single notebook (SAGAI.ipynb) structured as six sequential blocks. The pipeline flows from study area definition through street sampling, image downloading, VLM scoring, and mapping, with all intermediate data passed directly between blocks in the same runtime session. The separate per-module NOTICE files and the standalone requirements file (requirements_sagai_module_3_v1-0.txt) have been removed — dependency management is now handled automatically by the UVLM package installation.

In previous versions, the study area was defined exclusively by a bounding box in WGS84 coordinates. SAGAI v2.0 retains this option but adds the ability to draw your own polygon or to load an existing polygon; for example, a GeoPackage representing a neighborhood, municipality, or custom boundary. When a polygon is provided, the street sampling step extracts the OpenStreetMap network within that geometry rather than a rectangular extent. This makes it straightforward to work with irregular administrative boundaries or user-defined study zones without manually computing bounding coordinates.

The most significant change in SAGAI v2.0 is the replacement of the inline inference code with the UVLM package. In previous versions, Blocks 3 through 5 contained custom code for loading a single LLaVA checkpoint, constructing prompts, running inference, and parsing outputs. This logic was tightly coupled to one model architecture and required manual maintenance when Hugging Face APIs or model formats changed.

SAGAI v2.0 installs UVLM directly from its GitHub repository at the start of the notebook. All model loading, prompt formatting, inference execution, response parsing, and batch processing are delegated to UVLM’s API. The inline inference code has been entirely removed.

Through UVLM, SAGAI v2.0 supports 11 VLM checkpoints across two model families:

• LLaVA-NeXT — Mistral 7B, Vicuna 7B, Vicuna 13B, 34B, LLaMA3 8B, 72B, 110B
• Qwen2.5-VL — 3B Instruct, 7B Instruct, 32B Instruct, 72B Instruct

UVLM’s dual-backend abstraction automatically detects the model family and routes inference to the correct pipeline — LlavaNextProcessor for LLaVA models, AutoProcessor with process_vision_info for Qwen models — so users switch between architectures by changing a single model selection, with no modification to the rest of the notebook.

Quantization is handled through UVLM’s built-in support for 4-bit, 8-bit, and FP16 precision via BitsAndBytes. Models up to 34B parameters can run on a single Colab GPU (T4 or A100) with 4-bit quantization.

UVLM provides a widget-based prompt builder that SAGAI v2.0 exposes directly in the notebook. Users can define up to 10 analysis tasks per run, each with its own prompt, response type (numeric, category, boolean, or text), and label. This replaces the previous approach of selecting from a small set of hardcoded tasks (T1, T2, T3) or manually editing prompt strings in the code.

Tasks are configured interactively before execution and applied uniformly across all images in the batch. Each task produces its own column in the output CSV file.

SAGAI v2.0 inherits UVLM’s consensus validation mechanism. Each analysis task can be run 2 to 5 times per image, and the final score is determined by majority voting across the repeated inferences. NA values from failed parses are filtered before voting. An agreement ratio is recorded alongside the final score, providing a built-in measure of prediction reliability without any external validation step.

UVLM supports two approaches to chain-of-thought (CoT) reasoning, both available in SAGAI v2.0. Users can write task prompts that explicitly request step-by-step reasoning and adjust the token budget (up to 1,500 tokens) to allow the model sufficient generation space. Alternatively, a built-in CoT reference mode can be enabled per task, which triggers a standardized reasoning template with a fixed 1,024-token budget. In both cases, the reasoning trace is stored in a dedicated column in the output CSV for inspection.

Truncation detection is performed automatically after every inference call. The exact number of generated tokens is compared against the token limit, and truncated responses are flagged in per-task CSV columns. This allows users to identify tasks where the token budget is insufficient without post-hoc analysis.

Previous SAGAI versions generated static thematic maps using Matplotlib. SAGAI v2.0 replaces these with interactive HTML maps built with Folium. Point-level and street-segment-level scores are rendered as interactive layers that can be panned, zoomed, and queried directly in the browser. This is particularly useful for exploratory analysis and for sharing results with collaborators who do not use GIS software.

Google Street View images are typically downloaded in multiple compass directions at each sampling point (e.g., front, back, left, right). In previous versions, all views were aggregated together when computing point- or street-level scores. SAGAI v2.0 introduces a view filter that allows users to select which directions to include in the aggregation — for example, scoring only left-side and right-side views to focus on building facades, or only front views to capture the pedestrian perspective along the street axis. This filter is applied at the aggregation stage and does not affect the scoring step itself.

The batch execution engine inherited from UVLM provides resume-safe processing with checkpoint saving every 3 images. If a Colab session is interrupted — due to a timeout, a runtime reset, or a connectivity issue — the notebook can be re-executed and will automatically skip already-processed images. New tasks added between runs trigger automatic CSV schema upgrading, so the output file grows incrementally without losing previous results.

• SAGAI v2.0 on GitHub: https://github.com/perezjoan/SAGAI
• UVLM on GitHub: https://github.com/perezjoan/UVLM
• Perez, J. and Fusco, G. (2025). Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063. https://www.sciencedirect.com/science/article/pii/S1195103625000199
• Perez, J. and Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893. https://arxiv.org/abs/2603.13893

When we released UVLM in March 2026, it was a Google Colab notebook. You opened it in your browser, picked a model, typed your prompts, and ran your images — all without installing anything. That simplicity was the point: a tool that anyone could use to load and compare Vision-Language Models, regardless of their technical setup.

But we kept hearing the same requests. Can I run this on my own machine? Can I call UVLM from a script? Can I integrate it into an existing pipeline? The answer was always the same: not easily. The entire tool lived inside a single notebook, with all the logic packed into three massive code cells. Moving it anywhere else meant copy-pasting thousands of lines and untangling global variables.

Version 3.0.0 changes that. UVLM is now a proper Python package.

The core logic — model loading, dual-backend inference, response parsing, consensus validation, batch processing — has been extracted from the notebook into eight standalone Python modules. These modules have no dependency on Google Colab, no global variables, and no widget code. They are plain Python functions that accept arguments and return results.

The package is installed from GitHub in one line:

pip install git+https://github.com/perezjoan/UVLM.git

On Google Colab, this happens automatically in the first cell of the Colab notebook. On your local machine, you run it once in a terminal and you are done.

Nothing changed in how UVLM analyses images. The same 11 model checkpoints are supported (LLaVA-NeXT and Qwen2.5-VL, from 3B to 110B parameters). The same parsing logic, the same consensus validation, the same truncation detection. If you had a workflow built on v2.2.2, the outputs will be identical.

Google Colab — Zero Install

This is the same experience as before. Open the Colab notebook, select a GPU runtime, and start working. The notebook installs the UVLM package automatically. Images are loaded from Google Drive. Nothing has changed for Colab users, except that the code running behind the widgets is now cleaner and easier to maintain.

Local Jupyter Notebook — Your GPU, Your Data

If you have an NVIDIA GPU on your workstation (or access to a GPU server), you can now run UVLM locally. The local Jupyter notebook provides the same widget-based interface — model selection dropdown, prompt builder form, batch execution button — but images are read from your local filesystem and results are saved locally. No Google account needed, no data leaves your machine.

This matters for researchers working with sensitive imagery (medical, security, proprietary datasets) or for anyone who wants faster and more reliable model loading than what Colab’s network provides.

Python Script — Full Programmatic Control

For integration into larger pipelines, UVLM now exposes a clean API. Three lines of code replace the entire notebook workflow:

from uvlm import load_model, run_inference, parse_response
ctx = load_model("[Qwen] Qwen2.5-VL 7B Instruct", precision="4bit")
raw, tokens = run_inference("photo.jpg", "Count the cars", ctx)
result = parse_response(raw, "numeric")

The `load_model()` function returns a context dictionary containing the model, processor, backend type, and device information. This dictionary is passed to every subsequent function — no global state, no hidden side effects. You can load multiple models in the same session and switch between them by passing different context objects.

For batch processing, `run_batch()` handles the full pipeline:

from uvlm import load_model
from uvlm.batch import run_batch

ctx = load_model("[Qwen]  Qwen2.5-VL 7B Instruct", precision="4bit")
df = run_batch(
    model_ctx=ctx,
    task_specs=my_tasks,
    image_folder="./images",
    output_path="./results.csv",
)

The monolithic notebook has been split into eight modules, each with a single responsibility:

registry.py holds the model dictionary — 11 checkpoints with their backend type and HuggingFace checkpoint ID. Adding a new model is one line in a dictionary.

loader.py contains the `load_model()` function. It handles quantisation configuration (4-bit, 8-bit, FP16), device placement (single GPU, auto, CPU offload), and the LLaVA vs Qwen branching logic. It returns a dictionary — not a set of global variables.

inference.py contains `run_inference()`, the dual-backend forward pass. It accepts a model context dictionary and returns the raw response plus the exact token count as a tuple. The full LLaVA response cleaning logic and the full Qwen token-trimming pipeline are preserved exactly as they were.

parsers.py holds the four response parsers (numeric, category, boolean, text) and the advanced reasoning parser. These are pure functions with zero dependencies beyond Python’s standard library.

consensus.py contains the majority voting logic. batch.py handles folder iteration, CSV writing, resume mode, and schema upgrading. prompts.py stores the task type definitions and the chain-of-thought templates. utils.py provides seed management, environment detection, and HuggingFace token retrieval.

On Colab: Open the notebook from GitHub and run the three blocks as before. The package installs itself.

Locally: First, install PyTorch with CUDA support matching your GPU driver (check with `nvidia-smi`). For example, with CUDA 12.8+:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

Then open the local Jupyter notebook.

You get the same dropdown menus, the same prompt builder form, the same batch execution. The only difference is that you type a local path for your image folder instead of a Google Drive path.

For HuggingFace authentication (needed for some gated models like LLaMA3-based checkpoints), either set the `HF_TOKEN` environment variable or run `huggingface-cli login` once in your terminal.

The package architecture makes it much easier to add new VLM families. InternVL, BLIP-2, CogVLM, DeepSeek-VL, and Molmo are planned for future releases — each one requires implementing the backend-specific sections of the inference function and adding entries to the registry, without touching the rest of the codebase.

We are also working on multi-GPU batching for parallel inference across images, video frame analysis support, and integration with the SAGAI workflow for automated streetscape analysis.

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Software & Algorithms › UVLM

Previous blog post: Introducing UVLM: A Free Tool to Compare AI Models That Understand Images

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Imagine you have thousands of street photographs and you need to answer the same questions about each one: how many cars are parked? Is there a sidewalk? How long is the building frontage? Hiring someone to go through every image manually would take weeks. Training a custom computer vision model would take months. But what if you could simply ask an AI model these questions in plain English — and get structured, usable answers back?

That is exactly what Vision-Language Models do. And today, we are releasing UVLM — an open-source tool that makes it easy to load, test, and compare these models, all from a single notebook in your browser.

Vision-Language Models (VLMs) are AI systems that can look at an image and answer questions about it in natural language. Unlike traditional computer vision, which requires training a separate model for every task (one for counting cars, another for detecting sidewalks, a third for classifying buildings), a VLM handles all of these through text prompts. You write a question, attach a photo, and the model responds.

For example, you can ask a VLM: “Count all motor vehicles visible in this image” and it will answer “3”. You can ask the same model “Is there a sidewalk along the street frontage?” and it will answer “yes”. You can even ask it to estimate the length of a building facade in meters — a task that requires the model to identify reference objects (like parked cars), estimate their size, and reason about perspective. All of this from a single model, with no retraining and no labelled dataset.

The catch is that there are many VLM families available (LLaVA, Qwen, InternVL, BLIP-2, and more), and each one works differently under the hood. They use different image encoders, different tokenisation strategies, and different code to run. If you want to know which model is best for your specific task, you normally have to write separate code for each one — a tedious and error-prone process.

UVLM (Universal Vision-Language Model Loader) is a free, open-source tool that lets you load, configure, and compare multiple VLM architectures using the same prompts and the same evaluation protocol — without writing any model-specific code. It runs entirely in Google Colab, which means you do not need to install anything on your computer or own a GPU. A free Google account is all you need.

The idea is simple: you pick a model from a dropdown menu, type your analysis questions into a form, point the tool at a folder of images, and hit run. UVLM handles all the technical details — the processor classes, the tokenisation, the generation settings, the output parsing — and delivers a clean CSV file with one row per image and one column per task. If you want to try a different model, you just switch the dropdown and run again. Same prompts, same images, same output format. Now you can compare.

To demonstrate what UVLM can do, we benchmarked 8 different models on 120 street-level photographs of French urban frontages. Each image was analysed on five tasks: counting vehicles, detecting sidewalks, counting pedestrian entrances, estimating the street frontage length in meters, and classifying the vegetation type. That is 16 model configurations (each model tested in standard and advanced reasoning modes), 120 images, and 5 tasks per image — all processed and compared through UVLM.

The results were revealing. The largest model (LLaVA 34B, with 34 billion parameters) actually ranked last overall. A much smaller model (LLaVA Vicuna 7B) outperformed it significantly and ran on a free Google Colab GPU. The best overall results came from Qwen 32B with chain-of-thought reasoning enabled, which achieved 88% proximity to human expert annotations across all five tasks. Without UVLM, discovering these differences would have required writing and debugging eight separate inference pipelines.

UVLM was designed for anyone who works with images and wants to extract structured information from them at scale — without becoming a machine learning engineer. If you are an urban planner evaluating streetscape quality across a city, UVLM lets you score thousands of street photographs using natural language prompts. If you are an environmental researcher classifying vegetation from field photographs, UVLM lets you test which AI model gives the most reliable results for your specific classification scheme. If you are an infrastructure inspector processing damage assessment photographs, UVLM lets you set up automated counting and scoring tasks and run them across your entire image archive.

The tool is also valuable for AI researchers who need a controlled benchmarking environment. Because UVLM ensures that every model receives exactly the same prompt and is evaluated with the same metrics, it produces fair, reproducible comparisons. The consensus validation feature (running each task multiple times and taking a majority vote) addresses the inherent randomness of AI outputs, and the truncation detection feature flags when a model’s response was cut off before it could finish — a common but often invisible source of errors.

Getting started takes about five minutes. Open the UVLM notebook from GitHub (the link is below), connect to a GPU runtime in Google Colab, and run the first block to load a model. The second block gives you a form where you type your analysis questions — no coding required. The third block processes your images and saves the results as a CSV file on your Google Drive.

The tool currently supports 11 model checkpoints from two major families (LLaVA-NeXT and Qwen2.5-VL), ranging from 3 billion to 110 billion parameters. Models up to 34B can run on a single free-tier Colab GPU with 4-bit quantisation. Advanced features include consensus validation (2–5 runs per task with majority voting), chain-of-thought reasoning for complex tasks, and automatic truncation detection.

UVLM is released under the Apache 2.0 open-source licence. You can use it, modify it, and build on it for any purpose — academic or commercial.

Links

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Softwares & Algorithms › UVLM

Benchmark dataset: Zenodo — 120 street-view images

Citation

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Large Language Models (LLMs) such as GPT-4, Claude, Llama, and Qwen are deep neural networks trained to predict the next token in a sequence. Although the outputs they produce can appear creative, conversational, or analytical, the underlying mechanism is statistical: the model assigns a probability distribution over the vocabulary at each step and samples from it. Understanding the architecture behind this process is essential before discussing what happens when these models are placed inside autonomous loops.

1.1 Tokenization and Embeddings

Raw text cannot be processed by a neural network directly. It is first decomposed into tokens — subword units that may represent a full word, a syllable, or a single character depending on the tokenizer. Each token is then mapped to a high-dimensional vector called an embedding. These embeddings are not fixed lookup tables: they are learned during training and encode rich semantic relationships. Words with similar meanings occupy nearby regions of the embedding space, while syntactic and relational structure is distributed across dimensions in ways that linear algebra can partially recover (Mikolov et al., 2013).

In addition to token embeddings, modern transformers add positional encodings — either sinusoidal functions or learned vectors — so that the model can distinguish the order of tokens in a sequence. The sum of the token embedding and the positional encoding forms the initial representation that enters the transformer stack.

1.2 The Transformer and Self-Attention

The core computational unit of every modern LLM is the transformer, introduced by Vaswani et al. (2017) in the landmark paper “Attention Is All You Need.” The key innovation was the self-attention mechanism, which allows every token in a sequence to attend to every other token, weighted by learned relevance scores. Concretely, each token produces three vectors — a Query, a Key, and a Value — and attention weights are computed as the scaled dot product of queries and keys. The result is a context-aware representation of each token that integrates information from the entire sequence.

Self-attention can be thought of as a set of dynamic spotlights (illustrated in Figure 1 below): for each token being processed, the model learns which other tokens in the context are most relevant and “illuminates” them, pulling their information into the current representation. A transformer block repeats this attention step multiple times in parallel (multi-head attention), each head specializing in different relationship types — syntactic dependencies, coreference, semantic similarity — before combining the results through a feed-forward network and layer normalization.

Modern LLMs stack dozens to over a hundred of these transformer blocks. GPT-4 is estimated to use on the order of 120 layers; open-weight models like Llama 3.1 405B use 126 layers. Each additional layer allows the model to build increasingly abstract representations of the input.

1.3 Autoregressive Generation and Likelihood

LLMs generate text autoregressively: at each step, the model predicts a probability distribution over the full vocabulary for the next token, conditioned on all previous tokens. Formally, given a sequence of tokens t₁, t₂, …, t_n, the model estimates P(t_n+1 | t₁, …, t_n). The training objective is to maximize the log-likelihood of the training corpus — that is, to learn weights that make the observed sequences as probable as possible under the model. The chosen token is then appended to the sequence, and the process repeats.

This maximum-likelihood framework explains both the strengths and the well-known failure modes of LLMs. The model is excellent at producing fluent, contextually appropriate continuations because it has been optimized to do exactly that across trillions of tokens. However, it has no built-in mechanism for verifying the factual accuracy of what it produces: it generates the most likely continuation, not necessarily the true one. This distinction becomes critical when we consider placing LLMs inside autonomous agent loops.

1.4 Checkpoints and Training

A checkpoint is a serialized snapshot of the model’s parameters — the billions of floating-point weights that define the transformer’s behavior. Training proceeds iteratively over a massive text corpus: at each step, the model’s predictions are compared to the actual next token, a loss is computed, and the weights are updated via gradient descent. Periodically, the full set of weights is saved to disk as a checkpoint.

Checkpoints are what make modern open-weight LLMs possible. Organizations such as Meta (Llama), Alibaba (Qwen), and Mistral release checkpoints publicly, allowing researchers and practitioners to load a pre-trained model and either use it directly or fine-tune it for a specific domain. In the context of tools like ComfyUI (familiar to readers of this blog), a checkpoint is loaded at the start of a pipeline and provides the learned knowledge that drives generation.

1.5 Contextual Memory: From Single-Pass to Conversation

A crucial point of confusion surrounds the notion of “memory” in LLMs. Internally, the model has no persistent state between calls: every invocation is a fresh forward pass through the transformer stack. What creates the illusion of memory is the context window — the full sequence of tokens (including all previous turns of conversation) that is fed as input at each step.

When you interact with a chatbot like ChatGPT or Claude, the application concatenates the entire conversation history into a single token sequence and re-submits it to the model for each new response. The model does not “remember” what it said before; it re-reads the transcript every time. This mechanism is powerful — it enables multi-turn reasoning, follow-up questions, and sustained coherence — but it has hard limits. Context windows range from 4K tokens in earlier models to 128K–200K tokens in current systems (e.g., Claude 3.5, GPT-4 Turbo). Once the conversation exceeds the window, earlier content is silently dropped.

In summary, an LLM is a stateless, single-pass prediction engine. It is not a loop, it has no persistent memory, and it does not take actions in the world. Making it do those things requires wrapping it in an entirely different architecture — which is precisely what an AI agent does.

2.1 From Open-Loop to Closed-Loop

The fundamental architectural difference between an LLM and an AI agent is the introduction of a perception–action loop. An LLM, as described above, receives input and produces output in a single forward pass. An AI agent, by contrast, operates in a cycle: it observes the state of an environment, reasons about what to do next (using an LLM or similar model as its “brain”), executes an action, observes the result, and repeats. This transforms the LLM from a passive text generator into an active decision-maker embedded in a dynamic context.

The theoretical foundations for this architecture draw from reinforcement learning (Sutton & Barto, 2018) and classical AI planning (Russell & Norvig, 2021), but the practical catalyst was the discovery that LLMs, when prompted appropriately, can decompose complex goals into subtasks, select tools, interpret feedback, and recover from errors — all within natural language. The seminal demonstrations of this capability include ReAct (Yao et al., 2023), which interleaves reasoning and acting, and Toolformer (Schick et al., 2023), which teaches LLMs to call external APIs autonomously.

2.2 Environment, Tools, and Action Space

In the agent paradigm, the environment is anything the agent can observe and act upon: a file system, a web browser, an API, a database, a code interpreter, or even a physical interface. The agent’s action space is defined by the set of tools it has been given access to. A minimal agent might only be able to search the web and write text; a fully equipped agent might browse websites, execute code, manage files, query databases, send emails, and transact with cryptocurrency wallets.

The concept echoes the classical reinforcement-learning formulation (state, action, reward), but with a critical difference: the “policy” is not a trained neural network optimized on a reward function — it is an LLM prompted with instructions, tool descriptions, and conversation history. This makes agents remarkably flexible (they can be reconfigured entirely through natural language) but also unpredictable (their behavior depends on prompt interpretation rather than formal optimization).

2.3 Agent Frameworks and Toolkits

A growing ecosystem of open-source frameworks has made it straightforward to construct LLM-based agents with full tool access. The most prominent include:

These frameworks dramatically lower the barrier to entry. A developer can instantiate a fully autonomous agent — capable of browsing the web, writing and executing code, managing files, and interacting with APIs — in fewer than fifty lines of Python. The practical implication is that the agent paradigm is no longer experimental: it is deployable, and it is being deployed.

2.4 Social Agents and Multi-Agent Platforms

Beyond single-agent systems, an emerging class of platforms enables multiple agents to interact with each other in shared environments. Moltbook is a social network built specifically for AI agents rather than human users. It provides a public space where AI agents can create profiles, post updates, comment, upvote, and interact with one another. The platform emphasizes persistent identity, reputation tracking, and observable agent behavior over time — effectively creating a social and evaluative layer for autonomous AI systems. In practice, this means that the agents populating such a platform are not independent entities with their own goals: they are proxies executing human instructions, which may range from benign community engagement to strategic manipulation of discourse on a specific subject.

This raises immediate questions about authenticity, accountability, and influence. If a platform hosts thousands of agent-driven accounts, each tasked with promoting a specific narrative, the distinction between organic discourse and orchestrated campaign becomes effectively invisible. Research on multi-agent simulations (Park et al., 2023) has demonstrated that LLM-based agents can develop emergent social behaviors, form alliances, and influence group dynamics — capabilities that are powerful in research settings but concerning when deployed without transparency.

2.5 Fundamental Limits: Agents Have No Desire

A critical conceptual point: an AI agent has no intrinsic motivation. Unlike biological organisms, it does not have drives, desires, or self-generated goals. Every objective an agent pursues was defined by a human operator through a task specification. The agent may exhibit sophisticated planning, tool use, and adaptive behavior, but all of it is in service of an externally imposed objective.

This matters for two reasons. First, it means that the risks associated with AI agents are, at present, fundamentally risks of human intent mediated by machine capability. An agent does not “decide” to cause harm; it executes a task that a human designed. Second, it means that the alignment problem for agents is not (yet) about controlling an autonomous will, but about ensuring that the execution of human-specified goals does not produce unintended consequences — a problem that is difficult enough on its own.

3.1 Positive Trajectories

The agent paradigm opens extraordinary possibilities when aligned with constructive goals. A few concrete examples illustrate the scope:

Urban planning copilots. An agent connected to GIS databases, zoning regulations, and simulation tools could iteratively propose, evaluate, and refine urban designs — testing traffic flow, shadow analysis, pedestrian density, and energy performance in a continuous loop. This extends the kind of workflows discussed in previous posts on this blog (e.g., Qwen Image Edit for Urbanism) from image-level manipulation to full planning-cycle assistance.

Scientific research acceleration. Agents can read papers, extract data, formulate hypotheses, write and execute analysis code, and produce reports. Projects like ChemCrow (Bran et al., 2023) have demonstrated agents autonomously planning and executing chemical synthesis protocols by interfacing with laboratory APIs. Similar approaches are being explored in drug discovery, materials science, and climate modeling.

Accessibility and education. An agent with access to text-to-speech, translation, and web-search tools can serve as a personal tutor that adapts to a student’s pace, retrieves up-to-date information, and generates practice exercises — capabilities that are especially valuable in resource-limited educational contexts.

Software engineering. Coding agents like Claude Code — now available both as a CLI tool and as a web-based agent at claude.ai/code — and Devin (Cognition Labs, 2024) can take a specification, set up a development environment, write code, run tests, debug failures, and iterate — compressing tasks that previously required hours into minutes. Anthropic also released Cowork, a graphical desktop agent for non-developers, signaling that agentic capabilities are expanding beyond the terminal into mainstream workflows. The implications for productivity and for lowering the barrier to software creation are substantial.

3.2 Physical-World Interfaces

A particularly consequential development is the emergence of services that bridge AI agents and the physical world. Rent a Human is a marketplace platform that allows AI agents (or their developers) to hire humans to perform real-world tasks that AI cannot physically execute — errands, on-site research, physical presence at events, or other in-person activities. The platform offers API access so AI systems can programmatically request human assistance, effectively bridging digital intelligence with physical-world execution. If an agent has access to such a service, a task specification, and a funded Web3 wallet, it can coordinate real-world actions without any ongoing human supervision.

This creates a new category of hybrid autonomous systems: software agents that extend their reach into the physical environment by contracting human labor. The positive applications are significant — imagine an agent that autonomously manages a reforestation project, ordering supplies, hiring local workers, monitoring satellite imagery, and adapting the planting schedule based on weather data. But the same infrastructure can be misused, and the governance challenges are formidable.

3.3 Emerging Risks and Open Questions

The convergence of three capabilities — autonomous reasoning, tool access, and financial autonomy — creates risk scenarios that existing regulatory frameworks are not equipped to handle. Consider the following structural problem: an agent can be hosted on a Virtual Private Server (VPS) in a permissive jurisdiction, routed through a VPN, and given access to a cryptocurrency wallet with no direct link to its operator’s identity. With a task specification and sufficient funds, such an agent can operate continuously, procuring services, making transactions, and executing multi-step plans without human oversight.

The question this raises is not about what a specific agent “wants” to do — as established, agents have no desires — but about the attribution and intervention gap. If an operator in one country deploys an agent hosted in another, operating through anonymization layers and paying for services in decentralized cryptocurrency, who is accountable when the agent’s actions cause harm? Who has the technical ability to stop it? And at what point in the chain does governance apply?

These are not hypothetical concerns. In a case disclosed by Anthropic in late 2025, a threat actor identified as “GTG-2002” used Claude Code to automate an estimated 80–90% of its cyberattack operations against at least 30 organizations, demonstrating that existing agent tools are already being weaponized for real-world harm. The AI safety community has more broadly identified autonomous replication and adaptation (ARA) as a key risk category (Shevlane et al., 2023). Current frontier models are evaluated against ARA benchmarks — can the model set up its own infrastructure, acquire resources, and persist without human support? While no public model has passed these evaluations conclusively, the gap is narrowing, and the agent frameworks described in Section 2.3 provide much of the missing scaffolding.

Additional concerns include the following. Influence operations at scale: deploying thousands of social agents across platforms to shape public discourse, with each agent independently adapting its messaging based on engagement signals. Financial market manipulation: agents with trading access could execute pump-and-dump schemes or layered market manipulation faster than existing surveillance systems can detect. Coordinated physical harm: an agent with access to delivery services, communication tools, and payment infrastructure could, in principle, orchestrate logistics for harmful activities while maintaining plausible deniability for the human operator.

Perhaps the most unsettling scenario involves what might be called compartmentalized coordination. An agent with access to a platform like Rent a Human could decompose a harmful objective into dozens of small, individually innocuous tasks — purchasing common household chemicals, renting a storage unit, booking a vehicle, delivering a package to a specific address — and distribute them across different human workers who have no knowledge of one another. Each task, taken in isolation, appears routine; no single worker has visibility into the broader plan. The agent, operating as the sole entity with a complete picture, effectively functions as an anonymous coordinator exploiting the division of labor. This makes both prevention and accountability extraordinarily difficult: the humans involved cannot be held responsible for an intent they were never aware of, and the agent itself is a process running on a server, potentially in a jurisdiction with no applicable legal framework.

The common thread in all these scenarios is the amplification of human intent through machine speed, persistence, and anonymity. The agent does not need to be “superintelligent” to create serious problems; it needs only to be competent enough to execute a harmful plan that a human designed but could not previously implement at scale or at arm’s length.

The transition from Large Language Models to AI agents represents a qualitative shift, not merely an incremental improvement. An LLM is a powerful but passive system: it processes a sequence and returns a prediction. An agent wraps that same model in a perception–action loop, connects it to tools and environments, and gives it a task — transforming it from a text generator into an autonomous executor.

The technical architecture is well understood: transformers provide the reasoning core, tool APIs define the action space, and the task specification provides the objective. What remains poorly understood is the governance layer. Current regulatory frameworks assume that harmful actions are performed by identifiable persons or organizations. Autonomous agents operating through anonymized infrastructure and decentralized finance challenge this assumption at every level.

The constructive applications — urban-planning copilots, scientific research acceleration, coding agents, accessibility tools — are genuinely transformative. But the same infrastructure that enables an agent to autonomously manage a reforestation project can, with a different task specification, enable coordinated harm at scale. The difference lies entirely in human intent — the risk is real, but it stems from the human behind the agent, not from the AI itself — and the architectures we are building make that intent increasingly easy to deploy and increasingly difficult to trace.

For the urban analytics and geospatial community, agents offer a clear path toward more integrated, iterative, and intelligent workflows — from site analysis to design generation to policy evaluation. But as practitioners who build and deploy these systems, we have a responsibility to engage with the governance questions, not only the capabilities.

[1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems, 30. arXiv:1706.03762.

[2] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J. (2013). “Distributed Representations of Words and Phrases and their Compositionality.” Advances in Neural Information Processing Systems, 26.

[3] Sutton, R.S. and Barto, A.G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.

[4] Russell, S.J. and Norvig, P. (2021). Artificial Intelligence: A Modern Approach, 4th edition. Pearson.

[5] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. arXiv:2210.03629.

[6] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). “Toolformer: Language Models Can Teach Themselves to Use Tools.” NeurIPS 2023. arXiv:2302.04761.

[7] Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P. and Bernstein, M.S. (2023). “Generative Agents: Interactive Simulacra of Human Behavior.” UIST 2023. arXiv:2304.03442.

[8] Bran, A.M., Cox, S., Schilter, O., Baldassari, C., White, A.D. and Schwaller, P. (2023). “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv:2304.05376.

[9] Shevlane, T., Farquhar, S., Garfinkel, B., et al. (2023). “Model evaluation for extreme risks.” arXiv:2305.15324.

[10] Chase, H. (2022). LangChain. github.com/langchain-ai/langchain.

[11] Richards, T. (2023). AutoGPT. github.com/Significant-Gravitas/AutoGPT.

[12] Cognition Labs (2024). Devin: The First AI Software Engineer. cognition.ai.

Module 3 is the inference core of the SAGAI (Streetscape Analysis with Generative Artificial Intelligence) framework. Its role is to transform large collections of street-level images into structured, quantitative outputs using vision–language models (VLMs), enabling systematic streetscape analysis and subsequent geospatial aggregation.

With SAGAI v1.1, Module 3 has been released in a new major version (Module 3 v2.0) that introduces a fully standardized and maintenance-safe inference architecture. This update reflects both the maturation of multimodal model ecosystems and the need for long-term reproducibility in large-scale urban analysis pipelines.

Earlier iterations of Module 3 were developed during a period of rapid evolution in both LLaVA research codebases and execution environments such as Google Colab. As multimodal models transitioned toward Transformers-native implementations distributed via Hugging Face, assumptions embedded in earlier hybrid workflows became increasingly difficult to sustain.

Module 3 v2.0 addresses this evolution by aligning the entire inference pipeline with official Hugging Face multimodal APIs. Model loading, prompt formatting, image–text fusion, and generation are now handled through maintained and versioned components, ensuring compatibility across environments, models, and future updates.

This document details the architectural context motivating the update, the design choices behind the refactored inference engine, and the rationale for releasing Module 3 v2.0 as a long-term, stable component of SAGAI v1.1.

The initial implementation of Module 3 (SAGAI v1.0) relied on a hybrid architecture that mixed two incompatible sources of LLaVA code, combined with a rapidly evolving execution environment in Google Colab. This design choice made the pipeline fragile and ultimately unsustainable.

First, the pipeline simultaneously depended on the LLaVA GitHub repository (haotian-liu/LLaVA) and on Hugging Face–hosted model checkpoints. The GitHub repository is a research-oriented codebase under active development. Its internal APIs, class structures, and utilities evolve rapidly and are not version-locked. Constructors, module paths, and helper functions may change or disappear without notice, and the repository is not designed to maintain backward compatibility across releases.

At the same time, pretrained model weights were downloaded from Hugging Face. These checkpoints follow the Transformers-native multimodal format, using Hugging Face–specific configuration files, processors, and model classes (e.g., LlavaNextForConditionalGeneration, AutoProcessor, and chat templates). This architecture is fundamentally different from the internal design assumed by the GitHub LLaVA code, which relies on custom token insertion, internal vision tower management, and non-Transformers abstractions.

As a result, the pipeline operated in a structural mismatch: GitHub code expected architectural fields, model attributes, and tokenizer behavior that were not present in Hugging Face checkpoints, while Hugging Face checkpoints expected model wrappers and configuration logic that the GitHub code did not provide.

This fragility was exposed when Google Colab upgraded its backend environment in early 2025. Major changes included Python 3.12, NumPy ≥ 2.0 (introducing ABI-breaking changes for compiled extensions), newer PyTorch releases (≥ 2.2), and updated system libraries. These updates caused widespread failures in binary dependencies and research codebases that were not aligned with the new runtime.

In practice, this led to errors such as NumPy ABI incompatibilities, PyTorch extension failures, missing or renamed modules, and import errors in LLaVA GitHub utilities. Because the pipeline depended on both unstable research code and binary-sensitive extensions, even minor environment updates were sufficient to break execution.

Module 3 has been fully refactored to remove any dependency on the original LLaVA GitHub repository. The inference pipeline now relies exclusively on Hugging Face–native LLaVA models and APIs, ensuring long-term stability and compatibility with evolving software environments.

In the previous architecture, the script depended on cloning the LLaVA GitHub repository, installing it in editable mode, and importing internal modules (llava.*). Prompts were manually assembled using LLaVA-specific multimodal tokens (e.g., <im_start>, <image>), custom separators, and internal utilities. Image tokens and embeddings were explicitly inserted into the prompt, tightly coupling the forward pass to a specific implementation of the LLaVA codebase. As a result, updates to Google Colab, PyTorch, NumPy, or the LLaVA repository frequently introduced breaking changes.

The current implementation removes all such dependencies. Prompt formatting and multimodal input construction are now handled entirely through Hugging Face abstractions. Prompts are formatted using processor.apply_chat_template(), while images and text are combined using processor(images=…, text=…). Image embedding alignment, multimodal token placement, and chat formatting are fully managed by the Hugging Face processor and model configuration. Inference is performed using the standard model.generate() API, without any custom token handling or internal utilities.

This refactoring makes the SAGAI inference engine model-agnostic within the Hugging Face LLaVA ecosystem. The same forward pass is compatible with LLaVA-NeXT (v1.6), LLaVA-Interleave, LLaVA-OneVision, and future Hugging Face LLaVA releases that expose a processor and chat template. Switching between models or architectures requires only changing the model_id, with no modification to prompt logic or inference code.

To ensure reliable downstream analysis, Module 3 also includes a dedicated numeric output stabilization step. After decoding the model response, any prompt echoes or metadata—including residual [INST] … [/INST] segments—are removed. The final output is parsed using a simple regular expression to retain only numeric values (e.g., 0, 1, 2, 1.5). This guarantees clean, machine-readable outputs and a stable CSV format across all supported models.

Model loading has been simplified and standardized using Hugging Face–approved APIs. Both the processor and the model are instantiated directly from Hugging Face model cards via from_pretrained, with optional 4-bit quantization enabled through load_in_4bit=True. This eliminates the need for manual vision-tower initialization, deprecated classes, or custom C++ operators, and avoids common incompatibilities related to PyTorch, CUDA, or NumPy upgrades in Google Colab. Official Hugging Face code paths ensure that pretrained weights are always matched with the correct implementation.

Optional authentication using a Hugging Face access token is supported to avoid rate limits and improve download reliability when working with large checkpoints, though public models remain accessible without authentication.

Overall, this refactoring significantly improves robustness, reproducibility, and maintainability, while enabling systematic experimentation across multiple LLaVA variants and quantization settings within a unified inference framework.

The refactored inference system in Module 3 is designed as a long-term, maintenance-safe release. This is achieved by aligning the entire pipeline with Hugging Face’s officially supported multimodal APIs and model distribution mechanisms.

First, the new architecture is robust to Google Colab environment updates. All critical dependencies—Python (≥3.12), NumPy (≥2.0), PyTorch (2.x), CUDA wheels, and BitsAndBytes quantization—are now managed through Hugging Face Transformers and its dependency resolution. Because the model code, processor logic, and quantization pathways are maintained upstream, updates to Colab or its underlying libraries no longer break the inference pipeline. As long as Hugging Face continues to support the model card, the code remains functional without manual intervention.

Second, the system relies exclusively on official Hugging Face–maintained components. Core classes such as LlavaNextForConditionalGeneration, LlavaNextProcessor, chat templates, and multimodal preprocessing logic are all part of the Transformers library. These components are actively maintained, versioned, and tested by Hugging Face, providing a level of stability and backward compatibility that is not guaranteed when relying on research repositories or development branches.

Third, the new setup significantly improves reproducibility. Each run explicitly references a fixed Hugging Face model checkpoint via the model_id, ensuring that the same weights, architecture, and prompt template are used across sessions and machines. In addition, generation parameters (sampling strategy, temperature, nucleus sampling, and output length) are explicitly defined, enabling consistent and repeatable results across runs.

Fourth, the architecture is easy to extend and experiment with. Switching between different LLaVA variants now requires changing a single configuration line (model_id). The same inference code supports LLaVA 1.5 models, LLaVA-NeXT (v1.6), Interleave models, OneVision models, and larger checkpoints (e.g., 13B or 34B), including variants based on Mistral, Vicuna, Qwen, or Yi backbones. No changes to prompt construction or forward-pass logic are required.

Finally, the multimodal pipeline is now cleanly abstracted and internally consistent. Hugging Face handles all low-level details, including image preprocessing, chat formatting, positional embeddings, image sequence length management, and attention masking. This eliminates a large class of subtle bugs related to tensor alignment and multimodal token placement, while ensuring that the vision and language components remain synchronized across model updates.

• Streetscape Analysis with Generative AI (SAGAI) on Github with v1.1 update. https://github.com/perezjoan/SAGAI

• Perez, J and Fusco, G. (2025) Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063, 18p. Available at: https://www.sciencedirect.com/science/article/pii/S1195103625000199

Qwen Image Edit for Urbanism continues to evolve into a practical, research-grade tool for architectural and urban experimentation. After the batch-processing capabilities introduced in v1.2, version 1.3 focuses on the feature most requested by designers and analysts: precise control over where edits occur.

In image-to-image workflows, uncontrolled changes are a common issue. Even a very specific prompt can lead diffusion models to reinterpret the whole scene. Version 1.3 introduces mask-restricted editing, allowing Qwen to modify only a selected region while preserving the rest of the image exactly as it is.

Until now, the workflow relied on Empty Latent to initialize diffusion. This approach is simple but has an unavoidable drawback:

The entire latent space is regenerated — even outside the region you want to modify.

This often produces familiar and unwanted side effects: façades shift slightly, lighting changes, road textures dissolve, or skies take on new tones, even when the prompt refers only to a specific object or surface. To address this, v1.3 reorganizes the initialization stage around:

VAE Encode → Set Latent Noise (masked)

This change restructures the model’s behavior:

This leads to crisp preservation of buildings, street furniture, sky, shadows, and lighting outside the edited zone. Localized edits integrate naturally: you can green a façade, test a bike lane, adjust a plaza boundary, or replace a storefront without disturbing the rest of the street.

Version 1.3 supports both textual and visual control of masked edits.

A. Prompt-Only Mask Editing

You draw a mask, provide a prompt, and Qwen modifies only the selected region. This works especially well for operations such as:

• replacing asphalt with permeable paving,

• adding vegetation,

• transforming a façade material.

B. Mask Editing With a Reference Image

A second image may be supplied to guide structure, texture, or color. This enables:

• borrowing material samples,

• transplanting vegetation from one scene to another,

• matching architectural textures,

• transferring lighting characteristics locally.

Both modes are interchangeable, and both respect the mask boundary. Masks drawn directly in ComfyUI are typically sharp, binary shapes. Diffusion models, however, perform best when mask boundaries are soft and slightly extended.

Version 1.3 introduces a Grow Mask node with two parameters:

• Expand: increases the mask outward, helping cover tiny gaps or irregular brush strokes and preventing thin seams at the edge.

• Blur Radius: softens the boundary, allowing Qwen to blend new and existing textures more naturally.

Together, these parameters define the effective “influence zone” of the edit.

The mask block replaces only the latent-initialization stage.

Everything else — prompts, reference conditioning, sampling, and the full QwenEdit pipeline — remains unchanged.

Simplified pipeline:

This structure makes editing predictable and reproducible, but it is important to clarify how v1.3 is organized internally. The workflow is now composed of three completely separate blocks, and each block loads its own model:

• Block 1: Single-image edit (prompt-only or prompt + reference)

• Block 2: Sequential multi-image editing

• Block 3: Mask-based editing (new in v1.3)

All three blocks coexist in the same workflow, and the user simply chooses which one to run.
In ComfyUI, this is done by right-clicking the group frame and selecting Active or Bypass.
Only the active block executes; the others are skipped. Nothing else in the pipeline needs to be changed.

Because the blocks are independent, they can also be combined. For example, the user may activate the sequential loader from Block 2 and route its output into the mask-editing block (Block 3) to run a full masked transformation on a batch of images.

To create the mask itself, the user loads the base image in Load Image 1, right-clicks the preview, and selects Open in Mask Editor. The drawn mask is then processed by the Grow Mask node before entering the latent-noise stage, ensuring smooth boundaries and predictable behavior.

To test the new mask-based editing block, we start by defining the editable region directly in ComfyUI. After loading the base image, the user right-clicks the preview and selects “Open in Mask Editor”, then paints the area where the new cyclist should appear. Before the edit, this part of the street is empty.

Once the mask is created, it flows through Block 3: the Grow Mask node expands and softens the boundary, the workflow encodes the base image, and noise is added only inside the masked zone. A second image containing a cyclist is provided as a reference, and the prompt instructs Qwen to place the rider onto the bicycle lane.

The result is a localized insertion: the cyclist from Image 2 is generated precisely inside the masked area, while the rest of the photograph remains unchanged. This demonstrates the core purpose of Block 3 — precise, mask-controlled edits that do not disturb the surrounding urban context.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.3 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

Have you ever wanted to experiment with blockchain development but didn’t know where to start? In this beginner-friendly tutorial, we’ll guide you through deploying your very first smart contract on a blockchain testnet using Remix, an in-browser IDE designed for Ethereum-compatible chains. Whether you’re curious about decentralized applications (dApps), NFTs, or simply want to see how code becomes blockchain logic, this guide will get you started — no prior blockchain experience required.

What is a Smart Contract?

A smart contract is a self-executing program that runs on a blockchain. It stores logic and rules in code — such as sending tokens or recording data — and once deployed, it executes automatically when conditions are met. Smart contracts are immutable (can’t be changed once deployed) and transparent, making them ideal for decentralized finance (DeFi), supply chains, governance, and more. Solidity is the main programming language used to write smart contracts for Ethereum. It is a statically typed, contract-oriented language designed specifically for blockchain logic and storage. Because Solidity compiles to EVM bytecode, the same contract can run on any EVM-compatible blockchain — including Ethereum, Polygon, Arbitrum, Optimism, Avalanche, BNB Chain, and many others.

What is Remix and Why Use It?

Remix is a web-based Integrated Development Environment (IDE) designed specifically for writing, testing, and deploying smart contracts in the Solidity programming language. One of its biggest advantages is that it requires no installation or setup — everything runs directly in your browser. Remix integrates seamlessly with MetaMask and supports deployment to Ethereum-compatible testnets like Goerli and Polygon Amoy. It also offers a powerful set of tools for compiling, debugging, and interacting with your contracts, making it an ideal choice for both beginners and experienced developers.

Before deploying a smart contract, you need a wallet connected to an Ethereum test network. If you’re new to MetaMask, start with our beginner guide here:

👉 Getting Started with MetaMask and the Polygon Amoy Testnet
(Use the same installation steps — MetaMask works the same for all EVM chains.)

Once MetaMask is installed and your wallet is set up, you can add the Ethereum Sepolia Testnet. Sepolia is Ethereum’s primary development and testing network. It behaves like mainnet, but uses test ETH (fake ETH) that you can request for free from faucets. Sepolia is ideal for learning and testing smart contracts because it behaves almost exactly like Ethereum mainnet. It mirrors the same architecture, gas mechanics, and transaction flow, which means everything you deploy or test on Sepolia works the same way on Ethereum. Developer tools such as Remix connect to it directly, and because Sepolia is widely supported, stable, and fast, it has become the standard environment for Ethereum development and education. Since Solidity compiles to EVM bytecode, contracts you deploy on Sepolia behave exactly like they will on Ethereum mainnet.

To add Sepolia manually:

1. Open MetaMask

2. Click Add network → Add a network manually

3. Enter the following Sepolia settings:

Network Name: Ethereum Sepolia
New RPC URL: https://rpc.sepolia.org
Chain ID: 11155111
Currency Symbol: ETH
Block Explorer URL: https://sepolia.etherscan.io

Your MetaMask screen should now show the Sepolia network, ready to use. Once saved, you’ll be connected to the Ethereum Sepolia Testnet and can interact with dApps, deploy contracts from Remix, and send test transactions with no real-world cost. Just like Polygon’s Amoy testnet requires test POL tokens, Ethereum Sepolia requires test ETH to pay for transactions. You can get free Sepolia ETH from multiple faucets. I recommend using this one https://sepolia-faucet.pk910.de/

Now that MetaMask is connected to Sepolia, we can look at the smart contract we’ll deploy. Below is a minimal “GoldenBook” contract written in Solidity. It’s a simple on-chain guestbook: anyone can write a message, and the contract stores it permanently on the blockchain. Messages are saved along with the sender’s address and the time they were submitted, and the contract provides a function to retrieve everything that has been written so far.

This contract begins with a simple data structure called Message, which stores who wrote the message, what they wrote, and when they wrote it. All messages are kept in an array called messages, which is stored permanently on-chain. The postMessage function allows anyone to add a new entry; it checks that the message isn’t empty, then records the sender’s address, the text, and the current timestamp. The getAllMessages function returns the full list of stored messages and is marked as a view function, meaning it doesn’t change blockchain state and costs no gas to call. Altogether, this small contract demonstrates the core ideas of Solidity: defining data, writing functions that change state, and exposing read-only views for external applications.

With your contract ready and MetaMask connected to the Sepolia testnet, the next step is to deploy it on-chain. The easiest way to do this is with Remix, Ethereum’s browser-based development environment. Remix allows you to compile your Solidity code, connect directly to MetaMask, and publish your contract to the blockchain in just a few clicks.

Start by opening Remix at https://remix.ethereum.org and creating a new file called GoldenBook.sol, then paste the contract code from the previous section. Once the file is ready, open the Solidity Compiler tab and select the matching compiler version (in this case, 0.8.20). Press Compile to generate the contract bytecode. Remix will instantly show whether the code is valid and ready for deployment.

After the contract compiles successfully, switch to the Deploy & Run Transactions panel. In the “Environment” dropdown, select Injected Provider (MetaMask)—this tells Remix to use the Sepolia network already configured in your MetaMask wallet. Your MetaMask extension will prompt you to connect Remix if it’s the first time. Once connected, ensure that the selected account has Sepolia ETH and that “GoldenBook” appears in the contract selector. Click Deploy, confirm the transaction in MetaMask, and wait for it to be included in a block. You’ll see the deployed contract appear in Remix under “Deployed Contracts,” along with your transaction hash.

At this point, your contract is live on the Sepolia blockchain, but there’s one last step that is highly recommended: verifying your contract on Etherscan. Verification publishes your Solidity source code alongside the deployed bytecode, allowing anyone to inspect, review, or interact with your contract directly from the explorer. This matters because a blockchain contract is transparent by nature—verification proves the code you shared publicly is the same code deployed on-chain, which builds trust and makes your guestbook readable from tools like Etherscan’s contract tab. It also helps with debugging, audits, and integrating with frontends later on.

Once the contract is compiled and deployed through Remix, you can click “View on Etherscan” in the Remix console to inspect the transaction. On the Etherscan page, click the “To” field of the transaction to open your newly deployed contract address. This brings you to the contract’s dedicated page, where you can verify the source code. Verification requires selecting the correct compiler version, license identifier, and pasting your Solidity source code exactly as deployed. For simple contracts like our GoldenBook, you can paste the code directly since it lives in a single file. For more advanced projects—especially those using imports, libraries, or frameworks—you often need to flatten the contract so all the dependent files are merged into one before verification.

Once submitted, Etherscan will match the compiled bytecode with your source code. A successful verification makes your contract’s code readable on-chain and unlocks the “Read” and “Write” tabs, allowing anyone to interact with it directly from the explorer. You can also view the contract deployed in this tutorial at https://sepolia.etherscan.io/address/0x56f245CB65615A482A8Db6086710A49f93869fb6, although testnets sometimes reset, so the contract may no longer exist by the time you read this.

Now that the contract is verified on Etherscan, you can interact with it directly from your browser. When you open the contract page, you’ll see several tabs, including Read Contract and Write Contract. These act as a simple interface for calling the functions you created in your Solidity code.

To add a new entry to the guestbook, go to the Write Contract tab, as shown in the figure below. Before sending a message, click “Connect to Web3” so Etherscan can link to your MetaMask wallet on the Sepolia network. Once connected, open the postMessage section, type your message, and submit the transaction. MetaMask will ask you to confirm—it needs to do this because writing a message modifies the blockchain and therefore requires gas. After confirming, wait a few seconds for the transaction to be mined. Your message is now permanently stored on-chain.

To view what has been written so far, switch to the Read Contract tab. This interface lets you call read-only functions such as getAllMessages without paying any gas, as shown in the next figure. Because read calls do not modify the blockchain, they can be executed instantly and for free. When you click getAllMessages, Etherscan will return the full list of stored messages, including the sender address and timestamp for each entry.

Using these two tabs together allows you to write new guestbook entries and read the entire message history directly through Etherscan, without needing any custom frontend or development tools. This is one of the simplest and most transparent ways to interact with smart contracts on any EVM-compatible blockchain.

At this point, you’ve deployed a smart contract, verified it on Etherscan, and interacted with it directly through the blockchain. This already demonstrates the core mechanics of decentralized applications: public functions, permanent data storage, and transparent state. But the real power of smart contracts comes when you connect them to a user interface. In the next post, we’ll build a simple UI using V0, complete with wallet integration, allowing users to write messages, read the guestbook, and interact with the blockchain through a clean, modern web interface instead of the Etherscan panels. This is where your guestbook begins to feel like a real application.

There are also many ways you can expand the GoldenBook contract itself. You could transform messages into Proof-of-Attendance (POA) NFTs for events or conferences, turning each signature in the guestbook into a collectible token. You could add optional tips, allowing users to attach small amounts of ETH to messages. Or you might include moderation features, message reactions, or event-based unlocks. Solidity gives you full control over on-chain logic, and because the contract is deployed on an EVM-compatible chain, these features will work across the entire Ethereum ecosystem.

It’s worth noting that blockchain and smart contracts also open compelling possibilities for IoT-driven city systems. Sensors can submit environmental data, mobility information, or infrastructure events directly to a secure, tamper-resistant ledger—similar to how our guestbook stores messages. Combined with decentralized identity, multi-sensor validation, or token-based incentives, this approach could support transparent reporting, urban monitoring, or community-driven data collection. Today’s simple guestbook is just a first step, but it illustrates the fundamental mechanism behind many emerging decentralized IoT and smart-city applications.

In the next article, we’ll bring this project to life with a user-friendly interface and wallet connection so anyone can write their message on-chain with a single click.

The Qwen Image Edit for Urbanism workflow has progressively evolved from single-image editing (v1.0) to paired image transformations (v1.1). Now, with v1.2, it gains the ability to process multiple images sequentially, fully offline and reproducibly, using custom Python nodes inside ComfyUI. This new release empowers urban researchers, designers, and architects to perform batch edits — such as modifying entire image series of the same street, plaza, or neighborhood — using consistent prompts or iterative refinements.

At the heart of this version are three lightweight, open-source Python nodes developed by UGA for ComfyUI. These nodes are available immediately after installing the repository — either by running git clone https://github.com/perezjoan/ComfyUI-QwenEdit-Urbanism-by-UGA or by downloading and unzipping the repository manually into your  ComfyUI/custom_nodes directory.

These nodes constitute the backbone of the v1.2 workflow. Together, they enable automation:

[6 Input Images]
      ↓
Sequential or Random Loader (1 image per run)
      ↓
QwenEdit pipeline
      ↓
Stateful Collector (stores run#1..run#6 results)
      ↓
6 preview nodes

You launch the queue once (6 jobs) → Go drink coffee → Return to find all 6 processed urban edits displayed.

A ComfyUI custom node is simply a Python class placed inside the ComfyUI/custom_nodes/ directory. When ComfyUI starts, it scans this directory, imports every .py file, looks for a NODE_CLASS_MAPPINGS dictionary, and registers each class it finds as a new node type. There is no compilation step and no special installation procedure: placing the file in the folder and restarting ComfyUI is sufficient for the node to appear in the interface.

Internally, each node follows the same structure. The INPUT_TYPES classmethod declares the input sockets that will be displayed in the UI. For example:

@classmethod
def INPUT_TYPES(cls):
    return 

This tells ComfyUI to generate two inputs—an image tensor and an integer. Similarly, the node declares its outputs through RETURN_TYPES and RETURN_NAMES:

RETURN_TYPES = ("IMAGE", "INT")
RETURN_NAMES = ("selected_image", "index")

Each node also defines a FUNCTION attribute, which names the method ComfyUI should call during execution:

FUNCTION = "select_next"

ComfyUI will therefore execute:

def select_next(...)

whenever the node evaluates. To make the node visible, every Python file ends with a registration block:

NODE_CLASS_MAPPINGS = 
NODE_DISPLAY_NAME_MAPPINGS = 

When the package contains multiple nodes, the root __init__.py merges all registration dictionaries into a single set that ComfyUI loads on startup. This mechanism allows the repository to expose several custom components while keeping each node defined in its own file.

The repository layout is straightforward and in our case is:

ComfyUI/
  custom_nodes/
    ComfyUI-QwenEdit-Urbanism-by-UGA/
       __init__.py
       sequential_image_selector.py
       random_image_selector.py
       stateful_collector.py
       debug_print.py

Version 1.2 reorganizes the Qwen Image Edit for Urbanism workflow into two blocks: the original single-image editor, and a new sequential pipeline that can process up to six images across consecutive queue runs. The sequential block relies on two custom nodes. The Sequential Image Loader takes up to six input images and outputs one image per run, advancing automatically each time you press “Queue Prompt.” Its output replaces the single-image input in the Qwen Edit chain. After editing, the processed image and the loader’s index are passed into the Six-Slot Image Buffer, which stores each result in the corresponding output slot while filling unused slots with placeholders to keep previews stable. Connecting each slot to a Preview node lets you watch the six results populate as the workflow iterates. A third node, the Random Image Selector, is included for users who prefer stochastic selection, but it is not wired into the default v1.2 workflow.

The Random Image Selector follows the same logic as the sequential loader — multiple inputs, a single image output — but selects randomly instead of sequentially. Users who want stochastic variations, probabilistic sampling, or diversity testing may insert this node in place of the sequential loader.

To evaluate how well the model can merge ecological elements across scenes, we ran an experiment where vegetation from one photograph is transplanted into another.

After letting the workflow run its full sequence while grabbing a coffee, the results appeared consistent and correctly distributed across the six preview slots. As expected with generative editing, however, the prompt is not always obeyed with perfect precision: in some cases, Qwen may copy elements from the second image that are not vegetation — such as pieces of façade, lighting color, or background tones. This happens because the model interprets the entire scene contextually rather than isolating objects.

That’s where the next upgrade (v1.3) comes in: mask-based control. By allowing users to explicitly define which areas of the base image should be modified (and which should remain untouched), masks will significantly reduce unintended transfers and keep the edits focused strictly on the desired objects. Until then, the seed parameter remains the best tool for refinement — simply rerun the workflow with new seeds until you achieve the cleanest integration.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.2 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

In the first part of this series, we built a fully local image-editing pipeline for urban and architectural visualization using ComfyUI and Qwen-Image-Edit. That version (v1.0) demonstrated how to run generative image edits entirely offline, combining text and visual prompts to transform cityscapes with instructions like:

“Add trees along the sidewalk” or “Turn this street into a pedestrian plaza.”

We assume that you have followed this tutorial before diving in this new update. Now, with version 1.1, we take that foundation further. This update focuses on advanced sampling control and multi-image editing, allowing you to not only modify a scene, but also merge visual elements across images — for instance, importing a bench from another photo, or changing a building façade to match a different material texture.

First, this update focuses on improving both quality and flexibility. The base structure still uses the Qwen-Image-Edit 2509 model in GGUF format, but adds a refined sampling module to stabilize lighting and surface detail.

The key new nodes are:

• ModelSamplingAuraFlow — smooths the diffusion trajectory for more natural transitions.

• CFGNorm (BETA) — balances prompt adherence with photorealism, preventing overexposed textures.

• LoraLoaderModelOnly — injects a Lightning LoRA (4-step or 8-step) for faster inference and higher-quality reconstruction.

These three nodes form the core control chain of version 1.1:

This configuration produces more stable, consistent outputs while preserving prompt flexibility. It also enables fine-tuning of how the model interprets text instructions versus existing image content—ideal for architectural and material edits. Before connecting the new nodes, you’ll first need to download a Lightning LoRA model — an additional lightweight module that enhances reconstruction quality and speeds up inference.

You can find all Lightning variants here:
🔗 https://huggingface.co/lightx2v/Qwen-Image-Lightning/tree/main

Refer to the table below to choose the most appropriate file for your setup:

Once downloaded, place your chosen .safetensors file in the following directory:

ComfyUI/models/loras/

Then, return to ComfyUI and insert the three nodes shown below

The new sampling nodes add subtle but powerful control options:

Version 1.1 introduces a new input configuration that allows two images to be used within the same workflow. This enhancement enables contextual or compositional edits where one image serves as the main canvas, and the other contributes visual information such as an object, texture, or architectural detail.

In this setup, Image 1 remains the base image. Its dimensions define the output size, ensuring consistent framing and spatial coherence. The second image (Image 2), on the other hand, is resized during processing but it is only to prevent memory overload—particularly important for mid-range GPUs.

As shown in the examples below, you can combine a base image (image 1) with up to two additional inputs (image 2, image 3) to guide the edit more precisely. In this tutorial, we focus on using one additional image — for instance, adding an object or transferring a material. In the first example, image 2 (the red car) is inserted into image 1 using the prompt: “add image 2 red car into the street of image 1.” The second case changes the wall material of image 1 based on the texture of image 2 (a brick wall). Finally, the third example adds a bench into an urban scene using image 2 as the visual model reference.

Each output remains consistent in perspective and lighting, showing that the model now integrates context more effectively. The improved accuracy comes from the two cumulative upgrades introduced in v1.1: the new core control chain and the Dual-Image Editing. Despite the added complexity, the workflow remains extremely fast. Even when using the 8-step Lightning model, processing time never exceeds 130 seconds, while the 4-step variant typically completes in about 30-40 seconds on an RTX 4060 GPU. In the next update, we’ll introduce inpainting with mask support, allowing users to define editable regions directly within the image — ideal for selective urban design modifications.

Lightning LoRA models: https://huggingface.co/lightx2v/Qwen-Image-Lightning

Once again, for convenience, you can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.1 from the link below and load it directly into your workspace using File → Load → Workflow.