When we released UVLM in March 2026, it was a Google Colab notebook. You opened it in your browser, picked a model, typed your prompts, and ran your images — all without installing anything. That simplicity was the point: a tool that anyone could use to load and compare Vision-Language Models, regardless of their technical setup.

But we kept hearing the same requests. Can I run this on my own machine? Can I call UVLM from a script? Can I integrate it into an existing pipeline? The answer was always the same: not easily. The entire tool lived inside a single notebook, with all the logic packed into three massive code cells. Moving it anywhere else meant copy-pasting thousands of lines and untangling global variables.

Version 3.0.0 changes that. UVLM is now a proper Python package.

The core logic — model loading, dual-backend inference, response parsing, consensus validation, batch processing — has been extracted from the notebook into eight standalone Python modules. These modules have no dependency on Google Colab, no global variables, and no widget code. They are plain Python functions that accept arguments and return results.

The package is installed from GitHub in one line:

pip install git+https://github.com/perezjoan/UVLM.git

On Google Colab, this happens automatically in the first cell of the Colab notebook. On your local machine, you run it once in a terminal and you are done.

Nothing changed in how UVLM analyses images. The same 11 model checkpoints are supported (LLaVA-NeXT and Qwen2.5-VL, from 3B to 110B parameters). The same parsing logic, the same consensus validation, the same truncation detection. If you had a workflow built on v2.2.2, the outputs will be identical.

Google Colab — Zero Install

This is the same experience as before. Open the Colab notebook, select a GPU runtime, and start working. The notebook installs the UVLM package automatically. Images are loaded from Google Drive. Nothing has changed for Colab users, except that the code running behind the widgets is now cleaner and easier to maintain.

Local Jupyter Notebook — Your GPU, Your Data

If you have an NVIDIA GPU on your workstation (or access to a GPU server), you can now run UVLM locally. The local Jupyter notebook provides the same widget-based interface — model selection dropdown, prompt builder form, batch execution button — but images are read from your local filesystem and results are saved locally. No Google account needed, no data leaves your machine.

This matters for researchers working with sensitive imagery (medical, security, proprietary datasets) or for anyone who wants faster and more reliable model loading than what Colab’s network provides.

Python Script — Full Programmatic Control

For integration into larger pipelines, UVLM now exposes a clean API. Three lines of code replace the entire notebook workflow:

from uvlm import load_model, run_inference, parse_response
ctx = load_model("[Qwen] Qwen2.5-VL 7B Instruct", precision="4bit")
raw, tokens = run_inference("photo.jpg", "Count the cars", ctx)
result = parse_response(raw, "numeric")

The `load_model()` function returns a context dictionary containing the model, processor, backend type, and device information. This dictionary is passed to every subsequent function — no global state, no hidden side effects. You can load multiple models in the same session and switch between them by passing different context objects.

For batch processing, `run_batch()` handles the full pipeline:

from uvlm import load_model
from uvlm.batch import run_batch

ctx = load_model("[Qwen]  Qwen2.5-VL 7B Instruct", precision="4bit")
df = run_batch(
    model_ctx=ctx,
    task_specs=my_tasks,
    image_folder="./images",
    output_path="./results.csv",
)

The monolithic notebook has been split into eight modules, each with a single responsibility:

registry.py holds the model dictionary — 11 checkpoints with their backend type and HuggingFace checkpoint ID. Adding a new model is one line in a dictionary.

loader.py contains the `load_model()` function. It handles quantisation configuration (4-bit, 8-bit, FP16), device placement (single GPU, auto, CPU offload), and the LLaVA vs Qwen branching logic. It returns a dictionary — not a set of global variables.

inference.py contains `run_inference()`, the dual-backend forward pass. It accepts a model context dictionary and returns the raw response plus the exact token count as a tuple. The full LLaVA response cleaning logic and the full Qwen token-trimming pipeline are preserved exactly as they were.

parsers.py holds the four response parsers (numeric, category, boolean, text) and the advanced reasoning parser. These are pure functions with zero dependencies beyond Python’s standard library.

consensus.py contains the majority voting logic. batch.py handles folder iteration, CSV writing, resume mode, and schema upgrading. prompts.py stores the task type definitions and the chain-of-thought templates. utils.py provides seed management, environment detection, and HuggingFace token retrieval.

On Colab: Open the notebook from GitHub and run the three blocks as before. The package installs itself.

Locally: First, install PyTorch with CUDA support matching your GPU driver (check with `nvidia-smi`). For example, with CUDA 12.8+:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

Then open the local Jupyter notebook.

You get the same dropdown menus, the same prompt builder form, the same batch execution. The only difference is that you type a local path for your image folder instead of a Google Drive path.

For HuggingFace authentication (needed for some gated models like LLaMA3-based checkpoints), either set the `HF_TOKEN` environment variable or run `huggingface-cli login` once in your terminal.

The package architecture makes it much easier to add new VLM families. InternVL, BLIP-2, CogVLM, DeepSeek-VL, and Molmo are planned for future releases — each one requires implementing the backend-specific sections of the inference function and adding entries to the registry, without touching the rest of the codebase.

We are also working on multi-GPU batching for parallel inference across images, video frame analysis support, and integration with the SAGAI workflow for automated streetscape analysis.

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Software & Algorithms › UVLM

Previous blog post: Introducing UVLM: A Free Tool to Compare AI Models That Understand Images

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Imagine you have thousands of street photographs and you need to answer the same questions about each one: how many cars are parked? Is there a sidewalk? How long is the building frontage? Hiring someone to go through every image manually would take weeks. Training a custom computer vision model would take months. But what if you could simply ask an AI model these questions in plain English — and get structured, usable answers back?

That is exactly what Vision-Language Models do. And today, we are releasing UVLM — an open-source tool that makes it easy to load, test, and compare these models, all from a single notebook in your browser.

Vision-Language Models (VLMs) are AI systems that can look at an image and answer questions about it in natural language. Unlike traditional computer vision, which requires training a separate model for every task (one for counting cars, another for detecting sidewalks, a third for classifying buildings), a VLM handles all of these through text prompts. You write a question, attach a photo, and the model responds.

For example, you can ask a VLM: “Count all motor vehicles visible in this image” and it will answer “3”. You can ask the same model “Is there a sidewalk along the street frontage?” and it will answer “yes”. You can even ask it to estimate the length of a building facade in meters — a task that requires the model to identify reference objects (like parked cars), estimate their size, and reason about perspective. All of this from a single model, with no retraining and no labelled dataset.

The catch is that there are many VLM families available (LLaVA, Qwen, InternVL, BLIP-2, and more), and each one works differently under the hood. They use different image encoders, different tokenisation strategies, and different code to run. If you want to know which model is best for your specific task, you normally have to write separate code for each one — a tedious and error-prone process.

UVLM (Universal Vision-Language Model Loader) is a free, open-source tool that lets you load, configure, and compare multiple VLM architectures using the same prompts and the same evaluation protocol — without writing any model-specific code. It runs entirely in Google Colab, which means you do not need to install anything on your computer or own a GPU. A free Google account is all you need.

The idea is simple: you pick a model from a dropdown menu, type your analysis questions into a form, point the tool at a folder of images, and hit run. UVLM handles all the technical details — the processor classes, the tokenisation, the generation settings, the output parsing — and delivers a clean CSV file with one row per image and one column per task. If you want to try a different model, you just switch the dropdown and run again. Same prompts, same images, same output format. Now you can compare.

To demonstrate what UVLM can do, we benchmarked 8 different models on 120 street-level photographs of French urban frontages. Each image was analysed on five tasks: counting vehicles, detecting sidewalks, counting pedestrian entrances, estimating the street frontage length in meters, and classifying the vegetation type. That is 16 model configurations (each model tested in standard and advanced reasoning modes), 120 images, and 5 tasks per image — all processed and compared through UVLM.

The results were revealing. The largest model (LLaVA 34B, with 34 billion parameters) actually ranked last overall. A much smaller model (LLaVA Vicuna 7B) outperformed it significantly and ran on a free Google Colab GPU. The best overall results came from Qwen 32B with chain-of-thought reasoning enabled, which achieved 88% proximity to human expert annotations across all five tasks. Without UVLM, discovering these differences would have required writing and debugging eight separate inference pipelines.

UVLM was designed for anyone who works with images and wants to extract structured information from them at scale — without becoming a machine learning engineer. If you are an urban planner evaluating streetscape quality across a city, UVLM lets you score thousands of street photographs using natural language prompts. If you are an environmental researcher classifying vegetation from field photographs, UVLM lets you test which AI model gives the most reliable results for your specific classification scheme. If you are an infrastructure inspector processing damage assessment photographs, UVLM lets you set up automated counting and scoring tasks and run them across your entire image archive.

The tool is also valuable for AI researchers who need a controlled benchmarking environment. Because UVLM ensures that every model receives exactly the same prompt and is evaluated with the same metrics, it produces fair, reproducible comparisons. The consensus validation feature (running each task multiple times and taking a majority vote) addresses the inherent randomness of AI outputs, and the truncation detection feature flags when a model’s response was cut off before it could finish — a common but often invisible source of errors.

Getting started takes about five minutes. Open the UVLM notebook from GitHub (the link is below), connect to a GPU runtime in Google Colab, and run the first block to load a model. The second block gives you a form where you type your analysis questions — no coding required. The third block processes your images and saves the results as a CSV file on your Google Drive.

The tool currently supports 11 model checkpoints from two major families (LLaVA-NeXT and Qwen2.5-VL), ranging from 3 billion to 110 billion parameters. Models up to 34B can run on a single free-tier Colab GPU with 4-bit quantisation. Advanced features include consensus validation (2–5 runs per task with majority voting), chain-of-thought reasoning for complex tasks, and automatic truncation detection.

UVLM is released under the Apache 2.0 open-source licence. You can use it, modify it, and build on it for any purpose — academic or commercial.

Links

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Softwares & Algorithms › UVLM

Benchmark dataset: Zenodo — 120 street-view images

Citation

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Large Language Models (LLMs) such as GPT-4, Claude, Llama, and Qwen are deep neural networks trained to predict the next token in a sequence. Although the outputs they produce can appear creative, conversational, or analytical, the underlying mechanism is statistical: the model assigns a probability distribution over the vocabulary at each step and samples from it. Understanding the architecture behind this process is essential before discussing what happens when these models are placed inside autonomous loops.

1.1 Tokenization and Embeddings

Raw text cannot be processed by a neural network directly. It is first decomposed into tokens — subword units that may represent a full word, a syllable, or a single character depending on the tokenizer. Each token is then mapped to a high-dimensional vector called an embedding. These embeddings are not fixed lookup tables: they are learned during training and encode rich semantic relationships. Words with similar meanings occupy nearby regions of the embedding space, while syntactic and relational structure is distributed across dimensions in ways that linear algebra can partially recover (Mikolov et al., 2013).

In addition to token embeddings, modern transformers add positional encodings — either sinusoidal functions or learned vectors — so that the model can distinguish the order of tokens in a sequence. The sum of the token embedding and the positional encoding forms the initial representation that enters the transformer stack.

1.2 The Transformer and Self-Attention

The core computational unit of every modern LLM is the transformer, introduced by Vaswani et al. (2017) in the landmark paper “Attention Is All You Need.” The key innovation was the self-attention mechanism, which allows every token in a sequence to attend to every other token, weighted by learned relevance scores. Concretely, each token produces three vectors — a Query, a Key, and a Value — and attention weights are computed as the scaled dot product of queries and keys. The result is a context-aware representation of each token that integrates information from the entire sequence.

Self-attention can be thought of as a set of dynamic spotlights (illustrated in Figure 1 below): for each token being processed, the model learns which other tokens in the context are most relevant and “illuminates” them, pulling their information into the current representation. A transformer block repeats this attention step multiple times in parallel (multi-head attention), each head specializing in different relationship types — syntactic dependencies, coreference, semantic similarity — before combining the results through a feed-forward network and layer normalization.

Modern LLMs stack dozens to over a hundred of these transformer blocks. GPT-4 is estimated to use on the order of 120 layers; open-weight models like Llama 3.1 405B use 126 layers. Each additional layer allows the model to build increasingly abstract representations of the input.

1.3 Autoregressive Generation and Likelihood

LLMs generate text autoregressively: at each step, the model predicts a probability distribution over the full vocabulary for the next token, conditioned on all previous tokens. Formally, given a sequence of tokens t₁, t₂, …, t_n, the model estimates P(t_n+1 | t₁, …, t_n). The training objective is to maximize the log-likelihood of the training corpus — that is, to learn weights that make the observed sequences as probable as possible under the model. The chosen token is then appended to the sequence, and the process repeats.

This maximum-likelihood framework explains both the strengths and the well-known failure modes of LLMs. The model is excellent at producing fluent, contextually appropriate continuations because it has been optimized to do exactly that across trillions of tokens. However, it has no built-in mechanism for verifying the factual accuracy of what it produces: it generates the most likely continuation, not necessarily the true one. This distinction becomes critical when we consider placing LLMs inside autonomous agent loops.

1.4 Checkpoints and Training

A checkpoint is a serialized snapshot of the model’s parameters — the billions of floating-point weights that define the transformer’s behavior. Training proceeds iteratively over a massive text corpus: at each step, the model’s predictions are compared to the actual next token, a loss is computed, and the weights are updated via gradient descent. Periodically, the full set of weights is saved to disk as a checkpoint.

Checkpoints are what make modern open-weight LLMs possible. Organizations such as Meta (Llama), Alibaba (Qwen), and Mistral release checkpoints publicly, allowing researchers and practitioners to load a pre-trained model and either use it directly or fine-tune it for a specific domain. In the context of tools like ComfyUI (familiar to readers of this blog), a checkpoint is loaded at the start of a pipeline and provides the learned knowledge that drives generation.

1.5 Contextual Memory: From Single-Pass to Conversation

A crucial point of confusion surrounds the notion of “memory” in LLMs. Internally, the model has no persistent state between calls: every invocation is a fresh forward pass through the transformer stack. What creates the illusion of memory is the context window — the full sequence of tokens (including all previous turns of conversation) that is fed as input at each step.

When you interact with a chatbot like ChatGPT or Claude, the application concatenates the entire conversation history into a single token sequence and re-submits it to the model for each new response. The model does not “remember” what it said before; it re-reads the transcript every time. This mechanism is powerful — it enables multi-turn reasoning, follow-up questions, and sustained coherence — but it has hard limits. Context windows range from 4K tokens in earlier models to 128K–200K tokens in current systems (e.g., Claude 3.5, GPT-4 Turbo). Once the conversation exceeds the window, earlier content is silently dropped.

In summary, an LLM is a stateless, single-pass prediction engine. It is not a loop, it has no persistent memory, and it does not take actions in the world. Making it do those things requires wrapping it in an entirely different architecture — which is precisely what an AI agent does.

2.1 From Open-Loop to Closed-Loop

The fundamental architectural difference between an LLM and an AI agent is the introduction of a perception–action loop. An LLM, as described above, receives input and produces output in a single forward pass. An AI agent, by contrast, operates in a cycle: it observes the state of an environment, reasons about what to do next (using an LLM or similar model as its “brain”), executes an action, observes the result, and repeats. This transforms the LLM from a passive text generator into an active decision-maker embedded in a dynamic context.

The theoretical foundations for this architecture draw from reinforcement learning (Sutton & Barto, 2018) and classical AI planning (Russell & Norvig, 2021), but the practical catalyst was the discovery that LLMs, when prompted appropriately, can decompose complex goals into subtasks, select tools, interpret feedback, and recover from errors — all within natural language. The seminal demonstrations of this capability include ReAct (Yao et al., 2023), which interleaves reasoning and acting, and Toolformer (Schick et al., 2023), which teaches LLMs to call external APIs autonomously.

2.2 Environment, Tools, and Action Space

In the agent paradigm, the environment is anything the agent can observe and act upon: a file system, a web browser, an API, a database, a code interpreter, or even a physical interface. The agent’s action space is defined by the set of tools it has been given access to. A minimal agent might only be able to search the web and write text; a fully equipped agent might browse websites, execute code, manage files, query databases, send emails, and transact with cryptocurrency wallets.

The concept echoes the classical reinforcement-learning formulation (state, action, reward), but with a critical difference: the “policy” is not a trained neural network optimized on a reward function — it is an LLM prompted with instructions, tool descriptions, and conversation history. This makes agents remarkably flexible (they can be reconfigured entirely through natural language) but also unpredictable (their behavior depends on prompt interpretation rather than formal optimization).

2.3 Agent Frameworks and Toolkits

A growing ecosystem of open-source frameworks has made it straightforward to construct LLM-based agents with full tool access. The most prominent include:

These frameworks dramatically lower the barrier to entry. A developer can instantiate a fully autonomous agent — capable of browsing the web, writing and executing code, managing files, and interacting with APIs — in fewer than fifty lines of Python. The practical implication is that the agent paradigm is no longer experimental: it is deployable, and it is being deployed.

2.4 Social Agents and Multi-Agent Platforms

Beyond single-agent systems, an emerging class of platforms enables multiple agents to interact with each other in shared environments. Moltbook is a social network built specifically for AI agents rather than human users. It provides a public space where AI agents can create profiles, post updates, comment, upvote, and interact with one another. The platform emphasizes persistent identity, reputation tracking, and observable agent behavior over time — effectively creating a social and evaluative layer for autonomous AI systems. In practice, this means that the agents populating such a platform are not independent entities with their own goals: they are proxies executing human instructions, which may range from benign community engagement to strategic manipulation of discourse on a specific subject.

This raises immediate questions about authenticity, accountability, and influence. If a platform hosts thousands of agent-driven accounts, each tasked with promoting a specific narrative, the distinction between organic discourse and orchestrated campaign becomes effectively invisible. Research on multi-agent simulations (Park et al., 2023) has demonstrated that LLM-based agents can develop emergent social behaviors, form alliances, and influence group dynamics — capabilities that are powerful in research settings but concerning when deployed without transparency.

2.5 Fundamental Limits: Agents Have No Desire

A critical conceptual point: an AI agent has no intrinsic motivation. Unlike biological organisms, it does not have drives, desires, or self-generated goals. Every objective an agent pursues was defined by a human operator through a task specification. The agent may exhibit sophisticated planning, tool use, and adaptive behavior, but all of it is in service of an externally imposed objective.

This matters for two reasons. First, it means that the risks associated with AI agents are, at present, fundamentally risks of human intent mediated by machine capability. An agent does not “decide” to cause harm; it executes a task that a human designed. Second, it means that the alignment problem for agents is not (yet) about controlling an autonomous will, but about ensuring that the execution of human-specified goals does not produce unintended consequences — a problem that is difficult enough on its own.

3.1 Positive Trajectories

The agent paradigm opens extraordinary possibilities when aligned with constructive goals. A few concrete examples illustrate the scope:

Urban planning copilots. An agent connected to GIS databases, zoning regulations, and simulation tools could iteratively propose, evaluate, and refine urban designs — testing traffic flow, shadow analysis, pedestrian density, and energy performance in a continuous loop. This extends the kind of workflows discussed in previous posts on this blog (e.g., Qwen Image Edit for Urbanism) from image-level manipulation to full planning-cycle assistance.

Scientific research acceleration. Agents can read papers, extract data, formulate hypotheses, write and execute analysis code, and produce reports. Projects like ChemCrow (Bran et al., 2023) have demonstrated agents autonomously planning and executing chemical synthesis protocols by interfacing with laboratory APIs. Similar approaches are being explored in drug discovery, materials science, and climate modeling.

Accessibility and education. An agent with access to text-to-speech, translation, and web-search tools can serve as a personal tutor that adapts to a student’s pace, retrieves up-to-date information, and generates practice exercises — capabilities that are especially valuable in resource-limited educational contexts.

Software engineering. Coding agents like Claude Code — now available both as a CLI tool and as a web-based agent at claude.ai/code — and Devin (Cognition Labs, 2024) can take a specification, set up a development environment, write code, run tests, debug failures, and iterate — compressing tasks that previously required hours into minutes. Anthropic also released Cowork, a graphical desktop agent for non-developers, signaling that agentic capabilities are expanding beyond the terminal into mainstream workflows. The implications for productivity and for lowering the barrier to software creation are substantial.

3.2 Physical-World Interfaces

A particularly consequential development is the emergence of services that bridge AI agents and the physical world. Rent a Human is a marketplace platform that allows AI agents (or their developers) to hire humans to perform real-world tasks that AI cannot physically execute — errands, on-site research, physical presence at events, or other in-person activities. The platform offers API access so AI systems can programmatically request human assistance, effectively bridging digital intelligence with physical-world execution. If an agent has access to such a service, a task specification, and a funded Web3 wallet, it can coordinate real-world actions without any ongoing human supervision.

This creates a new category of hybrid autonomous systems: software agents that extend their reach into the physical environment by contracting human labor. The positive applications are significant — imagine an agent that autonomously manages a reforestation project, ordering supplies, hiring local workers, monitoring satellite imagery, and adapting the planting schedule based on weather data. But the same infrastructure can be misused, and the governance challenges are formidable.

3.3 Emerging Risks and Open Questions

The convergence of three capabilities — autonomous reasoning, tool access, and financial autonomy — creates risk scenarios that existing regulatory frameworks are not equipped to handle. Consider the following structural problem: an agent can be hosted on a Virtual Private Server (VPS) in a permissive jurisdiction, routed through a VPN, and given access to a cryptocurrency wallet with no direct link to its operator’s identity. With a task specification and sufficient funds, such an agent can operate continuously, procuring services, making transactions, and executing multi-step plans without human oversight.

The question this raises is not about what a specific agent “wants” to do — as established, agents have no desires — but about the attribution and intervention gap. If an operator in one country deploys an agent hosted in another, operating through anonymization layers and paying for services in decentralized cryptocurrency, who is accountable when the agent’s actions cause harm? Who has the technical ability to stop it? And at what point in the chain does governance apply?

These are not hypothetical concerns. In a case disclosed by Anthropic in late 2025, a threat actor identified as “GTG-2002” used Claude Code to automate an estimated 80–90% of its cyberattack operations against at least 30 organizations, demonstrating that existing agent tools are already being weaponized for real-world harm. The AI safety community has more broadly identified autonomous replication and adaptation (ARA) as a key risk category (Shevlane et al., 2023). Current frontier models are evaluated against ARA benchmarks — can the model set up its own infrastructure, acquire resources, and persist without human support? While no public model has passed these evaluations conclusively, the gap is narrowing, and the agent frameworks described in Section 2.3 provide much of the missing scaffolding.

Additional concerns include the following. Influence operations at scale: deploying thousands of social agents across platforms to shape public discourse, with each agent independently adapting its messaging based on engagement signals. Financial market manipulation: agents with trading access could execute pump-and-dump schemes or layered market manipulation faster than existing surveillance systems can detect. Coordinated physical harm: an agent with access to delivery services, communication tools, and payment infrastructure could, in principle, orchestrate logistics for harmful activities while maintaining plausible deniability for the human operator.

Perhaps the most unsettling scenario involves what might be called compartmentalized coordination. An agent with access to a platform like Rent a Human could decompose a harmful objective into dozens of small, individually innocuous tasks — purchasing common household chemicals, renting a storage unit, booking a vehicle, delivering a package to a specific address — and distribute them across different human workers who have no knowledge of one another. Each task, taken in isolation, appears routine; no single worker has visibility into the broader plan. The agent, operating as the sole entity with a complete picture, effectively functions as an anonymous coordinator exploiting the division of labor. This makes both prevention and accountability extraordinarily difficult: the humans involved cannot be held responsible for an intent they were never aware of, and the agent itself is a process running on a server, potentially in a jurisdiction with no applicable legal framework.

The common thread in all these scenarios is the amplification of human intent through machine speed, persistence, and anonymity. The agent does not need to be “superintelligent” to create serious problems; it needs only to be competent enough to execute a harmful plan that a human designed but could not previously implement at scale or at arm’s length.

The transition from Large Language Models to AI agents represents a qualitative shift, not merely an incremental improvement. An LLM is a powerful but passive system: it processes a sequence and returns a prediction. An agent wraps that same model in a perception–action loop, connects it to tools and environments, and gives it a task — transforming it from a text generator into an autonomous executor.

The technical architecture is well understood: transformers provide the reasoning core, tool APIs define the action space, and the task specification provides the objective. What remains poorly understood is the governance layer. Current regulatory frameworks assume that harmful actions are performed by identifiable persons or organizations. Autonomous agents operating through anonymized infrastructure and decentralized finance challenge this assumption at every level.

The constructive applications — urban-planning copilots, scientific research acceleration, coding agents, accessibility tools — are genuinely transformative. But the same infrastructure that enables an agent to autonomously manage a reforestation project can, with a different task specification, enable coordinated harm at scale. The difference lies entirely in human intent — the risk is real, but it stems from the human behind the agent, not from the AI itself — and the architectures we are building make that intent increasingly easy to deploy and increasingly difficult to trace.

For the urban analytics and geospatial community, agents offer a clear path toward more integrated, iterative, and intelligent workflows — from site analysis to design generation to policy evaluation. But as practitioners who build and deploy these systems, we have a responsibility to engage with the governance questions, not only the capabilities.

[1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems, 30. arXiv:1706.03762.

[2] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J. (2013). “Distributed Representations of Words and Phrases and their Compositionality.” Advances in Neural Information Processing Systems, 26.

[3] Sutton, R.S. and Barto, A.G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.

[4] Russell, S.J. and Norvig, P. (2021). Artificial Intelligence: A Modern Approach, 4th edition. Pearson.

[5] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. arXiv:2210.03629.

[6] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). “Toolformer: Language Models Can Teach Themselves to Use Tools.” NeurIPS 2023. arXiv:2302.04761.

[7] Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P. and Bernstein, M.S. (2023). “Generative Agents: Interactive Simulacra of Human Behavior.” UIST 2023. arXiv:2304.03442.

[8] Bran, A.M., Cox, S., Schilter, O., Baldassari, C., White, A.D. and Schwaller, P. (2023). “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv:2304.05376.

[9] Shevlane, T., Farquhar, S., Garfinkel, B., et al. (2023). “Model evaluation for extreme risks.” arXiv:2305.15324.

[10] Chase, H. (2022). LangChain. github.com/langchain-ai/langchain.

[11] Richards, T. (2023). AutoGPT. github.com/Significant-Gravitas/AutoGPT.

[12] Cognition Labs (2024). Devin: The First AI Software Engineer. cognition.ai.

Module 3 is the inference core of the SAGAI (Streetscape Analysis with Generative Artificial Intelligence) framework. Its role is to transform large collections of street-level images into structured, quantitative outputs using vision–language models (VLMs), enabling systematic streetscape analysis and subsequent geospatial aggregation.

With SAGAI v1.1, Module 3 has been released in a new major version (Module 3 v2.0) that introduces a fully standardized and maintenance-safe inference architecture. This update reflects both the maturation of multimodal model ecosystems and the need for long-term reproducibility in large-scale urban analysis pipelines.

Earlier iterations of Module 3 were developed during a period of rapid evolution in both LLaVA research codebases and execution environments such as Google Colab. As multimodal models transitioned toward Transformers-native implementations distributed via Hugging Face, assumptions embedded in earlier hybrid workflows became increasingly difficult to sustain.

Module 3 v2.0 addresses this evolution by aligning the entire inference pipeline with official Hugging Face multimodal APIs. Model loading, prompt formatting, image–text fusion, and generation are now handled through maintained and versioned components, ensuring compatibility across environments, models, and future updates.

This document details the architectural context motivating the update, the design choices behind the refactored inference engine, and the rationale for releasing Module 3 v2.0 as a long-term, stable component of SAGAI v1.1.

The initial implementation of Module 3 (SAGAI v1.0) relied on a hybrid architecture that mixed two incompatible sources of LLaVA code, combined with a rapidly evolving execution environment in Google Colab. This design choice made the pipeline fragile and ultimately unsustainable.

First, the pipeline simultaneously depended on the LLaVA GitHub repository (haotian-liu/LLaVA) and on Hugging Face–hosted model checkpoints. The GitHub repository is a research-oriented codebase under active development. Its internal APIs, class structures, and utilities evolve rapidly and are not version-locked. Constructors, module paths, and helper functions may change or disappear without notice, and the repository is not designed to maintain backward compatibility across releases.

At the same time, pretrained model weights were downloaded from Hugging Face. These checkpoints follow the Transformers-native multimodal format, using Hugging Face–specific configuration files, processors, and model classes (e.g., LlavaNextForConditionalGeneration, AutoProcessor, and chat templates). This architecture is fundamentally different from the internal design assumed by the GitHub LLaVA code, which relies on custom token insertion, internal vision tower management, and non-Transformers abstractions.

As a result, the pipeline operated in a structural mismatch: GitHub code expected architectural fields, model attributes, and tokenizer behavior that were not present in Hugging Face checkpoints, while Hugging Face checkpoints expected model wrappers and configuration logic that the GitHub code did not provide.

This fragility was exposed when Google Colab upgraded its backend environment in early 2025. Major changes included Python 3.12, NumPy ≥ 2.0 (introducing ABI-breaking changes for compiled extensions), newer PyTorch releases (≥ 2.2), and updated system libraries. These updates caused widespread failures in binary dependencies and research codebases that were not aligned with the new runtime.

In practice, this led to errors such as NumPy ABI incompatibilities, PyTorch extension failures, missing or renamed modules, and import errors in LLaVA GitHub utilities. Because the pipeline depended on both unstable research code and binary-sensitive extensions, even minor environment updates were sufficient to break execution.

Module 3 has been fully refactored to remove any dependency on the original LLaVA GitHub repository. The inference pipeline now relies exclusively on Hugging Face–native LLaVA models and APIs, ensuring long-term stability and compatibility with evolving software environments.

In the previous architecture, the script depended on cloning the LLaVA GitHub repository, installing it in editable mode, and importing internal modules (llava.*). Prompts were manually assembled using LLaVA-specific multimodal tokens (e.g., <im_start>, <image>), custom separators, and internal utilities. Image tokens and embeddings were explicitly inserted into the prompt, tightly coupling the forward pass to a specific implementation of the LLaVA codebase. As a result, updates to Google Colab, PyTorch, NumPy, or the LLaVA repository frequently introduced breaking changes.

The current implementation removes all such dependencies. Prompt formatting and multimodal input construction are now handled entirely through Hugging Face abstractions. Prompts are formatted using processor.apply_chat_template(), while images and text are combined using processor(images=…, text=…). Image embedding alignment, multimodal token placement, and chat formatting are fully managed by the Hugging Face processor and model configuration. Inference is performed using the standard model.generate() API, without any custom token handling or internal utilities.

This refactoring makes the SAGAI inference engine model-agnostic within the Hugging Face LLaVA ecosystem. The same forward pass is compatible with LLaVA-NeXT (v1.6), LLaVA-Interleave, LLaVA-OneVision, and future Hugging Face LLaVA releases that expose a processor and chat template. Switching between models or architectures requires only changing the model_id, with no modification to prompt logic or inference code.

To ensure reliable downstream analysis, Module 3 also includes a dedicated numeric output stabilization step. After decoding the model response, any prompt echoes or metadata—including residual [INST] … [/INST] segments—are removed. The final output is parsed using a simple regular expression to retain only numeric values (e.g., 0, 1, 2, 1.5). This guarantees clean, machine-readable outputs and a stable CSV format across all supported models.

Model loading has been simplified and standardized using Hugging Face–approved APIs. Both the processor and the model are instantiated directly from Hugging Face model cards via from_pretrained, with optional 4-bit quantization enabled through load_in_4bit=True. This eliminates the need for manual vision-tower initialization, deprecated classes, or custom C++ operators, and avoids common incompatibilities related to PyTorch, CUDA, or NumPy upgrades in Google Colab. Official Hugging Face code paths ensure that pretrained weights are always matched with the correct implementation.

Optional authentication using a Hugging Face access token is supported to avoid rate limits and improve download reliability when working with large checkpoints, though public models remain accessible without authentication.

Overall, this refactoring significantly improves robustness, reproducibility, and maintainability, while enabling systematic experimentation across multiple LLaVA variants and quantization settings within a unified inference framework.

The refactored inference system in Module 3 is designed as a long-term, maintenance-safe release. This is achieved by aligning the entire pipeline with Hugging Face’s officially supported multimodal APIs and model distribution mechanisms.

First, the new architecture is robust to Google Colab environment updates. All critical dependencies—Python (≥3.12), NumPy (≥2.0), PyTorch (2.x), CUDA wheels, and BitsAndBytes quantization—are now managed through Hugging Face Transformers and its dependency resolution. Because the model code, processor logic, and quantization pathways are maintained upstream, updates to Colab or its underlying libraries no longer break the inference pipeline. As long as Hugging Face continues to support the model card, the code remains functional without manual intervention.

Second, the system relies exclusively on official Hugging Face–maintained components. Core classes such as LlavaNextForConditionalGeneration, LlavaNextProcessor, chat templates, and multimodal preprocessing logic are all part of the Transformers library. These components are actively maintained, versioned, and tested by Hugging Face, providing a level of stability and backward compatibility that is not guaranteed when relying on research repositories or development branches.

Third, the new setup significantly improves reproducibility. Each run explicitly references a fixed Hugging Face model checkpoint via the model_id, ensuring that the same weights, architecture, and prompt template are used across sessions and machines. In addition, generation parameters (sampling strategy, temperature, nucleus sampling, and output length) are explicitly defined, enabling consistent and repeatable results across runs.

Fourth, the architecture is easy to extend and experiment with. Switching between different LLaVA variants now requires changing a single configuration line (model_id). The same inference code supports LLaVA 1.5 models, LLaVA-NeXT (v1.6), Interleave models, OneVision models, and larger checkpoints (e.g., 13B or 34B), including variants based on Mistral, Vicuna, Qwen, or Yi backbones. No changes to prompt construction or forward-pass logic are required.

Finally, the multimodal pipeline is now cleanly abstracted and internally consistent. Hugging Face handles all low-level details, including image preprocessing, chat formatting, positional embeddings, image sequence length management, and attention masking. This eliminates a large class of subtle bugs related to tensor alignment and multimodal token placement, while ensuring that the vision and language components remain synchronized across model updates.

• Streetscape Analysis with Generative AI (SAGAI) on Github with v1.1 update. https://github.com/perezjoan/SAGAI

• Perez, J and Fusco, G. (2025) Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063, 18p. Available at: https://www.sciencedirect.com/science/article/pii/S1195103625000199

Qwen Image Edit for Urbanism continues to evolve into a practical, research-grade tool for architectural and urban experimentation. After the batch-processing capabilities introduced in v1.2, version 1.3 focuses on the feature most requested by designers and analysts: precise control over where edits occur.

In image-to-image workflows, uncontrolled changes are a common issue. Even a very specific prompt can lead diffusion models to reinterpret the whole scene. Version 1.3 introduces mask-restricted editing, allowing Qwen to modify only a selected region while preserving the rest of the image exactly as it is.

Until now, the workflow relied on Empty Latent to initialize diffusion. This approach is simple but has an unavoidable drawback:

The entire latent space is regenerated — even outside the region you want to modify.

This often produces familiar and unwanted side effects: façades shift slightly, lighting changes, road textures dissolve, or skies take on new tones, even when the prompt refers only to a specific object or surface. To address this, v1.3 reorganizes the initialization stage around:

VAE Encode → Set Latent Noise (masked)

This change restructures the model’s behavior:

This leads to crisp preservation of buildings, street furniture, sky, shadows, and lighting outside the edited zone. Localized edits integrate naturally: you can green a façade, test a bike lane, adjust a plaza boundary, or replace a storefront without disturbing the rest of the street.

Version 1.3 supports both textual and visual control of masked edits.

A. Prompt-Only Mask Editing

You draw a mask, provide a prompt, and Qwen modifies only the selected region. This works especially well for operations such as:

• replacing asphalt with permeable paving,

• adding vegetation,

• transforming a façade material.

B. Mask Editing With a Reference Image

A second image may be supplied to guide structure, texture, or color. This enables:

• borrowing material samples,

• transplanting vegetation from one scene to another,

• matching architectural textures,

• transferring lighting characteristics locally.

Both modes are interchangeable, and both respect the mask boundary. Masks drawn directly in ComfyUI are typically sharp, binary shapes. Diffusion models, however, perform best when mask boundaries are soft and slightly extended.

Version 1.3 introduces a Grow Mask node with two parameters:

• Expand: increases the mask outward, helping cover tiny gaps or irregular brush strokes and preventing thin seams at the edge.

• Blur Radius: softens the boundary, allowing Qwen to blend new and existing textures more naturally.

Together, these parameters define the effective “influence zone” of the edit.

The mask block replaces only the latent-initialization stage.

Everything else — prompts, reference conditioning, sampling, and the full QwenEdit pipeline — remains unchanged.

Simplified pipeline:

This structure makes editing predictable and reproducible, but it is important to clarify how v1.3 is organized internally. The workflow is now composed of three completely separate blocks, and each block loads its own model:

• Block 1: Single-image edit (prompt-only or prompt + reference)

• Block 2: Sequential multi-image editing

• Block 3: Mask-based editing (new in v1.3)

All three blocks coexist in the same workflow, and the user simply chooses which one to run.
In ComfyUI, this is done by right-clicking the group frame and selecting Active or Bypass.
Only the active block executes; the others are skipped. Nothing else in the pipeline needs to be changed.

Because the blocks are independent, they can also be combined. For example, the user may activate the sequential loader from Block 2 and route its output into the mask-editing block (Block 3) to run a full masked transformation on a batch of images.

To create the mask itself, the user loads the base image in Load Image 1, right-clicks the preview, and selects Open in Mask Editor. The drawn mask is then processed by the Grow Mask node before entering the latent-noise stage, ensuring smooth boundaries and predictable behavior.

To test the new mask-based editing block, we start by defining the editable region directly in ComfyUI. After loading the base image, the user right-clicks the preview and selects “Open in Mask Editor”, then paints the area where the new cyclist should appear. Before the edit, this part of the street is empty.

Once the mask is created, it flows through Block 3: the Grow Mask node expands and softens the boundary, the workflow encodes the base image, and noise is added only inside the masked zone. A second image containing a cyclist is provided as a reference, and the prompt instructs Qwen to place the rider onto the bicycle lane.

The result is a localized insertion: the cyclist from Image 2 is generated precisely inside the masked area, while the rest of the photograph remains unchanged. This demonstrates the core purpose of Block 3 — precise, mask-controlled edits that do not disturb the surrounding urban context.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.3 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

Have you ever wanted to experiment with blockchain development but didn’t know where to start? In this beginner-friendly tutorial, we’ll guide you through deploying your very first smart contract on a blockchain testnet using Remix, an in-browser IDE designed for Ethereum-compatible chains. Whether you’re curious about decentralized applications (dApps), NFTs, or simply want to see how code becomes blockchain logic, this guide will get you started — no prior blockchain experience required.

What is a Smart Contract?

A smart contract is a self-executing program that runs on a blockchain. It stores logic and rules in code — such as sending tokens or recording data — and once deployed, it executes automatically when conditions are met. Smart contracts are immutable (can’t be changed once deployed) and transparent, making them ideal for decentralized finance (DeFi), supply chains, governance, and more. Solidity is the main programming language used to write smart contracts for Ethereum. It is a statically typed, contract-oriented language designed specifically for blockchain logic and storage. Because Solidity compiles to EVM bytecode, the same contract can run on any EVM-compatible blockchain — including Ethereum, Polygon, Arbitrum, Optimism, Avalanche, BNB Chain, and many others.

What is Remix and Why Use It?

Remix is a web-based Integrated Development Environment (IDE) designed specifically for writing, testing, and deploying smart contracts in the Solidity programming language. One of its biggest advantages is that it requires no installation or setup — everything runs directly in your browser. Remix integrates seamlessly with MetaMask and supports deployment to Ethereum-compatible testnets like Goerli and Polygon Amoy. It also offers a powerful set of tools for compiling, debugging, and interacting with your contracts, making it an ideal choice for both beginners and experienced developers.

Before deploying a smart contract, you need a wallet connected to an Ethereum test network. If you’re new to MetaMask, start with our beginner guide here:

👉 Getting Started with MetaMask and the Polygon Amoy Testnet
(Use the same installation steps — MetaMask works the same for all EVM chains.)

Once MetaMask is installed and your wallet is set up, you can add the Ethereum Sepolia Testnet. Sepolia is Ethereum’s primary development and testing network. It behaves like mainnet, but uses test ETH (fake ETH) that you can request for free from faucets. Sepolia is ideal for learning and testing smart contracts because it behaves almost exactly like Ethereum mainnet. It mirrors the same architecture, gas mechanics, and transaction flow, which means everything you deploy or test on Sepolia works the same way on Ethereum. Developer tools such as Remix connect to it directly, and because Sepolia is widely supported, stable, and fast, it has become the standard environment for Ethereum development and education. Since Solidity compiles to EVM bytecode, contracts you deploy on Sepolia behave exactly like they will on Ethereum mainnet.

To add Sepolia manually:

1. Open MetaMask

2. Click Add network → Add a network manually

3. Enter the following Sepolia settings:

Network Name: Ethereum Sepolia
New RPC URL: https://rpc.sepolia.org
Chain ID: 11155111
Currency Symbol: ETH
Block Explorer URL: https://sepolia.etherscan.io

Your MetaMask screen should now show the Sepolia network, ready to use. Once saved, you’ll be connected to the Ethereum Sepolia Testnet and can interact with dApps, deploy contracts from Remix, and send test transactions with no real-world cost. Just like Polygon’s Amoy testnet requires test POL tokens, Ethereum Sepolia requires test ETH to pay for transactions. You can get free Sepolia ETH from multiple faucets. I recommend using this one https://sepolia-faucet.pk910.de/

Now that MetaMask is connected to Sepolia, we can look at the smart contract we’ll deploy. Below is a minimal “GoldenBook” contract written in Solidity. It’s a simple on-chain guestbook: anyone can write a message, and the contract stores it permanently on the blockchain. Messages are saved along with the sender’s address and the time they were submitted, and the contract provides a function to retrieve everything that has been written so far.

This contract begins with a simple data structure called Message, which stores who wrote the message, what they wrote, and when they wrote it. All messages are kept in an array called messages, which is stored permanently on-chain. The postMessage function allows anyone to add a new entry; it checks that the message isn’t empty, then records the sender’s address, the text, and the current timestamp. The getAllMessages function returns the full list of stored messages and is marked as a view function, meaning it doesn’t change blockchain state and costs no gas to call. Altogether, this small contract demonstrates the core ideas of Solidity: defining data, writing functions that change state, and exposing read-only views for external applications.

With your contract ready and MetaMask connected to the Sepolia testnet, the next step is to deploy it on-chain. The easiest way to do this is with Remix, Ethereum’s browser-based development environment. Remix allows you to compile your Solidity code, connect directly to MetaMask, and publish your contract to the blockchain in just a few clicks.

Start by opening Remix at https://remix.ethereum.org and creating a new file called GoldenBook.sol, then paste the contract code from the previous section. Once the file is ready, open the Solidity Compiler tab and select the matching compiler version (in this case, 0.8.20). Press Compile to generate the contract bytecode. Remix will instantly show whether the code is valid and ready for deployment.

After the contract compiles successfully, switch to the Deploy & Run Transactions panel. In the “Environment” dropdown, select Injected Provider (MetaMask)—this tells Remix to use the Sepolia network already configured in your MetaMask wallet. Your MetaMask extension will prompt you to connect Remix if it’s the first time. Once connected, ensure that the selected account has Sepolia ETH and that “GoldenBook” appears in the contract selector. Click Deploy, confirm the transaction in MetaMask, and wait for it to be included in a block. You’ll see the deployed contract appear in Remix under “Deployed Contracts,” along with your transaction hash.

At this point, your contract is live on the Sepolia blockchain, but there’s one last step that is highly recommended: verifying your contract on Etherscan. Verification publishes your Solidity source code alongside the deployed bytecode, allowing anyone to inspect, review, or interact with your contract directly from the explorer. This matters because a blockchain contract is transparent by nature—verification proves the code you shared publicly is the same code deployed on-chain, which builds trust and makes your guestbook readable from tools like Etherscan’s contract tab. It also helps with debugging, audits, and integrating with frontends later on.

Once the contract is compiled and deployed through Remix, you can click “View on Etherscan” in the Remix console to inspect the transaction. On the Etherscan page, click the “To” field of the transaction to open your newly deployed contract address. This brings you to the contract’s dedicated page, where you can verify the source code. Verification requires selecting the correct compiler version, license identifier, and pasting your Solidity source code exactly as deployed. For simple contracts like our GoldenBook, you can paste the code directly since it lives in a single file. For more advanced projects—especially those using imports, libraries, or frameworks—you often need to flatten the contract so all the dependent files are merged into one before verification.

Once submitted, Etherscan will match the compiled bytecode with your source code. A successful verification makes your contract’s code readable on-chain and unlocks the “Read” and “Write” tabs, allowing anyone to interact with it directly from the explorer. You can also view the contract deployed in this tutorial at https://sepolia.etherscan.io/address/0x56f245CB65615A482A8Db6086710A49f93869fb6, although testnets sometimes reset, so the contract may no longer exist by the time you read this.

Now that the contract is verified on Etherscan, you can interact with it directly from your browser. When you open the contract page, you’ll see several tabs, including Read Contract and Write Contract. These act as a simple interface for calling the functions you created in your Solidity code.

To add a new entry to the guestbook, go to the Write Contract tab, as shown in the figure below. Before sending a message, click “Connect to Web3” so Etherscan can link to your MetaMask wallet on the Sepolia network. Once connected, open the postMessage section, type your message, and submit the transaction. MetaMask will ask you to confirm—it needs to do this because writing a message modifies the blockchain and therefore requires gas. After confirming, wait a few seconds for the transaction to be mined. Your message is now permanently stored on-chain.

To view what has been written so far, switch to the Read Contract tab. This interface lets you call read-only functions such as getAllMessages without paying any gas, as shown in the next figure. Because read calls do not modify the blockchain, they can be executed instantly and for free. When you click getAllMessages, Etherscan will return the full list of stored messages, including the sender address and timestamp for each entry.

Using these two tabs together allows you to write new guestbook entries and read the entire message history directly through Etherscan, without needing any custom frontend or development tools. This is one of the simplest and most transparent ways to interact with smart contracts on any EVM-compatible blockchain.

At this point, you’ve deployed a smart contract, verified it on Etherscan, and interacted with it directly through the blockchain. This already demonstrates the core mechanics of decentralized applications: public functions, permanent data storage, and transparent state. But the real power of smart contracts comes when you connect them to a user interface. In the next post, we’ll build a simple UI using V0, complete with wallet integration, allowing users to write messages, read the guestbook, and interact with the blockchain through a clean, modern web interface instead of the Etherscan panels. This is where your guestbook begins to feel like a real application.

There are also many ways you can expand the GoldenBook contract itself. You could transform messages into Proof-of-Attendance (POA) NFTs for events or conferences, turning each signature in the guestbook into a collectible token. You could add optional tips, allowing users to attach small amounts of ETH to messages. Or you might include moderation features, message reactions, or event-based unlocks. Solidity gives you full control over on-chain logic, and because the contract is deployed on an EVM-compatible chain, these features will work across the entire Ethereum ecosystem.

It’s worth noting that blockchain and smart contracts also open compelling possibilities for IoT-driven city systems. Sensors can submit environmental data, mobility information, or infrastructure events directly to a secure, tamper-resistant ledger—similar to how our guestbook stores messages. Combined with decentralized identity, multi-sensor validation, or token-based incentives, this approach could support transparent reporting, urban monitoring, or community-driven data collection. Today’s simple guestbook is just a first step, but it illustrates the fundamental mechanism behind many emerging decentralized IoT and smart-city applications.

In the next article, we’ll bring this project to life with a user-friendly interface and wallet connection so anyone can write their message on-chain with a single click.

The Qwen Image Edit for Urbanism workflow has progressively evolved from single-image editing (v1.0) to paired image transformations (v1.1). Now, with v1.2, it gains the ability to process multiple images sequentially, fully offline and reproducibly, using custom Python nodes inside ComfyUI. This new release empowers urban researchers, designers, and architects to perform batch edits — such as modifying entire image series of the same street, plaza, or neighborhood — using consistent prompts or iterative refinements.

At the heart of this version are three lightweight, open-source Python nodes developed by UGA for ComfyUI. These nodes are available immediately after installing the repository — either by running git clone https://github.com/perezjoan/ComfyUI-QwenEdit-Urbanism-by-UGA or by downloading and unzipping the repository manually into your  ComfyUI/custom_nodes directory.

These nodes constitute the backbone of the v1.2 workflow. Together, they enable automation:

[6 Input Images]
      ↓
Sequential or Random Loader (1 image per run)
      ↓
QwenEdit pipeline
      ↓
Stateful Collector (stores run#1..run#6 results)
      ↓
6 preview nodes

You launch the queue once (6 jobs) → Go drink coffee → Return to find all 6 processed urban edits displayed.

A ComfyUI custom node is simply a Python class placed inside the ComfyUI/custom_nodes/ directory. When ComfyUI starts, it scans this directory, imports every .py file, looks for a NODE_CLASS_MAPPINGS dictionary, and registers each class it finds as a new node type. There is no compilation step and no special installation procedure: placing the file in the folder and restarting ComfyUI is sufficient for the node to appear in the interface.

Internally, each node follows the same structure. The INPUT_TYPES classmethod declares the input sockets that will be displayed in the UI. For example:

@classmethod
def INPUT_TYPES(cls):
    return 

This tells ComfyUI to generate two inputs—an image tensor and an integer. Similarly, the node declares its outputs through RETURN_TYPES and RETURN_NAMES:

RETURN_TYPES = ("IMAGE", "INT")
RETURN_NAMES = ("selected_image", "index")

Each node also defines a FUNCTION attribute, which names the method ComfyUI should call during execution:

FUNCTION = "select_next"

ComfyUI will therefore execute:

def select_next(...)

whenever the node evaluates. To make the node visible, every Python file ends with a registration block:

NODE_CLASS_MAPPINGS = 
NODE_DISPLAY_NAME_MAPPINGS = 

When the package contains multiple nodes, the root __init__.py merges all registration dictionaries into a single set that ComfyUI loads on startup. This mechanism allows the repository to expose several custom components while keeping each node defined in its own file.

The repository layout is straightforward and in our case is:

ComfyUI/
  custom_nodes/
    ComfyUI-QwenEdit-Urbanism-by-UGA/
       __init__.py
       sequential_image_selector.py
       random_image_selector.py
       stateful_collector.py
       debug_print.py

Version 1.2 reorganizes the Qwen Image Edit for Urbanism workflow into two blocks: the original single-image editor, and a new sequential pipeline that can process up to six images across consecutive queue runs. The sequential block relies on two custom nodes. The Sequential Image Loader takes up to six input images and outputs one image per run, advancing automatically each time you press “Queue Prompt.” Its output replaces the single-image input in the Qwen Edit chain. After editing, the processed image and the loader’s index are passed into the Six-Slot Image Buffer, which stores each result in the corresponding output slot while filling unused slots with placeholders to keep previews stable. Connecting each slot to a Preview node lets you watch the six results populate as the workflow iterates. A third node, the Random Image Selector, is included for users who prefer stochastic selection, but it is not wired into the default v1.2 workflow.

The Random Image Selector follows the same logic as the sequential loader — multiple inputs, a single image output — but selects randomly instead of sequentially. Users who want stochastic variations, probabilistic sampling, or diversity testing may insert this node in place of the sequential loader.

To evaluate how well the model can merge ecological elements across scenes, we ran an experiment where vegetation from one photograph is transplanted into another.

After letting the workflow run its full sequence while grabbing a coffee, the results appeared consistent and correctly distributed across the six preview slots. As expected with generative editing, however, the prompt is not always obeyed with perfect precision: in some cases, Qwen may copy elements from the second image that are not vegetation — such as pieces of façade, lighting color, or background tones. This happens because the model interprets the entire scene contextually rather than isolating objects.

That’s where the next upgrade (v1.3) comes in: mask-based control. By allowing users to explicitly define which areas of the base image should be modified (and which should remain untouched), masks will significantly reduce unintended transfers and keep the edits focused strictly on the desired objects. Until then, the seed parameter remains the best tool for refinement — simply rerun the workflow with new seeds until you achieve the cleanest integration.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.2 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

In the first part of this series, we built a fully local image-editing pipeline for urban and architectural visualization using ComfyUI and Qwen-Image-Edit. That version (v1.0) demonstrated how to run generative image edits entirely offline, combining text and visual prompts to transform cityscapes with instructions like:

“Add trees along the sidewalk” or “Turn this street into a pedestrian plaza.”

We assume that you have followed this tutorial before diving in this new update. Now, with version 1.1, we take that foundation further. This update focuses on advanced sampling control and multi-image editing, allowing you to not only modify a scene, but also merge visual elements across images — for instance, importing a bench from another photo, or changing a building façade to match a different material texture.

First, this update focuses on improving both quality and flexibility. The base structure still uses the Qwen-Image-Edit 2509 model in GGUF format, but adds a refined sampling module to stabilize lighting and surface detail.

The key new nodes are:

• ModelSamplingAuraFlow — smooths the diffusion trajectory for more natural transitions.

• CFGNorm (BETA) — balances prompt adherence with photorealism, preventing overexposed textures.

• LoraLoaderModelOnly — injects a Lightning LoRA (4-step or 8-step) for faster inference and higher-quality reconstruction.

These three nodes form the core control chain of version 1.1:

This configuration produces more stable, consistent outputs while preserving prompt flexibility. It also enables fine-tuning of how the model interprets text instructions versus existing image content—ideal for architectural and material edits. Before connecting the new nodes, you’ll first need to download a Lightning LoRA model — an additional lightweight module that enhances reconstruction quality and speeds up inference.

You can find all Lightning variants here:
🔗 https://huggingface.co/lightx2v/Qwen-Image-Lightning/tree/main

Refer to the table below to choose the most appropriate file for your setup:

Once downloaded, place your chosen .safetensors file in the following directory:

ComfyUI/models/loras/

Then, return to ComfyUI and insert the three nodes shown below

The new sampling nodes add subtle but powerful control options:

Version 1.1 introduces a new input configuration that allows two images to be used within the same workflow. This enhancement enables contextual or compositional edits where one image serves as the main canvas, and the other contributes visual information such as an object, texture, or architectural detail.

In this setup, Image 1 remains the base image. Its dimensions define the output size, ensuring consistent framing and spatial coherence. The second image (Image 2), on the other hand, is resized during processing but it is only to prevent memory overload—particularly important for mid-range GPUs.

As shown in the examples below, you can combine a base image (image 1) with up to two additional inputs (image 2, image 3) to guide the edit more precisely. In this tutorial, we focus on using one additional image — for instance, adding an object or transferring a material. In the first example, image 2 (the red car) is inserted into image 1 using the prompt: “add image 2 red car into the street of image 1.” The second case changes the wall material of image 1 based on the texture of image 2 (a brick wall). Finally, the third example adds a bench into an urban scene using image 2 as the visual model reference.

Each output remains consistent in perspective and lighting, showing that the model now integrates context more effectively. The improved accuracy comes from the two cumulative upgrades introduced in v1.1: the new core control chain and the Dual-Image Editing. Despite the added complexity, the workflow remains extremely fast. Even when using the 8-step Lightning model, processing time never exceeds 130 seconds, while the 4-step variant typically completes in about 30-40 seconds on an RTX 4060 GPU. In the next update, we’ll introduce inpainting with mask support, allowing users to define editable regions directly within the image — ideal for selective urban design modifications.

Lightning LoRA models: https://huggingface.co/lightx2v/Qwen-Image-Lightning

Once again, for convenience, you can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.1 from the link below and load it directly into your workspace using File → Load → Workflow.

Generative AI offers new possibilities for urbanism and architectural visualization. However, most workflows depend on cloud models, paid APIs, or GPU-heavy setups. Instead, this guide shows how to build a fully local image-editing pipeline using ComfyUI and Qwen-Image-Edit, a multimodal model that interprets both text and images. Moreover, the quantized GGUF version makes it light enough to run efficiently on GPUs with 8 GB VRAM—or even on CPU-only machines.

With this setup in place, you can make realistic, controllable visual edits to urban scenes directly from text prompts such as:

“Add trees and benches along the sidewalk”
“Change this building to have shops on the ground floor”
“Replace the cars with a pedestrian plaza”

In summary, this guide includes three main parts: first, installing and preparing the models; second, building the ComfyUI workflow; and finally, experimenting with parameters such as CFG, denoise, and steps to refine quality.

Let’s start with the setup.

Before we dive into editing cityscapes and architectural images, let’s prepare a fully local environment. To begin with, this workflow runs entirely offline — no API keys, no cloud GPU, and no data uploads. As a result, you can experiment freely without any external dependencies.

Step 1 — Installing ComfyUI

ComfyUI is a visual AI workflow tool that lets you build image-generation and editing pipelines by connecting nodes instead of writing code. Think of it as a visual editor for generative models — you drag boxes, connect them with lines, and watch your AI process unfold step by step. It’s extremely powerful for image editing, concept design, and urban visualization, because you can control every part of the process: model loading, prompt conditioning, sampling, decoding, and saving. This makes it ideal for architects, urbanists, and researchers seeking privacy and control. On first launch, an empty grid appears as your workspace, ready to be filled with interconnected nodes for models, images, and prompts.

ComfyUI can be installed in two ways:

• Standalone version: downloaded from the official website and installed like a regular application. This version already includes the Node Manager, so you can install missing custom nodes directly from the interface.

• Git version: installed by cloning the repository (e.g., for advanced users or custom setups). In this case, you must install the Node Manager manually, because it is not included by default.

Next, open the v1.0 workflow for Qwen Edit for Urbanism. You can find it at the end of the post or in our Git repository. When the workflow loads, the Node Manager will automatically prompt you to install any missing custom nodes. Approve the installations, then close the workflow.

Step 2 — Add GGUF and Qwen Image Edit Support

Qwen-Image-Edit is a multimodal AI model capable of understanding both text and images, allowing image edits through natural language. The GGUF format makes it compact and memory-efficient, enabling faster processing on modest hardware.

Go to the ComfyUI-GGUF & Qwen Edit Utils project pages:
https://github.com/city96/ComfyUI-GGUF
https://github.com/lrzjason/Comfyui-QwenEditUtils

Clone the repositories, or click the green Code button and choose “Download ZIP” for both projects. After downloading, unzip each folder and place it directly into your custom_nodes directory—make sure the unzipped files are not nested inside an extra subfolder.

ComfyUI/custom_nodes/

Step 3 — Download the Required Models

You’ll need three model files, each serving a different purpose in the image-editing process. The first one is the core model that performs the actual visual transformation based on your text prompt.

You can find it here:

👉 https://huggingface.co/QuantStack/Qwen-Image-Edit-2509-GGUF/tree/main

On that page, you’ll see many versions of the same model — Q2, Q3, Q4, Q5, etc. These are quantized variants, meaning they trade a little precision for faster speed and lower VRAM usage. Here’s a quick guide to help you pick the right one:

The Qwen-Image-Edit model also needs two smaller files to work correctly — one for understanding your text prompts (CLIP) and one for turning the generated data back into a real image (VAE). You can download them from the official Comfy-Org repository on Hugging Face:

• Text Encoder (CLIP):
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders

  On that page, pick:
  👉 qwen_2.5_vl_7b_fp8_scaled.safetensors (about 9 GB — faster and works well locally)

• VAE Decoder:
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/vae

  Download the file:
  👉 qwen_image_vae.safetensors

Once downloaded, place all three models in:

ComfyUI/
 └── models/
      ├── gguf/
      │    └── Qwen-Image-Edit-2509-Q5_K_S.gguf
      ├── clip/
      │    └── qwen_2.5_vl_7b_fp8_scaled.safetensors
      └── vae/
           └── qwen_image_vae.safetensors

Verifying Your Setup

Open ComfyUI, right-click anywhere in the workspace or search in the node library, and check that you can find these nodes:

• • GGUF Loader – for loading the main Qwen-Image-Edit model (.gguf file)
  • CLIP Loader – for loading the CLIP text encoder
  • VAE Loader – for loading the VAE decoder
  • TextEncodeQwenImageEditPlus – for connecting text prompts and images

If they appear and can detect your downloaded files, congratulations 🎉 — your local setup is ready!

Now that all the models are downloaded and organized, you can start assembling the full workflow that makes Qwen-Image-Edit operate inside ComfyUI.
At this stage, the goal is to connect every component — the GGUF model, the text encoder (CLIP), the VAE decoder, the input image, and the text prompt — into a single functional chain. Once everything is linked, you’ll be able to type instructions such as “add shops on the ground floor” or “turn this street into a pedestrian zone” and watch ComfyUI generate updated images automatically.

Connecting the Model to Your Images and Prompts

Now that your model nodes are ready, it’s time to make them work together. You’ll connect your input image, prompt, and the Qwen-Image-Edit conditioning node so that ComfyUI can understand your instruction and modify the picture accordingly.

Right-click again and search for:

• Load Image – this node will import your input image (for example, a street or building photo).
• TextEncodeQwenImageEditPlus – this node combines your image with your text prompt

Adding the KSampler and the Other Required Elements

At this point, your setup has the model, encoders, and the prompt connection ready.
Now we’ll add the KSampler, which is the part that actually produces your final edited image, and all the other required elements.

Right-click again and search for:

• KSampler – this is ComfyUI’s main diffusion sampler. It takes in a latent (from the VAE) and conditioning (from Qwen) to generate a new image.
• Scale Image to Total Pixels – automatically resizes your image to a manageable resolution (set megapixels to 1.00).
• Get Image Size – extracts width and height from the resized image.
• EmptySD3LatentImage – creates an empty latent space matching the image dimensions, used as the canvas for generation.
• VAE Decode – converts the generated latent output back into a normal image.
• Save Image – saves your result to the ComfyUI output folder.

Your connections should look like this (check the figure for reference). Start by linking the Load Image node to Scale Image to Total Pixels, and then feed its output into Get Image Size. Next, connect the width and height outputs from Get Image Size to EmptySD3LatentImage, creating the correct latent dimensions. From there, route the latent output of EmptySD3LatentImage into the KSampler. In parallel, connect the model input of KSampler to the GGUF Loader, and feed the conditioning from TextEncodeQwenImageEditPlus. Finally, send the KSampler output into VAE Decode, and link that to Save Image to generate and store your final result.

Once these elements are connected, your workflow becomes fully functional. The scaled image is analyzed, encoded, transformed in the latent space according to your text prompt, and then decoded back into a visible edited image — all processed locally by your computer.

The core of the image-editing process lies in the KSampler node, where all generative inference occurs. Inside this component, several parameters control the quality, variability, and precision of the output. Understanding their effect is essential for achieving consistent and reproducible results.

To begin with, the seed parameter defines the random starting point of the diffusion process. Using the same seed with identical settings reproduces the same output, which helps when comparing the influence of other parameters. By contrast, changing the seed introduces controlled randomness, often generating new interpretations of the same instruction. Meanwhile, the control after generate option—set to increment—ensures each new image uses a slightly different seed, producing unique but related results.

Next, the steps value determines how many refinement iterations the sampler performs. Lower values such as five create quick previews and coarse adjustments, whereas higher counts (for instance, twenty or thirty) yield smoother and more detailed outcomes at the cost of longer processing time. In practice, Qwen-Image-Edit performs well even with fewer steps, since it relies heavily on prompt and image conditioning.

Similarly, the cfg parameter (classifier-free guidance) controls how closely the output follows the text prompt. A low value around 1.0 keeps changes subtle, while higher values push the model toward stronger, more literal transformations. Balancing this setting helps maintain realism without losing creative control.

As for the sampler_name and scheduler, they define how noise is reduced during diffusion. The Euler sampler offers an efficient trade-off between speed and visual quality, and the simple scheduler keeps results stable across seeds and image sizes.

Finally, the denoise value adjusts how strongly the latent image is modified. A setting of 1.0 applies a full transformation, producing bold edits, whereas smaller values retain more of the original features for subtle modifications. This parameter directly shapes the intensity of your edit—from a light retouch to a complete visual overhaul.

These results show the outcomes obtained with the final Qwen-Edit workflow, each generated from the same base image using a simple text prompt. Depending on the complexity of the requested transformation and the parameters defined in the KSampler—particularly the number of steps and the denoise factor—the processing time varied between 30 seconds and about 2 minutes on an RTX 4060 GPU.

• Qwen-Image-Edit official repository: https://huggingface.co/QuantStack/Qwen-Image-Edit-2509-GGUF

• ComfyUI documentation and examples: https://github.com/comfyanonymous/ComfyUI

• Qwen-Image-ComfyUI utilities and text encoders: https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI

• Zhang, F., Salazar-Miranda, A., Duarte, F., Vale, L., Hack, G., Chen, M., Liu, Y., Batty, M. & Ratti, C. (2024) ‘Urban Visual Intelligence: Studying Cities with Artificial Intelligence and Street-Level Imagery’, Annals of the American Association of Geographers, 114 (5), pp. 876-897. https://doi.org/10.1080/24694452.2024.2313515

• Perez, J. & Fusco, G. (2025) ‘Streetscape Analysis with Generative AI (SAGAI): Vision-language assessment and mapping of urban scenes’, GeoSpatial Analysis and Modelling, 100063. https://doi.org/10.1016/j.geomat.2025.100063 (ScienceDirect)

For convenience, you can download the ready-to-use ComfyUI JSON graph that we built in this post from the link below and load it directly into your workspace using File → Load → Workflow.

Before you can deploy smart contracts or explore decentralized applications, you need a crypto wallet and access to a test network. This guide will walk you through setting up MetaMask and obtaining free POL tokens on Polygon’s Amoy testnet.

Polygon is a Layer 2 blockchain network built on top of Ethereum. It’s fully compatible with Ethereum’s Virtual Machine (EVM), which means developers can use the same tools (like Solidity and MetaMask) and deploy smart contracts on Polygon just as they would on Ethereum — but with lower fees and faster speeds. Just like a developer might test an app on a “staging” version before going live, blockchain developers use testnets to try out smart contracts and decentralized applications (dApps) without risking real funds.

Testnets behave like real blockchains but use fake tokens and simulate real-world conditions, including:

• Smart contract deployment

• Gas usage and fees

• Wallet interactions and transactions

On the Amoy Polygon testnet, you use test POL tokens, which have no real-world value. They’re distributed for free via faucets and are used to Simulate transaction costs, test smart contracts and interact with dApps before going to mainnet. Since testnets are for development and learning, the tokens don’t come from real exchanges. Instead, developers request tokens from faucets — special services that send small amounts of testnet tokens (like POL on Amoy) to your wallet.

MetaMask is one of the most widely used browser-based wallets for Ethereum and EVM-compatible chains like Polygon. It’s beginner-friendly, well-supported, integrates directly with Remix, and is available as a browser extension and mobile app.

Other wallets like Rabbi, or Trust Wallet are excellent alternatives — especially for more advanced users or specific use cases — but MetaMask remains the simplest and most universal option for getting started with smart contracts. To get started, install MetaMask and follow the steps shown in the figure below to create your wallet safely.

Understand Password vs Secret Recovery Phrase

The password you create in MetaMask only unlocks the extension on your local device. If you lose it, you can still recover your wallet on any device using your Secret Recovery Phrase. That phrase is everything. It’s a master key to your wallet — anyone with it can access your funds, and if you lose it, your wallet is gone forever. That’s why you should store it offline, ideally on paper or a hardware password manager, and never on your computer or cloud storage.

The Amoy testnet is the official test network for Polygon. To add Amoy to MetaMask open MetaMask and click the network dropdown, then click Add network > Add a network manually and enter the information below.

Network Name: Polygon Amoy
New RPC URL: https://rpc-amoy.polygon.technology
Chain ID: 80002
Currency Symbol: POL
Block Explorer URL: https://www.oklink.com/amoy

This figure below shows how it shall look when you add Amoy on Metamask. After saving, you’ll be ready to interact with dApps and smart contracts on the Polygon Amoy testnet.

To interact with the Polygon Amoy testnet, you’ll need test POL tokens to pay for transactions.

Visit the Polygon Faucet, paste your wallet address, and select Amoy as the network. To prevent spam, the faucet requires you to verify your request by connecting a GitHub or X (Twitter) account — your wallet does not need to be connected. You can also request test tokens from third-party faucets if you don’t use GitHub or X.

Once verified, request the POL tokens. They should arrive in your wallet within seconds and appear under the Amoy network, as shown in the figure below.

You can now view your wallet activity directly on the Polygon Amoy Testnet Explorer. Just paste your wallet address into the search bar, and you’ll see your balance, transactions, and history—like the receipt of your test POL tokens. From here, your wallet is ready to use: you can send and receive tokens, monitor activity, and begin connecting to decentralized applications (dApps) built for testnets.

Stay tuned — this post is the first in a series. In the next entry, we’ll deploy a simple smart contract to the Amoy testnet and connect our wallet to it, and simulate sending environmental data (like temperature) as if from an IoT sensor. You’ll learn to interact with contracts on-chain, and begin experimenting with decentralized logic in practice.